OpenAI has launched web crawler GPTBot to improve artificial intelligence models.
“Web pages crawled with the GPTBot user agent may potentially be used to improve future models and are filtered to remove sources that require paywall access, are known to gather personally identifiable information (PII) or have text that violates our policies,” the company said in a post on its website.
“Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety,” OpenAI wrote.
A web crawler is a type of bot.
WHAT IS AI?
The OpenAI ChatGPT logo is seen on a mobile phone. (Jaap Arriens/NurPhoto via Getty Images)
It is usually operated by search engines that index the content of websites for the sites to appear in search results, according to internet company Cloudflare.
They are called “web crawlers” because crawling is the term for automatically accessing a website and obtaining data using software.
OpenAI also provided instructions on disallowing the GPTBot from accessing a website – either partially or fully.
Sam Altman, chief executive officer of OpenAI Inc., speaks with members of the media during the Allen & Co. Media and Technology Conference in Sun Valley, Idaho, on Wednesday, July 12, 2023. (David Paul Morris/Bloomberg via Getty Images)
WHAT IS CHATGPT?
Websites can block the crawler’s IP address or add the GPTBot to the site’s robots.txt file. The file essentially instructs web crawlers on what is accessible from a site.
“To allow GPTBot to access your only parts of your site you can add the GPTBot token to your site’s robots.txt,” it explained.
A man is seen using the OpenAI ChatGPT artificial intelligence chat website. (Jaap Arriens/NurPhoto via Getty Images)
CLICK HERE TO GET THE FOX NEWS APP
“For OpenAI’s crawler, calls to websites will be made from the IP address block documented on the OpenAI website,” OpenAI concluded.
Notably, AI companies, including OpenAI, previously signed an agreement with the White House to develop a watermarking system to let internet users know if something was generated by AI. However, the organizations have not pledged to stop using internet data for training.
IN TODAY’S NEWSLETTER: - Pentagon launches military AI platform powered by Google Gemini for defense…
ChatGPT went from novelty to necessity in less than two years. It is now part…
Holiday travel and winter storms create risky moments for drivers and families. Stress rises fast…
A long-running malware campaign quietly evolved over several years and turned trusted Chrome and Edge…
Sometimes you need someone to answer right away. Maybe you are locked out. Maybe those…
Surgeons at Rambam Eye Institute have made medical history. They restored sight to a legally…