OpenAI unveils web crawler 'GPTBot', allows websites to block it

OpenAI unveiled its web crawler 'GPTBot' and said the collected data could be used to improve future AI models. Website operators who are not interested in letting the bot crawl their sites can add the GPTBot to their sites' 'robots.txt' file, OpenAI said. It added that GPTBot is built to remove sources that require paywall access.