The impact of Artificial Intelligence (AI) on various industries, including the news sector, is becoming increasingly evident. As the use of AI bots for web crawling and data collection grows, news websites are facing a dilemma of whether to allow or block these bots. A recent analysis by data journalist Ben Welsh sheds light on this ongoing battle within the news industry.
Welsh’s analysis revealed some interesting statistics regarding the blocking of AI bots by news websites. While just over a quarter of the websites surveyed are blocking Applebot-Extended, a significant 53 percent are blocking OpenAI’s bot. Additionally, Google’s AI-specific bot, Google-Extended, is being blocked by nearly 43 percent of the surveyed sites. These numbers indicate a clear divide among news publishers in their approach to AI bots.
According to Welsh, a divide has emerged among news publishers on whether to block AI bots or not. Some publishers are open to allowing these bots, possibly due to licensing deals or strategic partnerships with the bot owners. For instance, major publishers like The New York Times have reportedly engaged in AI deals with companies like Apple, OpenAI, and Perplexity. This strategic approach highlights the evolving dynamics of the news industry in the AI age.
Jon Gillham, founder of Originality AI, suggests that there is a business strategy involved in partnering with AI bot owners. By withholding data until a partnership agreement is in place, publishers may be leveraging their content for financial gain. The recent example of Condé Nast unblocking OpenAI’s bots after announcing a partnership further supports this theory. This strategic maneuvering hints at the complexity of relationships between publishers and AI companies.
Despite the potential benefits of partnering with AI bot owners, publishers face challenges in managing and identifying these bots. With the manual editing of robots.txt files required to block bots, publishers often struggle to keep an up-to-date block list. Gavin King, founder of Dark Visitors, highlights this issue and offers a service to automate robots.txt updates for clients. The increasing number of AI agents further complicates the task of identifying and blocking unauthorized bots.
As the battle over AI bots intensifies, media executives are directly involved in decisions regarding which bots to block. CEO’s of major media companies are taking a stand on blocking AI scraping tools, particularly when partnerships with bot owners are not in place. This approach underscores the importance of maintaining control over data and content distribution in the digital landscape.
The ongoing battle of AI bots in the news industry reflects the changing dynamics of digital media. Publishers are balancing between the need for data protection and the potential benefits of partnerships with AI companies. As technology continues to advance, the influence of AI on news websites will only grow, necessitating strategic decisions by media executives to navigate this evolving landscape.
Leave a Reply