The Globe and Mail reports in its Tuesday, Sept. 2, edition that a foundational Internet rule has been the social contract between website owners and search engines like Alphabet's Google. The Globe's guest columnist Viet Vu writes that owners allow bots to crawl and index their sites for free, which drives traffic in return. If they prefer not to be indexed, they can specify this in a "robots.txt" file. Since 1994, most companies have followed this voluntary standard, but the emergence of large language models has challenged this agreement. The current arrangement forces websites to bear increased costs and receive fewer visitors, all while aiding in the training of AI chatbots. Without intervention, this could lead to a cycle where AI causes websites to shut down, ultimately depriving AI of the data it needs to improve. Cloudflare has begun treating AI bots as potential hackers threatening customer cybersecurity. In August, it claimed that Perplexity, a leading AI company, is working on methods to conceal its crawling activities to bypass cybersecurity measures. If all tech companies had to operate under common rules, it would level the playing field by removing the competitive pressure to race to the bottom.