Meta Platforms has faced intense scrutiny after announcing upcoming changes to its terms that, essentially, allow AI crawlers to run wild across business accounts.

Yet, the owner of Facebook and Instagram has been doing a version of that on websites for years, website managers say, with small online businesses often left to deal with the repercussions. 

Take California-based artist Chelsea Paulus, who says Meta’s crawlers crashed her website last week. 

A bot called Meta-ExternalAgent got stuck in a loop on Paulus’s website’s “remove from cart” function. It created more than a hundred hits per minute, growing to about 550,000 hits over the next few days. Her website’s CPU hit 80% and her host warned that the site could go down at 100%. 

Paulus blocked Meta’s bot, but two days later, another bot, this one called Meta-WebIndexer, showed up. It brought her CPU to 100% and crashed her website. 

So, why do these types of bots crawl the internet? Meta’s web crawler page explains, “The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.” 

“The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users,” Meta further states. “In doing so, Meta analyzes online content to enhance the relevance and accuracy of Meta AI.”

Crawlers allow many online companies to properly function. Companies use AI crawlers for things such as training large language models (LLMs) or being able to quickly locate answers for AI assistants.

Source link