If you'll recall, the company previously shared a planned update to its terms with language saying that Meta would have the right to use web scrapers and automated tools "to access, index, cache, analyze, collect, retrieve, or crawl the domains, web pages, or URLs (and any content available through them)" associated with business accounts on the platform. The updated terms would also state that Meta "may use any content from such domains, web pages, or URLs to provide, develop, improve, research, train, and maintain the integrity of, the Meta Company Products, services, and features.
Meta added in its statement that it will explain to its partners how it uses its web crawlers and make it clear that content owners and publishers will be able to use robots.txt. The plain text file will be able to signal the content owners' preferences to crawlers and AI bots and stop them from indexing their content. That said, Reuters previously reported that multiple AI companies' crawlers regularly ignore robots.txt instructions.
Source link







