robots.txt references are unfortunately quite prone to cargo-culting. I find ai.robots.txt to be generally misleading. Lots of the UAs they list aren’t used to train generative AI.

AdsBot-Google, for instance, doesn’t do anything if you don’t use Google Ads; Google uses GoogleBot-Extended to train its GenAI offerings.

For image2dataset, it’s better to use a noai meta robots directive and allow it to crawl to properly opt out of indexing, should it stumble upon a cached copy of your page.

Blocking certain bots

I don’t want my content on those sites in any form and I don’t want my content to feed their algorithms. Using robot.txt assumes they will ‘obey’ it. But they

^{Seirdy’s Home}