New page on seirdy.one: Scrapers I block (and allow), with explanations.

I’ve replaced all the comments in my robots.txt file with a more readable and detailed web page on scrapers I block. It includes info on the multiple blocking-approaches and criteria I use, commonly-blocked scrapers I allow, and more fact-checking than most of the more comprehensive alternatives.

#RobotsTxt #Scrapers #POSSE

Scrapers I block (and allow), with explanations

Here’s my thought process when deciding whether to block a scraper from seirdy.one, the scrapers I block, the scrapers I allow, and the ways I block them.

^{Seirdy’s Home}

New scraper just dropped (well, an old scraper was renamed):

Facebook/Meta updated its robots.txt entry for opting out of GenAI data scraping. If you blocked FacebookBot before, you should block meta-externalagent now:

User-Agent: meta-externalagent
Disallow: /

• Facebook developer documentation for FacebookBot no longer mentions GenAI.
• Facebook developer documentation for web crawlers, including Meta-ExternalAgent mentions “AI”.

#RobotsTxt #Scraper