The abstract of RFC 9309 defines the scope for robots.txt and an AP server is outside it:
This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers.
Sure, but the nature of the restriction seems to be against crawlers recursively hitting resources and such, rather than realising AP on behalf of (presumably human) users. I might agree botsin.space should honour robots.txt, but it doesn't seem otherwise relevant. Only my interpretation, but that's what I get from:
Crawlers are automated clients. Search engines, for instance, have crawlers to recursively traverse links for indexing as defined in [RFC8288].¶ It may be inconvenient for service owners if crawlers visit the entirety of their URI space. This document specifies the rules originally defined by the "Robots Exclusion Protocol" [ROBOTSTXT] that crawlers are requested to honor when accessing URIs.¶
modulux
in reply to Evan Prodromou • • •The abstract of RFC 9309 defines the scope for robots.txt and an AP server is outside it:
Evan Prodromou
in reply to modulux • • •modulux
in reply to Evan Prodromou • • •Sure, but the nature of the restriction seems to be against crawlers recursively hitting resources and such, rather than realising AP on behalf of (presumably human) users. I might agree botsin.space should honour robots.txt, but it doesn't seem otherwise relevant. Only my interpretation, but that's what I get from:
From the same RFC.