Thank you for the actionable comments, Ralf!
I think I addressed them and pushed it to github for your viewing pleasure:
Note on the examples in section 5: The examples on robotstxt.org are more geared towards site owners, while the draft's examples towards parser/matcher implementers, hence the significant differences. That said, I agree that we should have had a global user agent (*) example and so I extended the section 5.1 example with that.
On Sun, May 29, 2022 at 4:21 PM Ralf Weber via Datatracker <noreply@xxxxxxxx> wrote:
Reviewer: Ralf Weber
Review result: Ready with Issues
Moin!
I am an assigned INT directorate reviewer for draft-koster-rep.
These comments were written primarily for the benefit of the Internet Area
Directors. Document editors and shepherd(s) should treat these comments just
like they would treat comments from any other IETF contributors and resolve
them along with any other Last Call comments that have been received. For more
details on the INT Directorate, see
https://datatracker.ietf.org/group/intdir/about/
While the document technically defines the content of the robots.txt files it
could do a better job in describing with examples the semantic of how robots
interpret them. Especially in 2.2.1 the notation of "Crawlers MUST find the
group that matches the product token exactly" should be better explained. I
assume it does not mean being fully equal but instead a substring match in the
User-Agent Header, so in the example would also match a http user agent of
ExampleBotnet/1.2. Is that understanding correct at least?
Also the examples in 5 seem a lot more arbirtary than what the ROBOTSTXT
website has and it should explain all the outcomes, e.g in 5.1 it would allow
access to all crawlers and all paths, but the foobot, barbot and bazbot
/example/disallowed.gif. An example with a * group would be better and more
realistic.
So long
-Ralf
-- last-call mailing list last-call@xxxxxxxx https://www.ietf.org/mailman/listinfo/last-call
