It appears that Mark Nottingham <mnot@xxxxxxxx> said: > > >> On 16 Mar 2022, at 3:52 am, Murray S. Kucherawy <superuser@xxxxxxxxx> wrote: >> >> If the authors are prepared to make this change, I'd like to hear why "IETF Standards Track w/consensus" is (or >should be) off the table, especially if other search engines are going to indicate support. > >That's reasonable, if support eventuates. If it doesn't, it would be pretty awkward -- I think we'd want at least >some level of review or at least acknowledgement by other major implementers before declaring it Standards Track. A lot of search engines describe how they handle robots.txt. For example, Yahoo https://help.yahoo.com/kb/SLN22600.html Bing https://www.bing.com/webmasters/help/how-to-create-a-robots-txt-file-cb7c31ec Yandex https://yandex.com/support/webmaster/controlling-robot/robots-txt.html Baidu https://www.baidu.com/search/robots_english.html DataForSEO https://dataforseo.com/dataforseo-bot Petal Search https://webmaster.petalsearch.com/site/petalbot Mojeek https://www.mojeek.com/bot.html (Those last three I never heard of but their spiders show up in my logs.) >From what I can see, the current draft is OK, but I would make it clearer that the * and $ metacharacters aren't widely supported, and I would add Crawl-Delay and Siteindex directives which are. R's, John -- last-call mailing list last-call@xxxxxxxx https://www.ietf.org/mailman/listinfo/last-call
