On 30 December 2012 18:12, Kevin Fenzi <kevin@xxxxxxxxx> wrote: > On Sun, 30 Dec 2012 20:07:37 -0500 > Ricky Elrod <codeblock@xxxxxxxx> wrote: > >> We've been seeing load spikes on hostedXX, following >> df7e8578432b224d9576dc8359f0729763861526. This semi-reverts that >> commit and only allows /wiki/* to be crawled. >> >> diff --git a/configs/web/fedorahosted.org/fedorahosted-robots.txt >> b/configs/web/fedorahosted.org/fedorahosted-robots.txt >> index cd572f8..7782677 100644 >> --- a/configs/web/fedorahosted.org/fedorahosted-robots.txt >> +++ b/configs/web/fedorahosted.org/fedorahosted-robots.txt >> @@ -1,5 +1,5 @@ >> User-agent: * >> -Disallow: /*/browser >> -Disallow: /*/search >> +Allow: /wiki/* >> +Disallow: / >> user-agent: AhrefsBot >> disallow: / +1 Also. -- Stephen J Smoogen. "Don't derail a useful feature for the 99% because you're not in it." Linus Torvalds "Years ago my mother used to say to me,... Elwood, you must be oh so smart or oh so pleasant. Well, for years I was smart. I recommend pleasant. You may quote me." —James Stewart as Elwood P. Dowd _______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
