On Mon, 26 Jan 2015 07:53:27 -0700 Brandon Vincent <Brandon.Vincent@xxxxxxx> wrote: > On Mon, Jan 26, 2015 at 7:21 AM, Florian Weimer <fweimer@xxxxxxxxxx> > wrote: > > Any idea why? > > https://lists.fedoraproject.org/robots.txt > > User-agent: * > Crawl-delay: 10 > > From Bing, "This means the higher your crawl delay is, the fewer pages > BingBot will crawl. As crawling fewer pages may result in getting less > content indexed, we usually do not recommend it, although we also > understand that different web sites may have different bandwidth > constraints." [1]. > > [1] > http://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question/ Not sure that explains why there are no results at all though. I think we added the Crawl-delay several years ago when we were having storage issues. We could definitely try removing it and see if things improve. kevin
Attachment:
pgpVPOth3hHmM.pgp
Description: OpenPGP digital signature
-- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
