On Mon, Jan 26, 2015 at 7:21 AM, Florian Weimer <fweimer@xxxxxxxxxx> wrote: > Any idea why? https://lists.fedoraproject.org/robots.txt User-agent: * Crawl-delay: 10 >From Bing, "This means the higher your crawl delay is, the fewer pages BingBot will crawl. As crawling fewer pages may result in getting less content indexed, we usually do not recommend it, although we also understand that different web sites may have different bandwidth constraints." [1]. [1] http://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question/ Brandon Vincent -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
