On 26.1.2015 16:46, Kevin Fenzi wrote: > On Mon, 26 Jan 2015 07:53:27 -0700 Brandon Vincent > <Brandon.Vincent@xxxxxxx> wrote: > >> On Mon, Jan 26, 2015 at 7:21 AM, Florian Weimer <fweimer@xxxxxxxxxx> >> wrote: >>> Any idea why? >> >> https://lists.fedoraproject.org/robots.txt >> >> User-agent: * Crawl-delay: 10 >> >> From Bing, "This means the higher your crawl delay is, the fewer pages >> BingBot will crawl. As crawling fewer pages may result in getting less >> content indexed, we usually do not recommend it, although we also >> understand that different web sites may have different bandwidth >> constraints." [1]. >> >> [1] >> http://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question/ > >> > Not sure that explains why there are no results at all though. > > I think we added the Crawl-delay several years ago when we were having > storage issues. We could definitely try removing it and see if things > improve. Yes please. We definitely should get archives indexed by search engines! -- Petr Spacek @ Red Hat -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
