Re: FBR: Change to category based crawling

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Thu, Apr 19, 2018 at 08:54:21AM +0200, Adrian Reber wrote:
> As discussed previously I would like to change the crawler to crawl each
> category separately. The goal is to reduce the load on the database by
> distributing the crawling better over the whole day and to reduce the
> chance of mirrors being disabled because of the high database load.
> 
> This should also remove the need for mirror administrators to create
> multiple hosts in MirrorManager to work around the 4 hours timeout per
> host.
> 
> Attached is my patch. Please +1. This affects mm-crawler01 and
> mm-crawler02.

Thanks for the '+1's. This change is active since yesterday and so far
it seems to work. If we still see hosts timing out, especially when
crawling the archive hosts, we can increase the timeout for archive
crawling to 6 hours. Another option to decrease the number of wrongly
auto-deactivated mirrors is to increase CRAWLER_AUTO_DISABLE from 4 to 6
or 8 crawls.

I will look at those changes after the freeze.

		Adrian

Attachment: signature.asc
Description: PGP signature

_______________________________________________
infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx

[Index of Archives]     [Fedora Development]     [Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [KDE Users]

  Powered by Linux