On Wed, May 13, 2015 at 03:43:55PM +0200, Pierre-Yves Chibon wrote: > On Wed, May 13, 2015 at 09:40:49AM -0400, Ralph Bean wrote: > > On Wed, May 13, 2015 at 02:17:24PM +0200, Adrian Reber wrote: > > > The new MM2 crawler disables mirrors which have failed to be > > > successfully crawled for 4 consecutive crawls. This seems to be a good > > > idea to reduce the total number of crawls by removing mirrors which are > > > just too slow. Unfortunately the current default timeout of 2 hours is > > > not enough. Especially for mirrors which mirror more than one category > > > as the timeout is per host and not per category. The problem is also not > > > network bound but it seems to be related to two crawlers updating the > > > directories of all mirrors on the same database at the same time. To > > > workaround this timeout problem I am now starting the crawler on the > > > second crawler 3 hours later and I have also increased the timeout from > > > 2 hours to 3 hours. Additionally a small fix is included to also crawl > > > the last mirror in the database which was ignored until now. > > > After this is applied I would also re-enable the auto-disabled hosts > > > in the database. > > > > > > Can I get two +1 for these changes? > > > > This looks good and sane to me. +1. > > > > If things go wrong, we can easily back out by just reverting the > > commits. > > It looks sane to me as well, +1 Thanks. Changes are a now active on the crawlers and I have re-enabled 141 mirror which have been auto-disabled. update host set user_active=true where crawl_failures > 3; update host set crawl_failures=0 where crawl_failures > 3; Adrian
Attachment:
pgpXhLIQlLmVb.pgp
Description: PGP signature
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
