On Tue, Apr 21, 2015 at 07:33:27AM +0200, Pierre-Yves Chibon wrote: > > > > ! Crons: iirc there are still a couple of crons that are not running > > > > properly and returning errors > > > > Which ones? > > From this night: > ---- > Subject: Cron <root@mm-crawler01> /usr/bin/mm2_crawler --threads 50 > /dev/null > 2>&1 > > /bin/sh: line 1: 18523 Killed /usr/bin/mm2_crawler --threads 50 > > /dev/null 2>&1 This is OOM. One crawler did not finish within 12 hours. The second crawler started and Linux killed one of them. The atomic tree is really killing the crawler with its over 370000 files. If a mirror has http as the only protocol and is geographically far and slow it is not possible to crawl in a reasonable time. The atomic tree needs a different setup/rule how to be crawled. > ---- > Subject: Cron <root@mm-backend01> cd /usr/share/mirrormanager2 && /usr/bin/mm2_get_internet2_netblocks > /var/lib/mirrormanager/internet2_netblocks.txt > > gzip: /tmp/get_i2_netblocks.RS1rA07F/rib.gz: unexpected end of file > short file (empty packet) at zebra-dump-parser/zebra-dump-parser.pl line 98. [...] > gzip: /tmp/get_i2_netblocks.RS1rA07F/rib.gz: unexpected end of file > short file (empty packet) at zebra-dump-parser/zebra-dump-parser.pl line 98. > unable to retrieve netblock list. I think I see these also on my MM1 installation. Probably easy to fix. > These are the only two I can see right now. > > > > So, that would be another, higher level test: > > > > > > Make staging scan directories for latest content. > > > Make staging make a pkl based on that. > > > Check it against the prod one > > > > I am not sure if the resulting pickles can be that easily compared as > > there have been changes all around MM2 which might return slightly > > different results which are nevertheless both valid results. > > I don't think we changed the pickles themselves, for example mirrorlist barely > changed and is currently working with MM1's pickle. I was talking about the content of the pickles. As the crawler is making slightly different decisions and umdl maybe also I do not think we can compare the content of the pickles. Adrian
Attachment:
pgp5dGvYti7pw.pgp
Description: PGP signature
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
