On Mon, Jan 2, 2012 at 6:55 PM, Dennis Gilmore <dennis@xxxxxxxx> wrote: > El Sat, 24 Dec 2011 23:25:56 -0600 > Dennis Gilmore <dennis@xxxxxxxx> escribió: >> When we were testing build were happening really fast, once we loaded >> up the build jobs things have become really slow >> >> http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=238672 started >> at 1:19 utc and at 5:18 utc four hours later the buildrequires are >> still being installed. australia is completely io bound. i think we >> need to see how we can spread the io load. ~30 hosts reading and >> writing to the 4 spindles just saturates the disk io. i guess options >> are find a way to add more spindles. move /var/lib/mock to sdcard, see >> if we can get some kind of san that has alot of smaller fast disks. >> get some 2.5" usb drives one for each builder. some other idea? is >> there anyway we could add 4-8 more disks to australia. the size of the >> matters little. gaining more iops by adding more spindles would help. >> >> Just throwing out what im seeing. lets see what ideas we can come up >> with. performance is better than before. and seneca has done a great >> job and put a lot of hard work into the reorg and im thankful for >> that. We just have another bottleneck to address. >> _______________________________________________ >> arm mailing list >> arm@xxxxxxxxxxxxxxxxxxxxxxx >> https://admin.fedoraproject.org/mailman/listinfo/arm > > http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=245410 has been > running 10 hours now. its inited the chroot and installing the srpm. it > is a big srpm but still. performance on the hfp builders while not > great is better. when looking at a sfp builder the mock process is > using 100% cpu and in a D state. there is one of two issues here. the > hfp builders are running a slightly older kernel. so it could be a > kernel ext3 nfs or loop device issue, upgrading a hfp builder for > testing would confirm it. or its some issue with decompresion of the > rpms. I do think that disk io on the nfs server is partly to blame. the > box sits at a load of 9 all the time with 30% or so io wait. Id like to > take a layer or 2 of the complexity out. switching over to nfsv4 we > should be able to run mock on it. im going to do some compression tests > with the latest 2.6.41.6 kernel. Nearly two days after I submitted this build its still not actually to the point of building. This is a dire situation and doesn't give me much hope of being able to follow closely behind mainline at this rate. Peter _______________________________________________ arm mailing list arm@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/arm
