On 12/25/2011 06:16 AM, Brendan Conoboy wrote:
On 12/24/2011 09:25 PM, Dennis Gilmore wrote:
Just throwing out what im seeing. lets see what ideas we can come up
with. performance is better than before. and seneca has done a great
job and put a lot of hard work into the reorg and im thankful for that.
We just have another bottleneck to address.
Allocating builders to individual rather than a single raid volume will
help dramatically.
Care to explain why?
Since hfp/sfp F15 builds happen at the same time,
having 2 hfp disks and 2 sfp disks is a good start. Split it up again by
some other criteria- you want to try to ensure that any one builder only
causes one spindle to be used and any 2 builders each only cause 2
spindles to be used (1-to-1), and so forth. As long as any single
spindle isn't doing multiple simultaneous mock inits it should scale
pretty well.
You can largely avoid this by making sure your RAID and FS are aligned.
Most mock IOPS are small, and provided layers are aligned right, should
scale very well on RAID 0/10 (RAID5/6 will inevitably cripple just about
anything when it comes to writes regardless of what you do, unless the
vast majority of your writes is a multiple of your stripe-width, which
is unlikely).
Allocate the large files the builders are using all at once
(no sparse files) to ensure large read/writes don't require seeks.
Is this a "proper" SAN or just another Linux box with some disks in it?
Is NFS backed by a SAN "volume"?
Use the async nfs export option if it isn't already in use.
+1
Disable fs
journaling (normally dangerous, but this is throw-away space).
Not really dangerous - the only danger is that you might have to wait
for fsck to do it's thing on an unclean shutdown (which can take hours
on a full TB scale disk, granted).
Speaking of "dangerous" tweaks, you could LD_PRELOAD libeatmydata (add
to a profile.d file in the mock config, and add the package to the
buildsys-build group). That will eat fsync() calls which will smooth out
commits and make a substantial difference to performance. Since it's
scratch space anyway it doesn't matter WRT safety.
Use
noatime mounts if that won't break package builds checking filesystem
conformance.
Build of zsh will break on NFS whateveryou do. It will also break on a
local FS with noatime. There may be other packages that suffer from this
issue but I don't recall them off the top of my head. Anyway, that is an
issue for a build policy - have one builder using block level storage
with atime and the rest on NFS.
Once all that is done, tweak the number of nfsds such that
there are as many as possible without most of them going into deep
sleep. Perhaps somebody else can suggest some optimal sysctl and ext4fs
settings?
As mentioned in a previous post, have a look here:
http://www.altechnative.net/?p=96
Deadline scheduler might also help on the NAS/SAN end, plus all the
usual tweaks (e.g. make sure write caches on the disks are enabled, if
the disks support write-read-verify disable it, etc.)
Gordan
_______________________________________________
arm mailing list
arm@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/arm
