On Wed, 2008-11-26 at 19:08 +0100, Patrice Dumas wrote:
> Indeed, that's a lot of steps. But what steps take so long? There are
> some steps that I don't really understand, but I can't see which one
> would take more than some minutes, except maybe 9) if there are a lot of
> packages -- and 4) if 4) involves transferring the packages. In any case
> I guess that you know what you are doing, so my questions are certainly
> very naive.
The parts that take the longest are sorting out the signed packages from
koji output into a directory structure that is suitable for updates via
hardlinks, and then the createrepo runs. Sorting the package takes a
little time as koji puts packages into dirs of just <arch>, which can
include noarch. debuginfo is put along side the other arch packages.
We have to sort out the debuginfo, copy around any noarch, but also
examine the srpm the noarch package came from to determine if the noarch
package has any Exclude or ExclusiveArch settings. This involves a lot
of filesystem work and tends to be a lengthy process, as is just the
mere querying of koji for the 'latest-tagged' for each of the tags.
As for repodata, for each tag (dist-f{8,9,10}-{updates,updates-testing})
we have to create repodata for:
i386, x86_64, ppc, pp64
debuginfo
source
that's 6 repos worth of packages to run createrepo on, across an NFS
mount. For two of those repos (x86_64, ppc) in order to make multilib
we first copy in all the compat arch packages (i386, ppc64), run
createrepo, run through an algorithm to select certain ones as being
"multilib" and then depsolve the rest, throwing out what's not needed,
and then running createrepo again.
So to recap, for each tag, we have 11 createrepo calls, and since we're
doing 6 tags at this point, that's 66 calls to createrepo, again all
over NFS (since we don't want to put the software that pushes updates on
the same machine that holds the filesystem that koji uses, and we want
to use hardlinks to avoid adding time to copy all those packages over
NFS).
Right now, I do believe there is a faulty switch between the machine(s)
capable of doing the repo creations and the nfs server, which is capping
bandwidth around 100mb/s rather than 1000mb/s. We've been trying to get
that fixed for nearly a year, and will continue trying, Mike McGrath is
scheduled to be onsite in the colo to do various things such as fixing
this in December.
As stated in the previous email, there are steps that are done in
parallel, like many of the createrepo calls. But there is only so much
bandwidth between the caller and the NFS share, and there is only so
much memory in the caller, and there are only so many tasks that can
truly be done in parallel.
--
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating
Attachment:
signature.asc
Description: This is a digitally signed message part
-- fedora-devel-list mailing list fedora-devel-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/fedora-devel-list
