On Thu, 5 Apr 2007, Linus Torvalds wrote:
> Without "--full", it doesn't actually really do anything much, since it
> will basically ignore objects that are in the pack.
>
> With --full, there are certainly things that we could improve upon. We
> currently tend to walk things a few times for pack contents:
> - first we do the SHA1 of the full pack
> - then we go back, and unpack and fsck each entry in the pack.
>
> So if the pack-file is too big to fit in memory, we'll basically always
> read it at least twice (and that's ignoring the fact that delta lookup
> will obviously seek back and forth, which makes access patterns worse).
>
> On the other hand, there's a perfectly good reason why we don't actually
> fsck pack-files by default. They're "stable storage". You don't normally
> need to. So I'd not worry too much about fsck performance.
Well.... still it certainly can be helped a bit. I wouldn't mind it
spending half an hour of CPU if it needs to. But I just interrupted it
with ^C with the following result so far:
real 75m44.374s
user 2m5.318s
sys 0m54.059s
(I should have used /usr/bin/time to see the number of page faults).
> I suspect you'll find that with 1GB or RAM you'll have other
> performance problems that are more pressing ("git clone" comes to mind
> ;)
Well... same issue actually. git-pack-objects spent about 40 secs
firmly at 100% CPU usage counting objects.
Then it got stuck on:
remote: Done counting 4111366 objects.
again spending 3% CPU and the rest waiting for IO with the disk
definitely trashing. It didn't allocate more than 47% of memory during
that phase which lasted a few minutes.
Then, the "Indexing 4111366 objects." message appeared and CPU usage
went up to 6% CPU with 67% memory for pack-objects and 30% CPU and 7%
memory for index-pack while the rest was spent waiting for IO. This
also took maybe two minutes.
And now it reached the "Resolving 3305158 deltas." phase with only
index-pack on the radar with approx 10% CPU and 19% memory, and the rest
of the time waiting for IO again.
It has been probably half an our now and the thing is at:
21% (710502/3305158) done
So it will work and eventually complete. And the good news is that the
worst part performance wise is on the client side. But it looks like
we're definitely trashing the kernel buffer cache.
Nicolas
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html