On Wed, 12 Dec 2007, Nicolas Pitre wrote:
>
> So... my conclusion is that the glibc allocator has fragmentation issues
> with this work load, given the notable difference with the Google
> allocator, which itself might not be completely immune to fragmentation
> issues of its own.
Yes.
Note that delta following involves patterns something like
allocate (small) space for delta
for i in (1..depth) {
allocate large space for base
allocate large space for result
.. apply delta ..
free large space for base
free small space for delta
}
so if you have some stupid heap algorithm that doesn't try to merge and
re-use free'd spaces very aggressively (because that takes CPU time!), you
might have memory usage be horribly inflated by the heap having all those
holes for all the objects that got free'd in the chain that don't get
aggressively re-used.
Threaded memory allocators then make this worse by probably using totally
different heaps for different threads (in order to avoid locking), so they
will *all* have the fragmentation issue.
And if you *really* want to cause trouble for a memory allocator, what you
should try to do is to allocate the memory in one thread, and free it in
another, and then things can really explode (the freeing thread notices
that the allocation is not in its thread-local heap, so instead of really
freeing it, it puts it on a separate list of areas to be freed later by
the original thread when it needs memory - or worse, it adds it to the
local thread list, and makes it effectively totally impossible to then
ever merge different free'd allocations ever again because the freed
things will be on different heap lists!).
I'm not saying that particular case happens in git, I'm just saying that
it's not unheard of. And with the delta cache and the object lookup, it's
not at _all_ impossible that we hit the "allocate in one thread, free in
another" case!
Linus
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
