guo tang wrote:
On Mon, Sep 22, 2008 at 12:32 AM, Andreas Ericsson <ae@xxxxxx> wrote:
Guo Tang wrote:
Gentlemen,
I try to run "git gc" on linux kernel tree. The virtual memory keeps going
up until over 3GB, then crash. Tried twice with the v1.6.0.2, same result.
Then I used the git coming with FC9 (v1.5.5.1), the peak virutal memory
usage is about 1.5GB. "git gc" finished without any trouble.
Could there be a memory leak in v1.6.0.2?
There could be, but most likely it's commit
38bd64979a2a3ffa178af801c6a62e6fcd658274 (Enable threaded delta
search on BSD and Linux). Do you have multiple cpu's in the
computer where 'git gc' was running? If so, and if you've set
pack.threads = 0, or --threads=0 it will autodetect the number
of CPU's you have and then saturate all of them with work. Each
thread will however consume memory close to that of a single
process running the repack, so for large repositories you might
want to set pack.threads = 1 in such large repositories.
It is a Pentium M single core machine. But I am not sure whether it is using
just a single thread or
multiple threads. I will try setting pack.threads parameter next I run into
trouble.
Unless you explicitly told it to run multiple threads (which
would be a bit silly on a single-core machine), it just ran
one thread.
It's a shame you didn't save the unpacked repository, or this could
have been properly debugged. While it's possible there is a memory
leak, it's a dismal project trying to locate it by staring at the
code, and the time it takes to repack huge repositories with memory
intensive parameters is sort of prohibitive for finding the possible
leak by bisection.
Yes, the repository is already packed now. One question, beside the
bisecting method, do we have
this ability built into kernel:
1. Turn a flag on for a process.
2. OS will keep track off process malloc(), free() calls and the call stack.
3. For the malloc() calls without the the free() call (a memory leak), OS
will keep it count based on malloc() call stack.
4. After some time, be able to dump this information out based on biggest
leak spot.
No, there's not. The kernel isn't the one handing out the memory when you
call malloc(). That's handled by the C library, which can (and usually does)
allocate a larger area of memory than the application needs, so that it
doesn't have to run a system call for every malloc() call you do.
You can pre-load a different memory allocator though, which can do whatever
it wants with calls to malloc(), including ofcourse logging which function
called them and how much memory was requested.
Google for "memory leak check linux" and you'll get something like 750000
results.
The complain when I ran out of memory if from mmap failure. Is it the same
as malloc() failure?
Sort of. Read 'man 2 mmap' for a more exhaustive description.
This kind of tool is available in Windows with its umdh (user mode heap
dump) tool.
There are a number of tools to detect leaks under Linux/Unix as well.
valgrind is probably the most frequently used of all such leak checkers.
--
Andreas Ericsson andreas.ericsson@xxxxxx
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
