On Thu, Jan 24, 2019 at 7:11 PM Jeff King <peff@xxxxxxxx> wrote: > > On Tue, Jan 22, 2019 at 06:07:14PM -0400, Joey Hess wrote: > > > joey@darkstar:~/tmp/t> ls -l big-file > > -rw-r--r-- 1 joey joey 11811160064 Jan 22 17:48 big-file > > joey@darkstar:~/tmp/t> git status > > fatal: Out of memory, realloc failed > > > > This file is checked into git, but using a smudge/clean filter, so the actual > > data checked into git is a hash. I did so using git-annex v7 mode, but I > > suppose git lfs would cause the same problem. > > > > [pid 6573] mmap(NULL, 11811164160, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory) > > > > Why status needs to mmap a large file that is not modified > > and that is configured to pass through smudge/clean, I don't know. > > It seems like it should be possible for status work in this situation. > > One minor point: I don't think this is us mmap-ing the file. The > descriptor is -1, and Git never uses PROT_WRITE. This is likely your > libc using mmap to fulfill a malloc() request. > > That said, it just turns the question into: why did Git try to malloc > that many bytes? If I reproduce your example (using a 100MB file) and > set GIT_ALLOC_LIMIT in the environment, the backtrace to die() is: > > #1 0x0000555555786d65 in memory_limit_check (size=104857601, gentle=0) at wrapper.c:27 > #2 0x0000555555787084 in xrealloc (ptr=0x0, size=104857601) at wrapper.c:137 > #3 0x000055555575612e in strbuf_grow (sb=0x7fffffffdbf0, extra=104857600) at strbuf.c:98 > #4 0x000055555575731a in strbuf_read (sb=0x7fffffffdbf0, fd=4, hint=104857600) at strbuf.c:429 > #5 0x0000555555664a1f in apply_single_file_filter (path=0x5555558c787c "foo.rand", ...) > #6 0x0000555555665321 in apply_filter (path=0x5555558c787c "foo.rand", ...) > ... > > Looking at apply_single_file_filter(), it's not the _original_ file that > it's trying to store, but rather the data coming back from the filter. > It's just that we use the original file size as a hint! Really cool! I guessed as far as malloc() but did not actually test it, let alone examine the problem closely like this. -- Duy
