Re: git status OOM on mmap of large file

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Thu, Jan 24, 2019 at 12:39:49AM +0000, brian m. carlson wrote:

> > [pid  6573] mmap(NULL, 11811164160, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
> > 
> > Why status needs to mmap a large file that is not modified
> > and that is configured to pass through smudge/clean, I don't know.
> 
> I believe that currently, Git stores the smudge/clean output in memory
> until it writes it out. When using the persistent filter process, it's
> possible for the process to choose to abort the operation, so we store
> the data in memory until we get the status.

For the clean step, that should be OK, since that filter output is tiny
(but see my other message for a silly heuristic and an easy fix). And
that should be all that "git status" needs. But...

> Theoretically, it should be possible for us to write this to a temporary
> file, and if necessary, rename into place, although I'm not sure how
> well that will work on Windows. File modes may also be tricky here.
> Patches are of course welcome.

I didn't experiment with the smudge side, but I think it uses the same
apply_filter() code. Which means that yes, it would try to store the
11GB in memory before writing it out. And I agree writing it out to a
file and moving it directly into place is the sanest option there. If
that doesn't work, spooling to a tempfile and then streaming it into
place would also work.

-Peff



[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]

  Powered by Linux