Re: git fast-import leaks memory drastically, so crashes with out of memory by attempt to import 22MB export dump

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Am 15.10.20 um 13:50 schrieb Dipl. Ing. Sergey Brester:
> ```
> # clone or copy git repository (we'll use it for export and import):
> git clone https://github.com/git/git.git
> cd git
>
> # make 1st fast-export in order to generate marks (don't need the dump, only marks are needed):
> git fast-export --reencode=yes --export-marks=.tmp.exp-1.tmp e83c5163316f89bfbde7d9ab23ca2e25604af290..dc04167d378fb29d30e1647ff6ff51dd182bc9a3 > /dev/null
>
> # make 2nd fast-export in order to generate partial export dump (file ".tmp.dump" will be ca. 87MB):
> git fast-export --reencode=yes --import-marks=.tmp.exp-1.tmp --export-marks=.tmp.exp-2.tmp 61addb841f2a6d74a1737a01e03df1f773e04944..master > .tmp.dump

In case someone else is wondering about the meaning of those hashes,
here are their reference strings:

e83c516331 (Initial revision of "git", the information manager from hell, 2005-04-07)
dc04167d37 (Fourth batch, 2020-08-04)
61addb841f (Merge branch 'jk/strvec' into next, 2020-08-04)

> # now try to import this dump, using first marks as import marks (we have all revs in git-repo):
> git fast-import --import-marks=.tmp.exp-1.tmp --export-marks=.tmp.imp.tmp < .tmp.dump
>
> ```

So you use the marks generated by the first export to import the second
export.  I wonder if that's relevant to trigger the memory allocation
issue.

> And see how git-fast-import eating your whole memory and enjoy the crash :)
>
> ```
> fatal: Out of memory, malloc failed (tried to allocate 2097152 bytes)
> fast-import: dumping crash report to .git/fast_import_crash_6684
> ```

I can reproduce this on Debian -- but don't get a crash report, just a
notice from the OOM killer.  It bisects to ddddf8d7e2 (fast-import:
permit reading multiple marks files, 2020-02-22).

René




[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]

  Powered by Linux