Re: [PATCH 2/2] index-pack: reduce memory usage when the pack has large blobs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Sat, Feb 25, 2012 at 08:49:55AM +0700, Nguyen Thai Ngoc Duy wrote:
> 2012/2/24 Ian Kumlien <pomac@xxxxxxxxx>:
> > Writing objects: 100% (1425/1425), 56.06 MiB | 4.62 MiB/s, done.
> > Total 1425 (delta 790), reused 1425 (delta 790)
> > fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
> > fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
> > fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
> > fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
> > To ../test_data/
> >  ! [remote rejected] master -> master (missing necessary objects)
> >  ! [remote rejected] origin/HEAD -> origin/HEAD (missing necessary objects)
> >  ! [remote rejected] origin/master -> origin/master (missing necessary objects)
> > error: failed to push some refs to '../test_data/'
> >
> > So there are additional code paths to look at... =(
> 
> I can't say where that came from. Does this help? (Space damaged, may
> need manual application)

> If not, you might need to apply this to generate coredump, then look
> and see where that failed malloc comes from

Actually, i added a backtrace and used addr2line to confirm my
suspicion... which is:
builtin/index-pack.c:414

ie get_data_from_pack... 

It looks to me like, if we are to support this kind of things, we need a
slightly different approach - instead of passing the data around, it
feels like passing a function pointer around would be beneficial.

Looking at the code i see alot of places where this would be a issue,
just the fact that get_data_from_pack is used in several functions that
might do some small operation and then just free it.

I understand and recognize that my "problem" is not what git was
designed for; it was designed for small files, which is very evident in
how it approaches the data... And I'd most definetly have to look alot
closer to this code... =)

> -- 
> Duy
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]