On Fri, Feb 24, 2012 at 07:23:21PM +0700, Nguyễn Thái Ngọc Duy wrote: > This command unpacks every non-delta objects in order to: > > 1. calculate sha-1 > 2. do byte-to-byte sha-1 collision test if we happen to have objects > with the same sha-1 > 3. validate object content in strict mode > > All this requires the entire object to stay in memory, a bad news for > giant blobs. This patch lowers memory consumption by not saving the > object in memory whenever possible, calculating SHA-1 while unpacking > the object. > > This patch assumes that the collision test is rarely needed. The > collision test will be done later in second pass if necessary, which > puts the entire object back to memory again (We could even do the > collision test without putting the entire object back in memory, by > comparing as we unpack it). > > In strict mode, it always keeps non-blob objects in memory for > validation (blobs do not need data validation). "--strict --verify" > also keeps blobs in memory. > > Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@xxxxxxxxx> Writing objects: 100% (1425/1425), 56.06 MiB | 4.62 MiB/s, done. Total 1425 (delta 790), reused 1425 (delta 790) fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes) fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes) fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes) fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes) To ../test_data/ ! [remote rejected] master -> master (missing necessary objects) ! [remote rejected] origin/HEAD -> origin/HEAD (missing necessary objects) ! [remote rejected] origin/master -> origin/master (missing necessary objects) error: failed to push some refs to '../test_data/' So there are additional code paths to look at... =( -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html