Here is a solution that avoids problems with OS-specific implementations of SHA_Update() by limiting the size of each update request to 1GiB and calling the function repeatedly in a loop. Atousa -- [PATCH] Limit the size of the data block passed to SHA1_Update() This avoids issues where OS-specific implementations use a 32-bit integer to specify block size. Limit currently set to 1GiB. --- cache.h | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/cache.h b/cache.h index 79066e5..c305985 100644 --- a/cache.h +++ b/cache.h @@ -14,10 +14,28 @@ #ifndef git_SHA_CTX #define git_SHA_CTX SHA_CTX #define git_SHA1_Init SHA1_Init -#define git_SHA1_Update SHA1_Update #define git_SHA1_Final SHA1_Final #endif +#define SHA1_MAX_BLOCK_SIZE (1024*1024*1024) + +static inline int git_SHA1_Update(SHA_CTX *c, const void *data, size_t len) +{ + size_t nr; + size_t total = 0; + char *cdata = (char*)data; + while(len > 0) { + nr = len; + if(nr > SHA1_MAX_BLOCK_SIZE) + nr = SHA1_MAX_BLOCK_SIZE; + SHA1_Update(c, cdata, nr); + total += nr; + cdata += nr; + len -= nr; + } + return total; +} + #include <zlib.h> typedef struct git_zstream { z_stream z; -- 2.4.9 (Apple Git-60) On Thu, Oct 29, 2015 at 8:15 AM, Filipe Cabecinhas <filcab@xxxxxxxxx> wrote: > Defining BLK_SHA1 = YesPlease (when calling make) should just change > the SHA functions, instead of completely removing OpenSSL or > CommonCrypto. > > Regards, > Filipe > > > On Thu, Oct 29, 2015 at 3:46 AM, Rafael Espíndola > <rafael.espindola@xxxxxxxxx> wrote: >> Awesome, building with >> >> NO_OPENSSL = 1 >> NO_GETTEXT = 1 >> >> produces a working git :-) >> >> Cheers, >> Rafael >> >> >> On 28 October 2015 at 23:37, Filipe Cabecinhas <filcab@xxxxxxxxx> wrote: >>> I did some debugging, and it seems CC_SHA1_Update (used by >>> write_sha1_file_prepare if APPLE_COMMON_CRYPTO is defined in the Makefile) >>> takes a uint32_t as a "length" parameter, which explains why it stops >>> working at 4GiB (UINT_MAX+1). >>> >>> In the OS X 10.11 SDK header CommonCrypto/CommonDigest.h, we have: >>> >>> typedef uint32_t CC_LONG; /* 32 bit unsigned integer */ >>> //... >>> extern int CC_SHA1_Update(CC_SHA1_CTX *c, const void *data, CC_LONG len) >>> >>> A possible fix would be to either call SHA1_Update with a maximum of >>> UINT_MAX, looping if necessary. Or have a compatibility SHA1_Update for OS X >>> which can handle data longer than UINT_MAX. >>> >>> I'm not sure what the git maintainers would prefer. >>> >>> Regards, >>> >>> Filipe >>> >>> On Wed, Oct 28, 2015 at 4:10 PM, Rafael Espíndola >>> <rafael.espindola@xxxxxxxxx> wrote: >>>> >>>> I first noticed this with "2.4.9 (Apple Git-60)", but it reproduces >>>> with git built from 37023ba381b6d251d7140a997b39b566dbc63c42. >>>> >>>> Create two files with just 0s: >>>> >>>> -rw-r--r-- 1 espindola staff 4294967296 28 Oct 11:09 exactly-4gib >>>> -rw-r--r-- 1 espindola staff 4294967295 28 Oct 11:09 one-less-than-4gib >>>> >>>> >>>> and run >>>> >>>> git init >>>> git add one-less-than-4gib >>>> git commit -m bar >>>> git fsck >>>> git add exactly-4gib >>>> git commit -m bar >>>> git fsck >>>> >>>> The first fsck will run with no problems, but the second one fails: >>>> >>>> error: packed cfdaf54c9ccfd8f5e4cee562f7d5f92df13d3106 from >>>> .git/objects/pack/pack-ff08480fd7f767b6bd0aeb559f0f5dea2245b0b3.pack >>>> is corrupt >>>> >>>> Using the very same revision on freebsd doesn't cause any errors. >>>> >>>> Cheers, >>>> Rafael >>> >>> > -- > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Atousa Pahlevan, PhD M.Math. University of Waterloo, Canada Ph.D. Department of Computer Science, University of Victoria, Canada Voice: 415-341-6206 Email: apahlevan@xxxxxxxx Website: www.apahlevan.org -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html