Here is a solution that avoids problems with OS-specific
implementations of SHA_Update() by limiting the size of each update
request to 1GiB and calling the function repeatedly in a loop.
Atousa
--
[PATCH] Limit the size of the data block passed to SHA1_Update()
This avoids issues where OS-specific implementations use
a 32-bit integer to specify block size. Limit currently
set to 1GiB.
---
cache.h | 20 +++++++++++++++++++-
1 file changed, 19 insertions(+), 1 deletion(-)
diff --git a/cache.h b/cache.h
index 79066e5..c305985 100644
--- a/cache.h
+++ b/cache.h
@@ -14,10 +14,28 @@
#ifndef git_SHA_CTX
#define git_SHA_CTX SHA_CTX
#define git_SHA1_Init SHA1_Init
-#define git_SHA1_Update SHA1_Update
#define git_SHA1_Final SHA1_Final
#endif
+#define SHA1_MAX_BLOCK_SIZE (1024*1024*1024)
+
+static inline int git_SHA1_Update(SHA_CTX *c, const void *data, size_t len)
+{
+ size_t nr;
+ size_t total = 0;
+ char *cdata = (char*)data;
+ while(len > 0) {
+ nr = len;
+ if(nr > SHA1_MAX_BLOCK_SIZE)
+ nr = SHA1_MAX_BLOCK_SIZE;
+ SHA1_Update(c, cdata, nr);
+ total += nr;
+ cdata += nr;
+ len -= nr;
+ }
+ return total;
+}
+
#include <zlib.h>
typedef struct git_zstream {
z_stream z;
--
2.4.9 (Apple Git-60)
On Thu, Oct 29, 2015 at 8:15 AM, Filipe Cabecinhas <filcab@xxxxxxxxx> wrote:
> Defining BLK_SHA1 = YesPlease (when calling make) should just change
> the SHA functions, instead of completely removing OpenSSL or
> CommonCrypto.
>
> Regards,
> Filipe
>
>
> On Thu, Oct 29, 2015 at 3:46 AM, Rafael Espíndola
> <rafael.espindola@xxxxxxxxx> wrote:
>> Awesome, building with
>>
>> NO_OPENSSL = 1
>> NO_GETTEXT = 1
>>
>> produces a working git :-)
>>
>> Cheers,
>> Rafael
>>
>>
>> On 28 October 2015 at 23:37, Filipe Cabecinhas <filcab@xxxxxxxxx> wrote:
>>> I did some debugging, and it seems CC_SHA1_Update (used by
>>> write_sha1_file_prepare if APPLE_COMMON_CRYPTO is defined in the Makefile)
>>> takes a uint32_t as a "length" parameter, which explains why it stops
>>> working at 4GiB (UINT_MAX+1).
>>>
>>> In the OS X 10.11 SDK header CommonCrypto/CommonDigest.h, we have:
>>>
>>> typedef uint32_t CC_LONG; /* 32 bit unsigned integer */
>>> //...
>>> extern int CC_SHA1_Update(CC_SHA1_CTX *c, const void *data, CC_LONG len)
>>>
>>> A possible fix would be to either call SHA1_Update with a maximum of
>>> UINT_MAX, looping if necessary. Or have a compatibility SHA1_Update for OS X
>>> which can handle data longer than UINT_MAX.
>>>
>>> I'm not sure what the git maintainers would prefer.
>>>
>>> Regards,
>>>
>>> Filipe
>>>
>>> On Wed, Oct 28, 2015 at 4:10 PM, Rafael Espíndola
>>> <rafael.espindola@xxxxxxxxx> wrote:
>>>>
>>>> I first noticed this with "2.4.9 (Apple Git-60)", but it reproduces
>>>> with git built from 37023ba381b6d251d7140a997b39b566dbc63c42.
>>>>
>>>> Create two files with just 0s:
>>>>
>>>> -rw-r--r-- 1 espindola staff 4294967296 28 Oct 11:09 exactly-4gib
>>>> -rw-r--r-- 1 espindola staff 4294967295 28 Oct 11:09 one-less-than-4gib
>>>>
>>>>
>>>> and run
>>>>
>>>> git init
>>>> git add one-less-than-4gib
>>>> git commit -m bar
>>>> git fsck
>>>> git add exactly-4gib
>>>> git commit -m bar
>>>> git fsck
>>>>
>>>> The first fsck will run with no problems, but the second one fails:
>>>>
>>>> error: packed cfdaf54c9ccfd8f5e4cee562f7d5f92df13d3106 from
>>>> .git/objects/pack/pack-ff08480fd7f767b6bd0aeb559f0f5dea2245b0b3.pack
>>>> is corrupt
>>>>
>>>> Using the very same revision on freebsd doesn't cause any errors.
>>>>
>>>> Cheers,
>>>> Rafael
>>>
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Atousa Pahlevan, PhD
M.Math. University of Waterloo, Canada
Ph.D. Department of Computer Science, University of Victoria, Canada
Voice: 415-341-6206
Email: apahlevan@xxxxxxxx
Website: www.apahlevan.org
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html