> On 23/12/2019, at 2:13 PM, brian m. carlson <sandals@xxxxxxxxxxxxxxxxxxxx> wrote: > > On 2019-12-22 at 06:41:33, Michael Clark wrote: >> Recommendations >> =============== >> >> After performing multiple benchmarks and thinking about this for >> several weeks I have come to the conclusion that SHA512/256 is a >> very good alternative to SHA256, both from the perspective of its >> length extension resistance, and performance. It seems that NewHash >> implementation is some way off so it is not too late to change. >> >> - Consider SHA512/256 for NewHash > > This has been discussed extensively on the list. Git is not in general > vulnerable to length extension attacks because it hashes the length of > the object as part of the object header. This was documented as one of > the known aspects in the transition plan in > Documentation/technical/hash-function-transition.txt. > > We've considered a lot of hash algorithms, including BLAKE2b-256, > SHA-256, SHA-512/256, SHA-3-256, SHAKE256 with a 256-bit length, and > others. This was no doubt controversial, but we discussed the pros and > cons of many options and this is what we decided. I will state that > while our choice was not one of my more preferred options, I value that > we have a consensus on the way forward. > > Among the reasons we chose SHA-256 is performance on newer x86-64 > processors with the SHA extensions, which support only SHA-1 and > SHA-256, and compatibility with a wide variety of cryptographic > libraries. I have no desire for Git to provide assembly implementations > of NewHash, and that means people must rely on existing cryptographic > libraries. the SHA extensions that are not generally available, which include SHA1 on the cusp of its deprecation. It seems silly not to support SHA512 when implementing SHA2 support especially considering it’s faster on 64-bit. It’s used in PBKDF2-SHA512 for iterated password hashes. SHA512/256 is just a different IV. Don’t be discouraged by my comments. Thanks for taking the time to describe the state. The rationale of pragmatism should be in the hash migration plan. SHA256 will always take more energy than SHA512 which actually means is possibly slightly weaker, which could be used as a rationale for choosing SHA256. I don’t think the other rationale is acceptable though because the choice of purity of choice should lead what goes into hardware. I want pure hashes of the unpacked object so I will have to think about how I do that. Possibly SHA-3 too. git doesn’t really support large files that well either so I’ll have to keep that in mind... If the algorithm is not vulnerable to length extensions, then the length prefix is not required. Anyhow... I’ll have to hack up some shell scripts or something in the meantime. > Cryptographic libraries overwhelmingly do not support SHA-512/256. > While OpenSSL does, it's only in the latest versions, and people > generally cannot legally distribute both Git and OpenSSL linked > together, so they must rely on system libraries for good performance. > Apple Common Crypto, for example, does not document any SHA-512/256 > implementations. > > There is also a benefit to chaining using a single algorithm for > signatures, which therefore necessitates using SHA-256 or one of the > other original SHA-2 algorithms until the next version of OpenPGP is > implemented which supports SHA-3. > > Because we decided some time ago, I've sent in a bunch of patches to our > testsuite to make it work with SHA-256. Some of these patches are > general, in that they make the tests generate values which are used, or > they are specific to the length of the hash algorithm. Others use > specific hash values, and changing the hash algorithm will require > recomputing all of these values. > > Absent a compelling security reason to abandon SHA-256, such as a > significant previously unknown cryptographic weakness, I don't plan to > reimplement all of this work. Updating our testsuite to work > successfully with SHA-256 has taken a huge amount of time, and this work > has been entirely done on my own free time because I want the Git > project to be successful. That doesn't even include the contributions > of others who have reviewed, discussed, and contributed to the current > work and transition plan. While I appreciate the benefits of > alternatives to SHA-256, I'm a bit annoyed that you're proposing to > throw away all of this work after we explicitly sought to build > consensus about an algorithm so that this stage of the work could begin. > > Of course, nothing prevents you from proposing a new algorithm and > fixing the testsuite to make it work, but I don't believe that we're > likely to adopt a different algorithm at this point, given the previous > discussion on the list. The code is designed to be pluggable to support > arbitrary algorithms, but there are pieces which assume that the > algorithm can be distinguished by length (e.g., the dumb HTTP protocol) > and therefore you'd need to devise a different way to distinguish these > versions. > > If you'd like to see what a Git binary that supports multiple hash > algorithms looks like at the moment, you're welcome to look at my > transition-stage-4 branch at https://github.com/bk2204/git.git so you > can get a fuller understanding of what's involved. You can run the > testsuite with that code and GIT_TEST_DEFAULT_HASH=<hashname> to see > what passes and fails. Thanks. I’ll check it out and try it. My comments aren’t directed at you. I’m interested in your work. Keep it up. Well, take a break. idk. Whatever. I’ll probably be spending my holidays reading git code, or maybe libgit2 :-p >> SHA algorithm patches >> ===================== >> >> I will be sending a patch series which adds SHA Algorithms which >> are in the sha-algorithms branch of my git tree. >> >> - https://github.com/michaeljclark/git.git >> - https://github.com/michaeljclark/git/tree/sha-algorithms/sha >> >> This is a precursor to asking some questions about the proposed >> implementation of multiple or compat hashs (N-hash or 2-hash?). >> I am curious about the pack format and directory structure for >> loose objects. It seems there needs to be a single hash chosen >> for loose object filenames, and auxilliary hashes will need >> hash to hash map to go from a compat hash to new hash. > > That's correct. There's documentation in the transition plan about what > this will look like. > > A stage 4 implementation (where repositories are either SHA-1 only or > SHA-256 only) has been written and I'll send it out once the remaining > test fixes are merged. > > I should point out that we specifically chose not to support a plethora > of algorithms because the goal is to encourage everybody to adopt a > single, secure algorithm. The network protocol will need to > interoperate using a single algorithm, which means the repository will > need to contain a mapping for that algorithm. Since the current plan is > to support no more than two algorithms at a time, one of which will need > to be SHA-1, allowing users to pick and choose from a variety of > algorithms harms interoperability. Seamless interoperability between > SHA-1 and NewHash repositories was considered extremely important in the > transition plan. That won't be coming immediately with the stage 4 > implementation, but it is planned for the future. > > My hope is that whatever we pick now will be secure for some time to > come and that when we need to adopt a newer hash, whether due to > cryptographic advance or post-quantum cryptography, there will be no > existing SHA-1 repositories and transition can take place from NewHash > to NewNewHash. > -- > brian m. carlson: Houston, Texas, US > OpenPGP: https://keybase.io/bk2204
