Hi Jonathan, On Tue, 28 Aug 2018, Jonathan Nieder wrote: > Johannes Schindelin wrote: > > On Thu, 23 Aug 2018, Jonathan Nieder wrote: > > > Ævar Arnfjörð Bjarmason wrote: > > >>> Are we going to need a midx version of these mapping files? How does > >>> midx fit into this picture? Perhaps it's too obscure to worry > >>> about... > >> > >> That's a great question! I think the simplest answer is to have a > >> midx only for the primary object format and fall back to using > >> ordinary idx files for the others. > >> > >> The midx format already has a field for hash function (thanks, > >> Derrick!). > > > > Related: I wondered whether we could simply leverage the midx code for > > the bidirectional SHA-1 <-> SHA-256 mapping, as it strikes me as very > > similar in concept and challenges. > > Interesting: tell me more. > > My first instinct is to prefer the idx-based design that is already > described in the design doc. If we want to change that, we should > have a motivating reason. > > Midx is designed to be optional and to not necessarily cover all > objects, so it doesn't seem like a good fit. Right. What I meant was to leverage the midx code, not the .midx files. My comment was motivated by my realizing that both the SHA-1 <-> SHA-256 mapping and the MIDX code have to look up (in a *fast* way) information with hash values as keys. *And* this information is immutable. *And* the amount of information should grow with new objects being added to the database. I know that Stolee performed a bit of performance testing regarding different data structures to use in MIDX. We could benefit from that testing by using not only the results from those tests, but also the code. IIRC one of the insights was that packs are a natural structure that can be used for the MIDX mapping, too (you could, for example, store the SHA-1 <-> SHA-256 mapping *only* for objects inside packs, and re-generate them on the fly for loose objects all the time). Stolee can speak with much more competence and confidence about this, though, whereas all of what I said above is me waving my hands quite frantically. Ciao, Dscho
