On 11/14/2017 10:04 AM, Junio C Hamano wrote:
Ben Peart <peartben@xxxxxxxxx> writes:
How about I add the logic to write out a special extension right
before the SHA1 that contains an offset to the beginning of the
extensions section. I will also add the logic in do_read_index() to
search for and load this special extension if it exists.
This will provide a common framework for any future extension to take
advantage of if it wants to be loaded/processed before or in parallel
with the cache entries or other extensions.
For all existing extensions that assume they are loaded _after_ the
cache entries, in do_read_index() I'll add the logic to use the offset
(if it exists) to adjust the src_offset and then load them normally.
Given the IEOT extension is just another list of offsets into the
index to enable out of order processing, I'll add those offsets into
the same extension so that it is a more generic "table of contents"
for the entire index. This enables us to have common/reusable way to
have random access to _all_ sections in the index while maintaining
backwards comparability with the existing index formats and code.
These additional offsets will initially only be used to parallelize
the loading of cache entries and only if the user explicitly enables
that option but I can think of other interesting uses for them in the
future.
If we freeze the format of IEOT extension so that we can guarantee
that the very first version of Git that understands IEOT can always
find the beginning of extension section in an index file that was
written by future versions of Git, then I'm all for that plan, but
my impression was that you are planning to make incompatible change
in the future to IEOT, judging from the way that IEOT records its own
version number in the section and the reader uses it to reject an
unknown one.
I have no thoughts or plans for changes in the future of IEOT (which I
plan to rename ITOC). At this point in time, I can't even imagine what
else we'd want as the index only contains cache entries, extensions and
the trailing SHA1 and with the TOC we have random access to each of
those. If nothing else, it gives another signature to verify to ensure
we actually have a valid trailing extension. :)
I only added the version because I've learned my ability to predict the
future is pretty bad and it leaves us that option _if_ something ever
comes up that we need it _and_ are willing to live with the significant
negative trade-offs you correctly point out.
With that plan, what I suspect would happen is that a version of Git
that understands another optional extension section that wants to be
findable without scanning the main table and the then-current
version of IEOT would not be able to use an index file written by a
new version of Git that enhances the format of the IEOT extension
bumps its extension version.
And if that is the case I would have to say that I strongly suspect
that you would regret the design decision to mix it into IEOT. That
is why I keep suggesting that the back pointer extension should be
on its own, minimizing what it does and minimizing the need to be
updated across versions of Git.
I understand the risk but the list of offsets into the cache entries is
pretty simple as well. I prefer the simplicity of a single TOC extension
that gives us random access to the entire index rather than having to
piece one together using multiple extensions. That model has its own
set of risks and tradeoffs.
