This series enables pack reverse-indexes to be written to disk by
default instead of computed on-the-fly in memory.
For repositories with large packs, this can have a significant benefit
for both end-users and Git forges. Extensive performance tests appear in
the fifth and sixth commit, but here are some highlights:
- The time it takes to generate a pack containing objects for the 10
most-recent commits in linux.git (starting from 68047c48b228) drops
from ~540ms to ~240ms, yielding a ~2.22x improvement.
- The time it takes to compute the on-disk object size of a single
object drops from ~300ms to 4ms, yielding a ~76.96x improvement.
Reverse indexes are a trade-off between the time they take to be
computed versus the time it takes to access a single record. In-memory
reverse indexes take time to generate proportional to the number of
packed objects in the repository, but have instantaneous lookup time.
On-disk reverse indexes can be "generated" instantly (the time it takes
to generate them is a one-time cost, loading them is instantaneous).
But individual record lookup time is slower, since it involves multiple
disk I/O operations.
In the vast majority of cases, this trade-off favors the on-disk ".rev"
files. But in certain cases, the in-memory variant performs more
favorably. Since these cases are narrow, and performance is machine- and
repository-dependent, this series also introduces a new configuration
option to disable reading ".rev" files in the third commit.
The series is structured as follows:
- A couple of cleanup patches to plug a leak in stage_tmp_packfiles().
- Three patches to enable `pack.readReverseIndex`.
- Another patch to change the default of `pack.writeReverseIndex` from
"false" to "true".
- A final patch to enable the test suite to be run in a mode that does
not use on-disk ".rev" files.
Thanks in advance for your review. I'm really excited to get this in the
hands of users after a couple of years of running this at GitHub (and
being opt-in otherwise).
Taylor Blau (7):
pack-write.c: plug a leak in stage_tmp_packfiles()
t5325: mark as leak-free
pack-revindex: make `load_pack_revindex` take a repository
pack-revindex: introduce GIT_TEST_REV_INDEX_DIE_ON_DISK
pack-revindex: introduce `pack.readReverseIndex`
config: enable `pack.writeReverseIndex` by default
t: invert `GIT_TEST_WRITE_REV_INDEX`
Documentation/config/pack.txt | 8 +++++++-
builtin/index-pack.c | 5 +++--
builtin/pack-objects.c | 5 +++--
ci/run-build-and-tests.sh | 1 -
pack-bitmap.c | 5 +++--
pack-revindex.c | 12 +++++++++---
pack-revindex.h | 6 ++++--
pack-write.c | 2 ++
packfile.c | 2 +-
repo-settings.c | 1 +
repository.h | 1 +
t/README | 2 +-
t/perf/p5312-pack-bitmaps-revs.sh | 3 +--
t/t5325-reverse-index.sh | 16 +++++++++++++++-
14 files changed, 51 insertions(+), 18 deletions(-)
--
2.40.0.323.g9c80379958
