Writing the index is a critical action that takes place in multiple Git commands. The recent performance improvements available with the sparse index show how often the I/O costs around the index can affect different Git commands, although reading the index takes place more often than a write. The index is written through the hashfile API, both as a buffered write and as a computation of its trailing hash. This trailing hash is an expectation of the file format: several optional index extensions are provided using indicators immediately preceding the trailing hash. 'git fsck' will complain if the trailing hash does not match the contents of the file up to that point. However, computing the hash is expensive. This is even more expensive now that we've moved to sha1dc instead of hardware-accelerated SHA1 computations. This series provides a new config option, index.skipHash, that allows users to disable computing the hash as Git writes the index. In this case, the trailing hash is stored as the null hash (all bytes are zero). The implementation is rather simple, but the patches organize different aspects in a way that we could split things out: * Patch 1 adds a 'skip_hash' option to 'struct hashfile' that creates this as a general option to the hashwrite API. * The motivation in Patch 1 avoids the talk about the chunk-format API and instead focuses on the index as the first example that could make use of this. * Patch 2 creates the index.skipHash config option and updates 'git fsck' to ignore a null hash on the index. This is demonstrated by a very simple test. * Patch 3 creates a test helper to get the trailing hash of a file, and adds a concrete check on the trailing hash when index.skipHash=true. * Patch 4 (which could be deferred until later) adds index.skipHash=true as an implication of feature.manyFiles and as a setting in Scalar. The end result is that index writes speed up significantly, enough that 'git update-index --force-write' improves by 1.75x. Similar patches appeared in my earlier refs RFC [1]. [1] https://lore.kernel.org/git/pull.1408.git.1667846164.gitgitgadget@xxxxxxxxx/ Updates since v3 ================ * Patch 1 uses an "int" instead of "unsigned int". This was a carry-over from Kevin's patch in microsoft/git, but our use of it in patch 2 is different and thus is better with a signed int. * Patch 2 uses a fallback to the_repository if istate->repo is NULL. This allows the setting to be applied to more cases where istate->repo is not set. * Patch 2 also uses 'start' in more places, since it already computed the beginning of the hash. * Patch 4 slightly modifies its use of prepare_repo_settings() due to Patch 2's fallback to the_repository. Updates since v2 ================ * Patch 2 now has additional details about why to use the config option in its message. The discussion about why the index is special is included in the commit message. Updates since v1 ================ * In Patch 1, use hashcler() instead of memset(). * In Patches 2 and 4, be explicit about which Git versions will exhibit different behavior when reading an index with a null trailing hash. * In Patch 2, used a more compact way of loading from config. * In Patch 2 (and following) dropped the sub-shell in t1600-index.sh. * In Patch 3, dropped back-ticks when using a multi-line pipe. * In Patch 4, use "! cmp" instead of "! test_cmp" Updates since RFC ================= * The config name is now index.skipHash to make it more clear that the default is to not skip the hash, and choosing this option requires a true value. * Use oidread() instead of an ad-hoc loop. * Patches 3 and 4 are new. 3 adds some extra test verification that wasn't in the RFC. * I think that patch 4 is helpful to see now, but I could understand if we wanted to defer that patch until after index.skipHash has had more time to bake. Thanks, -Stolee Derrick Stolee (4): hashfile: allow skipping the hash function read-cache: add index.skipHash config option test-lib-functions: add helper for trailing hash features: feature.manyFiles implies fast index writes Documentation/config/feature.txt | 5 +++++ Documentation/config/index.txt | 11 +++++++++++ csum-file.c | 14 +++++++++++--- csum-file.h | 7 +++++++ read-cache.c | 14 +++++++++++++- repo-settings.c | 2 ++ repository.h | 1 + scalar.c | 1 + t/t1600-index.sh | 20 ++++++++++++++++++++ t/test-lib-functions.sh | 8 ++++++++ 10 files changed, 79 insertions(+), 4 deletions(-) base-commit: 805265fcf7a737664a8321aaf4a0587b78435184 Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1439%2Fderrickstolee%2Fhashfile-skip-hash-v4 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1439/derrickstolee/hashfile-skip-hash-v4 Pull-Request: https://github.com/gitgitgadget/git/pull/1439 Range-diff vs v3: 1: b582d681581 ! 1: c99470d4676 hashfile: allow skipping the hash function @@ csum-file.h: struct hashfile { unsigned char *check_buffer; + + /** -+ * If set to 1, skip_hash indicates that we should ++ * If non-zero, skip_hash indicates that we should + * not actually compute the hash for this hashfile and + * instead only use it as a buffered write. + */ -+ unsigned int skip_hash; ++ int skip_hash; }; /* Checkpoint */ 2: 00738c81a12 ! 2: aae047cbc9f read-cache: add index.skipHash config option @@ Commit message [1] https://github.com/microsoft/git/commit/21fed2d91410f45d85279467f21d717a2db45201 + We load this config from the repository config given by istate->repo, + with a fallback to the_repository if it is not set. + While older Git versions will not recognize the null hash as a special case, the file format itself is still being met in terms of its structure. Using this null hash will still allow Git operations to @@ read-cache.c: static int verify_hdr(const struct cache_header *hdr, unsigned lon the_hash_algo->update_fn(&c, hdr, size - the_hash_algo->rawsz); the_hash_algo->final_fn(hash, &c); - if (!hasheq(hash, (unsigned char *)hdr + size - the_hash_algo->rawsz)) -+ if (!hasheq(hash, end - the_hash_algo->rawsz)) ++ if (!hasheq(hash, start)) return error(_("bad index file sha1 signature")); return 0; } @@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempfile *tempfile, + int ieot_entries = 1; + struct index_entry_offset_table *ieot = NULL; + int nr, nr_threads; ++ struct repository *r = istate->repo ? istate->repo : the_repository; f = hashfd(tempfile->fd, tempfile->filename.buf); -+ git_config_get_maybe_bool("index.skiphash", (int *)&f->skip_hash); ++ repo_config_get_maybe_bool(r, "index.skiphash", &f->skip_hash); + for (i = removed = extended = 0; i < entries; i++) { if (cache[i]->ce_flags & CE_REMOVE) 3: 86370af1351 = 3: 27fbe52e748 test-lib-functions: add helper for trailing hash 4: 6490bd445eb ! 4: 075921514f2 features: feature.manyFiles implies fast index writes @@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempf f = hashfd(tempfile->fd, tempfile->filename.buf); -- git_config_get_maybe_bool("index.skiphash", (int *)&f->skip_hash); -+ if (istate->repo) { -+ prepare_repo_settings(istate->repo); -+ f->skip_hash = istate->repo->settings.index_skip_hash; -+ } +- repo_config_get_maybe_bool(r, "index.skiphash", &f->skip_hash); ++ prepare_repo_settings(r); ++ f->skip_hash = r->settings.index_skip_hash; for (i = removed = extended = 0; i < entries; i++) { if (cache[i]->ce_flags & CE_REMOVE) -- gitgitgadget
