I received an internal bug report that after upgrading from v2.39.2 to v2.40.0, some users ran into the following error message: BUG: fsmonitor.c:21: fsmonitor_dirty has more entries than the index (57 > 0) It sounds very much like the report we received in https://lore.kernel.org/git/CAC7ZvybvykKQyMWcZoKXxFDu_amnkxZCDq2C6KHoyhmHN2tcKw@xxxxxxxxxxxxxx/, but sadly that thread petered out when the reporter stopped being able to reproduce the problem. After a few days of investigating, I am convinced that this is due to some old bugs, and not actually a regression in v2.40.0 (although I can believe that some improvements in v2.40.0 might make it easier to run into these bugs). This patch series addresses those bugs. Note: While the Git maintainer has stated a strong preference to introduce regression tests in the same patch that fixes the corresponding regression, this patch series starts with a stand-alone patch that demonstrates a problematic scenario via a new test_expect_failure test case. The reason why I specifically split out the test into its own commit is that there is a lot of information to unpack in the commit message that is larger than any of the subsequent bug fixes. Besides, it motivates not only the second patch (which marks the test case as test_expect_success) but paints the larger picture necessary to understand also the need for the remaining two patches. This patch series is based on maint-2.37, the oldest maintenance branch it applies without merge conflicts. When merging with next, there are only trivial conflicts in unpack-trees.c due to en/dir-api-cleanup where o->result is now o->internal.result. Changes since v1: * Fix a double "the" in a commit message * Replace enum strip_extensions by the bit field enum write_extensions, inverting the meaning of the values to avoid double negatives * Leave a trailing comma at the definition of the enum values Johannes Schindelin (4): split-index & fsmonitor: demonstrate a bug split-index; stop abusing the `base_oid` to strip the "link" extension fsmonitor: avoid overriding `cache_changed` bits unpack-trees: take care to propagate the split-index flag fsmonitor.h | 2 +- read-cache.c | 49 +++++++++++++++++++++++------------- t/t7527-builtin-fsmonitor.sh | 37 +++++++++++++++++++++++++++ unpack-trees.c | 2 ++ 4 files changed, 72 insertions(+), 18 deletions(-) base-commit: eb88fe1ff5ceb34845f0919b8bdc60d8a1703cf6 Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1497%2Fdscho%2Ffix-split-index-v2 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1497/dscho/fix-split-index-v2 Pull-Request: https://github.com/gitgitgadget/git/pull/1497 Range-diff vs v1: 1: c025fccbdde = 1: c025fccbdde split-index & fsmonitor: demonstrate a bug 2: f1897b88072 ! 2: 8cc075f6325 split-index; stop abusing the `base_oid` to strip the "link" extension @@ Commit message One might be tempted to simply call `discard_split_index()` instead, under the assumption that Git decided to write a non-split index and - therefore the the `split_index` structure might no longer be wanted. + therefore the `split_index` structure might no longer be wanted. However, that is not possible because that would release index entries in `split_index->base` that are likely to still be in use. Therefore we cannot do that. - The next best thing we _can_ do is to introduce a flag, specifically - indicating when the "link" extension should be skipped. So that's what - we do here. + The next best thing we _can_ do is to introduce a bit field to indicate + specifically which index extensions (not) to write. So that's what we do + here. Signed-off-by: Johannes Schindelin <johannes.schindelin@xxxxxx> @@ read-cache.c: static int record_ieot(void) return !git_config_get_index_threads(&val) && val != 1; } -+enum strip_extensions { -+ WRITE_ALL_EXTENSIONS = 0, -+ STRIP_ALL_EXTENSIONS = 1, -+ STRIP_LINK_EXTENSION_ONLY = 2 ++enum write_extensions { ++ WRITE_NO_EXTENSION = 0, ++ WRITE_SPLIT_INDEX_EXTENSION = 1<<0, ++ WRITE_CACHE_TREE_EXTENSION = 1<<1, ++ WRITE_RESOLVE_UNDO_EXTENSION = 1<<2, ++ WRITE_UNTRACKED_CACHE_EXTENSION = 1<<3, ++ WRITE_FSMONITOR_EXTENSION = 1<<4, +}; ++#define WRITE_ALL_EXTENSIONS ((enum write_extensions)-1) + /* * On success, `tempfile` is closed. If it is the temporary file @@ read-cache.c: static int record_ieot(void) */ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, - int strip_extensions, unsigned flags) -+ enum strip_extensions strip_extensions, unsigned flags) ++ enum write_extensions write_extensions, unsigned flags) { uint64_t start = getnanotime(); struct hashfile *f; @@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempf } - if (!strip_extensions && istate->split_index && -+ if (strip_extensions == WRITE_ALL_EXTENSIONS && istate->split_index && - !is_null_oid(&istate->split_index->base_oid)) { +- !is_null_oid(&istate->split_index->base_oid)) { ++ if (write_extensions & WRITE_SPLIT_INDEX_EXTENSION && ++ istate->split_index) { struct strbuf sb = STRBUF_INIT; + if (istate->sparse_index) @@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempfile *tempfile, if (err) return -1; } - if (!strip_extensions && !drop_cache_tree && istate->cache_tree) { -+ if (strip_extensions != STRIP_ALL_EXTENSIONS && !drop_cache_tree && istate->cache_tree) { ++ if (write_extensions & WRITE_CACHE_TREE_EXTENSION && ++ !drop_cache_tree && istate->cache_tree) { struct strbuf sb = STRBUF_INIT; cache_tree_write(&sb, istate->cache_tree); @@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempf return -1; } - if (!strip_extensions && istate->resolve_undo) { -+ if (strip_extensions != STRIP_ALL_EXTENSIONS && istate->resolve_undo) { ++ if (write_extensions & WRITE_RESOLVE_UNDO_EXTENSION && ++ istate->resolve_undo) { struct strbuf sb = STRBUF_INIT; resolve_undo_write(&sb, istate->resolve_undo); @@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempf return -1; } - if (!strip_extensions && istate->untracked) { -+ if (strip_extensions != STRIP_ALL_EXTENSIONS && istate->untracked) { ++ if (write_extensions & WRITE_UNTRACKED_CACHE_EXTENSION && ++ istate->untracked) { struct strbuf sb = STRBUF_INIT; write_untracked_extension(&sb, istate->untracked); @@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempf return -1; } - if (!strip_extensions && istate->fsmonitor_last_update) { -+ if (strip_extensions != STRIP_ALL_EXTENSIONS && istate->fsmonitor_last_update) { ++ if (write_extensions & WRITE_FSMONITOR_EXTENSION && ++ istate->fsmonitor_last_update) { struct strbuf sb = STRBUF_INIT; write_fsmonitor_extension(&sb, istate); @@ read-cache.c: static int commit_locked_index(struct lock_file *lk) return commit_lock_file(lk); } -+/* -+ * Write the Git index into a `.lock` file -+ * -+ * If `strip_link_extension` is non-zero, avoid writing any "link" extension -+ * (used by the split-index feature). -+ */ - static int do_write_locked_index(struct index_state *istate, struct lock_file *lock, +-static int do_write_locked_index(struct index_state *istate, struct lock_file *lock, - unsigned flags) -+ unsigned flags, int strip_link_extension) ++static int do_write_locked_index(struct index_state *istate, ++ struct lock_file *lock, ++ unsigned flags, ++ enum write_extensions write_extensions) { int ret; int was_full = istate->sparse_index == INDEX_EXPANDED; @@ read-cache.c: static int do_write_locked_index(struct index_state *istate, struc trace2_region_enter_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); - ret = do_write_index(istate, lock->tempfile, 0, flags); -+ ret = do_write_index(istate, lock->tempfile, strip_link_extension ? STRIP_LINK_EXTENSION_ONLY : 0, flags); ++ ret = do_write_index(istate, lock->tempfile, write_extensions, flags); trace2_region_leave_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); @@ read-cache.c: static int write_split_index(struct index_state *istate, int ret; prepare_to_write_split_index(istate); - ret = do_write_locked_index(istate, lock, flags); -+ ret = do_write_locked_index(istate, lock, flags, 0); ++ ret = do_write_locked_index(istate, lock, flags, WRITE_ALL_EXTENSIONS); finish_writing_split_index(istate); return ret; } +@@ read-cache.c: static int write_shared_index(struct index_state *istate, + + trace2_region_enter_printf("index", "shared/do_write_index", + the_repository, "%s", get_tempfile_path(*temp)); +- ret = do_write_index(si->base, *temp, 1, flags); ++ ret = do_write_index(si->base, *temp, WRITE_NO_EXTENSION, flags); + trace2_region_leave_printf("index", "shared/do_write_index", + the_repository, "%s", get_tempfile_path(*temp)); + @@ read-cache.c: int write_locked_index(struct index_state *istate, struct lock_file *lock, if ((!si && !test_split_index_env) || alternate_index_output || @@ read-cache.c: int write_locked_index(struct index_state *istate, struct lock_fil - if (si) - oidclr(&si->base_oid); - ret = do_write_locked_index(istate, lock, flags); -+ ret = do_write_locked_index(istate, lock, flags, 1); ++ ret = do_write_locked_index(istate, lock, flags, ++ ~WRITE_SPLIT_INDEX_EXTENSION); goto out; } @@ read-cache.c: int write_locked_index(struct index_state *istate, struct lock_fil if (!temp) { - oidclr(&si->base_oid); - ret = do_write_locked_index(istate, lock, flags); -+ ret = do_write_locked_index(istate, lock, flags, 1); ++ ret = do_write_locked_index(istate, lock, flags, ++ ~WRITE_SPLIT_INDEX_EXTENSION); goto out; } ret = write_shared_index(istate, &temp, flags); 3: c1c35f0f026 = 3: 89b3cd9a668 fsmonitor: avoid overriding `cache_changed` bits 4: 3963d3e5428 = 4: df61146eaf5 unpack-trees: take care to propagate the split-index flag -- gitgitgadget
