This is the first "payoff" series in the sparse-index work. It makes 'git status' very fast when a sparse-index is enabled on a repository with cone-mode sparse-checkout (and a small populated set). This is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout. The latter branch is needed because it changes the behavior of 'git add' around sparse entries, which changes the expectations of a test added in patch 1. The approach here is to audit the places where ensure_full_index() pops up while doing normal commands with pathspecs within the sparse-checkout definition. Each of these are checked and tested. In the end, the sparse-index is integrated with these features: * git status * FS Monitor index extension. The performance tests in p2000-sparse-operations.sh improve by 95% or more, even when compared with the full-index cases, not just the sparse-index cases that previously had extra overhead. Hopefully this is the first example of how ds/sparse-index-protections has done the basic work to do these conversions safely, making them look easier than they seemed when starting this adventure. Thanks, -Stolee Update in V8 ============ * The directory/file conflict patch is removed and delayed to the next series where it will be required. (It will also be improved in that series.) * Some comments have been improved, including a new assert() that helps document the situation. Derrick Stolee (15): sparse-index: skip indexes with unmerged entries sparse-index: include EXTENDED flag when expanding t1092: replace incorrect 'echo' with 'cat' t1092: expand repository data shape t1092: add tests for status/add and sparse files unpack-trees: preserve cache_bottom unpack-trees: compare sparse directories correctly unpack-trees: rename unpack_nondirectories() unpack-trees: unpack sparse directory entries dir.c: accept a directory as part of cone-mode patterns diff-lib: handle index diffs with sparse dirs status: skip sparse-checkout percentage with sparse-index status: use sparse-index throughout wt-status: expand added sparse directory entries fsmonitor: integrate with sparse index builtin/commit.c | 3 + diff-lib.c | 19 +++ dir.c | 24 +++- read-cache.c | 10 +- sparse-index.c | 27 +++- t/t1092-sparse-checkout-compatibility.sh | 158 ++++++++++++++++++++++- t/t7519-status-fsmonitor.sh | 49 +++++++ unpack-trees.c | 142 +++++++++++++++++--- wt-status.c | 65 +++++++++- wt-status.h | 1 + 10 files changed, 464 insertions(+), 34 deletions(-) base-commit: d486ca60a51c9cb1fe068803c3f540724e95e83a Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-932%2Fderrickstolee%2Fsparse-index%2Fstatus-and-add-v8 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-932/derrickstolee/sparse-index/status-and-add-v8 Pull-Request: https://github.com/gitgitgadget/git/pull/932 Range-diff vs v7: 1: 2a4a7256304 = 1: 1815c148e8c sparse-index: skip indexes with unmerged entries 2: f5bae86014d = 2: 7bcde075d8d sparse-index: include EXTENDED flag when expanding 3: d965669c766 = 3: 05981e30b97 t1092: replace incorrect 'echo' with 'cat' 4: e10fa11cfdb ! 4: d38b66e9ee4 t1092: expand repository data shape @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'diff --staged' ' for branch in rename-out-to-out rename-out-to-in rename-in-to-out do test_all_match git checkout rename-base && - test_all_match git checkout $branch -- .&& + test_all_match git checkout $branch -- . && + test_all_match git status --porcelain=v2 && + test_all_match git diff --staged --no-renames && + test_all_match git diff --staged --find-renames || return 1 5: e94ffa07d46 = 5: 95ddd3abe4e t1092: add tests for status/add and sparse files 6: a8dda933567 = 6: b182b456613 unpack-trees: preserve cache_bottom 7: e52166f6e4c = 7: 988ddce4d45 unpack-trees: compare sparse directories correctly 8: d04b62381b8 = 8: d67ad048b08 unpack-trees: rename unpack_nondirectories() 9: 237ccf4e43d ! 9: c0b0b58584c unpack-trees: unpack sparse directory entries @@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info, + * Check for a sparse-directory entry named "path/". + * Due to the input p->path not having a trailing + * slash, the negative 'pos' value overshoots the -+ * expected position by at least one, hence "-2" here. ++ * expected position, hence "-2" instead of "-1". + */ + pos = -pos - 2; + @@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info, return NULL; + + /* -+ * We might have multiple entries between 'pos' and -+ * the actual sparse-directory entry, so start walking -+ * back until finding it or passing where it would be. ++ * Due to lexicographic sorting and sparse directory ++ * entried ending with a trailing slash, our path as a ++ * sparse directory (e.g "subdir/") and our path as a ++ * file (e.g. "subdir") might be separated by other ++ * paths (e.g. "subdir-"). + */ + while (pos >= 0) { + ce = o->src_index->cache[pos]; 10: 9f31c691af6 < -: ----------- unpack-trees: handle dir/file conflict of sparse entries 11: 2a43287c47e = 10: 76c7528f78f dir.c: accept a directory as part of cone-mode patterns 12: f83aa08ff6b ! 11: d875a7f8585 diff-lib: handle index diffs with sparse dirs @@ diff-lib.c: static int show_modified(struct rev_info *revs, unsigned dirty_submodule = 0; struct index_state *istate = revs->diffopt.repo->index; ++ assert(S_ISSPARSEDIR(old_entry->ce_mode) == ++ S_ISSPARSEDIR(new_entry->ce_mode)); ++ + /* + * If both are sparse directory entries, then expand the -+ * modifications to the file level. ++ * modifications to the file level. If only one was a sparse ++ * directory, then they appear as an add and delete instead of ++ * a modification. + */ -+ if (old_entry && new_entry && -+ S_ISSPARSEDIR(old_entry->ce_mode) && -+ S_ISSPARSEDIR(new_entry->ce_mode)) { ++ if (S_ISSPARSEDIR(new_entry->ce_mode)) { + diff_tree_oid(&old_entry->oid, &new_entry->oid, new_entry->name, &revs->diffopt); + return 0; + } 13: 35063ffb8ed = 12: 2b72cc2d985 status: skip sparse-checkout percentage with sparse-index 14: b4033a9bf36 = 13: 1c1feef3733 status: use sparse-index throughout 15: 717a3f49f97 = 14: dada1b91bdc wt-status: expand added sparse directory entries 16: 1d744848ee6 = 15: bdc771cf373 fsmonitor: integrate with sparse index -- gitgitgadget
