Here is the second patch series submission coming out of the sparse-index RFC [1]. [1] https://lore.kernel.org/git/pull.847.git.1611596533.gitgitgadget@xxxxxxxxx/ This is based on v3 of the format series [2]. [2] https://lore.kernel.org/git/pull.883.v3.git.1615912983.gitgitgadget@xxxxxxxxx/ The point of this series is to insert protections for the consumers of the in-memory index. We mark certain regions of code as needing a full index, so we call ensure_full_index() to expand a sparse index to a full one, if necessary. These protections are inserted file-by-file in every loop over all cache entries. Well, "most" loops, because some are going to be handled in the very next series so I leave them out. Many callers use index_name_pos() to find a path by name. In these cases, we can check if that position resolves to a sparse directory instance. In those cases, we just expand to a full index and run the search again. The last few patches deal with the name-hash hashtable for doing O(1) lookups. These protections don't do much right now, since the previous series created the_repository->settings.command_requires_full_index to guard all index reads and writes to ensure the in-memory copy is full for commands that have not been tested with the sparse index yet. However, after this series is complete, we now have a straight-forward plan for making commands "sparse aware" one-by-one: 1. Disable settings.command_requires_full_index to allow an in-memory sparse-index. 2. Run versions of that command under a debugger, breaking on ensure_full_index(). 3. Examine the call stack to determine the context of that expansion, then implement the proper behavior in those locations. 4. Add tests to ensure we are checking this logic in the presence of sparse directory entries. I will admit that mostly it is the writing of the test cases that takes the most time in the conversions I've done so far. Thanks, -Stolee Derrick Stolee (27): *: remove 'const' qualifier for struct index_state read-cache: expand on query into sparse-directory entry sparse-index: API protection strategy cache: move ensure_full_index() to cache.h add: ensure full index checkout-index: ensure full index checkout: ensure full index commit: ensure full index difftool: ensure full index fsck: ensure full index grep: ensure full index ls-files: ensure full index merge-index: ensure full index rm: ensure full index sparse-checkout: ensure full index update-index: ensure full index diff-lib: ensure full index dir: ensure full index entry: ensure full index merge-ort: ensure full index merge-recursive: ensure full index pathspec: ensure full index read-cache: ensure full index resolve-undo: ensure full index revision: ensure full index sparse-index: expand_to_path() name-hash: use expand_to_path() Documentation/technical/sparse-index.txt | 32 ++++++++++- attr.c | 14 ++--- attr.h | 4 +- builtin/add.c | 1 + builtin/checkout-index.c | 1 + builtin/checkout.c | 2 + builtin/commit.c | 2 + builtin/difftool.c | 2 + builtin/fsck.c | 1 + builtin/grep.c | 1 + builtin/ls-files.c | 12 ++-- builtin/merge-index.c | 3 + builtin/rm.c | 1 + builtin/stash.c | 1 + builtin/update-index.c | 1 + cache.h | 7 ++- convert.c | 26 ++++----- convert.h | 20 +++---- diff-lib.c | 1 + dir.c | 13 +++-- dir.h | 8 +-- entry.c | 1 + merge-ort.c | 1 + merge-recursive.c | 3 +- name-hash.c | 10 ++++ pathspec.c | 7 ++- pathspec.h | 6 +- read-cache.c | 33 +++++++++-- resolve-undo.c | 2 + revision.c | 1 + sparse-index.c | 72 ++++++++++++++++++++++++ sparse-index.h | 14 ++++- submodule.c | 6 +- submodule.h | 6 +- 34 files changed, 243 insertions(+), 72 deletions(-) base-commit: 3db06ac46dd5c61e83d7fc4747615d616fdbbdda Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-906%2Fderrickstolee%2Fsparse-index%2Fprotections-v1 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-906/derrickstolee/sparse-index/protections-v1 Pull-Request: https://github.com/gitgitgadget/git/pull/906 -- gitgitgadget
