Base Ref: master Web-Diff: https://github.com/benpeart/git/commit/a0300882d4 Checkout: git fetch https://github.com/benpeart/git read-index-multithread-v6 && git checkout a0300882d4 This iteration brings back the Index Entry Offset Table (IEOT) extension which enables us to multi-thread the cache entry parsing without having the primary thread have to scan all the entries first. In cases where the cache entry parsing is the most expensive part, this yields some additional savings. Using p0002-read-cache.sh to generate some performance numbers shows how each of the various patches contribute to the overall performance win. Test w/100,000 files Baseline Optimize V4 Extensions Entries ---------------------------------------------------------------------------- 0002.1: read_cache 22.36 18.74 -16.2% 18.64 -16.6% 12.63 -43.5% Test w/1,000,000 files Baseline Optimize V4 Extensions Entries ----------------------------------------------------------------------------- 0002.1: read_cache 304.40 270.70 -11.1% 195.50 -35.8% 204.82 -32.7% Note that on the 1,000,000 files case, multi-threading the cache entry parsing does not yield a performance win. This is because the cost to parse the index extensions in this repo, far outweigh the cost of loading the cache entries. Name First Last Elapsed load_index_extensions() 629.001 870.244 241.243 load_cache_entries_thread() 683.911 723.199 39.288 load_cache_entries_thread() 686.206 723.512 37.306 load_cache_entries_thread() 686.43 722.596 36.166 load_cache_entries_thread() 684.998 718.74 33.742 load_cache_entries_thread() 685.035 718.698 33.663 load_cache_entries_thread() 686.557 709.545 22.988 load_cache_entries_thread() 684.533 703.536 19.003 load_cache_entries_thread() 684.537 703.521 18.984 load_cache_entries_thread() 685.062 703.774 18.712 load_cache_entries_thread() 685.42 703.416 17.996 load_cache_entries_thread() 648.604 664.496 15.892 293.74 Total load_cache_entries_thread() The high cost of parsing the index extensions is driven by the cache tree and the untracked cache extensions. As this is currently the longest pole, any reduction in this time will reduce the overall index load times so is worth further investigation in another patch series. Name First Last Elapsed | + git!read_index_extension 684.052 870.244 186.192 | + git!cache_tree_read 684.052 797.801 113.749 | + git!read_untracked_extension 797.801 870.244 72.443 One option would be to load each extension on a separate thread but I believe that is overkill for the vast majority of repos. Instead, some optimization of the loading code for these two extensions is probably worth looking into as a quick examination shows that the bulk of the time for both of them is spent in xcalloc(). ### Patches Ben Peart (6): read-cache: clean up casting and byte decoding eoie: add End of Index Entry (EOIE) extension config: add new index.threads config setting read-cache: load cache extensions on a worker thread ieot: add Index Entry Offset Table (IEOT) extension read-cache: load cache entries on worker threads Nguyễn Thái Ngọc Duy (1): read-cache.c: optimize reading index format v4 Documentation/config.txt | 7 + Documentation/technical/index-format.txt | 41 ++ config.c | 18 + config.h | 1 + read-cache.c | 741 +++++++++++++++++++---- t/README | 10 + t/t1700-split-index.sh | 2 + 7 files changed, 705 insertions(+), 115 deletions(-) base-commit: fe8321ec057f9231c26c29b364721568e58040f7 -- 2.18.0.windows.1
