On Wed, Dec 1, 2021 at 2:38 AM Derrick Stolee <stolee@xxxxxxxxx> wrote: > > I used a pack-file from an internal repo. It happened to be using > partial clone, so here is a repro with the git/git repository > after cloning this way: > > $ git clone --no-checkout --filter=blob:none https://github.com/git/git > > (copy the large .pack from git/.git/objects/pack/ to big.pack) > > $ hyperfine \ > --prepare 'rm -rf dest.git && git init --bare dest.git' \ > -n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <big.pack' \ > -n 'new' '~/_git/git/git -C dest.git unpack-objects <big.pack' \ > -n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <big.pack' > > Benchmark 1: old > Time (mean ± σ): 82.748 s ± 0.445 s [User: 50.512 s, System: 32.049 s] > Range (min … max): 82.042 s … 83.587 s 10 runs > > Benchmark 2: new > Time (mean ± σ): 101.644 s ± 0.524 s [User: 67.470 s, System: 34.047 s] > Range (min … max): 100.866 s … 102.633 s 10 runs > > Benchmark 3: new (small threshold) > Time (mean ± σ): 101.093 s ± 0.269 s [User: 67.404 s, System: 33.559 s] > Range (min … max): 100.639 s … 101.375 s 10 runs > > Summary > 'old' ran > 1.22 ± 0.01 times faster than 'new (small threshold)' > 1.23 ± 0.01 times faster than 'new' > > I'm also able to repro this with a smaller repo (microsoft/scalar) > so the tests complete much faster: > > $ hyperfine \ > --prepare 'rm -rf dest.git && git init --bare dest.git' \ > -n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <small.pack' \ > -n 'new' '~/_git/git/git -C dest.git unpack-objects <small.pack' \ > -n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <small.pack' > > Benchmark 1: old > Time (mean ± σ): 3.295 s ± 0.023 s [User: 1.063 s, System: 2.228 s] > Range (min … max): 3.269 s … 3.351 s 10 runs > > Benchmark 2: new > Time (mean ± σ): 3.592 s ± 0.105 s [User: 1.261 s, System: 2.328 s] > Range (min … max): 3.378 s … 3.679 s 10 runs > > Benchmark 3: new (small threshold) > Time (mean ± σ): 3.584 s ± 0.144 s [User: 1.241 s, System: 2.339 s] > Range (min … max): 3.359 s … 3.747 s 10 runs > > Summary > 'old' ran > 1.09 ± 0.04 times faster than 'new (small threshold)' > 1.09 ± 0.03 times faster than 'new' > > It's not the same relative overhead, but still significant. > > These pack-files contain (mostly) small objects, no large blobs. > I know that's not the target of your efforts, but it would be > good to avoid a regression here. > > Thanks, > -Stolee With your help, I did catch this performance problem, which was introduced in this patch: https://lore.kernel.org/git/20211122033220.32883-4-chiyutianyi@xxxxxxxxx/ This patch changes the original data reading ino to stream reading, but its problem is that even for the original reading of the whole object data, it still generates an additional git_deflate() and subsequent transfer. I will fix it in a follow-up patch. Thanks, -Han Xin
