I hadn't sent a shameless plug for my "git hyperfine" script to the list, perhaps this is a good time. It's just a thin shellscript wrapper around "hyperfine" that I wrote the other day, which... On Tue, Nov 30 2021, Derrick Stolee wrote: > [...] > I used a pack-file from an internal repo. It happened to be using > partial clone, so here is a repro with the git/git repository > after cloning this way: > > $ git clone --no-checkout --filter=blob:none https://github.com/git/git > > (copy the large .pack from git/.git/objects/pack/ to big.pack) > > $ hyperfine \ > --prepare 'rm -rf dest.git && git init --bare dest.git' \ > -n 'old' '~/_git/git-upstream/git -C dest.git unpack-objects <big.pack' \ > -n 'new' '~/_git/git/git -C dest.git unpack-objects <big.pack' \ > -n 'new (small threshold)' '~/_git/git/git -c core.bigfilethreshold=64k -C dest.git unpack-objects <big.pack' > > Benchmark 1: old > Time (mean ± σ): 82.748 s ± 0.445 s [User: 50.512 s, System: 32.049 s] > Range (min … max): 82.042 s … 83.587 s 10 runs > > Benchmark 2: new > Time (mean ± σ): 101.644 s ± 0.524 s [User: 67.470 s, System: 34.047 s] > Range (min … max): 100.866 s … 102.633 s 10 runs > > Benchmark 3: new (small threshold) > Time (mean ± σ): 101.093 s ± 0.269 s [User: 67.404 s, System: 33.559 s] > Range (min … max): 100.639 s … 101.375 s 10 runs > > Summary > 'old' ran > 1.22 ± 0.01 times faster than 'new (small threshold)' > 1.23 ± 0.01 times faster than 'new' ...adds enough sugar around "hyperfine" itself to do this as e.g. (the "-s" is a feature I submitted to hyperfine itself, it's not in a release yet[1], but in this case you could also use "-p"): git hyperfine -L rev v2.20.0,origin/master \ -s 'if ! test -d redis.git; then git clone --bare --filter=blob:none https://github.com/redis/redis; fi && make' \ -p 'rm -rf dest.git; git init --bare dest.git' \ './git -C dest.git unpack-objects <$(echo redis.git/objects/pack/*.pack)' The sugar being that for each named "rev" parameter it'll set up "git worktree" for you, so under the hood each of those is chdir-ing to the respective revision of: $ git worktree list [...] /run/user/1001/git-hyperfine/origin/master abe6bb39053 (detached HEAD) /run/user/1001/git-hyperfine/v2.33.0 225bc32a989 (detached HEAD) That they're named revisions and not git-rev-parse'd is intentional, since you'll benefit from faster incremental "make" (even if using "ccache"). I'm typically benchmarking HEAD~1,HEAD~0. The output will then use those "rev" parameters, and be e.g.: Benchmark 1: ./git -C dest.git unpack-objects <$(echo redis.git/objects/pack/*.pack)' in 'v2.20.0 Time (mean ± σ): 6.678 s ± 0.046 s [User: 4.525 s, System: 2.117 s] Range (min … max): 6.619 s … 6.765 s 10 runs Benchmark 2: ./git -C dest.git unpack-objects <$(echo redis.git/objects/pack/*.pack)' in 'origin/master Time (mean ± σ): 6.756 s ± 0.074 s [User: 4.586 s, System: 2.134 s] Range (min … max): 6.691 s … 6.941 s 10 runs Summary './git -C dest.git unpack-objects <$(echo redis.git/objects/pack/*.pack)' in 'v2.20.0' ran 1.01 ± 0.01 times faster than './git -C dest.git unpack-objects <$(echo redis.git/objects/pack/*.pack)' in 'origin/master' I think if you're routinely benchmarking N different git versions you'll find it handy, it also has configurable hook support (using git config), so e.g. it's easy to copy your config.mak in-place in the worktrees. E.g. my config is: $ git -P config --get-regexp '^hyperfine' hyperfine.run-dir $XDG_RUNTIME_DIR/git-hyperfine hyperfine.xargs-options -r hyperfine.hook.setup ~/g/git.meta/config.mak.sh It's hosted at https://github.com/avar/git-hyperfine/ and https://gitlab.com/avar/git-hyperfine/; It's implemented in (portable) POSIX shell script. There's surely some bugs in it, one known one is that unlike hyperfine it doesn't accept there being spaces in the parameters to -L, because I'm screwing up some quoting-within-quoting in the (shellscript) implementation (suggestions for that particular one most welcome). I hacked it up after this suggestion from Jeff King[2] of moving t/perf over to it. I haven't done any of that legwork, but I think a wrapper like "git-hyperfine" that prepares worktrees for the N revisions we're benchmarking is a good direction to go in. We don't use git-worktrees in t/perf, but probably could for most/all tests. In any case it would be easy to have the script setup the revs to be benchmarked in some hookable custom manner to have it do exactly what t/perf/run is doing now. 1. https://github.com/sharkdp/hyperfine/commit/017d55a 2. https://lore.kernel.org/git/YV+zFqi4VmBVJYex@xxxxxxxxxxxxxxxxxxxxxxx/
