Hi,
this is another patch series with the aim to speed up mirror fetches. It
applies on top of e6ebfd0e8c (The sixth batch, 2022-02-18) with
3824153b23 (Merge branch 'ps/fetch-atomic' into next, 2022-02-18) merged
into it to fix a conflict.
The only change compared to v2 is an update to the benchmarks so that
they're less verbose, as proposed by Derrick. I also had a look at
introducing a new helper `parse_object_probably_commit()`, but I didn't
find the end result to be much of an improvement compared to the ad-hoc
`lookup_commit_in_graph() || parse_object()` dance we do right now.
Thanks!
Patrick
Patrick Steinhardt (5):
upload-pack: look up "want" lines via commit-graph
fetch: avoid lookup of commits when not appending to FETCH_HEAD
refs: add ability for backends to special-case reading of symbolic
refs
remote: read symbolic refs via `refs_read_symbolic_ref()`
refs/files-backend: optimize reading of symbolic refs
builtin/fetch.c | 42 +++++++++++++++++++++++++++---------------
builtin/remote.c | 8 +++++---
refs.c | 17 +++++++++++++++++
refs.h | 3 +++
refs/debug.c | 1 +
refs/files-backend.c | 33 ++++++++++++++++++++++++++++-----
refs/packed-backend.c | 1 +
refs/refs-internal.h | 16 ++++++++++++++++
remote.c | 14 +++++++-------
upload-pack.c | 20 +++++++++++++++++---
10 files changed, 122 insertions(+), 33 deletions(-)
Range-diff against v1:
1: ca5e136cca ! 1: b5c696bd8e upload-pack: look up "want" lines via commit-graph
@@ Commit message
Refactor parsing of both "want" and "want-ref" lines to do so.
The following benchmark is executed in a repository with a huge number
- of references. It uses cached request from git-fetch(1) as input and
- contains about 876,000 "want" lines:
+ of references. It uses cached request from git-fetch(1) as input to
+ git-upload-pack(1) that contains about 876,000 "want" lines:
- Benchmark 1: git-upload-pack (HEAD~)
+ Benchmark 1: HEAD~
Time (mean ± σ): 7.113 s ± 0.028 s [User: 6.900 s, System: 0.662 s]
Range (min … max): 7.072 s … 7.168 s 10 runs
- Benchmark 2: git-upload-pack (HEAD)
+ Benchmark 2: HEAD
Time (mean ± σ): 6.622 s ± 0.061 s [User: 6.452 s, System: 0.650 s]
Range (min … max): 6.535 s … 6.727 s 10 runs
Summary
- 'git-upload-pack (HEAD)' ran
- 1.07 ± 0.01 times faster than 'git-upload-pack (HEAD~)'
+ 'HEAD' ran
+ 1.07 ± 0.01 times faster than 'HEAD~'
Signed-off-by: Patrick Steinhardt <ps@xxxxxx>
2: 80f993dddd ! 2: fbe76b78c3 fetch: avoid lookup of commits when not appending to FETCH_HEAD
@@ Commit message
Skip this busywork in case we're not writing to FETCH_HEAD. The
following benchmark performs a mirror-fetch in a repository with about
- two million references:
+ two million references via `git fetch --prune --no-write-fetch-head
+ +refs/*:refs/*`:
- Benchmark 1: git fetch --prune --no-write-fetch-head +refs/*:refs/* (HEAD~)
+ Benchmark 1: HEAD~
Time (mean ± σ): 75.388 s ± 1.942 s [User: 71.103 s, System: 8.953 s]
Range (min … max): 73.184 s … 76.845 s 3 runs
- Benchmark 2: git fetch --prune --no-write-fetch-head +refs/*:refs/* (HEAD)
+ Benchmark 2: HEAD
Time (mean ± σ): 69.486 s ± 1.016 s [User: 65.941 s, System: 8.806 s]
Range (min … max): 68.864 s … 70.659 s 3 runs
Summary
- 'git fetch --prune --no-write-fetch-head +refs/*:refs/* (HEAD)' ran
- 1.08 ± 0.03 times faster than 'git fetch --prune --no-write-fetch-head +refs/*:refs/* (HEAD~)'
+ 'HEAD' ran
+ 1.08 ± 0.03 times faster than 'HEAD~'
Signed-off-by: Patrick Steinhardt <ps@xxxxxx>
3: 28cacbdbe2 = 3: 29eb81d37c refs: add ability for backends to special-case reading of symbolic refs
4: 1d24101fe4 = 4: 0489380e00 remote: read symbolic refs via `refs_read_symbolic_ref()`
5: 7213ffdbdd ! 5: b6eca63d3b refs/files-backend: optimize reading of symbolic refs
@@ Commit message
need to skip updating local symbolic references during a fetch, which is
why the change results in a significant speedup when doing fetches in
repositories with huge numbers of references. The following benchmark
- executes a mirror-fetch in a repository with about 2 million references:
+ executes a mirror-fetch in a repository with about 2 million references
+ via `git fetch --prune --no-write-fetch-head +refs/*:refs/*`:
- Benchmark 1: git fetch --prune --no-write-fetch-head +refs/*:refs/* (HEAD~)
+ Benchmark 1: HEAD~
Time (mean ± σ): 68.372 s ± 2.344 s [User: 65.629 s, System: 8.786 s]
Range (min … max): 65.745 s … 70.246 s 3 runs
- Benchmark 2: git fetch --prune --no-write-fetch-head +refs/*:refs/* (HEAD)
+ Benchmark 2: HEAD
Time (mean ± σ): 60.259 s ± 0.343 s [User: 61.019 s, System: 7.245 s]
Range (min … max): 60.003 s … 60.649 s 3 runs
Summary
- 'git fetch --prune --no-write-fetch-head +refs/*:refs/* (HEAD)' ran
- 1.13 ± 0.04 times faster than 'git fetch --prune --no-write-fetch-head +refs/*:refs/* (HEAD~)'
+ 'HEAD' ran
+ 1.13 ± 0.04 times faster than 'HEAD~'
Signed-off-by: Patrick Steinhardt <ps@xxxxxx>
--
2.35.1
Attachment:
signature.asc
Description: PGP signature
