@derrickstolee With Bundles Without Bundles Full Clone 491.9s 964.9s Partial
Clone 132.3s 171.7s
I recruited GitHub employees from across the globe to test this experimental
deployment of bundle servers and found mixed results. Some users had better
performance with bundle servers, and many had very close results. Some had
much worse connections to bundle servers than the GitHub remote, because of
the use of the cheapest form of Azure Storage services compared to highly
optimized GitHub infrastructure.
I plan to keep these servers running for a while, so this test should be
possible throughout the review of this RFC and the patch series as they are
reviewed. I'd love to hear if anyone else does this experiment and has
anything to say about the results.
Implementation Plan
===================
The first patch contains a design document that is "aspirational": It
describes the feature as it should be when the implementation is complete.
Most of that implementation is included in this RFC, though there are some
minor tweaks to the functionality I want to change or add before the patches
are under full review. In particular, things are poorly tested and
undocumented the further you look in the RFC.
Here is a potential plan for splitting this RFC into digestible pieces that
can be reviewed in sequence:
0. Update the git bundle create command to take a --filter option, allowing
bundles to store packfiles restricted to an object filter. This is
necessary for using bundle URIs to benefit partial clones. This step was
already submitted for full review [4]. These patches are based on those.
1. Integrate bundle URIs into git clone with a --bundle-uri option. This
will include the full understanding of a table of contents, but will not
integrate with git fetch or allow the server to advertise URIs.
2. Integrate bundle URIs into git fetch, triggered by config values that
are set during git clone if the server indicates that the bundle
strategy works for fetches.
3. Create a new "recommended features" capability in protocol v2 where the
server can recommend features such as bundle URIs, partial clone, and
sparse-checkout. These features will be extremely limited in scope and
blocked by opt-in config options. The design for this portion could be
replaced by a "bundle-uri" capability that only advertises bundle URIs
and no other information.
Intended Focus of this RFC
==========================
This RFC is very large, and that's even with many patches not including full
documentation or tests. These commits are not intended to be reviewed as if
I intended to merge this as-is.
One thing this feature establishes is a new standard by which the Git client
will communicate with external servers. The goal of this RFC is to determine
if this standard is well designed, and whether we need to make it more
robust. Alternatively, the design might need to be changed for reasons I
cannot predict.
For that reason, hopefully most of the feedback is directly on the first
patch, which contains the design document. In particular, the design
document repeats the implementation plan, and I'd like extra eyes on that,
too.
[0]
https://github.com/git/git/blob/master/Documentation/technical/packfile-uri.txt
The packfile URI feature in Git (Created June 2020)
[1]
https://lore.kernel.org/git/RFC-cover-00.13-0000000000-20210805T150534Z-avarab@xxxxxxxxx/
An earlier RFC for a bundle URI feature. (August 2021)
[2]
https://lore.kernel.org/git/cover-0.3-00000000000-20211025T211159Z-avarab@xxxxxxxxx/
An earlier patch series creating a 'bundle-uri' protocol v2 capability.
(October 2021)
[3]
https://lore.kernel.org/git/e7fe220b-2877-107e-8f7e-ea507a65feff@xxxxxxxxx/
My earlier thoughts on the previous RFCs, many of which are integrated into
this RFC (August 2021)
[4]
https://lore.kernel.org/git/pull.1159.git.1645638911.gitgitgadget@xxxxxxxxx/
Add object filters to bundles (February 2022)
Thanks, -Stolee
Derrick Stolee (24):
docs: document bundle URI standard
bundle: alphabetize subcommands better
dir: extract starts_with_dot[_dot]_slash()
remote: move relative_url()
remote: allow relative_url() to return an absolute url
http: make http_get_file() external
remote-curl: add 'get' capability
bundle: implement 'fetch' command for direct bundles
bundle: parse table of contents during 'fetch'
bundle: add --filter option to 'fetch'
bundle: allow relative URLs in table of contents
bundle: make it easy to call 'git bundle fetch'
clone: add --bundle-uri option
clone: --bundle-uri cannot be combined with --depth
config: add git_config_get_timestamp()
bundle: only fetch bundles if timestamp is new
fetch: fetch bundles before fetching original data
protocol-caps: implement cap_features()
serve: understand but do not advertise 'features' capability
serve: advertise 'features' when config exists
connect: implement get_recommended_features()
transport: add connections for 'features' capability
clone: use server-recommended bundle URI
t5601: basic bundle URI test
�var Arnfjörð Bjarmason (1):
connect.c: refactor sending of agent & object-format
Documentation/gitremote-helpers.txt | 6 +
Documentation/technical/bundle-uri.txt | 404 +++++++++++++++++++++
builtin/bundle.c | 478 ++++++++++++++++++++++++-
builtin/clone.c | 51 +++
builtin/fetch.c | 17 +
builtin/submodule--helper.c | 129 -------
bundle.c | 21 ++
bundle.h | 9 +
config.c | 39 ++
config.h | 14 +
connect.c | 69 +++-
dir.h | 11 +
fsck.c | 14 +-
http.c | 4 +-
http.h | 9 +
protocol-caps.c | 66 ++++
protocol-caps.h | 1 +
remote-curl.c | 32 ++
remote.c | 104 ++++++
remote.h | 35 ++
serve.c | 23 ++
t/t5601-clone.sh | 12 +
t/t5701-git-serve.sh | 9 +
transport-helper.c | 14 +
transport-internal.h | 9 +
transport.c | 38 ++
transport.h | 5 +
27 files changed, 1467 insertions(+), 156 deletions(-)
create mode 100644 Documentation/technical/bundle-uri.txt
base-commit: ec51d0a50e6e64ae37795d77f7d33204b9b71ecd
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1160%2Fderrickstolee%2Fbundle%2Frfc-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1160/derrickstolee/bundle/rfc-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1160
--
gitgitgadget
