We're in the 2.33.0 rc cycle, and I'd hoped to have some more prep work for this integrated already, but for now here's something interesting I've been working on for early commentary/feedback. This adds the the ability to protocol v2 for servers to optimistically pre-seed supporting clients with one or more bundles via a new "bundle-uri" protocol extension. Right now only "clone" supports this, but it's a rather easy change on top to add incremental "fetch" support as well. The elevator pitch for this feature is (I guess it's a long elevator ride..); * Allows for offloading most/all of a PACK "fetch" to "dumb" CDN's that *don't* have very close coordination with the server running "git-upload-pack" (unlike packfile-uri, more on that below). I.e. distributing an up-to-date-enough bundle via something like Debian's FTP mirror system, or a best-effort occasionally updated CDN. Should the bundle(s) be outdated, corrupt or whatever the client gracefully recovers by either ignoring the bad data, or catching up via negotiation with whatever data it did get. Server operators should be confident in using bundle URIs, even if the CDN they're pointing to is flaky, out of date, or even sometimes outright broken or unreachable. The client will recover in all those cases. * Makes performant git infrastructure more accessible, i.e. this feature helps the last with an up-to-date repack with up-to-date bitmaps when talking to a network-local git server, but a lot of users have more option for scaling or distributing things via dumb CDNs than a server that can run "git-upload-pack". * You can even bootstrap a clone of a remote server that doesn't support bundle-uri with a local or remote bundle with the "transfer.injectBundleURI" config. * Trivial path to resumable clones. Note that that's "resumable" in the sense that curl(1) will resume a partially downloaded bundle, we don't resume whatever state index-pack was in when the connection was broken. I have a POC of this working locally, it's just a matter of invoking curl(1) with "--continue-at -". The hindrance for resumably clones is now just the UI for git-clone (i.e. stashing the partial data somewhere), not protocol limitations. This goes on top of the outstanding series I have for serve.c API cleanup & fixes at https://lore.kernel.org/git/cover-v4-00.10-00000000000-20210805T011823Z-avarab@xxxxxxxxx I also needed to grab one patch from my "bundle unbundle progress" series: https://lore.kernel.org/git/cover-0.4-0000000000-20210727T004015Z-avarab@xxxxxxxxx/ Something like this approach had been suggested before in late 2011 by Jeff King, see: https://lore.kernel.org/git/20111110074330.GA27925@xxxxxxxxxxxxxxxxxxxxx/; There's significant differences in the approach, mainly due to protocol v2 not existing at the time. I wrote most of this before finding/seeing Jeff's earlier patches. For a demo of how this works head over to 12/13: https://lore.kernel.org/git/RFC-patch-12.13-8dc5613e87-20210805T150534Z-avarab@xxxxxxxxx In 13/13 there's a design doc discussing the approach, and major differences with the existing packfile-uri mechanism: https://lore.kernel.org/git/RFC-patch-13.13-1e657ed27a-20210805T150534Z-avarab@xxxxxxxxx This can also be grabbed from the "avar/bundle-uri-client-clone" branch of https://github.com/avar/git/ Ævar Arnfjörð Bjarmason (13): serve: add command to advertise bundle URIs bundle-uri client: add "bundle-uri" parsing + tests connect.c: refactor sending of agent & object-format bundle-uri client: add minimal NOOP client bundle-uri client: add "git ls-remote-bundle-uri" bundle-uri client: add transfer.injectBundleURI support bundle-uri client: add boolean transfer.bundleURI setting bundle.h: make "fd" version of read_bundle_header() public fetch-pack: add a deref_without_lazy_fetch_extended() fetch-pack: move --keep=* option filling to a function index-pack: add --progress-title option bundle-uri client: support for bundle-uri with "clone" bundle-uri docs: add design notes Documentation/config/transfer.txt | 26 ++ Documentation/git-index-pack.txt | 6 + Documentation/git-ls-remote-bundle-uri.txt | 63 +++ Documentation/git-ls-remote.txt | 1 + Documentation/technical/bundle-uri.txt | 119 ++++++ Documentation/technical/protocol-v2.txt | 145 +++++++ Makefile | 3 + builtin.h | 1 + builtin/clone.c | 7 + builtin/index-pack.c | 6 + builtin/ls-remote-bundle-uri.c | 90 +++++ bundle-uri.c | 151 ++++++++ bundle-uri.h | 30 ++ bundle.c | 8 +- bundle.h | 2 + command-list.txt | 1 + connect.c | 80 +++- fetch-pack.c | 304 ++++++++++++++- fetch-pack.h | 6 + git.c | 1 + remote.h | 4 + serve.c | 6 + t/helper/test-bundle-uri.c | 80 ++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/lib-t5730-protocol-v2-bundle-uri.sh | 425 +++++++++++++++++++++ t/t5701-git-serve.sh | 124 +++++- t/t5730-protocol-v2-bundle-uri-file.sh | 36 ++ t/t5731-protocol-v2-bundle-uri-git.sh | 17 + t/t5732-protocol-v2-bundle-uri-http.sh | 17 + t/t5750-bundle-uri-parse.sh | 98 +++++ transport-helper.c | 13 + transport-internal.h | 7 + transport.c | 120 ++++++ transport.h | 22 ++ 35 files changed, 1988 insertions(+), 33 deletions(-) create mode 100644 Documentation/git-ls-remote-bundle-uri.txt create mode 100644 Documentation/technical/bundle-uri.txt create mode 100644 builtin/ls-remote-bundle-uri.c create mode 100644 bundle-uri.c create mode 100644 bundle-uri.h create mode 100644 t/helper/test-bundle-uri.c create mode 100644 t/lib-t5730-protocol-v2-bundle-uri.sh create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh create mode 100755 t/t5750-bundle-uri-parse.sh -- 2.33.0.rc0.646.g585563e77f
