Re-send to the Git mailing-list as setting a font on gmail switched plain-text to HTML and thus, got blocked by mailing-list. On Sun, Feb 12, 2023 at 3:09 PM Son Luong Ngoc <sluongng@xxxxxxxxx> wrote: > > Hi Andrew, > > On Sat, Feb 11, 2023 at 11:49 PM Andrew Wansink <andy@xxxxxxxxxxx> wrote: > > > > This unlocks `git clone --shallow-exclude=<commit-sha1>` > > > > git-clone only accepts --shallow-excude arguments where > > the argument is a branch or tag because upload_pack only > > searches deepen-not arguments for branches and tags. > > > > Make process_deepen_not search for commit objects if no > > branch or tag is found then add them to the deepen_not > > list. > > > > Signed-off-by: Andrew Wansink <wansink@xxxxxxxx> > > --- > > > > At Uber we have a lot of patches in CI simultaneously, > > the CI jobs will frequently clone the monorepo multiple > > times for each patch. They do this to calculate diffs > > between a patch and its parent commit. > > > > I used to manage a CI system that support monorepo use cases not so long ago. > We had several hosts(VM/Baremetal) on which we spin up containers for CI to run. > > We maintain a bare copy of the monorepo on the host level (cron job / systemd / DaemonSet) and mount this as read-only into each of the CI containers. > > Each of the CI containers would attempt to clone/fetch the monorepo with `--reference-if-able ./path/to/read-only-mount/repo.git` (1) > So that most of the needed objects are already on disk in the shared bare repo. > > > +-----------+ +-----------+ +-----------+ > | container | | container | | container | > +-----------+ +-----------+ +-----------+ > \ | / > (mount) \ | / > +------------+ +--------+ > | bare-repo | <-------------- | Remote | > +------------+ (git-fetch) +--------+ > | > | (maintain) > | > +----------+ > | cron-job | > +----------+ > > (forgive my horrible drawing) > > With this setup, we did not have a need to shallow clone any longer, > and our git-clone in each container is simply a combination of git-ls-remote and a very light-weighted git-fetch. > In some cases, such as a job in the later stages of a CI pipeline, > the host would already download all the needed objects into the bare copy of the repository. > This lets us skip git-fetch entirely when the CI container executes. > > Compared to the shallow clone approach, > our "local cache" approach sped up the clone speed drastically > while allowing developers to interact with git history inside tests a lot easier. > > > One optimisation in this flow is to clone only to a specific > > depth, this may or may not work, depending on how old the > > patch is. In this case we have to --unshallow or discard > > the shallow clone and fully clone the repo. > > > > This patch would allow us to clone to exactly the depth we > > need to find a patch's parent commit. > > Hope it helps, > Son Luong. > > (1): https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---reference-if-ableltrepositorygt
