On Mon, Apr 25, 2016 at 3:39 AM, Lars Schneider
<larsxschneider@xxxxxxxxx> wrote:
> Hi,
>
> a few folks from the Git LFS project and I try to make cloning of repositories
> with a lot of LFS files faster.
>
> The core problem is that Git LFS uses a Git smudge filter to replace LFS
> pointers with the actual file content. Right now, a smudge filter can only
> be executed on an individual file which makes the operation slow for many
> files [1].
>
> We solved this issue by temporarily disabling the smudge filter for the clone
> command via Git config (optimized in 1a8630 [2]):
>
> git -c filter.lfs.smudge= -c filter.lfs.required=false clone <url> <path>
>
> Afterwards Git LFS runs a special command to download and replace all LFS
> content in bulk [3]. This works great for LFS repositories.
>
> However, I noticed that git config command line instructions such as
> "-c filter.lfs.smudge=" are not passed to Git submodule operations. Thus
> this does not work as expected:
>
> git -c filter.lfs.smudge= -c filter.lfs.required=false clone --recursive <url> <path>
I have cc'd Jacob Keller, who authored origin/jk/submodule-c-credential,
which does work in that area (deciding which config option to pass down
into the submodule commands).
>
> I tried to work around that by copying the relevant pieced from the Git
> Submodule command [4] and applying the command line Git config
> manually (look closely at the modified checkout command):
>
> git -c filter.lfs.smudge= -c filter.lfs.required=false clone $@
> if [[ -z $2 ]]; then
> CLONE_PATH=$(basename ${1%.git});
> else
> CLONE_PATH=$2;
> fi
> pushd "$CLONE_PATH"
> git submodule init
> wt_prefix=$(git rev-parse --show-prefix)
> git submodule--helper list --prefix "$wt_prefix" | {
> while read mode sha1 stage sm_path
> do
> name=$(git submodule--helper name "$sm_path") || exit
> url=$(git config submodule."$name".url)
> if ! test -d "$sm_path"/.git && ! test -f "$sm_path"/.git
> then
> git submodule--helper clone --prefix "$wt_prefix" --path "$sm_path" --name "$name" --url "$url"
The init and then clone should be covered by
"git submodule--helper update-clone", which may be better named as
"list-and-clone-if-necessary", then you get parallel cloning for free
as well. ;)
> pushd "$sm_path"
> git -c filter.lfs.smudge= -c filter.lfs.required=false checkout -q $sha1 || exit
> git-lfs pull || exit
> popd
> fi
> done
> }
> popd
>
> Do you see an easier way to pass command line Git config instructions to the
> underlaying Git Submodule commands? If not, do you think a patch adding this
> would be worth working on?
I would build on top of origin/jk/submodule-c-credential at least, and using
"git submodule--helper update-clone" (origin/sb/submodule-parallel-update)
>
> I also started a discussion about that on the Git LFS issue page [5].
Unrelated to this, but about LFS:
Currently it is only possible to store the big blobs at a $VENDOR hosting site.
How would you backup you whole repository including all binariers?
I think we should add an option to store one blob in a dedicated ref
(refs/lfs/$blobsha1 or such). That way you can make a backup of the
repository including all large files using "git clone --mirror" and
you don't have
to rely on the $VENDOR hosting your files.
Thanks,
Stefan
>
> Thanks,
> Lars
>
>
> [1] https://github.com/github/git-lfs/issues/931
> [2] https://github.com/git/git/commit/1a8630dc3b1cc6f1361a4e5d94630133c24c97d9
> [3] https://developer.atlassian.com/blog/2016/04/git-lfs-12-clone-faster/
> [4] https://github.com/git/git/blob/6a6636270fbaf74609cd3e1bd207dd2c420d640a/git-submodule.sh#L686-L731
> [5] https://github.com/github/git-lfs/issues/1172
>
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html