Partial clone demo for large files (Re: Why Git LFS is not a built-in feature)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Sat, Nov 14, 2020 at 7:25 PM Ævar Arnfjörð Bjarmason
<avarab@xxxxxxxxx> wrote:
>
>
> On Sat, Nov 14 2020, Konstantin Ryabitsev wrote:
>
> > On Sat, Nov 14, 2020 at 12:29:02AM +0000, brian m. carlson wrote:
> >> Additionally, in many cases, projects can avoid the need for storing
> >> large files at all by using repository best practices, like not storing
> >> build products or binary dependencies in the repository and instead
> >> using an artifact server or a standard packaging system.  If possible,
> >> that will almost always provide a better experience than any solution
> >> for storing large files in the repository.
> >
> > Well, I would argue that if the goal is ongoing archival and easy
> > replication, then storing objects in a repository like git makes a lot
> > more sense than keeping them on a central server that may or may not be
> > there a few years down the line. Having large file support native in git
> > is a laudable goal and I quite often wish that it existed.
>
> That native support does exist right now in the form of partial clones,
> the packfile-uris support, core.bigFileThreshold etc.
>
> It's got a lot of rough edges currently, but if it's something you're
> interested in you should try it out and see if the subset of features
> that works well now is something that would work for you.

I have been working on a partial clone demo that stores large files on
an HTTP server:

https://gitlab.com/chriscool/partial-clone-demo/-/blob/master/http-promisor/demo.txt

It has a lot of rough edges indeed. Fetching from the HTTP server
promisor remote is very slow. For the last part you need a hacked Git.
The scripts have a lot of bugs and limitations and are not finished
(to say the least).

The goal for now is just to give people (especially product managers,
developers and managers inside GitLab) an outlook about how it could
work.




[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]

  Powered by Linux