git torrent - sane deterministic pack files

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Please CC me when replying, if you think of it. Thanks.

LWN discussion, this particular thread starts here:
https://lwn.net/Articles/646758/

Some extracts:
Deterministic packfile creation is required for parallel git downloads.

Example git pack file parameter/ configuration variations:
- compression on X number of CPUs
- maximum packfile size
- more trees/ branches in this repo than on that repo

A way is needed to capture these packfile config variations and
distribute them to other git servers (perhaps on a standardized branch
name or ??).
---
Related mail from 4 years ago:
http://article.gmane.org/gmane.comp.version-control.git/164643

Conceptually it may not be hard, but implementation is hard. By
forcing certain object layout rules, you may have lower compression
ratio, or slower pack access, and may consume more power. Git tries to
reuse deltas from existing packs to produce a new pack. This makes it
quick to assemble a pack, but also underterministic. There's also
threads stealing jobs from one another in the above link. Resumable
clone is a frequent request, and we still don't have it now.
---
Deterministic pack file parameter sets and compression can always be
tuned over time even though they change the format - that's just local
policy for the authoritative git torrent server.

Also for scenarios which benefit from pack file torrents, the marginal
reduction in compression (increase in pack file size) due to the need
for determinism may very well be valuable (marginal increase in local
storage in order to distribute downloads) - local policy strikes
again.
...
It's up to the "authoritative git server" admin to make the policy
decision of how long to keep with a current deterministic torrentable
pack file parameter set, and when to update to a new/tuned set. This
is always a local policy matter! "We can't do that because it's not
the best policy for maximum compression" is not the right answer
here...
...
If I am a torrent repo mirror, the "authoritative torrent upstream" is
merely a local config.

This not only sounds easy, it is easy - even in the face of
compression technology changes and "tuning" over time - that's merely
a "version" increase or new parameter set provided by the
"authoritative" repo server, and is local policy to that server.

Regards
Zenaan
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]