This patch series is upstreaming work made by GitHub and available in: https://github.com/peff/git/commits/jk/delta-islands The above work has been already described in the following article: https://githubengineering.com/counting-objects/ The above branch contains only one patch. In this patch series the patch has been split into 5 patches (1/8, 2/8, 4/8, 5/8 and 6/8) with their own commit message, and on top of that 3 new patches (3/8, 7/8 and 8/8) have been added. The new patches implement things that were requested after the previous iterations. I kept Peff as the author of the original 5 patches and took the liberty to add his Signed-off-by to them. As explained in details in the Counting Object article referenced above, the goal of the delta island mechanism is for a hosting provider to make it possible to have the "forks" of a repository share as much storage as possible while preventing object packs to contain deltas between different forks. If deltas between different forks are not prevented, when users clone or fetch a fork, preparing the pack that should be sent to them can be very costly CPU wise, as objects from a different fork should not be sent, which means that a lot of deltas might need to be computed again (instead of reusing existing deltas). The following changes have been made since the previous iteration: * suggested by Duy: move the code computing the write order for a layer to a new separate function called compute_layer_order() in builtin/pack-objects.c in new patch 3/8 I think that indeed this makes the following patch (4/8) shorter and easier to understand as well as the overall result nicer. * suggested by Duy and Peff: rework the way the 'tree_depth' field is moved from 'struct object_entry' to 'struct packing_data' in pack-object.h in patch 7/8 * suggested by Duy and Peff: move field 'layer' from 'struct object_entry' to 'struct packing_data' in pack-object.h in new patch 8/8 This patch series is also available on GitHub in: https://github.com/chriscool/git/commits/delta-islands The previous versions are available there: V2: https://github.com/chriscool/git/commits/delta-islands19 V1: https://github.com/chriscool/git/commits/delta-islands6 V2: https://public-inbox.org/git/20180805172525.15278-1-chriscool@xxxxxxxxxxxxx/ V1: https://public-inbox.org/git/20180722054836.28935-1-chriscool@xxxxxxxxxxxxx/ Christian Couder (3): pack-objects: refactor code into compute_layer_order() pack-objects: move tree_depth into 'struct packing_data' pack-objects: move 'layer' into 'struct packing_data' Jeff King (5): packfile: make get_delta_base() non static Add delta-islands.{c,h} pack-objects: add delta-islands support repack: add delta-islands support t: add t5319-delta-islands.sh Documentation/config.txt | 19 ++ Documentation/git-pack-objects.txt | 97 ++++++ Documentation/git-repack.txt | 5 + Makefile | 1 + builtin/pack-objects.c | 137 +++++--- builtin/repack.c | 9 + delta-islands.c | 506 +++++++++++++++++++++++++++++ delta-islands.h | 11 + pack-objects.c | 12 + pack-objects.h | 39 +++ packfile.c | 10 +- packfile.h | 7 + t/t5319-delta-islands.sh | 143 ++++++++ 13 files changed, 948 insertions(+), 48 deletions(-) create mode 100644 delta-islands.c create mode 100644 delta-islands.h create mode 100755 t/t5319-delta-islands.sh -- 2.18.0.555.g17f9c4abba
