On Mon, Jun 13, 2016 at 3:55 PM, Christian Couder <christian.couder@xxxxxxxxx> wrote: > Future work > ~~~~~~~~~~~ > > From the discussions it appear that using the bundle v3 mechanism to > tranfer external ODB data could work, but only if the server has access > to its own external ODB. > > Another possible mechanism to transfer external ODB data would be some > kind of replace refs. This would be slower but the mechanism for the > transfer already fully exists. > > So I think I am going to experiment with some kind of replace refs. Or we can go "fault-in" all the way. 1) Extend rev-list to support large blob exclusion, e.g. blobs whose size above a certain limit are excluded, or blobs in certain paths are excluded (maybe except those at the tip) 2) Add a new command, similar to "shallow", in the git protocol for the client to tell the server that it does not want the server to send certain large blobs (and the server probably should acknowledge with "ok you can get them from this url...") 3) upload-pack passes the relevant rev-list blob filtering arguments to pack-objects, no big blobs are transferred 4) External odb setting must be done (temporarily) before index-pack is run (as part of the clone/fetch process). index-pack will notice that certain blobs are missing from the received pack, but it will also notice that it already has them via standard has_sha1_file() (we just need to make sure it does not read blob content back, i think we're ok here). I'm assuming here that EODB in fact contains _all_ blobs, so no extra setup is required at the server side. 5) If index-pack says "ok" then you can make this new external odb permanent, else remove it. Subsequent fetches need to send the same "do not include these blobs" instructions, of course. Not sure how well it interacts with pack-bitmap, but I guess when you use this, you probably already use shallow clone (which disables pack bitmap). > One interesting thing also would be to use the streaming api when > reading from or writing to the external ODB. If EODB is about large blobs only, you probably want to stream them directly to a pack as you get them and not keep in loose object form. Infrastructure is already in place (I think it's used by git-add), but hooking it at sha1_file.c layer might be tricky. This is just cherry on top, not strictly needed of course. -- Duy -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html