On Wed, 2018-01-10 at 08:06 -0800, Kevin Fenzi wrote: > On 01/09/2018 02:09 PM, Tim Landscheidt wrote: > > BTW, are there technical reasons why the metadata is updated > > en bloc and not incrementally like for example delta RPMs, > > or is it just that nobody bothered to implement something > > like that yet for metadata? > > There is no implementation that I know of. It's been talked about for > many years, but not been implemented. We talked a bit about using casync for delta metadata at Flock last summer, and I even did a small proof of concept to see what kind of savings we'd see, but there were a couple of issues that came up: * casync creates a large number of very small files (roughly 4k files for yesterday's metadata) that mirrors would need to sync * Users would need to download a significant portion of those files to delta against (anywhere from 0 to all of them, depending on how similar their metadata is) Downloading any significant number of tiny files over http/1.x is really, really slow as the downloads are serial. On reflection, one idea that might fix both problems would be if casync could concatenate the cacnk files into one large block (cablk, perhaps), and have the caidx file contain the location of each cacnk in the block file. Using http ranges, you would then be able to download just the parts of the file you wanted (though I have not actually tested whether specifying a large list of http ranges will download faster than serially downloading each individual file). FWIW, casyncing Jan 4th's metadata over Dec 15th's downloads 9.6MB (in just over 1k cacnk files), while downloading the same metadata directly would total 16MB. More frequent updates would be smaller, but I'm not convinced the difference from an average updates push to another would be less than 2MB. Jonathan
Attachment:
signature.asc
Description: This is a digitally signed message part
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
