On Mon, 7 Feb 2005, Dag Wieers wrote: > On Sun, 29 Jan 2005, Alexandre Oliva wrote: > > On Jan 29, 2005, "Charles R. Anderson" <cra@xxxxxxx> wrote: > > > > > Why all the complication when a general-purpose algorithm, rsync, > > > already exists to solve this problem? > > > > It doesn't do very well on compressed files, and there aren't a lot of > > rsync-enabled web proxies out there. > > You may like this RFE: > > https://devel.linux.duke.edu/bugzilla/show_bug.cgi?id=391 > > gzip has an --rsyncable option that is missing from the python zlib > implementation. If python/zlib could be taught this, and createrepo > is able to use it, the metadata would be rsyncable (improves > transferspeed for repo maintainers/mirrors). My repo-generation script now unzips the createrepo metadata, recompresses with --rsyncable and recreates the repomd.xml after createrepo's run. I synced the repository. I then added a single update package to my repository And then rsynced again (with a 12kB upstream limitation), and got: fedora/3/en/i386/dag/repodata/ fedora/3/en/i386/dag/repodata/filelists.xml.gz 961353 100% 306.10kB/s 0:00:03 (68, 16.8% of 208800) fedora/3/en/i386/dag/repodata/other.xml.gz 412335 100% 89.30kB/s 0:00:04 (69, 16.8% of 208800) fedora/3/en/i386/dag/repodata/primary.xml.gz 668753 100% 154.79kB/s 0:00:04 (70, 16.8% of 208800) fedora/3/en/i386/dag/repodata/repomd.xml 1128 100% 4.87kB/s 0:00:00 (71, 16.8% of 208800) and fedora/3/en/x86_64/dag/repodata/ fedora/3/en/x86_64/dag/repodata/filelists.xml.gz 1466742 100% 273.93kB/s 0:00:05 (86, 20.9% of 208800) fedora/3/en/x86_64/dag/repodata/other.xml.gz 674072 100% 79.47kB/s 0:00:08 (87, 20.9% of 208800) fedora/3/en/x86_64/dag/repodata/primary.xml.gz 1086239 100% 141.68kB/s 0:00:07 (88, 20.9% of 208800) fedora/3/en/x86_64/dag/repodata/repomd.xml 1128 100% 1.46kB/s 0:00:00 (89, 20.9% of 208800) Which is a between 6.6x and 25.6x speed improvement for something I have to update every single sync. (Normally these files are synced with a 12kB/sec rate limitation taking on average 1min per file, now only 6secs) Now imagine what this would do if Yum had a client-side rsync implementation. zsync may be something to look at. -- dag wieers, dag@xxxxxxxxxx, http://dag.wieers.com/ -- [all I want is a warm bed and a kind word and unlimited power]
