Once upon a time, John.Florian@xxxxxxxx <John.Florian@xxxxxxxx> said: > Is there anything that could be done to make it unnecessary to pull the > complete metadata for every update? For example, IIRC this is all sqlite > data, but what if this was in a plain-text data dump form where something > like rsync could be used to efficiently transfer only those bits that have > changed. Client CPU time to reconstruct the DB is probably cheaper than > the bandwidth. Maybe such a mode would only be used if the DB size > exceeded some threshold. The metadata starts in XML before being loaded into an SQLite DB file, and the XML is in the repodata directory with the DB. However, both are compressed, as they are large. For example, the current updates/18/x86_64 XML is over 34M (5M gzip compressed), and the DB is 41M (9M bzip2 compressed). I'm guessing there are historical reasons why different compression is used; both could be made noticeably smaller with xz (XML to just over 3M, DB to 7M), but that's still a lot of data to download (and there are also other metadata files that have to be downloaded sometimes, especially the filelists.xml.gz, which is 10M gzip compressed). I'm not sure when the XML is downloaded instead of (or in addition to) the DB, but it does appear to happen (I see one example in my mirror server web logs this morning for example). All the metadata changes with every push (usually once per day for the updates repo), so it has to be downloaded constantly. Your only practical choice for distribution is HTTP; rsync has much higher server overhead and only available on some mirrors. If you want anything other than the full download, it'll have to be in the form of additional repodata files generated as part of the push. -- Chris Adams <cmadams@xxxxxxxxxx> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble. -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel
