Re: Proposed zchunk file format - V3

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Thu, Mar 22, 2018 at 9:45 AM, Michal Domonkos <mdomonko@xxxxxxxxxx> wrote:
> On Thu, Mar 22, 2018 at 12:39 PM, Jonathan Dieter <jdieter@xxxxxxxxx> wrote:
>> CC'ing fedora-infrastructure, as I think they got lost somewhere along
>> the way.
>
> Oh, thanks.  I screwed up (again), this time by hitting "Reply"
> instead of "Reply to all" in gmail (*facepalm*).
>
>> Just to be clear, zchunk *could* use buzhash.  There's no rule about
>> where the boundaries need to be, only that the application creating the
>> zchunk file is consistent.  I'd actually like to make the command-line
>> utility use buzhash, but I'm trying to keep the code BSD 2-clause, so I
>> can't just lift casync's buzhash code, and I haven't had time to write
>> that part myself.
>
> Makes sense, thanks for the clarification!
>
> For completeness, I'm also copying below the "git concept" idea that I
> elaborated on in the "lost" email:
>
> ---8<-----
>
> The git concept is basically just a generalization of the chunking
> idea we're talking about.
>
> As long as your data semantically represents a tree, you can chunk up
> the content and get a structure like this:
>
> tree
>     +-- tree
>         +-- blob1
>         +-- blob2
>         +-- blob3
>     +-- tree
>         +-- blob1
>         +-- blob2
>         +-- blob3
>     +-- ...
>
> Now, to sync this structure locally, a simple recursive algorithm is
> used:  look at the root tree to see what objects it needs (i.e. gather
> a list of hashes), then download them and do the same with those
> recursively until you have no more incomplete trees left.  In order to
> avoid having too many files, blobs could be stored in one file (maybe
> per tree) and accessed via HTTP ranges, the same way as in zchunk.
>
> The point is, you will only have to fetch those subtrees where some
> objects along the path have changed.  The effectiveness is then a
> (logarithmic) function of how deep and how well you do the chunking of
> your content.
>
> Applying this to our domain, we have:
>
> repomd (tree)
>     +-- primary (tree)
>         +-- srpm1 (tree)
>             +-- rpm1 (blob)
>             +-- rpm2 (blob)
>         +-- srpm2 (tree)
>         +-- srpm3 (tree)
>     +-- filelists (tree)
>         +-- ...
>
> Doing a "checkout" of such a structure would result in the traditional
> metadata files we're using now.  That's just for backward
> compatibility; we could, of course, have a different structure that's
> better suited for our use case.
>
> As you can see, this is really just zchunk, only generalized (not sure
> if compressions plays a role here, I haven't considered it).
>
> ---8<-----
>

One thing I'm concerned about is handling appended metadata. For
example, both Mageia and openSUSE append AppStream metadata to the
repodata, using a combination of appstream-builder[1] (or
appstream-generator[2]) and modifyrepo_c[3]. How does this scale to
handling that?

[1]: https://www.mankier.com/1/appstream-builder
[2]: https://www.mankier.com/1/appstream-generator
[3]: https://www.mankier.com/8/modifyrepo_c

-- 
真実はいつも一つ!/ Always, there's only one truth!
_______________________________________________
infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx




[Index of Archives]     [Fedora Development]     [Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [KDE Users]

  Powered by Linux