Re: Can we change the xz block size in our cloud images?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, Nov 23, 2018 at 12:48:30PM +0000, Peter Robinson wrote:
> On Fri, Nov 23, 2018 at 12:43 PM Richard W.M. Jones <rjones@xxxxxxxxxx> wrote:
> >
> > It looks like the raw format xz-compressed cloud images that we ship
> > use a very large block size, possibly 192M.  This is not ideal and it
> > would be better to use a smaller block size such as 16M so that they
> > can be consumed without having to be uncompressed by nbdkit, or even
> > be used remotely without even downloading them.
> >
> > (Long story here: https://rwmj.wordpress.com/2018/11/23/nbdkit-xz-curl/ )
> >
> > I recompressed the Fedora 29 cloud image using a 16M block size and
> > there is about 4% overhead to doing this:
> >
> > before 194278292
> > after  202874868
> >
> > At the moment I'm not clear what actual component does the xz
> > compression step, so I don't even know where I could file a bug or who
> > I could discuss this with, nor what the current code looks like.
> > Apparently it's no longer done using appliance-tools.
> 
> The cloud images haven't used appliance-tools for years. It uses
> imagefactory and some bits in koji.
> 
> Looking at the output of the logs it just runs:
> $ /usr/bin/xz -z9T2
> /var/tmp/koji/tasks/2418/31062418/Fedora-Cloud-Base-Rawhide-20181123.n.0.aarch64.raw
> 
> Example task: https://koji.fedoraproject.org/koji/taskinfo?taskID=31062418
> 
> So at a guess it should be a straight forward patch to koji.

Thanks Peter, I found the source now:

  https://pagure.io/koji/blob/master/f/builder/kojid#_3887

More generally, what are the goals for these cloud images?  Example
questions for the mailing list:

Must they be absolutely as small as possible?  (I assume small size is
a goal to some extent because of the cost of bandwidth and mirroring
space).

Is it important to let people download them and run them without
uncompressing them?  For me, unxz is pretty slow as well as the
obvious problem with consuming disk space, so I can get started on a
cloud image faster if it can be uncompressed on the fly (faster still
if I didn't have to download it up front, but that has other issues
like causing load on the mirror sites).

  $ time unxz Fedora-Cloud-Base-29-1.2.x86_64.raw.xz 
  real   0m23.760s
  user   0m21.564s
  sys    0m0.729s

What do people do with the cloud images?  Do you download them and put
them in local a Glance store?  Do you ignore the published cloud
images and instead get Fedora on clouds through some other method like
AMIs published by 3rd parties?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux