Re: [lorax] how to decrease squashfs.img size, it increase more than install size

[耳东 <volcanowill@xxxxxxxxx>]

hi guys, I have refered to the minimal ks config of cloud-kickstarts.git:

    https://git.fedorahosted.org/cgit/cloud-kickstarts.git/tree/container/container-small-20.ks

I have clean unused locales and some file.

- compress cracklib pw_dict
- clean unused locale files
- rm some static pictures and log files

the different ks config:

    +gzip -9 /usr/share/cracklib/pw_dict.pwd
    +localedef --list-archive | grep -v en_US | xargs localedef --delete-from-archive
    +mv -f /usr/lib/locale/locale-archive /usr/lib/locale/locale-archive.tmpl
    +/usr/sbin/build-locale-archive
    +mv -f /usr/share/locale/en /usr/share/locale/en_US /tmp
    +rm -rf /usr/share/locale/*
    +mv /tmp/en /tmp/en_US /usr/share/locale/
    +mv /usr/share/i18n/locales/en_US /tmp
    +rm -rf /usr/share/i18n/locales/*
    +mv /tmp/en_US /usr/share/i18n/locales/
    +echo '%_install_langs C:en:en_US:en_US.UTF-8' >> /etc/rpm/macros.imgcreate
    +
    +truncate -c -s 0 /var/log/lastlog

     rpm -qa|sort > /root/rpm-qa.txt
     yum clean all
     rm -f /etc/resolv.conf
     rm -f /var/lib/rpm/__db*
    +rm -rf /tmp/*
    +rm -fr /var/log/anaconda
     rm -fr /usr/share/doc/*
    +rm -fr /usr/share/backgrounds/*
    +rm -fr /usr/share/kde4/*
    +rm -fr /var/lib/yum/yumdb/*

I mount the different squashfs.img file:

    /dev/loop0      312M  312M     0 100% /mnt/1
    /dev/loop1       21G  806M   20G   4% /media/1          <-- before clean

    /dev/loop2      315M  315M     0 100% /mnt/2
    /dev/loop3       21G  619M   21G   3% /media/2          <-- after clean

the install size was saved 806M - 619M = 187M, but squashfs.img only save 315M - 312M = 3M
where has the spare space gone ? 

    # losetup -l
    NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
    /dev/loop0         0      0         1  0 /var/tmp/tmp3vk1TM/LiveOS/squashfs.img
    /dev/loop1         0      0         1  1 /mnt/1/LiveOS/rootfs.img
    /dev/loop2         0      0         1  0 /var/tmp/tmpWRiKra/LiveOS/squashfs.img
    /dev/loop3         0      0         1  1 /mnt/2/LiveOS/rootfs.img

    # du -sh /var/tmp/tmp3vk1TM/LiveOS/squashfs.img
    312M    /var/tmp/tmp3vk1TM/LiveOS/squashfs.img

    # du -sh /var/tmp/tmpWRiKra/LiveOS/squashfs.img
    315M    /var/tmp/tmpWRiKra/LiveOS/squashfs.img

    # find . -name rootfs.img|xargs du -sh
    1.1G    ./1/LiveOS/rootfs.img
    1.1G    ./2/LiveOS/rootfs.img


the different of mounted rootfs size was mainly cleaned locale directory:

# du -sh /media/1/*|sort -hr                            # du -sh /media/2/*|sort -hr
727M    /media/1/usr            <-- cleaned -->         540M    /media/2/usr
42M     /media/1/boot                                   42M     /media/2/boot
21M     /media/1/var                                    21M     /media/2/var
9.6M    /media/1/etc                                    9.6M    /media/2/etc
3.2M    /media/1/opt                                    3.2M    /media/2/opt
144K    /media/1/root                                   144K    /media/2/root
12K     /media/1/lost+found                             12K     /media/2/lost+found
11K     /media/1/tmp                                    9.0K    /media/2/tmp
1.0K    /media/1/sys                                    1.0K    /media/2/sys
1.0K    /media/1/srv                                    1.0K    /media/2/srv
1.0K    /media/1/run                                    1.0K    /media/2/run
1.0K    /media/1/proc                                   1.0K    /media/2/proc
1.0K    /media/1/mnt                                    1.0K    /media/2/mnt
1.0K    /media/1/media                                  1.0K    /media/2/media
1.0K    /media/1/home                                   1.0K    /media/2/home
1.0K    /media/1/dev                                    1.0K    /media/2/dev
0       /media/1/sbin                                   0       /media/2/sbin
0       /media/1/lib64                                  0       /media/2/lib64
0       /media/1/lib                                    0       /media/2/lib
0       /media/1/bin                                    0       /media/2/bin

# du -sh /media/1/usr/lib/*|sort -hr|head               # du -sh /media/2/usr/lib/*|sort -hr|head
103M    /media/1/usr/lib/modules                        103M    /media/2/usr/lib/modules
102M    /media/1/usr/lib/locale       <-- locale -->    2.0M    /media/2/usr/lib/locale
52M     /media/1/usr/lib/firmware                       52M     /media/2/usr/lib/firmware
9.1M    /media/1/usr/lib/udev                           9.1M    /media/2/usr/lib/udev
8.1M    /media/1/usr/lib/python2.7                      8.1M    /media/2/usr/lib/python2.7
4.2M    /media/1/usr/lib/grub                           4.2M    /media/2/usr/lib/grub
3.9M    /media/1/usr/lib/systemd                        3.9M    /media/2/usr/lib/systemd
1.8M    /media/1/usr/lib/kbd                            1.8M    /media/2/usr/lib/kbd
710K    /media/1/usr/lib/dracut                         710K    /media/2/usr/lib/dracut
280K    /media/1/usr/lib/rpm                            280K    /media/2/usr/lib/rpm

# du -sh /media/1/usr/share/*|sort -hr|head             # du -sh /media/2/usr/share/*|sort -hr|head
79M     /media/1/usr/share/locale     <-- locale -->    22M     /media/2/usr/share/vim
22M     /media/1/usr/share/vim                          17M     /media/2/usr/share/perl5
17M     /media/1/usr/share/perl5                        13M     /media/2/usr/share/nmap
13M     /media/1/usr/share/nmap                         13M     /media/2/usr/share/man
13M     /media/1/usr/share/man                          5.2M    /media/2/usr/share/hwdata
9.0M    /media/1/usr/share/cracklib                     4.3M    /media/2/usr/share/cracklib
8.6M    /media/1/usr/share/i18n                         4.2M    /media/2/usr/share/mime
5.2M    /media/1/usr/share/hwdata                       3.0M    /media/2/usr/share/i18n
4.2M    /media/1/usr/share/mime                         2.9M    /media/2/usr/share/misc
2.9M    /media/1/usr/share/misc                         2.7M    /media/2/usr/share/info

how could I optimize the cleaned spare space of squashfs.img

thanks

2014-07-19 12:06 GMT+08:00 耳东 <volcanowill@xxxxxxxxx>:

@Vratislav Podzimek

the difference of ks files was only 'dracut-network' package :

    # find  . -name 'anaconda-ks.cfg'|xargs ls
    ./1/root/anaconda-ks.cfg  ./2/root/anaconda-ks.cfg

    # find  . -name 'anaconda-ks.cfg'|xargs diff
    115a116
    > dracut-network

second liveos only add 'dracut-network' package to build 'livenet' module into initrd.img.

@Brian C. Lane, I have sort all installed packges at the end of %post install section:

    %post --logfile /dev/stdout
    ...
    rpm -qa|sort > /root/rpm-qa.txt
    yum clean all
    rm -f /var/lib/rpm/__db*
    rm -fr /usr/share/doc
    %end

    # find  . -name 'rpm-qa.txt'|xargs ls
    ./1/root/rpm-qa.txt  ./2/root/rpm-qa.txt

    # find  . -name 'rpm-qa.txt'|xargs diff
    65a66
    > dracut-network-033-161.el7.x86_64

Because dracut rebuild initramfs.img after post install, so I boot from this livecd,
and diff the livcos's rpm packags with %post end 'rpm -qa' sort. they are all the same.
so the dracut rebuild initramfs.img not install extra packages.

This is my ks.cfg paste :

    http://ix.io/dsS

mount rootfs.img :

    # losetup -l
    NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
    /dev/loop0         0      0         1  0 /var/tmp/tmpc4zHMy/images/install.img
    /dev/loop1         0      0         1  1 /mnt/1/LiveOS/rootfs.img
    /dev/loop2         0      0         1  0 /var/tmp/tmp4Gzruj/images/install.img
    /dev/loop3         0      0         1  1 /mnt/2/LiveOS/rootfs.img

    # du -sh /mnt/1/LiveOS/rootfs.img /mnt/2/LiveOS/rootfs.img
    1000M   /mnt/1/LiveOS/rootfs.img
    1.1G    /mnt/2/LiveOS/rootfs.img

    # df -h
    /dev/loop0      296M  296M     0 100% /mnt/1
    /dev/loop1       21G  817M   20G   4% /media/1      <-- root size
    /dev/loop2      346M  346M     0 100% /mnt/2
    /dev/loop3       21G  828M   20G   4% /media/2      <-- root size

then I 'du -sh *' the root filesystem of rootfs.img:

    # du -sh /media/*
    813M    /media/1
    824M    /media/2

    # du -sh /media/1/*|sort -hr                 # du -sh /media/2/*|sort -hr
    748M    /media/1/usr                         748M    /media/2/usr
    31M     /media/1/boot    <-- diff size -->   42M     /media/2/boot
    22M     /media/1/var                         22M     /media/2/var
    9.6M    /media/1/etc                         9.6M    /media/2/etc
    3.2M    /media/1/opt                         3.2M    /media/2/opt
    143K    /media/1/root                        143K    /media/2/root
    12K     /media/1/lost+found                  12K     /media/2/lost+found
    11K     /media/1/tmp                         11K     /media/2/tmp
    1.0K    /media/1/sys                         1.0K    /media/2/sys
    1.0K    /media/1/srv                         1.0K    /media/2/srv
    1.0K    /media/1/run                         1.0K    /media/2/run
    1.0K    /media/1/proc                        1.0K    /media/2/proc
    1.0K    /media/1/mnt                         1.0K    /media/2/mnt
    1.0K    /media/1/media                       1.0K    /media/2/media
    1.0K    /media/1/home                        1.0K    /media/2/home
    1.0K    /media/1/dev                         1.0K    /media/2/dev
    0       /media/1/sbin                        0       /media/2/sbin
    0       /media/1/lib64                       0       /media/2/lib64
    0       /media/1/lib                         0       /media/2/lib
    0       /media/1/bin                         0       /media/2/bin

I suspect livemedia-creator can NOT release the temp space under install or dracut generate initramfs.img.
mksquashfs could NOT ignore these sparse space.

2014-07-19 1:20 GMT+08:00 Brian C. Lane <bcl@xxxxxxxxxx>:

On Fri, Jul 18, 2014 at 01:12:44PM +0800, 耳东 wrote:
> hi guys. I want to build a cutom liveos by lorax.
> my first liveos , dracut miss 'livenet' module, so it can **NOT** boot
> under PXE
> boot failed message:
>
>     dracut: FATAL: Don't Know how to handle 'root=live:http://
> ...squashfs.img'
>
> first create command:
>
>     livemedia-creator --no-virt --make-iso --ks=centos7.liveos.ks.cfg
>
> After I add 'dracut-network' package in centos7.liveos.ks.cfg '%package'
> section,
> the second create liveos with 'livenet' module can boot from PXE
> successfully:
>
>     livemedia-creator --dracut-arg="--no-hostonly" --no-virt --make-iso
> --ks=centos7.liveos.ks.cfg
>
> But squashfs.img size increased was larger than the size initramfs.img
> increased.

Adding packages changes the squashfs.img (the root filesystem). dracut
uses some of these to create the initrd. dracut-network will pull in
other packages (eg. dhclient, iputils, iproute) which may not have been
present.

The easiest way to tell is to mount both squashfs.img files and diff
their contents.

--
Brian C. Lane | Anaconda Team | IRC: bcl #anaconda | Port Orchard, WA (PST8PDT)

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/anaconda-devel-list

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/anaconda-devel-list