Re: Remove gcc, gcc-c++ and make from minimal build root

[Vít Ondruch <vondruch@xxxxxxxxxx>]

Forgot to add this was measured on my Lenovo T440s, with SSD and 12 GB RAM, running F21 with mock 1.2.3, configured to use LVM plugin and DNF (see attached config files). And network? Who knows, probably fast. Should be 1Gb/s if I am not mistaken.

Not sure how similar/different the tests could look on Fedoras Koji builders.


Vít



Dne 13.1.2015 v 15:56 Vít Ondruch napsal(a):

Dne 13.1.2015 v 14:02 Richard W.M. Jones napsal(a):

On Tue, Jan 13, 2015 at 01:31:06PM +0100, Sven Lankes wrote:

On Tue, Jan 13, 2015 at 12:16:42PM +0000, Richard W.M. Jones wrote:

That's simply an idiotic thing to say.  What is the progress involved
in adding a new BR to thousands of packages?

Am I wrong assuming that it would be rather easy to add the BR to all
packages semi-automatically previous to a mass rebuild so that people
who want to get rid of gcc for their packages can easily remove it?

What am I missing?

What is missing is there is no quantifcation yet of the benefit.
How much faster will builds be?

Since you are asking I did 5 times:

$ mock -r minimal --scrub all && time mock -r minimal --init

Where the "Total" is DNF's output of package download speed and amount of downloaded data obtained from mock output. Other DNF information are shown on top of each section as well.


== with current minimal build root, i.e. "install @buildsys-build"

Install  164 Packages

Total download size: 119 M
Installed size: 445 M

1)
Total                                            26 MB/s | 119 MB     00:04
real    0m51.364s
user    0m33.050s
sys    0m6.454s

2)
Total                                            13 MB/s | 119 MB     00:09
real    0m51.479s
user    0m33.024s
sys    0m3.400s

3)
Total                                            19 MB/s | 119 MB     00:06
real    0m52.373s
user    0m33.171s
sys    0m7.121s

4)
Total                                            21 MB/s | 119 MB     00:05
real    0m54.213s
user    0m32.999s
sys    0m7.122s

5)
Total                                            13 MB/s | 119 MB     00:09
real    0m56.093s
user    0m32.708s
sys    0m5.745s


== without gcc, gcc-c++ and make, i.e. "install rpm-build shadow-utils util-linux which"

Install  153 Packages

Total download size: 81 M
Installed size: 350 M

1)
Total                                            14 MB/s |  81 MB     00:05
real    0m49.789s
user    0m28.926s
sys    0m6.864s

2)
Total                                            17 MB/s |  81 MB     00:04
real    0m44.010s
user    0m28.965s
sys    0m3.434s

3)
Total                                            18 MB/s |  81 MB     00:04
real    0m45.860s
user    0m29.032s
sys    0m5.730s

4)
Total                                            19 MB/s |  81 MB     00:04
real    0m42.047s
user    0m28.944s
sys    0m3.099s

5)
Total                                            14 MB/s |  81 MB     00:05
real    0m47.079s
user    0m29.839s
sys    0m3.978s


== without gcc, gcc-c++, make and without perl (rhbz#1158860, note that the rpm packages are local in this case), i.e. "install rpm-build shadow-utils util-linux which"

Install  124 Packages

Total size: 69 M
Total download size: 68 M
Installed size: 310 M

1)
Total                                            11 MB/s |  68 MB     00:06
real    0m44.725s
user    0m27.644s
sys    0m6.126s

2)
Total                                            12 MB/s |  68 MB     00:05
real    0m46.253s
user    0m27.624s
sys    0m7.056s

3)
Total                                           6.5 MB/s |  68 MB     00:10
real    0m50.296s
user    0m27.567s
sys    0m6.064s

4)
Total                                            12 MB/s |  68 MB     00:05
real    0m46.078s
user    0m27.581s
sys    0m7.698s

5)
Total                                            16 MB/s |  68 MB     00:04
real    0m44.307s
user    0m27.830s
sys    0m5.293s

== Only rpmbuild install "install rpm-build shadow-utils"

Install  111 Packages

Total size: 61 M
Total download size: 60 M
Installed size: 286 M

1)
Total                                            20 MB/s |  60 MB     00:03
real    0m37.848s
user    0m26.158s
sys    0m2.676s

2)
Total                                            13 MB/s |  60 MB     00:04
real    0m44.756s
user    0m26.442s
sys    0m5.185s

3)
Total                                            16 MB/s |  60 MB     00:03
real    0m42.703s
user    0m26.118s
sys    0m3.174s

4)
Total                                            13 MB/s |  60 MB     00:04
real    0m42.365s
user    0m26.439s
sys    0m4.193s

5)
Total                                           5.9 MB/s |  60 MB     00:10
real    0m48.940s
user    0m26.608s
sys    0m4.576s

== Minimal root I can shell in, i.e. "install bash shadow-utils"

Install  48 Packages

Total download size: 31 M
Installed size: 176 M

1)
Total                                           8.3 MB/s |  31 MB     00:03
real    0m36.745s
user    0m22.019s
sys    0m4.936s

2)
Total                                           9.5 MB/s |  31 MB     00:03
real    0m32.608s
user    0m21.805s
sys    0m2.719s

3)
Total                                            11 MB/s |  31 MB     00:02
real    0m34.111s
user    0m22.170s
sys    0m3.651s

4)
Total                                            12 MB/s |  31 MB     00:02
real    0m41.141s
user    0m22.595s
sys    0m5.535s

5)
Total                                           8.4 MB/s |  31 MB     00:03
real    0m34.303s
user    0m21.851s
sys    0m3.839s


And several times during the test I encountered bug similar to:

Downloading Packages:
Error: Error downloading packages:
  Curl error: Failure when receiving data from the peer for ftp://mirror.slu.cz/fedora/linux/development/rawhide/x86_64/os/Packages/e/elfutils-libelf-0.161-1.fc22.x86_64.rpm [response reading failed]

Less packages to download, less possible issues.


You can interpret these data yourself, but with less packages in build root, I can see:

1) Saved build time
2) Saved network bandwidth
3) Saved storage
4) Less things to break

  Versus how long will it take to
either modify all these packages / write and run the script to do
that?  Is A > B?

There are things which are hard to quantify. If you want to prohibit any change, this is probably right question to ask. Of course answering such questions also takes some time, I hope you will count it into the final equation at the end, although hard to say on which side it should belong.


Vít

config_opts['root'] = 'minimal'
config_opts['target_arch'] = 'x86_64'
config_opts['legal_host_arches'] = ('x86_64',)
#config_opts['chroot_setup_cmd'] = 'install @buildsys-build'
#config_opts['chroot_setup_cmd'] = 'install rpm-build shadow-utils util-linux which'
#config_opts['chroot_setup_cmd'] = 'install rpm-build shadow-utils'
config_opts['chroot_setup_cmd'] = 'install bash shadow-utils'

#rpm-build
#shadow-utils

#bash
#bzip2
#coreutils
#cpio
#diffutils
#fedora-release
#findutils
#gawk
#gcc
#gcc-c++
#grep
#gzip
#info
#make
#patch
#redhat-rpm-config
#sed
#tar
#unzip
#util-linux
#which
#xz

config_opts['dist'] = 'rawhide'  # only useful for --resultdir variable subst
config_opts['extra_chroot_dirs'] = [ '/run/lock', ]
config_opts['releasever'] = '22'

config_opts['yum.conf'] = """
[main]
cachedir=/var/cache/yum
debuglevel=1
reposdir=/dev/null
logfile=/var/log/yum.log
retries=20
obsoletes=1
gpgcheck=0
assumeyes=1
syslog_ident=mock
syslog_device=

# repos

[fedora]
name=fedora
metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=$basearch
failovermethod=priority

[local]
name=local
baseurl=http://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/
cost=2000
enabled=0

[debug]
name=Fedora Rawhide - x86_64 - Debug
failovermethod=priority
metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide-debug&arch=$basearch
enabled=0

[rpm]
name=rpm
baseurl=file:///var/lib/mock/fedora-rawhide-x86_64/result/
enabled=0

"""

# mock defaults
# vim:tw=0:ts=4:sw=4:et:
#
# This config file is for site-specific default values that apply across all
# configurations. Options specified in this config file can be overridden in
# the individual mock config files.
#
# The site-defaults.cfg delivered by default has NO options set. Only set
# options here if you want to override the defaults.
#
# Entries in this file follow the same format as other mock config files.
# config_opts['foo'] = bar

#############################################################################
#
# Things that we recommend you set in site-defaults.cfg:
#
# config_opts['basedir'] = '/var/lib/mock/'
# config_opts['cache_topdir'] = '/var/cache/mock'
#  Note: the path pointed to by basedir and cache_topdir must be owned
#        by group 'mock' and must have mode: g+rws
# config_opts['rpmbuild_timeout'] = 0
# config_opts['use_host_resolv'] = True

# You can configure log format to pull from logging.ini formats of these names:
# config_opts['build_log_fmt_name'] = "unadorned"
# config_opts['root_log_fmt_name']  = "detailed"
# config_opts['state_log_fmt_name'] = "state"
#
# mock will normally set up a minimal chroot /dev.
# If you want to use a pre-configured /dev, disable this and use the bind-mount
# plugin to mount your special /dev
# config_opts['internal_dev_setup'] = True
#
# internal_setarch defaults to 'True' if the python 'ctypes' package is
# available. It is in the python std lib on >= python 2.5. On older versions,
# it is available as an addon. On systems w/o ctypes, it will default to 'False'
# config_opts['internal_setarch'] = False
#
# the cleanup_on_* options allow you to automatically clean and remove the
# mock build directory, but only take effect if --resultdir is used.
# config_opts provides fine-grained control. cmdline only has big hammer
#
# config_opts['cleanup_on_success'] = 1
# config_opts['cleanup_on_failure'] = 1

# The default package manager is Yum
config_opts['package_manager'] = 'dnf'
# If you want to use DNF, set it to 'dnf'. To use DNf you need to have dnf and
# dnf-plugins-core installed

# You can configure Yum, DNF, rpm and rpmbuild executable paths if you need to
# use different versions that the system-wide ones
# config_opts['yum_command'] = '/usr/bin/yum'
# config_opts['yum_builddep_command'] = '/usr/bin/yum-builddep'
# config_opts['dnf_command'] = '/usr/bin/dnf'
# config_opts['rpm_command'] = '/bin/rpm'
# config_opts['rpmbuild_command'] = '/usr/bin/rpmbuild'
#
# By default a Yum/DNF update is performed before each rebuild
# config_opts['update_before_build'] = True

# if you want mock to automatically run createrepo on the rpms in your
# resultdir.
# config_opts['createrepo_on_rpms'] = False
# config_opts['createrepo_command'] = '/usr/bin/createrepo_c -d -q -x *.src.rpm'

# if you want mock to backup the contents of a result dir before clean
# config_opts['backup_on_clean'] = False
# config_opts('backup_base_dir'] = config_opts['basedir'] + "backup"

# if you want to speed up the package installation and the build process, mock
# can use nosync library to skip fsync and related calls from programs called
# from within mock. It needs nosync library to be installed and for multilib
# target, it requires both architectures of nosync to be present. If these
# conditions aren't met, it won't be enabled
config_opts['nosync'] = True
# if you cannot or don't want to install both architectures of nosync and still
# want mock to use it, you can force it, but then expect a lot of (generally
# harmless) error messages from ld.so when a 32bit program is executed
# config_opts['nosync_force'] = False


#############################################################################
#
# plugin related. Below are the defaults. Change to suit your site
# policy. site-defaults.cfg is a good place to do this.
#
# NOTE: Some of the caching options can theoretically affect build
#  reproducability. Change with care.
#
# config_opts['plugin_conf']['package_state_enable'] = False
# config_opts['plugin_conf']['ccache_enable'] = True
# config_opts['plugin_conf']['ccache_opts'] = {}
# config_opts['plugin_conf']['ccache_opts']['max_cache_size'] = '4G'
# config_opts['plugin_conf']['ccache_opts']['compress'] = None
# config_opts['plugin_conf']['ccache_opts']['dir'] = "%(cache_topdir)s/%(root)s/ccache/"
# config_opts['plugin_conf']['yum_cache_enable'] = True
# config_opts['plugin_conf']['yum_cache_opts'] = {}
# config_opts['plugin_conf']['yum_cache_opts']['max_age_days'] = 30
# config_opts['plugin_conf']['yum_cache_opts']['dir'] = "%(cache_topdir)s/%(root)s/yum_cache/"
config_opts['plugin_conf']['root_cache_enable'] = False
# config_opts['plugin_conf']['root_cache_opts'] = {}
# config_opts['plugin_conf']['root_cache_opts']['max_age_days'] = 15
# config_opts['plugin_conf']['root_cache_opts']['dir'] = "%(cache_topdir)s/%(root)s/root_cache/"
# config_opts['plugin_conf']['root_cache_opts']['compress_program'] = "pigz"
# config_opts['plugin_conf']['root_cache_opts']['extension'] = ".gz"
# config_opts['plugin_conf']['root_cache_opts']['exclude_dirs'] = ["./proc", "./sys", "./dev",
#                                                                  "./tmp/ccache", "./var/cache/yum" ]
#
# bind mount plugin is enabled by default but has no configured directories to
# mount
# config_opts['plugin_conf']['bind_mount_enable'] = True
# config_opts['plugin_conf']['bind_mount_opts']['dirs'].append(('/host/path', '/bind/mount/path/in/chroot/' ))
#
# config_opts['plugin_conf']['tmpfs_enable'] = False
# config_opts['plugin_conf']['tmpfs_opts'] = {}
# config_opts['plugin_conf']['tmpfs_opts']['required_ram_mb'] = 1024
# config_opts['plugin_conf']['tmpfs_opts']['max_fs_size'] = '768m'
# config_opts['plugin_conf']['tmpfs_opts']['mode'] = '0755'
# config_opts['plugin_conf']['tmpfs_opts']['keep_mounted'] = False
#
# config_opts['plugin_conf']['chroot_scan_enable'] = False
# config_opts['plugin_conf']['chroot_scan_opts'] = [ "core(\.\d+)?", "\.log$",]
#
# lvm_root plugin is not enabled by default and is distributed in separate
# subpackage mock-lvm. If you want to use it, it's recommended to disable the
# root_cache plugin, otherwise you'd be caching twice.
config_opts['plugin_conf']['lvm_root_enable'] = True
# config_opts['plugin_conf']['lvm_root_opts'] = {}
# You need to give it a volume group with sufficient space. It won't touch any
# of the existing logical volumes, so you can use the same volume group you use
# for other purposes. It requires a name of the VG (not device path).
# config_opts['plugin_conf']['lvm_root_opts']['volume_group'] = 'my_vg'
# You need to specify the size that will mock's thinpool occupy. For regular
# packages with small set of dependencies, 2G should suffice. For large packages
# such as libreoffice, you should set it to at least 5 GB otherwise you may run
# out of space and the build process will be blocked
# config_opts['plugin_conf']['lvm_root_opts']['size'] = '2G'
# You can specify alternative pool metadata size, format is the same as size.
# Default value is determined by lvcreate based on size
# config_opts['plugin_conf']['lvm_root_opts']['poolmetadatasize'] = None
config_opts['plugin_conf']['lvm_root_opts'] = {
    'volume_group': 'vg_fedora',
    'size': '15G',
    'pool_name': 'mock',
}
#
### If you want to compress mock log, enable this plugin
# config_opts['plugin_conf']['compress_logs_enable'] = False
# config_opts['plugin_conf']['compress_logs_opts'] = {}
### Command used to compress logs - e.g. "/usr/bin/xz -9"
# config_opts['plugin_conf']['compress_logs_opts']['command'] = ""

#############################################################################
#
# environment for chroot
#
# config_opts['environment']['TERM'] = 'vt100'
# config_opts['environment']['SHELL'] = '/bin/bash'
# config_opts['environment']['HOME'] = '/builddir'
# config_opts['environment']['HOSTNAME'] = 'mock'
# config_opts['environment']['PATH'] = '/usr/bin:/bin:/usr/sbin:/sbin'
# config_opts['environment']['PROMPT_COMMAND'] = 'printf "\033]0;<mock-chroot>\007<mock-chroot>"'
# config_opts['environment']['LANG'] = os.environ.setdefault('LANG', 'en_US.UTF-8')
# config_opts['environment']['TZ'] = os.environ.setdefault('TZ', 'EST5EDT')

#############################################################################
#
# Things that you can change, but we dont recommend it:
# config_opts['chroothome'] = '/builddir'
# config_opts['clean'] = True

#############################################################################
#
# Things that must be adjusted if SCM integration is used:
#
# config_opts['scm'] = True
# config_opts['scm_opts']['method'] = 'git'
# config_opts['scm_opts']['cvs_get'] = 'cvs -d /srv/cvs co SCM_BRN SCM_PKG'
# config_opts['scm_opts']['git_get'] = 'git clone SCM_BRN git://localhost/SCM_PKG.git SCM_PKG'
# config_opts['scm_opts']['svn_get'] = 'svn co file:///srv/svn/SCM_PKG/SCM_BRN SCM_PKG'
# config_opts['scm_opts']['spec'] = 'SCM_PKG.spec'
# config_opts['scm_opts']['ext_src_dir'] = '/dev/null'
# config_opts['scm_opts']['write_tar'] = True
# config_opts['scm_opts']['git_timestamps'] = True
# config_opts['scm_opts']['exclude_vcs'] = True

# These options are also recognized but usually defined in cmd line
# with --scm-option package=<pkg> --scm-option branch=<branch>
# config_opts['scm_opts']['package'] = 'mypkg'
# config_opts['scm_opts']['branch'] = 'master'

#############################################################################
#
# Things that are best suited for individual chroot config files:
#
# MUST SET (in individual chroot cfg file):
# config_opts['root'] = 'name-of-yum-build-dir'
# config_opts['target_arch'] = 'i386'
# config_opts['yum.conf'] = ''
# config_opts['yum_common_opts'] = []
#
# CAN SET, defaults usually work ok:
# config_opts['chroot_setup_cmd'] = 'install buildsys-build'
# config_opts['log_config_file'] = 'logging.ini'
# config_opts['more_buildreqs']['srpm_name-version-release'] = 'dependencies'
# config_opts['macros']['%Add_your_macro_name_here'] = "add macro value here"
# config_opts['files']['path/name/no/leading/slash'] = "put file contents here."
# config_opts['chrootuid'] = os.getuid()
# config_opts['releasever'] = '20'

# If you change chrootgid, you must also change "mock" to the correct group
# name in this line of the mock PAM config:
#   auth  sufficient pam_succeed_if.so user ingroup mock use_uid quiet
# config_opts['chrootgid'] = grp.getgrnam("mock")[2]

# config_opts['useradd'] = '/usr/sbin/useradd -m -u %(uid)s -g %(gid)s -d %(home)s -n %(user)s' # Fedora/RedHat
#
# Security related
# config_opts['no_root_shells'] = False
#
# Proxy settings (https_proxy, ftp_proxy, and no_proxy can also be set)
# config_opts['http_proxy'] = 'http://localhost:3128'
#
# Extra dirs to be created when the chroot is initialized
# This is just a list of strings representing chroot paths such as:
#  [ '/run/lock', ]
# config_opts['extra_chroot_dirs'] = []

-- 
devel mailing list
devel@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct