On Fri, Jan 12, 2024 at 7:48 AM Mirsad Todorovac
<mirsad.todorovac@xxxxxxxxxxxx> wrote:
>
> On 11. 01. 2024. 16:37, Nicolas Schier wrote:
> > Hi Mirsad,
> >
> > On Thu 11 Jan 2024 13:22:39 GMT, Mirsad Todorovac wrote:
> >> Hi,
> >>
> >> With this new release, it seems that Debian kernel build uses "xz" in single-
> >> threaded mode:
New release of what?
> >>
> >> Tasks: 484 total, 2 running, 481 sleeping, 0 stopped, 1 zombie
> >> %Cpu(s): 2.5 us, 2.2 sy, 6.3 ni, 85.1 id, 2.3 wa, 0.0 hi, 1.7 si, 0.0 st
> >> MiB Mem : 64128.3 total, 524.3 free, 5832.0 used, 58540.9 buff/cache
> >> MiB Swap: 32760.0 total, 32758.7 free, 1.2 used. 58296.3 avail Mem
> >>
> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> >> COMMAND
> >>
> >> 978084 marvin 30 10 112440 97792 2432 R 100.0 0.1 29:30.23 xz
> >>
> >>
> >> Before dpkg-deb was using up to 3200% of CPU time on a 16 core SMT CPU.
> >>
> >> Can it be something with dpkg-deb --thread-max=%n option?
> >
> > I cannot find any --thread-max option in Linux tree. Do you call
> > dpkg-deb manually or somehow induce a thread maximum?
> >
> >> Waiting for half an hour just for the build of linux-image-...-dbg package
> >> seems like an overkill ...
> >
> > With current v6.7 release tree I do not see the reported slow-downs
> > when building bindeb-pkg; I tested by cross-compiling for arm64 on
> > amd64 with CONFIG_MODULE_COMPRESS_XZ=y and =n).
> >
> > Both take roughly 5mins on my 24-core i9 system.
> >
> > Kind regards,
> > Nicolas
>
> I am perplexed too, but you can see from the top output the
> single-threaded xz with 29:30m processor time.
>
> On my laptop with the sam Ubuntu 23.10 mantic minotaur, I have
> dpkg-deb version 1.20.12 and it shows things like 400% and 3200%
> CPU time, so it is working multithreaded.
>
> On desktop machine with the same Ubuntu 23.10 and the same git
> torvalds tree, it starts single-threaded xz from dpkg-deb instead.
You built the same mainline git tree on your laptop and desktop.
The former runs xz multi-threaded, the latter single-threaded.
So, this is not about the kernel code, but about your environment,
isn't it?
You mentioned you were using Ubuntu 23.10, where
dpkg-deb version is 1.22.0
Your dpkg-deb version, 1.20.12, is too old for Ubuntu 23.10.
Is it a self-built one?
dpkg-deb usually compresses binary packages, but the default
compression type depends on the distro.
(It is determined at "./configure" time)
On Ubuntu, the default compression type for dpkg-deb is "zstd"
(while it is "xz" on Debian)
Check "man dpkg-deb" on your Ubuntu machine.
-Zcompress‐type
Specify which compression type to use when building a package.
Allowed values are gzip, xz (since dpkg 1.15.6),
zstd (since dpkg 1.21.18) and none (default is zstd).
You are still allowed to use xz with "make KDEB_COMPRESS=xz deb-pkg".
Is this your case?
Overall, your report is not sensible.
You should check what you are seeing.
> I tried things like this:
>
> diff --git a/scripts/package/builddeb b/scripts/package/builddeb
> index d7dd0d04c70c..b2319c23db34 100755
> --- a/scripts/package/builddeb
> +++ b/scripts/package/builddeb
> @@ -38,7 +38,7 @@ create_package() {
>
> # Fix ownership and permissions
> if [ "$DEB_RULES_REQUIRES_ROOT" = "no" ]; then
> - dpkg_deb_opts="--root-owner-group"
> + dpkg_deb_opts="--threads-max=0 --root-owner-group"
> else
> chown -R root:root "$pdir"
> fi
>
> and it didn't work either - dpkg-deb --threads-max=0 still spawned a
> single-threaded xz that ran 30 minutes.
>
> Then the workaround was a very simple xz shell script that adds option --threads=0
> and calls system xz:
>
> ~/bin/xz:
> ----------------------------------------------------------------
> #!/bin/bash -f
>
> /usr/bin/xz --threads=0 "$@"
> ----------------------------------------------------------------
>
> This finally worked, but sometimes I get:
>
> marvin@defiant:~/linux/kernel$ xz -9 --memlimit-compress=8000MiB linux-image-6.7.0-rc8-dbg_6.7.0-rc8-6_amd64.deb
> /usr/bin/xz: Reduced the number of threads from 8 to 6 to not exceed the memory usage limit of 8000 MiB
>
> (This is of course just an example of compressing a large file, as .deb is already compressed.)
>
> I used the default Ubuntu 23.10 config, with .pems excluded, and I think module compression
> did not work either. I had to turn it off ...
>
> Hope this helps.
>
> Regards,
> Mirsad
>
> --
> Mirsad Goran Todorovac
> Sistem inženjer
> Grafički fakultet | Akademija likovnih umjetnosti
> Sveučilište u Zagrebu
>
> System engineer
> Faculty of Graphic Arts | Academy of Fine Arts
> University of Zagreb, Republic of Croatia
> The European Union
>
> "I see something approaching fast ... Will it be friends with me?"
>
>
--
Best Regards
Masahiro Yamada
