On 11. 01. 2024. 16:37, Nicolas Schier wrote:
> Hi Mirsad,
>
> On Thu 11 Jan 2024 13:22:39 GMT, Mirsad Todorovac wrote:
>> Hi,
>>
>> With this new release, it seems that Debian kernel build uses "xz" in single-
>> threaded mode:
>>
>> Tasks: 484 total, 2 running, 481 sleeping, 0 stopped, 1 zombie
>> %Cpu(s): 2.5 us, 2.2 sy, 6.3 ni, 85.1 id, 2.3 wa, 0.0 hi, 1.7 si, 0.0 st
>> MiB Mem : 64128.3 total, 524.3 free, 5832.0 used, 58540.9 buff/cache
>> MiB Swap: 32760.0 total, 32758.7 free, 1.2 used. 58296.3 avail Mem
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> COMMAND
>>
>> 978084 marvin 30 10 112440 97792 2432 R 100.0 0.1 29:30.23 xz
>>
>>
>> Before dpkg-deb was using up to 3200% of CPU time on a 16 core SMT CPU.
>>
>> Can it be something with dpkg-deb --thread-max=%n option?
>
> I cannot find any --thread-max option in Linux tree. Do you call
> dpkg-deb manually or somehow induce a thread maximum?
>
>> Waiting for half an hour just for the build of linux-image-...-dbg package
>> seems like an overkill ...
>
> With current v6.7 release tree I do not see the reported slow-downs
> when building bindeb-pkg; I tested by cross-compiling for arm64 on
> amd64 with CONFIG_MODULE_COMPRESS_XZ=y and =n).
>
> Both take roughly 5mins on my 24-core i9 system.
>
> Kind regards,
> Nicolas
I am perplexed too, but you can see from the top output the
single-threaded xz with 29:30m processor time.
On my laptop with the sam Ubuntu 23.10 mantic minotaur, I have
dpkg-deb version 1.20.12 and it shows things like 400% and 3200%
CPU time, so it is working multithreaded.
On desktop machine with the same Ubuntu 23.10 and the same git
torvalds tree, it starts single-threaded xz from dpkg-deb instead.
I tried things like this:
diff --git a/scripts/package/builddeb b/scripts/package/builddeb
index d7dd0d04c70c..b2319c23db34 100755
--- a/scripts/package/builddeb
+++ b/scripts/package/builddeb
@@ -38,7 +38,7 @@ create_package() {
# Fix ownership and permissions
if [ "$DEB_RULES_REQUIRES_ROOT" = "no" ]; then
- dpkg_deb_opts="--root-owner-group"
+ dpkg_deb_opts="--threads-max=0 --root-owner-group"
else
chown -R root:root "$pdir"
fi
and it didn't work either - dpkg-deb --threads-max=0 still spawned a
single-threaded xz that ran 30 minutes.
Then the workaround was a very simple xz shell script that adds option --threads=0
and calls system xz:
~/bin/xz:
----------------------------------------------------------------
#!/bin/bash -f
/usr/bin/xz --threads=0 "$@"
----------------------------------------------------------------
This finally worked, but sometimes I get:
marvin@defiant:~/linux/kernel$ xz -9 --memlimit-compress=8000MiB linux-image-6.7.0-rc8-dbg_6.7.0-rc8-6_amd64.deb
/usr/bin/xz: Reduced the number of threads from 8 to 6 to not exceed the memory usage limit of 8000 MiB
(This is of course just an example of compressing a large file, as .deb is already compressed.)
I used the default Ubuntu 23.10 config, with .pems excluded, and I think module compression
did not work either. I had to turn it off ...
Hope this helps.
Regards,
Mirsad
--
Mirsad Goran Todorovac
Sistem inženjer
Grafički fakultet | Akademija likovnih umjetnosti
Sveučilište u Zagrebu
System engineer
Faculty of Graphic Arts | Academy of Fine Arts
University of Zagreb, Republic of Croatia
The European Union
"I see something approaching fast ... Will it be friends with me?"
