On Sun, Jan 14, 2024 at 6:09 AM Mirsad Todorovac <mirsad.todorovac@xxxxxxxxxxxx> wrote: > > On 1/13/24 16:16, Masahiro Yamada wrote: > > On Sat, Jan 13, 2024 at 8:28 PM Mirsad Todorovac > > <mirsad.todorovac@xxxxxxxxxxxx> wrote: > >> > >> On 1/13/24 06:40, Masahiro Yamada wrote: > >>> On Fri, Jan 12, 2024 at 7:48 AM Mirsad Todorovac > >>> <mirsad.todorovac@xxxxxxxxxxxx> wrote: > >>>> > >>>> On 11. 01. 2024. 16:37, Nicolas Schier wrote: > >>>>> Hi Mirsad, > >>>>> > >>>>> On Thu 11 Jan 2024 13:22:39 GMT, Mirsad Todorovac wrote: > >>>>>> Hi, > >>>>>> > >>>>>> With this new release, it seems that Debian kernel build uses "xz" in single- > >>>>>> threaded mode: > >> > >> Hi, Masahiro, > >> > >> Thank you for your reply. > >> > >>> New release of what? > >> > >> Forgive me for not being precise. It was the new release of torvalds tree mainline > >> kernel 6.7+ (with merge commits up-to-date). I sort of mistakenly assumed that > >> I wrote it, but it was declared in some mail on LKML. > > > > When you report a regression in the kernel code in the future, > > please try to do "git bisect" and pin-point an offending commit. > > > > That is more helpful to figure out how to fix the issue. > > > > And, you will be sure whether or not the root cause exists > > in the kernel. > > Good point. Thanks. > > But I did not notice any regression on Ubuntu 22.04 LTS system, and neither on > the Ubuntu 22.10 Mantic system which was upgraded. So I assumed it was not related to > particular commit, and I did not think of bisect. > > >> One environment was Ubuntu 23.10 Mantic Minotaur on the desktop at work. > >> > >> The laptop also has Mantic Minotaur, but for some reason it spawns multithreaded > >> dpkg-deb when building packages, unlike desktop which spawned single-threaded "xz". > >> > >> This is at least what the "top" displayed when turning on "H" (show threads). > >> > >> On one system (upgraded 23.10 from 22.04 LTS) the same dpkg-deb shows as multi-threaded > >> dpkg-deb, while on the other it calls xz which didn't respect XT_OPT=-T0 (visible > >> from the copy+paste from top output). > >> > >> I am perplexed myself. > >> > >>>>>> Tasks: 484 total, 2 running, 481 sleeping, 0 stopped, 1 zombie > >>>>>> %Cpu(s): 2.5 us, 2.2 sy, 6.3 ni, 85.1 id, 2.3 wa, 0.0 hi, 1.7 si, 0.0 st > >>>>>> MiB Mem : 64128.3 total, 524.3 free, 5832.0 used, 58540.9 buff/cache > >>>>>> MiB Swap: 32760.0 total, 32758.7 free, 1.2 used. 58296.3 avail Mem > >>>>>> > >>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > >>>>>> COMMAND > >>>>>> > >>>>>> 978084 marvin 30 10 112440 97792 2432 R 100.0 0.1 29:30.23 xz > >>>>>> > >>>>>> > >>>>>> Before dpkg-deb was using up to 3200% of CPU time on a 16 core SMT CPU. > >>>>>> > >>>>>> Can it be something with dpkg-deb --thread-max=%n option? > >>>>> > >>>>> I cannot find any --thread-max option in Linux tree. Do you call > >>>>> dpkg-deb manually or somehow induce a thread maximum? > >>>>> > >>>>>> Waiting for half an hour just for the build of linux-image-...-dbg package > >>>>>> seems like an overkill ... > >>>>> > >>>>> With current v6.7 release tree I do not see the reported slow-downs > >>>>> when building bindeb-pkg; I tested by cross-compiling for arm64 on > >>>>> amd64 with CONFIG_MODULE_COMPRESS_XZ=y and =n). > >>>>> > >>>>> Both take roughly 5mins on my 24-core i9 system. > >>>>> > >>>>> Kind regards, > >>>>> Nicolas > >>>> > >>>> I am perplexed too, but you can see from the top output the > >>>> single-threaded xz with 29:30m processor time. > >>>> > >>>> On my laptop with the sam Ubuntu 23.10 mantic minotaur, I have > >>>> dpkg-deb version 1.20.12 and it shows things like 400% and 3200% > >>>> CPU time, so it is working multithreaded. > >>>> > >>>> On desktop machine with the same Ubuntu 23.10 and the same git > >>>> torvalds tree, it starts single-threaded xz from dpkg-deb instead. > >>> > >>> You built the same mainline git tree on your laptop and desktop. > >>> The former runs xz multi-threaded, the latter single-threaded. > >>> > >>> So, this is not about the kernel code, but about your environment, > >>> isn't it? > >> > >> It should be. Somehow it doesn't behave rationally. To be true to the facts, > >> one is 23.10 Mantic Minotaur upgraded from 22.04 LTS, and the other is > >> fresh Mantic from install. > >> > >>> You mentioned you were using Ubuntu 23.10, where > >>> dpkg-deb version is 1.22.0 > >> > >> This is correct. > >> > >>> Your dpkg-deb version, 1.20.12, is too old for Ubuntu 23.10. > >>> Is it a self-built one? > >> > >> Yes, it is the build from Debian 11 source package. > >> > >> This had the similar rationale because Ubuntu version ran single-threaded. > >> > >> dpkg-deb from Debian 11 source package worked well for months. > >> > >>> dpkg-deb usually compresses binary packages, but the default > >>> compression type depends on the distro. > >>> (It is determined at "./configure" time) > >> > >> I see. But the home-built dpkg-deb also resorted to running "xz", and > >> xz did not recognise XZ_OPT environment variable. Very odd. :-/ > > > > It depends on the environment whether or not dpkg-deb > > spawns the external 'xz' command. > > > > With WITH_LIBLZMA defined, dpkg-deb uses internal code > > to compress the data with xz. [1] > > > > Without WITH_LIBLZMA, dpkg-deb sparns "xz" processes. [2] > > > > [1]: https://salsa.debian.org/dpkg-team/dpkg/-/blob/1.20.12/lib/dpkg/compress.c?ref_type=tags#L412 > > [2]: https://salsa.debian.org/dpkg-team/dpkg/-/blob/1.20.12/lib/dpkg/compress.c?ref_type=tags#L663 > > > > Since you said you saw "xz" in the "ps" command output, > > I believe your case is the latter. > > Thanks for your insight. It is obviously a compile-time define. > > In previous build the defaults were apparently different. The Debian defaulting to multi-threaded > dpkg-deb was the exact reason why I did the hand build in the first place. > > In the long run, it saved an awful lot of time. > > But I can't see the logic of enforcing the single-threaded "xz" from dpkg-deg :-/ > > > When dpkg-deb swarns the external "xz" command, > > dpkg-deb unsets "XZ_OPT". [3] > > > > I believe that's why you do not see XZ_OPT propagated. > > > > [3]: https://salsa.debian.org/dpkg-team/dpkg/-/blob/1.20.12/lib/dpkg/compress.c?ref_type=tags#L643 > > Your insight really casts light on this. Now it is very clear. > > But IMHO single-threaded 30-min xz compression on a 16-core or 32-SMT machine is kind of weird. > I wish there was something we could do about it. > > The same happened with the rpmbuild with similar defaults on Fedora. :-( > > >>> On Ubuntu, the default compression type for dpkg-deb is "zstd" > >>> (while it is "xz" on Debian) > >>> > >>> Check "man dpkg-deb" on your Ubuntu machine. > >>> > >>> -Zcompress‐type > >>> Specify which compression type to use when building a package. > >>> Allowed values are gzip, xz (since dpkg 1.15.6), > >>> zstd (since dpkg 1.21.18) and none (default is zstd). > >> > >> Verified. "man dpkg-deb" confirms that. > >> > >>> You are still allowed to use xz with "make KDEB_COMPRESS=xz deb-pkg". > >>> Is this your case? > >> > >> Somehow, my Mantic's "xz" did not react to the environment variables. Maybe it is > >> enchanted? :-/ > >>> Overall, your report is not sensible. > >> > >>> You should check what you are seeing. > >> > >> I was seeing exactly what I copy+pasted (top output). Give me the benefit of doubt > >> that I did not falsify copy+paste. Ideally, I should have submitted JPG terminal > >> screenshot. > >> > >> Home-built dpkg-deb worked well on Ubuntu 22.04 LTS and 23.10 (upgraded from 22.04). > >> So it was a surprise that it started showing quirks on this new box with fresh 23.10 > >> from .iso. > >> > >> But I am only 16 months in the Linux kernel development, so it is probably my fault. > >> > >> I saw from the output of "ps xewww" that "xz" had "XZ_OPT=-T0", but it refused to obey > >> the environment variable. > >> > >> This is probably worth digging and delving into to find the culprit, but eventually the > >> workaround was the script manually prepending "--threads=0" to the parameter list and > >> calling system xz: > >> > >> /usr/bin/xz --threads=0 "$@" > >> > >> Certainly, I need to establish "normality" before building complex kernels and applying > >> patches or the results will be non-reproducible and useless. > >> > >> I did the background search on your valuable contributions to the LK, but as we have this > >> problem with the vanilla installation of Ubuntu, maybe we can think of something to > >> optimise the LK build time? > > > > > > I do not think so. > > > > There already exists a solution to control the number of threads. > > > > See "man dpkg-deb" > > > > DPKG_DEB_THREADS_MAX > > Sets the maximum number of threads allowed for compressors that > > support multi‐threaded operations (since dpkg 1.21.9). > > > > The --threads-max option overrides this value. > > By setting this env variable, you should be able to control the multi-threading. > > > > > > However, your dpkg-deb is 1.20.12, so you need to use a newer version. > > > > You locally hacked builddeb to add --threads-max, but it does not > > work for you for the same reason; it requires dpkg 1.21.9+ > > > > --threads-max=threads > > Sets the maximum number of threads allowed for compressors that support > > multi‐threaded operations (since dpkg 1.21.9). > > > > Why do you stick to the old home-built dpkg-deb? > > > > If you use the default dpkg-deb bundled with Ubuntu 23.10, > > you will have a better experience. > > Yes, but I did it because Ubuntu dpkg-deb was single-threaded by default. I did not > know so many kbuild options at the time ... No. The dpkg-deb on Ubuntu works multi-threaded by default. Just try /usr/bin/dpkg-deb installed by APT, then you will see more than 100% CPU resources used. With proper library installed, "./configure" for dpkg should show: checking for lzma_stream_encoder_mt in -llzma... yes With HAVE_LZMA_MT_ENCODER defined, filter_xz_get_cputhreads() returns the number of threads to use. The default is the number of available CPU cores. [1]: https://salsa.debian.org/dpkg-team/dpkg/-/blob/main/lib/dpkg/compress.c?ref_type=heads#L717 > > > Or, if you still insist on a home-built dpkg-deb, > > why don't you build a newer version? > > That's a good idea. But with Debian defaults, I suppose :-) > > Thank you for all your insight you provided, it really cleared a lot of dubious things. > > Except that I thought I saw "XZ_OPT=-T0" in "ps xewww" environment listing of the xz > process itself. But right now I cannot access that box until Monday, so I rest my case. > > IMHO, the build process should have optimal defaults. This would eventually reduce the > load on the kbuild developers and maintainers, and increase overall user satisfaction. As above, dpkg-deb works in parallel. It is still possible to force the single-threading by "export DPKG_DEB_THREADS_MAX=1", but many people already benefit from multi-threaded compression, which is the default behavior on recent distributions. > > I will do what it takes, but for the real Linux newbies maybe it would be more user-friendly > to have the build system use all the cores out-of-the-box. But maybe not. > > There might be some cons to this, dunno. This way it is certainly more mystery ;-) > > Thanks again, and have a blessed weekend! > > Best regards, > Mirsad > > >> To recall better from Thursday, I actually tried to prepend "XZ_OPT=--threads=0" to > >> system xz, and it worked. But when called from dpkg-deb, it suddenly got amnesia, despite > >> XZ_OPT=-T0 being seen in "ps xewww" output :-/ > >> > >> I believe there should be a rational explanation rather than black or red magic. > >> > >> Thank you again for considering this problem report and have a nice weekend! > >> > >> Best regards, > >> Mirsad Todorovac > >> > >>>> I tried things like this: > >>>> > >>>> diff --git a/scripts/package/builddeb b/scripts/package/builddeb > >>>> index d7dd0d04c70c..b2319c23db34 100755 > >>>> --- a/scripts/package/builddeb > >>>> +++ b/scripts/package/builddeb > >>>> @@ -38,7 +38,7 @@ create_package() { > >>>> > >>>> # Fix ownership and permissions > >>>> if [ "$DEB_RULES_REQUIRES_ROOT" = "no" ]; then > >>>> - dpkg_deb_opts="--root-owner-group" > >>>> + dpkg_deb_opts="--threads-max=0 --root-owner-group" > >>>> else > >>>> chown -R root:root "$pdir" > >>>> fi > >>>> > >>>> and it didn't work either - dpkg-deb --threads-max=0 still spawned a > >>>> single-threaded xz that ran 30 minutes. > >>>> > >>>> Then the workaround was a very simple xz shell script that adds option --threads=0 > >>>> and calls system xz: > >>>> > >>>> ~/bin/xz: > >>>> ---------------------------------------------------------------- > >>>> #!/bin/bash -f > >>>> > >>>> /usr/bin/xz --threads=0 "$@" > >>>> ---------------------------------------------------------------- > >>>> > >>>> This finally worked, but sometimes I get: > >>>> > >>>> marvin@defiant:~/linux/kernel$ xz -9 --memlimit-compress=8000MiB linux-image-6.7.0-rc8-dbg_6.7.0-rc8-6_amd64.deb > >>>> /usr/bin/xz: Reduced the number of threads from 8 to 6 to not exceed the memory usage limit of 8000 MiB > >>>> > >>>> (This is of course just an example of compressing a large file, as .deb is already compressed.) > >>>> > >>>> I used the default Ubuntu 23.10 config, with .pems excluded, and I think module compression > >>>> did not work either. I had to turn it off ... > >>>> > >>>> Hope this helps. > >>>> > >>>> Regards, > >>>> Mirsad > >>>> > >>>> -- > >>>> Mirsad Goran Todorovac > >>>> Sistem inženjer > >>>> Grafički fakultet | Akademija likovnih umjetnosti > >>>> Sveučilište u Zagrebu > >>>> > >>>> System engineer > >>>> Faculty of Graphic Arts | Academy of Fine Arts > >>>> University of Zagreb, Republic of Croatia > >>>> The European Union > >>>> > >>>> "I see something approaching fast ... Will it be friends with me?" > >>>> > >>>> > >>> > >>> > > > > > > > -- Best Regards Masahiro Yamada
