On 1/22/25 14:54, Jann Horn wrote:
On Wed, Jan 22, 2025 at 2:31 PM Ard Biesheuvel <ardb@xxxxxxxxxx> wrote:
Hi Jann,
On Tue, 21 Jan 2025 at 23:16, Jann Horn <jannh@xxxxxxxxxx> wrote:
Support storing the kernel uncompressed for developers who want to quickly
iterate with one-off kernel builds.
Store it in the usual format with a 4-byte length suffix and keep this new
codepath as close as possible to the normal path where decompression
happens.
The other compression methods offered by the kernel take some time;
even LZ4 (which the kernel uses at compression level 9) takes ~2.8
seconds to compress a 110M large vmlinux.bin on my machine.
An alternate approach to this would be to offer customization of the LZ4
compression level through a kconfig variable; and yet another approach
would be to abuse the existing gzip decompression logic by storing the
kernel as "non-compressed" DEFLATE blocks, so that the decompression code
will essentially end up just doing a bunch of memcpy() calls.
This all seems pretty complicated, and adding yet another
(pseudo-)compression method is not great in terms of maintenance
burden, especially because there are other consumers of the compressed
images (both for bzImage and EFI zboot)
Did you try running gzip with -1 instead of -9? On my build machine,
this reduces the compression time of a defconfig bzImage build from
4.3 seconds to 0.9 seconds.
I tried lz4 with -1; that is very fast (240ms wall clock time on my
machine, and just 120ms user time):
$ ls -lh arch/x86/boot/compressed/vmlinux.bin
-rwxr-x--- 1 [...] 110M Jan 22 00:01 arch/x86/boot/compressed/vmlinux.bin
$ cat arch/x86/boot/compressed/vmlinux.bin | time lz4 -l -9 - - | wc -c
2.86user 0.04system 0:02.96elapsed 97%CPU (0avgtext+0avgdata 15756maxresident)k
0inputs+0outputs (0major+220minor)pagefaults 0swaps
46309676
$ cat arch/x86/boot/compressed/vmlinux.bin | time lz4 -l -1 - - | wc -c
0.12user 0.06system 0:00.24elapsed 75%CPU (0avgtext+0avgdata 15524maxresident)k
0inputs+0outputs (0major+94minor)pagefaults 0swaps
56029608
But I wasn't sure how to wire that up in a nice way. I guess the
nicest option would be to create a separate kconfig variable for the
compression level to use for any cmd_lz4/cmd_lz4_with_size invocations
in the build process; and then maybe only make this option visible if
LZ4 is selected as kernel compression method?
Another option would be to create a new option in the "Kernel
compression mode" choice menu with a name like "LZ4 (fast)", turn
CONFIG_KERNEL_LZ4 into an internal flag that is selected by both LZ4
variants shown in the choice menu, and duplicate some of the make
rules, but that seems overly complicated.
Hello,
In my opinion 'lz4 -9' doesn't make much sense.
It's terribly slow and the compression ratio is also not exactly good.
Instead, zstd seems to be a much better choice. Not quite as ultra fast
as lz4 levels 1 to 3, but much better compression.
As an example, I compressed a kernel source tarball (zstd is
multithreaded with 4 threads here, which are not fully used with
small-ish files like vmlinux):
- zstd -3: from 1.32 GB to 199 MB in 5.23 seconds
- zstd -6: to 173 MB in 10.77 seconds
- zstd -10: to 165 MB in 24.52 seconds
- lz4 -1: to 373 MB in 1.60 seconds
- lz4 -3: to 278 MB in 6.45 seconds
- lz4 -9: to 266 MB in 22.03 seconds
And a vmlinux from my installed kernel 6.13:
- zstd -3: from 51.8 MB to 16.7 MB in 0.60 seconds
- zstd -6: to 15.8 MB in 1.39 seconds
- zstd -10: to 15.3 MB in 3.54 seconds
- lz4 -1: to 23.7 MB in 0.08 seconds
- lz4 -3: to 20.2 MB in 0.51 seconds
- lz4 -9: to 19.7 MB in 1.23 seconds
For my kernel, I use a Kconfig option to choose a zstd compression
level. I could submit it if there is interest.
Tor
