On Tue, Jul 7, 2020 at 6:03 AM Sedat Dilek <sedat.dilek@xxxxxxxxx> wrote: > > On Tue, Jul 7, 2020 at 5:50 AM Nick Terrell <nickrterrell@xxxxxxxxx> wrote: > > > > From: Nick Terrell <terrelln@xxxxxx> > > > > Please pull from > > > > git@xxxxxxxxxx:terrelln/linux.git tags/v6-zstd > > > > to get these changes. Alternatively the patchset is included. > > > > Hi Nick, > > cool, I just pulled from their. > Yesterday, I switched over from Linux v5.7.y to v5.8-rc4. > What a good timing :-). > > I will report later. > Looks good. $ scripts/diffconfig /boot/config-5.8.0-rc4-3-amd64-llvm11-ias /boot/config-5.8.0-rc4-4-amd64-llvm11-ias BUILD_SALT "5.8.0-rc4-3-amd64-llvm11-ias" -> "5.8.0-rc4-4-amd64-llvm11-ias" KERNEL_XZ y -> n +DECOMPRESS_ZSTD y +HAVE_KERNEL_ZSTD y +KERNEL_ZSTD y +RD_ZSTD y # du -s -k initrd.img-5.8.0-rc4-3-amd64-llvm11-ias initrd.img-5.8.0-rc4-4-amd64-llvm11-ias 31208 initrd.img-5.8.0-rc4-3-amd64-llvm11-ias <--- GZIP 24768 initrd.img-5.8.0-rc4-4-amd64-llvm11-ias <--- ZSTD - Sedat - > Regards, > - Sedat - > > > Hi all, > > > > This patch set adds support for a ZSTD-compressed kernel, ramdisk, and > > initramfs in the kernel boot process. ZSTD-compressed ramdisk and initramfs > > are supported on all architectures. The ZSTD-compressed kernel is only > > hooked up to x86 in this patch set. > > > > Zstandard requires slightly more memory during the kernel decompression > > on x86 (192 KB vs 64 KB), and the memory usage is independent of the > > window size. > > > > Zstandard requires memory proprortional to the window size used during > > compression for decompressing the ramdisk image, since streaming mode is > > used. Newer versions of zstd (1.3.2+) list the window size of a file > > with `zstd -lv <file>'. The absolute maximum amount of memory required > > is just over 8 MB, but it can be controlled at compression time. > > > > This patch set has been boot tested with buildroot and QEMU based off > > of linux-5.8-rc4. > > > > On i386 and x86_64 I have tested the following configurations: > > * zstd compressed kernel and a separate zstd compressed initramfs > > * zstd compressed kernel and a built-in zstd compressed initramfs > > * gzip compressed kernel and a separate gzip compressed initramfs > > * gzip compressed kernel and a built-in gzip compressed initramfs > > > > On arm and aarch64 I tested the same configurations, except that the kernel is > > always gzip compressed. > > > > Facebook has been using v1 of these patches on x86_64 devices for more than 6 > > months. When we switched from a xz compressed initramfs to a zstd compressed > > initramfs decompression time shrunk from 12 seconds to 3 seconds. When we > > switched from a xz compressed kernel to a zstd compressed kernel we saved 2 > > seconds of boot time. > > > > Facebook has been using v2 of these patches on aarch64 devices for a few weeks. > > When we switched from an lzma compressed initramfs to a zstd compressed initramfs > > decompression time shrunk from 27 seconds to 8 seconds. > > > > The zstd compressed kernel is smaller than the gzip compressed kernel but larger > > than the xz or lzma compressed kernels, and it decompresses faster than > > everything except lz4. See the table below for the measurement of an x86_64 > > kernel ordered by compressed size: > > > > algo size > > xz 6,509,792 > > lzma 6,856,576 > > zstd 7,399,157 > > gzip 8,522,527 > > bzip 8,629,603 > > lzo 9,808,035 > > lz4 10,705,570 > > none 32,565,672 > > > > Alex Xu ran benchmarks in https://lkml.org/lkml/2020/7/1/722. > > > > v1 -> v2: > > - Rebase > > - usr/Makefile and init/Kconfig were changed so the patches were updated > > - No functional changes except to rebase > > - Split the patches up into smaller chunks > > > > v2 -> v3: > > - Add *.zst to the .gitignore in patch 8 > > - Style nits in patch 3 > > - Rename the PREBOOT macro to ZSTD_PREBOOT and XXH_PREBOOT in patches > > 1 through 3 > > > > v3 -> v4: > > - Increase the ZSTD_IOBUF_SIZE from 4KB to 128KB to improve performance. > > With this change I switch from malloc() to large_malloc() for the > > buffers. > > - Increase the maximum allowed window size from 8 MB to 128 MB, which is > > the max that zstd in the kernel supports. > > > > v4 -> v5: > > - Update commit message for patch 6 in response to comments > > - Rebase onto next-20200408 > > > > v5 -> v6: > > - Rebase onto v5.8-rc4 > > > > Best, > > Nick Terrell > > > > Adam Borowski (1): > > .gitignore: add ZSTD-compressed files > > > > Nick Terrell (7): > > lib: prepare zstd for preboot environment > > lib: prepare xxhash for preboot environment > > lib: add zstd support to decompress > > init: add support for zstd compressed kernel > > usr: add support for zstd compressed initramfs > > x86: bump ZO_z_extra_bytes margin for zstd > > x86: Add support for ZSTD compressed kernel > > > > .gitignore | 1 + > > Documentation/x86/boot.rst | 6 +- > > arch/x86/Kconfig | 1 + > > arch/x86/boot/compressed/Makefile | 5 +- > > arch/x86/boot/compressed/misc.c | 4 + > > arch/x86/boot/header.S | 8 +- > > arch/x86/include/asm/boot.h | 6 +- > > include/linux/decompress/unzstd.h | 11 + > > init/Kconfig | 15 +- > > lib/Kconfig | 4 + > > lib/Makefile | 1 + > > lib/decompress.c | 5 + > > lib/decompress_unzstd.c | 342 ++++++++++++++++++++++++++++++ > > lib/xxhash.c | 21 +- > > lib/zstd/decompress.c | 2 + > > lib/zstd/fse_decompress.c | 9 +- > > lib/zstd/zstd_internal.h | 14 +- > > scripts/Makefile.lib | 15 ++ > > usr/Kconfig | 20 ++ > > usr/Makefile | 1 + > > 20 files changed, 464 insertions(+), 27 deletions(-) > > create mode 100644 include/linux/decompress/unzstd.h > > create mode 100644 lib/decompress_unzstd.c > > > > -- > > 2.27.0 > >
