> On Nov 20, 2021, at 8:53 AM, Alex Xu (Hello71) <alex_y_xu@xxxxxxxx> wrote: > > Otherwise, it allocates 2 GB of memory at once. Even though the majority > of this memory is never touched, the default heuristic overcommit > refuses this request if less than 2 GB of RAM+swap is currently > available. This results in "zstd: error 11 : Allocation error : not > enough memory" and the kernel failing to build. > > When the size is specified, zstd will reduce the memory request > appropriately. For typical kernel sizes of ~32 MB, the largest mmap > request will be reduced to 512 MB, which will succeed on all but the > smallest devices. > > For inputs around this size, --stream-size --no-content-size may > slightly decrease the compressed size, or slightly increase it: > https://github.com/facebook/zstd/issues/2848. I like the change in general. However, I have a question of portability. The --stream-size option was added in zstd-1.4.4 released November 2019 [0]. That means the build will fail for people with older zstd versions. As a reference, Ubuntu Bionic (18.04) includes zstd-1.3.3, and Ubuntu Focal (20.04) has zstd-1.4.4. So Bionic users will have to install zstd manually to build the kernel. I’d prefer to err on the side of caution for portability. I’d say once Bionic hits end of life, it’d be safe to require zstd-1.4.4 or newer. However, I see the need to reduce memory usage, as this issue has come up before. Could we detect the zstd version with `zstd --version` and pass --stream-size if we detect the version is >= v1.4.4? That add some complexity, but I think it would be worth it. Just be sure to document why we are detecting support for --stream-size, and why it is important to pass to zstd. Best, Nick Terrell [0] https://github.com/facebook/zstd/releases/tag/v1.4.4 > Signed-off-by: Alex Xu (Hello71) <alex_y_xu@xxxxxxxx> > --- > scripts/Makefile.lib | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib > index ca901814986a..13d756fbcdc7 100644 > --- a/scripts/Makefile.lib > +++ b/scripts/Makefile.lib > @@ -468,10 +468,10 @@ quiet_cmd_xzmisc = XZMISC $@ > # be used because it would require zstd to allocate a 128 MB buffer. > > quiet_cmd_zstd = ZSTD $@ > - cmd_zstd = { cat $(real-prereqs) | $(ZSTD) -19; $(size_append); } > $@ > + cmd_zstd = { cat $(real-prereqs) | $(ZSTD) --stream-size=$(total_size) --no-content-size -19; $(size_append); } > $@ > > quiet_cmd_zstd22 = ZSTD22 $@ > - cmd_zstd22 = { cat $(real-prereqs) | $(ZSTD) -22 --ultra; $(size_append); } > $@ > + cmd_zstd22 = { cat $(real-prereqs) | $(ZSTD) --stream-size=$(total_size) --no-content-size -22 --ultra; $(size_append); } > $@ > > # ASM offsets > # --------------------------------------------------------------------------- > -- > 2.34.0 >
