Re: # Toplevel Makefile doesn't process module build correctly on recursive make calls

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, Oct 20, 2023 at 06:21:40AM +0200, Nicolas Schier wrote:
> On Thu, Oct 19, 2023 at 03:50:05PM -0300, Marcos Paulo de Souza wrote:
> > On Sat, Oct 14, 2023 at 05:35:55PM +0900, Masahiro Yamada wrote:
> > > On Tue, Oct 10, 2023 at 5:43 AM Marcos Paulo de Souza <mpdesouza@xxxxxxx> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I found an issue while moving the livepatch kselftest modules to be built on the
> > > > fly, instead of building them on kernel building.
> > > >
> > > > If, for some reason, there is a recursive make invocation that starts from the
> > > > top level Makefile and in the leaf Makefile it tries to build a module (using M=
> > > > in the make invocation), it doesn't produce the module. This happens because the
> > > > toplevel Makefile checks for M= only once. This is controlled by the
> > > > sub_make_done variable, which is exported after checking the command line
> > > > options are passed to the top level Makefile. Once this variable is set it's
> > > > the M= setting is never checked again on the recursive call.
> > > >
> > > > This can be observed when cleaning the bpf kselftest dir. When calling
> > > >
> > > >         $ make TARGETS="bpf" SKIP_TARGETS="" kselftest-clean
> > > >
> > > > What happens:
> > > >
> > > >         1. It checks for some command line settings (like M=) was passed (it wasn't),
> > > >         set some definitions and exports sub_make_done.
> > > >
> > > >         2. Jump into tools/testing/selftests/bpf, and calls the clean target.
> > > >
> > > >         3. The clean target is overwritten to remove some files and then jump to
> > > >         bpf_testmod dir and call clean there
> > > >
> > > >         4. On bpf_testmod/Makefile, the clean target will execute
> > > >                 $(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean
> > > >
> > > >         5. The KDIR is to toplevel dir. The top Makefile will check that sub_make_done was
> > > >         already set, ignoring the M= setting.
> > > >
> > > >         6. As M= wasn't checked, KBUILD_EXTMOD isn't set, and the clean target applies
> > > >         to the kernel as a whole, making it clean all generated code/objects and
> > > >         everything.
> > > >
> > > > One way to avoid it is to call "unexport sub_make_done" on
> > > > tools/testing/selftests/bpf/bpf_testmod/Makefile before processing the all
> > > > target, forcing the toplevel Makefile to process the M=, producing the module
> > > > file correctly.
> > > >
> > > > If the M=dir points to /lib/modules/.../build, then it fails with "m2c: No such
> > > > file", which I already reported here[1]. At the time this problem was treated
> > > > like a problem with kselftest infrastructure.
> > > >
> > > > Important: The process works fine if the initial make invocation is targeted to a
> > > > different directory (using -C), since it doesn't goes through the toplevel
> > > > Makefile, and sub_make_done variable is not set.
> > > >
> > > > I attached a minimal reproducer, that can be used to better understand the
> > > > problem. The "make testmod" and "make testmod-clean" have the same effect that
> > > > can be seem with the bpf kselftests. There is a unexport call commented on
> > > > test-mods/Makefile, and once that is called the process works as expected.
> > > >
> > > > Is there a better way to fix this? Is this really a problem, or am I missing
> > > > something?
> > > 
> > > 
> > > Or, using KBUILD_EXTMOD will work too.
> > 
> > Yes, that works, only if set to /lib/modules:
> > 
> > $ make kselftest TARGETS=bpf SKIP_TARGETS=""
> > make[3]: Entering directory '/home/mpdesouza/git/linux/tools/testing/selftests/bpf'
> >   MOD      bpf_testmod.ko
> > warning: the compiler differs from the one used to build the kernel
> >   The kernel was built by: gcc (SUSE Linux) 13.2.1 20230803 [revision cc279d6c64562f05019e1d12d0d825f9391b5553]
> >   You are using:           gcc (SUSE Linux) 13.2.1 20230912 [revision b96e66fd4ef3e36983969fb8cdd1956f551a074b]
> >   CC [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.o
> >   MODPOST /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/Module.symvers
> >   CC [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.mod.o
> >   LD [M]  /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko
> >   BTF [M] /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko
> > Skipping BTF generation for /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko due to unavailability of vmlinux
> >   BINARY   xdp_synproxy
> > ...
> > 
> > But if we set the KBUILD_EXTMOD to toplevel Makefile, it fails with a different
> > strange issue:
> > 
> > $ make kselftest TARGETS=bpf SKIP_TARGETS=""
> >   BINARY   urandom_read
> >   MOD      bpf_testmod.ko
> > m2c    -o scripts/Makefile.build -e scripts/Makefile.build scripts/Makefile.build.mod
> > make[6]: m2c: No such file or directory
> > make[6]: *** [<builtin>: scripts/Makefile.build] Error 127
> > make[5]: *** [Makefile:1913: /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod] Error 2
> > make[4]: *** [Makefile:19: all] Error 2
> > make[3]: *** [Makefile:229: /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod.ko] Error 2
> > make[3]: Leaving directory '/home/mpdesouza/git/linux/tools/testing/selftests/bpf'
> > make[2]: *** [Makefile:175: all] Error 2
> > make[1]: *** [/home/mpdesouza/git/linux/Makefile:1362: kselftest] Error 2
> > 
> > I attached a patch that can reproduce the case where it works, and the case
> > where it doesn't by changing the value of KDIR.
> > 
> > I understand that KBUILD_EXTMOD, as the name implies, was designed to build
> > "external" modules, and not ones that live inside kernel, but how could this be
> > solved?
> 
> It seems to me as if there is some confusion about in-tree vs.
> out-of-tree kmods.
> 
> KBUILD_EXTMOD and M are almost the same and indicate that you want to
> build _external_ (=out-of-tree) kernel modules.  In-tree modules are
> only those that stay in-tree _and_ are built along with the kernel.
> Thus, 'make modules KBUILD_EXTMOD=fs/ext4' could be used to build ext4
> kmod as "out-of-tree" kernel module, that even taints the kernel if it
> gets loaded.
> 
> If you want bpf_testmod.ko to be an in-tree kmod, it has to be build
> during the usual kernel build, not by running 'make kselftest'.
> 
> If you use 'make -C $(KDIR)' for building out-of-tree kmods, KDIR has to
> point to the kernel build directory.  (Or it may point to the source
> tree if you give O=$(BUILDDIR) as well).

Thanks for the explanation Nicolas. In this, I believe that the BPF module
should be moved into lib/, like lib/livepatch, when then be built along with
other in-tree modules.

Currently there is a bug when running the kselftests-clean target with bpf:

	make kselftest-clean TARGETS=bpf SKIP_TARGETS=""

As the M= argument is ignore on the toplevel Makefile, this make invocation
applies the clean to all built kernel objects/modules/everything, which is bug
IMO.

There is a statement in the BPF docs saying that the selftests should be run
inside the tools/testing/selftests/bpf directory. At the same time, kselftests
should comply with all the targets defined in the documention, like gen_tar, and
run_tests. In this case should the build process be fixed, or just make
kselftests less restrict?

(CCing kselftests and bpf ML)

> 
> HTH.
> 
> Kind regards,
> Nicolas
> 
> 
> > For the sake of my initial about livepatch kselftests, KBUILD_EXTMOD
> > will suffice, since we will target /lib/modules, but I would like to know what
> > we can do in this case. Do you have other suggestions?
> > 
> > Thanks in advance,
> >   Marcos
> > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > --
> > > Best Regards
> > > Masahiro Yamada
> 
> > diff --git a/tools/testing/selftests/bpf/bpf_testmod/Makefile b/tools/testing/selftests/bpf/bpf_testmod/Makefile
> > index 15cb36c4483a..1dce76f35405 100644
> > --- a/tools/testing/selftests/bpf/bpf_testmod/Makefile
> > +++ b/tools/testing/selftests/bpf/bpf_testmod/Makefile
> > @@ -1,5 +1,6 @@
> >  BPF_TESTMOD_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
> > -KDIR ?= $(abspath $(BPF_TESTMOD_DIR)/../../../../..)
> > +#KDIR ?= $(abspath $(BPF_TESTMOD_DIR)/../../../../..)
> > +KDIR ?= /lib/modules/$(shell uname -r)/build
> >  
> >  ifeq ($(V),1)
> >  Q =
> > @@ -12,9 +13,10 @@ MODULES = bpf_testmod.ko
> >  obj-m += bpf_testmod.o
> >  CFLAGS_bpf_testmod.o = -I$(src)
> >  
> > +export KBUILD_EXTMOD := $(BPF_TESTMOD_DIR)
> > +
> >  all:
> > -	+$(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) modules
> > +	+$(Q)make -C $(KDIR) modules
> >  
> >  clean:
> > -	+$(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean
> > -
> > +	+$(Q)make -C $(KDIR) clean
> 




[Index of Archives]     [Linux Samsung SoC]     [Linux Rockchip SoC]     [Linux Actions SoC]     [Linux for Synopsys ARC Processors]     [Linux NFS]     [Linux NILFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]


  Powered by Linux