On Thu, Oct 19, 2023 at 03:50:05PM -0300, Marcos Paulo de Souza wrote: > On Sat, Oct 14, 2023 at 05:35:55PM +0900, Masahiro Yamada wrote: > > On Tue, Oct 10, 2023 at 5:43 AM Marcos Paulo de Souza <mpdesouza@xxxxxxx> wrote: > > > > > > Hi all, > > > > > > I found an issue while moving the livepatch kselftest modules to be built on the > > > fly, instead of building them on kernel building. > > > > > > If, for some reason, there is a recursive make invocation that starts from the > > > top level Makefile and in the leaf Makefile it tries to build a module (using M= > > > in the make invocation), it doesn't produce the module. This happens because the > > > toplevel Makefile checks for M= only once. This is controlled by the > > > sub_make_done variable, which is exported after checking the command line > > > options are passed to the top level Makefile. Once this variable is set it's > > > the M= setting is never checked again on the recursive call. > > > > > > This can be observed when cleaning the bpf kselftest dir. When calling > > > > > > $ make TARGETS="bpf" SKIP_TARGETS="" kselftest-clean > > > > > > What happens: > > > > > > 1. It checks for some command line settings (like M=) was passed (it wasn't), > > > set some definitions and exports sub_make_done. > > > > > > 2. Jump into tools/testing/selftests/bpf, and calls the clean target. > > > > > > 3. The clean target is overwritten to remove some files and then jump to > > > bpf_testmod dir and call clean there > > > > > > 4. On bpf_testmod/Makefile, the clean target will execute > > > $(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean > > > > > > 5. The KDIR is to toplevel dir. The top Makefile will check that sub_make_done was > > > already set, ignoring the M= setting. > > > > > > 6. As M= wasn't checked, KBUILD_EXTMOD isn't set, and the clean target applies > > > to the kernel as a whole, making it clean all generated code/objects and > > > everything. > > > > > > One way to avoid it is to call "unexport sub_make_done" on > > > tools/testing/selftests/bpf/bpf_testmod/Makefile before processing the all > > > target, forcing the toplevel Makefile to process the M=, producing the module > > > file correctly. > > > > > > If the M=dir points to /lib/modules/.../build, then it fails with "m2c: No such > > > file", which I already reported here[1]. At the time this problem was treated > > > like a problem with kselftest infrastructure. > > > > > > Important: The process works fine if the initial make invocation is targeted to a > > > different directory (using -C), since it doesn't goes through the toplevel > > > Makefile, and sub_make_done variable is not set. > > > > > > I attached a minimal reproducer, that can be used to better understand the > > > problem. The "make testmod" and "make testmod-clean" have the same effect that > > > can be seem with the bpf kselftests. There is a unexport call commented on > > > test-mods/Makefile, and once that is called the process works as expected. > > > > > > Is there a better way to fix this? Is this really a problem, or am I missing > > > something? > > > > > > Or, using KBUILD_EXTMOD will work too. > > Yes, that works, only if set to /lib/modules: > > $ make kselftest TARGETS=bpf SKIP_TARGETS="" > make[3]: Entering directory '/home/mpdesouza/git/linux/tools/testing/selftests/bpf' > MOD bpf_testmod.ko > warning: the compiler differs from the one used to build the kernel > The kernel was built by: gcc (SUSE Linux) 13.2.1 20230803 [revision cc279d6c64562f05019e1d12d0d825f9391b5553] > You are using: gcc (SUSE Linux) 13.2.1 20230912 [revision b96e66fd4ef3e36983969fb8cdd1956f551a074b] > CC [M] /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.o > MODPOST /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/Module.symvers > CC [M] /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.mod.o > LD [M] /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko > BTF [M] /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko > Skipping BTF generation for /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.ko due to unavailability of vmlinux > BINARY xdp_synproxy > ... > > But if we set the KBUILD_EXTMOD to toplevel Makefile, it fails with a different > strange issue: > > $ make kselftest TARGETS=bpf SKIP_TARGETS="" > BINARY urandom_read > MOD bpf_testmod.ko > m2c -o scripts/Makefile.build -e scripts/Makefile.build scripts/Makefile.build.mod > make[6]: m2c: No such file or directory > make[6]: *** [<builtin>: scripts/Makefile.build] Error 127 > make[5]: *** [Makefile:1913: /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod] Error 2 > make[4]: *** [Makefile:19: all] Error 2 > make[3]: *** [Makefile:229: /home/mpdesouza/git/linux/tools/testing/selftests/bpf/bpf_testmod.ko] Error 2 > make[3]: Leaving directory '/home/mpdesouza/git/linux/tools/testing/selftests/bpf' > make[2]: *** [Makefile:175: all] Error 2 > make[1]: *** [/home/mpdesouza/git/linux/Makefile:1362: kselftest] Error 2 > > I attached a patch that can reproduce the case where it works, and the case > where it doesn't by changing the value of KDIR. > > I understand that KBUILD_EXTMOD, as the name implies, was designed to build > "external" modules, and not ones that live inside kernel, but how could this be > solved? It seems to me as if there is some confusion about in-tree vs. out-of-tree kmods. KBUILD_EXTMOD and M are almost the same and indicate that you want to build _external_ (=out-of-tree) kernel modules. In-tree modules are only those that stay in-tree _and_ are built along with the kernel. Thus, 'make modules KBUILD_EXTMOD=fs/ext4' could be used to build ext4 kmod as "out-of-tree" kernel module, that even taints the kernel if it gets loaded. If you want bpf_testmod.ko to be an in-tree kmod, it has to be build during the usual kernel build, not by running 'make kselftest'. If you use 'make -C $(KDIR)' for building out-of-tree kmods, KDIR has to point to the kernel build directory. (Or it may point to the source tree if you give O=$(BUILDDIR) as well). HTH. Kind regards, Nicolas > For the sake of my initial about livepatch kselftests, KBUILD_EXTMOD > will suffice, since we will target /lib/modules, but I would like to know what > we can do in this case. Do you have other suggestions? > > Thanks in advance, > Marcos > > > > > > > > > > > > > -- > > Best Regards > > Masahiro Yamada > diff --git a/tools/testing/selftests/bpf/bpf_testmod/Makefile b/tools/testing/selftests/bpf/bpf_testmod/Makefile > index 15cb36c4483a..1dce76f35405 100644 > --- a/tools/testing/selftests/bpf/bpf_testmod/Makefile > +++ b/tools/testing/selftests/bpf/bpf_testmod/Makefile > @@ -1,5 +1,6 @@ > BPF_TESTMOD_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST))))) > -KDIR ?= $(abspath $(BPF_TESTMOD_DIR)/../../../../..) > +#KDIR ?= $(abspath $(BPF_TESTMOD_DIR)/../../../../..) > +KDIR ?= /lib/modules/$(shell uname -r)/build > > ifeq ($(V),1) > Q = > @@ -12,9 +13,10 @@ MODULES = bpf_testmod.ko > obj-m += bpf_testmod.o > CFLAGS_bpf_testmod.o = -I$(src) > > +export KBUILD_EXTMOD := $(BPF_TESTMOD_DIR) > + > all: > - +$(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) modules > + +$(Q)make -C $(KDIR) modules > > clean: > - +$(Q)make -C $(KDIR) M=$(BPF_TESTMOD_DIR) clean > - > + +$(Q)make -C $(KDIR) clean
