On Tue, Jan 31, 2023 at 10:08 AM Ian Rogers <irogers@xxxxxxxxxx> wrote:
>
> On Tue, Jan 31, 2023 at 9:21 AM Nathan Chancellor <nathan@xxxxxxxxxx> wrote:
> >
> > Hi Ian,
> >
> > On Mon, Jan 23, 2023 at 10:43:24PM -0800, Ian Rogers wrote:
> > > HOSTCC is always wanted when building. Setting CC to HOSTCC happens
> > > after tools/scripts/Makefile.include is included, meaning flags are
> > > set assuming say CC is gcc, but then it can be later set to HOSTCC
> > > which may be clang. tools/scripts/Makefile.include is needed for host
> > > set up and common macros in objtool's Makefile. Rather than override
> > > CC to HOSTCC, just pass CC as HOSTCC to Makefile.build, the libsubcmd
> > > builds and the linkage step. This means the Makefiles don't see things
> > > like CC changing and tool flag determination, and similar, work
> > > properly.
> > >
> > > Also, clear the passed subdir as otherwise an outer build may break by
> > > inadvertently passing an inappropriate value.
> > >
> > > Signed-off-by: Ian Rogers <irogers@xxxxxxxxxx>
> > > ---
> > > tools/bpf/resolve_btfids/Makefile | 17 +++++++----------
> > > 1 file changed, 7 insertions(+), 10 deletions(-)
> > >
> > > diff --git a/tools/bpf/resolve_btfids/Makefile b/tools/bpf/resolve_btfids/Makefile
> > > index 1fe0082b2ecc..daed388aa5d7 100644
> > > --- a/tools/bpf/resolve_btfids/Makefile
> > > +++ b/tools/bpf/resolve_btfids/Makefile
> > > @@ -18,14 +18,11 @@ else
> > > endif
> > >
> > > # always use the host compiler
> > > -AR = $(HOSTAR)
> > > -CC = $(HOSTCC)
> > > -LD = $(HOSTLD)
> > > -ARCH = $(HOSTARCH)
> > > +HOST_OVERRIDES := AR="$(HOSTAR)" CC="$(HOSTCC)" LD="$(HOSTLD)" ARCH="$(HOSTARCH)" \
> > > + EXTRA_CFLAGS="$(HOSTCFLAGS) $(KBUILD_HOSTCFLAGS)"
> > > +
> > > RM ?= rm
> > > CROSS_COMPILE =
> > > -CFLAGS := $(KBUILD_HOSTCFLAGS)
> > > -LDFLAGS := $(KBUILD_HOSTLDFLAGS)
> > >
> > > OUTPUT ?= $(srctree)/tools/bpf/resolve_btfids/
> > >
> > > @@ -56,12 +53,12 @@ $(OUTPUT) $(OUTPUT)/libsubcmd $(LIBBPF_OUT):
> > >
> > > $(SUBCMDOBJ): fixdep FORCE | $(OUTPUT)/libsubcmd
> > > $(Q)$(MAKE) -C $(SUBCMD_SRC) OUTPUT=$(SUBCMD_OUT) \
> > > - DESTDIR=$(SUBCMD_DESTDIR) prefix= \
> > > + DESTDIR=$(SUBCMD_DESTDIR) $(HOST_OVERRIDES) prefix= subdir= \
> > > $(abspath $@) install_headers
> > >
> > > $(BPFOBJ): $(wildcard $(LIBBPF_SRC)/*.[ch] $(LIBBPF_SRC)/Makefile) | $(LIBBPF_OUT)
> > > $(Q)$(MAKE) $(submake_extras) -C $(LIBBPF_SRC) OUTPUT=$(LIBBPF_OUT) \
> > > - DESTDIR=$(LIBBPF_DESTDIR) prefix= EXTRA_CFLAGS="$(CFLAGS)" \
> > > + DESTDIR=$(LIBBPF_DESTDIR) $(HOST_OVERRIDES) prefix= subdir= \
> > > $(abspath $@) install_headers
> > >
> > > LIBELF_FLAGS := $(shell $(HOSTPKG_CONFIG) libelf --cflags 2>/dev/null)
> > > @@ -80,11 +77,11 @@ export srctree OUTPUT CFLAGS Q
> > > include $(srctree)/tools/build/Makefile.include
> > >
> > > $(BINARY_IN): fixdep FORCE prepare | $(OUTPUT)
> > > - $(Q)$(MAKE) $(build)=resolve_btfids
> > > + $(Q)$(MAKE) $(build)=resolve_btfids $(HOST_OVERRIDES)
> > >
> > > $(BINARY): $(BPFOBJ) $(SUBCMDOBJ) $(BINARY_IN)
> > > $(call msg,LINK,$@)
> > > - $(Q)$(CC) $(BINARY_IN) $(LDFLAGS) -o $@ $(BPFOBJ) $(SUBCMDOBJ) $(LIBS)
> > > + $(Q)$(HOSTCC) $(BINARY_IN) $(KBUILD_HOSTLDFLAGS) -o $@ $(BPFOBJ) $(SUBCMDOBJ) $(LIBS)
> > >
> > > clean_objects := $(wildcard $(OUTPUT)/*.o \
> > > $(OUTPUT)/.*.o.cmd \
> > > --
> > > 2.39.0.246.g2a6d74b583-goog
> > >
> >
> > I just bisected a linking failure when building resolve_btfids with
> > clang to this change as commit 13e07691a16f ("tools/resolve_btfids:
> > Alter how HOSTCC is forced") in the bpf-next tree.
> >
> > It appears to be related to whether or not CROSS_COMPILE is specified,
> > which we have to do for certain architectures and configurations still.
> > arm64 is not one of those but it helps demonstrate the issue.
> >
> > # Turn off CONFIG_DEBUG_INFO_REDUCED and turn on CONFIG_DEBUG_INFO_BTF
> > $ make -skj"$(nproc)" ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- HOSTLDFLAGS=-fuse-ld=lld LLVM=1 defconfig menuconfig
> >
> > $ make -skj"$(nproc)" ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- HOSTLDFLAGS=-fuse-ld=lld LLVM=1 prepare
> > ld.lld: error: $LINUX_SRC/tools/bpf/resolve_btfids//resolve_btfids-in.o is incompatible with elf64-x86-64
> > clang-17: error: linker command failed with exit code 1 (use -v to see invocation)
> > ...
> >
> > Before your change, with V=1, I see:
> >
> > clang -Wp,-MD,$LINUX_SRC/tools/bpf/resolve_btfids/.main.o.d -Wp,-MT,$LINUX_SRC/tools/bpf/resolve_btfids/main.o -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu11 -Wdeclaration-after-statement -g -I$LINUX_SRC/tools/include -I$LINUX_SRC/tools/include/uapi -I$LINUX_SRC/tools/bpf/resolve_btfids/libbpf/include -I$LINUX_SRC/tools/bpf/resolve_btfids/libsubcmd/include -D"BUILD_STR(s)=#s" -c -o $LINUX_SRC/tools/bpf/resolve_btfids/main.o main.c
> >
> > After, I see:
> >
> > clang -Wp,-MD,$LINUX_SRC/tools/bpf/resolve_btfids/.main.o.d -Wp,-MT,$LINUX_SRC/tools/bpf/resolve_btfids/main.o --target=aarch64-linux-gnu -g -I$LINUX_SRC/tools/include -I$LINUX_SRC/tools/include/uapi -I$LINUX_SRC/tools/bpf/resolve_btfids/libbpf/include -I$LINUX_SRC/tools/bpf/resolve_btfids/libsubcmd/include -D"BUILD_STR(s)=#s" -c -o $LINUX_SRC/tools/bpf/resolve_btfids/main.o main.c
> >
> > We seem to have taken on a '--target=aarch64-linux-gnu' (changing the
> > target of resolve_btfids-in.o) and we dropped the warning flags.
> >
> > I think this comes from the clang block in
> > tools/scripts/Makefile.include, which is included into the
> > resolve_btfids Makefile via tools/lib/bpf/Makefile.
> >
> > I am not super familiar with the tools build system, otherwise I would
> > try to provide a patch. I tried moving CROSS_COMPILE from a recursive to
> > simple variable ('=' -> ':=') and moving it to HOST_OVERRIDES but those
> > did not appear to resolve it for me.
> >
> > If there is any other information I can provide or patches I can test,
> > please let me know.
> >
> > Cheers,
> > Nathan
>
> Thanks Nathan, and thanks for all the details in the bug report. I'm
> looking into this.
>
> Ian
Given the somewhat complicated cross compile I wasn't able to get a
reproduction. Could you see if the following addresses the problem:
```
diff --git a/tools/bpf/resolve_btfids/Makefile
b/tools/bpf/resolve_btfids/Makefile
index daed388aa5d7..a06966841df4 100644
--- a/tools/bpf/resolve_btfids/Makefile
+++ b/tools/bpf/resolve_btfids/Makefile
@@ -19,10 +19,9 @@ endif
# always use the host compiler
HOST_OVERRIDES := AR="$(HOSTAR)" CC="$(HOSTCC)" LD="$(HOSTLD)"
ARCH="$(HOSTARCH)" \
- EXTRA_CFLAGS="$(HOSTCFLAGS) $(KBUILD_HOSTCFLAGS)"
+ EXTRA_CFLAGS="$(HOSTCFLAGS) $(KBUILD_HOSTCFLAGS)"
CROSS_COMPILE=""
RM ?= rm
-CROSS_COMPILE =
OUTPUT ?= $(srctree)/tools/bpf/resolve_btfids/
```
Thanks,
Ian
