2019-11-27 15:15 UTC+0100 ~ Jiri Olsa <jolsa@xxxxxxxxxx> > On Wed, Nov 27, 2019 at 01:38:55PM +0000, Quentin Monnet wrote: >> 2019-11-27 10:48 UTC+0100 ~ Jiri Olsa <jolsa@xxxxxxxxxx> >>> Currently we support only static linking with kernel's libbpf >>> (tools/lib/bpf). This patch adds LIBBPF_DYNAMIC compile variable >>> that triggers libbpf detection and bpf dynamic linking: >>> >>> $ make -C tools/bpf/bpftool make LIBBPF_DYNAMIC=1 >>> >>> If libbpf is not installed, build (with LIBBPF_DYNAMIC=1) stops with: >>> >>> $ make -C tools/bpf/bpftool LIBBPF_DYNAMIC=1 >>> Auto-detecting system features: >>> ... libbfd: [ on ] >>> ... disassembler-four-args: [ on ] >>> ... zlib: [ on ] >>> ... libbpf: [ OFF ] >>> >>> Makefile:102: *** Error: libbpf-devel is missing, please install it. Stop. >>> >>> Adding specific bpftool's libbpf check for libbpf_netlink_open (LIBBPF_0.0.6) >>> which is the latest we need for bpftool at the moment. >>> >>> Adding LIBBPF_DIR compile variable to allow linking with >>> libbpf installed into specific directory: >>> >>> $ make -C tools/lib/bpf/ prefix=/tmp/libbpf/ install_lib install_headers >>> $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 LIBBPF_DIR=/tmp/libbpf/ >>> >>> It might be needed to clean build tree first because features >>> framework does not detect the change properly: >>> >>> $ make -C tools/build/feature clean >>> $ make -C tools/bpf/bpftool/ clean >>> >>> Signed-off-by: Jiri Olsa <jolsa@xxxxxxxxxx> >>> --- >>> tools/bpf/bpftool/Makefile | 40 ++++++++++++++++++++++++++++++- >>> tools/build/feature/test-libbpf.c | 9 +++++++ >>> 2 files changed, 48 insertions(+), 1 deletion(-) >>> >>> diff --git a/tools/bpf/bpftool/Makefile b/tools/bpf/bpftool/Makefile >>> index 39bc6f0f4f0b..2b6ed08cb31e 100644 >>> --- a/tools/bpf/bpftool/Makefile >>> +++ b/tools/bpf/bpftool/Makefile >> >>> @@ -55,7 +64,7 @@ ifneq ($(EXTRA_LDFLAGS),) >>> LDFLAGS += $(EXTRA_LDFLAGS) >>> endif >>> >>> -LIBS = $(LIBBPF) -lelf -lz >>> +LIBS = -lelf -lz >> >> Hi Jiri, >> >> This change seems to be breaking the build with the static library for >> me. I know you add back $(LIBBPF) later in the Makefile, see at the end >> of this email... >> >>> >>> INSTALL ?= install >>> RM ?= rm -f >>> @@ -64,6 +73,23 @@ FEATURE_USER = .bpftool >>> FEATURE_TESTS = libbfd disassembler-four-args reallocarray zlib >>> FEATURE_DISPLAY = libbfd disassembler-four-args zlib >>> >>> +ifdef LIBBPF_DYNAMIC >>> + # Add libbpf check with the flags to ensure bpftool >>> + # specific version is detected. >> >> Nit: We do not check for a specific bpftool version, we check for a >> recent enough libbpf version? > > hi, > we check for a version that has the latest exported function > that bpftool needs, which is currently libbpf_netlink_open > > please check the '#ifdef BPFTOOL' in feature/test-libbpf.c > > it's like that because there's currently no support to check > for particular library version in the build/features framework > > I'll make that comment more clear Thanks! >>> + FEATURE_CHECK_CFLAGS-libbpf := -DBPFTOOL >>> + FEATURE_TESTS += libbpf >>> + FEATURE_DISPLAY += libbpf >>> + >>> + # for linking with debug library run: >>> + # make LIBBPF_DYNAMIC=1 LIBBPF_DIR=/opt/libbpf >>> + ifdef LIBBPF_DIR >>> + LIBBPF_CFLAGS := -I$(LIBBPF_DIR)/include >>> + LIBBPF_LDFLAGS := -L$(LIBBPF_DIR)/$(libdir_relative) >>> + FEATURE_CHECK_CFLAGS-libbpf := $(LIBBPF_CFLAGS) >>> + FEATURE_CHECK_LDFLAGS-libbpf := $(LIBBPF_LDFLAGS) >>> + endif >>> +endif >>> + >>> check_feat := 1 >>> NON_CHECK_FEAT_TARGETS := clean uninstall doc doc-clean doc-install doc-uninstall >>> ifdef MAKECMDGOALS >>> @@ -88,6 +114,18 @@ ifeq ($(feature-reallocarray), 0) >>> CFLAGS += -DCOMPAT_NEED_REALLOCARRAY >>> endif >>> >>> +ifdef LIBBPF_DYNAMIC >>> + ifeq ($(feature-libbpf), 1) >>> + LIBS += -lbpf >>> + CFLAGS += $(LIBBPF_CFLAGS) >>> + LDFLAGS += $(LIBBPF_LDFLAGS) >>> + else >>> + dummy := $(error Error: No libbpf devel library found, please install libbpf-devel) >> >> libbpf-devel sounds like a RH/Fedora package name, but other >> distributions might have different names (Debian/Ubuntu would go by >> libbpf-dev I suppose, although I don't believe such package exists at >> the moment). Maybe use a more generic message? > > sure, actually in perf we use both package names like: > > Error: No libbpf devel library found, please install libbpf-devel or libbpf-dev. > > or we can go with generic message: > > Error: No libbpf devel library found, please install. Either is fine with me, thanks! >>> + endif >>> +else >>> + LIBS += $(LIBBPF) >> >> ... I believe the order of the libraries is relevant, and it seems the >> static libbpf should be passed before the dynamic libs. Here I could fix >> the build with the static library on my setup by prepending the library >> path instead, like this: >> >> LIBS := $(LIBBPF) $(LIBS) > > could you please paste the build error? I don't see any on Fedora, Sure, here it is. On top of your branch (on Ubuntu 18.04): ---------------- $ make -C tools/bpf/bpftool/ make: Entering directory '/root/linux/bpf-next/tools/bpf/bpftool' Auto-detecting system features: ... libbfd: [ on ] ... disassembler-four-args: [ on ] ... zlib: [ on ] CC map_perf_ring.o CC xlated_dumper.o CC btf.o CC tracelog.o CC perf.o CC cfg.o CC btf_dumper.o CC net.o CC netlink_dumper.o CC common.o CC cgroup.o CC main.o CC json_writer.o CC prog.o CC map.o CC feature.o CC jit_disasm.o CC disasm.o make[1]: Entering directory '/root/linux/bpf-next/tools/lib/bpf' Auto-detecting system features: ... libelf: [ on ] ... bpf: [ on ] Parsed description of 117 helper function(s) MKDIR staticobjs/ CC staticobjs/libbpf.o CC staticobjs/bpf.o CC staticobjs/nlattr.o CC staticobjs/btf.o CC staticobjs/libbpf_errno.o CC staticobjs/str_error.o CC staticobjs/netlink.o CC staticobjs/bpf_prog_linfo.o CC staticobjs/libbpf_probes.o CC staticobjs/xsk.o CC staticobjs/hashmap.o CC staticobjs/btf_dump.o LD staticobjs/libbpf-in.o LINK libbpf.a make[1]: Leaving directory '/root/linux/bpf-next/tools/lib/bpf' LINK bpftool /root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__init_prog_names': /root/linux/bpf-next/tools/lib/bpf/libbpf.c:466: undefined reference to `gelf_getsym' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:473: undefined reference to `elf_strptr' /root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__elf_finish': /root/linux/bpf-next/tools/lib/bpf/libbpf.c:570: undefined reference to `elf_end' /root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__elf_init': /root/linux/bpf-next/tools/lib/bpf/libbpf.c:600: undefined reference to `elf_memory' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:613: undefined reference to `elf_begin' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:623: undefined reference to `gelf_getehdr' /root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object_search_section_size': /root/linux/bpf-next/tools/lib/bpf/libbpf.c:714: undefined reference to `gelf_getshdr' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:720: undefined reference to `elf_strptr' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:730: undefined reference to `elf_getdata' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:708: undefined reference to `elf_nextscn' /root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__variable_offset': /root/linux/bpf-next/tools/lib/bpf/libbpf.c:784: undefined reference to `gelf_getsym' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:790: undefined reference to `elf_strptr' /root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__init_user_maps': /root/linux/bpf-next/tools/lib/bpf/libbpf.c:937: undefined reference to `elf_getscn' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:939: undefined reference to `elf_getdata' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:957: undefined reference to `gelf_getsym' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:981: undefined reference to `gelf_getsym' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:990: undefined reference to `elf_strptr' /root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__init_user_btf_maps': /root/linux/bpf-next/tools/lib/bpf/libbpf.c:1359: undefined reference to `elf_getscn' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:1361: undefined reference to `elf_getdata' /root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `section_have_execinstr': /root/linux/bpf-next/tools/lib/bpf/libbpf.c:1428: undefined reference to `elf_getscn' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:1432: undefined reference to `gelf_getshdr' /root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__elf_collect': /root/linux/bpf-next/tools/lib/bpf/libbpf.c:1608: undefined reference to `elf_getscn' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:1608: undefined reference to `elf_rawdata' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:1619: undefined reference to `gelf_getshdr' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:1625: undefined reference to `elf_strptr' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:1632: undefined reference to `elf_getdata' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:1613: undefined reference to `elf_nextscn' /root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_program__collect_reloc': /root/linux/bpf-next/tools/lib/bpf/libbpf.c:1928: undefined reference to `gelf_getrel' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:1932: undefined reference to `gelf_getsym' /root/linux/bpf-next/tools/lib/bpf/libbpf.c:1941: undefined reference to `elf_strptr' /root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `__bpf_object__open': /root/linux/bpf-next/tools/lib/bpf/libbpf.c:3934: undefined reference to `elf_version' /root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `btf__parse_elf': /root/linux/bpf-next/tools/lib/bpf/btf.c:413: undefined reference to `elf_version' /root/linux/bpf-next/tools/lib/bpf/btf.c:427: undefined reference to `elf_begin' /root/linux/bpf-next/tools/lib/bpf/btf.c:432: undefined reference to `gelf_getehdr' /root/linux/bpf-next/tools/lib/bpf/btf.c:440: undefined reference to `elf_getscn' /root/linux/bpf-next/tools/lib/bpf/btf.c:440: undefined reference to `elf_rawdata' /root/linux/bpf-next/tools/lib/bpf/btf.c:450: undefined reference to `gelf_getshdr' /root/linux/bpf-next/tools/lib/bpf/btf.c:455: undefined reference to `elf_strptr' /root/linux/bpf-next/tools/lib/bpf/btf.c:462: undefined reference to `elf_getdata' /root/linux/bpf-next/tools/lib/bpf/btf.c:470: undefined reference to `elf_getdata' /root/linux/bpf-next/tools/lib/bpf/btf.c:445: undefined reference to `elf_nextscn' /root/linux/bpf-next/tools/lib/bpf/btf.c:500: undefined reference to `elf_end' collect2: error: ld returned 1 exit status Makefile:158: recipe for target 'bpftool' failed make: *** [bpftool] Error 1 make: Leaving directory '/root/linux/bpf-next/tools/bpf/bpftool' ---------------- Adding the V=1 flag shows the compilation is attempted with the following command: gcc -O2 -W -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wbad-function-cast -Wdeclaration-after-statement -Wformat-security -Wformat-y2k -Winit-self -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wno-system-headers -Wold-style-definition -Wpacked -Wredundant-decls -Wstrict-prototypes -Wswitch-default -Wundef -Wwrite-strings -Wformat -Wstrict-aliasing=3 -Wshadow -DPACKAGE='"bpftool"' -D__EXPORTED_HEADERS__ -I/root/linux/bpf-next/kernel/bpf/ -I/root/linux/bpf-next/tools/include -I/root/linux/bpf-next/tools/include/uapi -I/root/linux/bpf-next/tools/lib/bpf -I/root/linux/bpf-next/tools/perf -DBPFTOOL_VERSION='"5.4.0"' -DDISASM_FOUR_ARGS_SIGNATURE -DHAVE_LIBBFD_SUPPORT -o bpftool map_perf_ring.o perf.o net.o cgroup.o tracelog.o btf_dumper.o xlated_dumper.o json_writer.o prog.o map.o common.o btf.o cfg.o main.o netlink_dumper.o feature.o jit_disasm.o disasm.o -lelf -lz /root/linux/bpf-next/tools/lib/bpf/libbpf.a -lbfd -ldl -lopcodes If I move -lelf and -lz after <path>/libbpf.a, then it builds fine. Hope this helps, Quentin
