Hi,
On Thu, May 26, 2022 at 04:59:07PM +0100, Andre Przywara wrote:
> On Thu, 26 May 2022 16:30:56 +0100
> Alexandru Elisei <alexandru.elisei@xxxxxxx> wrote:
>
> Hi Alex,
>
> > I found it very difficult to cross-build kvmtool for all arches because for
> > some architectures I couldn't get libfdt to install: either dtc wasn't
> > compiling (x86 machine, Arch Linux, missing yaml.h include when building
> > for arm and powerpc), or libfdt was not being found by Makefile after
> > installing it (x86 machine, Debian 11, building for powerpc). It was
> > possible to compile only libfdt and teach Makefile where to look for it,
> > which is what I ended up doing, but in my opinion, that's not an ideal user
> > experience.
>
> Meh. Nobody said that userland cross compilation would be a walk in the
> park, and this approach (cross-compiling and putting in SYSROOT) is the
> canonical approach. When using a multiarch capable distro, this is easy
> enough.
Except when it doesn't work. On debian 11 I was not able to make it work for
powerpc by following the instructions, while I was able to make it work for
aarch64 following the same instructions. That's the kind of thing that I would
like to avoid.
> But I see that this is some major pain point for an otherwise rather "low
> maintenance" tool, and libfdt is designed to be embedded, so I guess this
> is fine, with the caveat mentioned below.
>
> > This is my attempt at making compilation easier by pulling in the libfdt
> > library and having kvmtool compile it when an architecture requires it. All
> > the user needs to do now is to find a suitable cross-compiler toolchain
> > with glibc included [1]*.
> >
> > When making lkvm-static, the static version of libfdt is built (libfdt.a);
> > for all other make targets the shared library is built.
>
> So does that mean that the .so gets shipped with kvmtool? And it should be
Err.. I screwed up badly here, I just checked with ldd and lkvm was still
using the system proved libfdt.so, that's why it was still working.
Uninstalling the libfdt package broke lkvm.
> installed as part of "make install"? I think that might bring up more
> questions and problems, so do we want to leave the user some choice?
> - The user brings their own libfdt. This should be the case for any clean
> userland cross compilation, where you install target libraries in SYSROOT,
> just for compilation, and have the .so file also on the actual target
> system. This would also be best for native compilation. Current situation.
> - The user decides to use the built-in libfdt, in which case that gets
> always statically linked into the lkvm binary (which might still be
> dynamically linked against libc), to avoid any conflict between the just
> built version (.so), and the installed library on the target system.
>
> Some projects choose this approach, where it gets decided at build time
> which version to use ("system library" vs. built-in version). This is
> either done automatically, because a system library is not found, or using
> some user provided flag for configure/make.
>
> If that sounds too involved, we could just always link in our libfdt copy
> statically. And later revisit this when people complain.
I would prefer this approach, as it keeps things the Makefile simple and it
makes the build reproducible.
> Can you check by how much an otherwise dynamically linked kvmtool binary
> grows when we link against libfdt.a?
I wrote a patch to always link against libfdt.a, here are the results of
running make on a rockpro64 (I renamed the executables to tell which is
which).
Built from current master:
$ /usr/bin/ls -l lkvm-libfdt-dynamically-linked
-rwxr-xr-x 1 alex users 2127112 May 26 16:31 lkvm-libfdt-dynamically-linked
$ ldd lkvm-libfdt-dynamically-linked
linux-vdso.so.1 (0x0000ffffb52ae000)
libz.so.1 => /usr/lib/libz.so.1 (0x0000ffffb5140000)
libaio.so.1 => /usr/lib/libaio.so.1 (0x0000ffffb5120000)
libfdt.so.1 => /usr/lib/libfdt.so.1 (0x0000ffffb5100000)
libc.so.6 => /usr/lib/libc.so.6 (0x0000ffffb4f50000)
/lib/ld-linux-aarch64.so.1 => /usr/lib/ld-linux-aarch64.so.1 (0x0000ffffb5275000)
Built with Makefile modified to link against the static version:
$ /usr/bin/ls -l lkvm-libfdt-statically-linked
-rwxr-xr-x 1 alex users 2238920 May 26 16:32 lkvm-libfdt-statically-linked
$ ldd lkvm-libfdt-statically-linked
linux-vdso.so.1 (0x0000ffff99a6d000)
libz.so.1 => /usr/lib/libz.so.1 (0x0000ffff99900000)
libaio.so.1 => /usr/lib/libaio.so.1 (0x0000ffff998e0000)
libc.so.6 => /usr/lib/libc.so.6 (0x0000ffff99730000)
/lib/ld-linux-aarch64.so.1 => /usr/lib/ld-linux-aarch64.so.1 (0x0000ffff99a34000)
A difference of 109KB (111808 bytes to be more exact).
I'll send the changes to the Makefile in case people want to play with it.
Thanks,
Alex
>
> Cheers,
> Andre
>
> > This only happens
> > for those arches that require libfdt (arm, arm64, riscv, powerpc); libfdt
> > is not built for MIPS and x86.
> >
> > I am pretty happy with how libfdt has been integrated with the kvmtool
> > build system. The only niggle is that the libfdt_clean target echoes the
> > commands to remove the build artifacts:
> >
> > $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- clean
> > Makefile:378: Skipping optional libraries: bfd zlib aio
> > CLEAN (libfdt)
> > rm -f libfdt/*.o libfdt/libfdt.a libfdt/libfdt-1.6.1.so
> > rm -f libfdt/libfdt.so.1
> > CLEAN
> >
> > but I believe that's impossible to change without modifying the libfdt
> > Makefile, which I don't want to do to keep it as easy as possible to update
> > libfdt.
> >
> > Made some quick measurement about how libfdt affects the build time when
> > building from a clean directory (incremental builds do not rebuild libfdt):
> >
> > - On a rockpro64, on the little core (Cortex-A53), running
> >
> > $ time taskset -c 0 make
> >
> > with libfdt takes an average of 90.721s (averaged over 3 runs, standard
> > deviation 0.040s); without libfdt it takes 84.003s (averaged over 3 runs,
> > stddev of 0.105s). That's a 6.718s absolute increase, or a 7.997% relative
> > increase.
> >
> > - On the same rockpro64, on the big core (Cortex-A72), running
> >
> > $ time taskset -c 5 make
> >
> > with libfdt takes an average of 45.078s (averaged over 3 runs, stddev of
> > 0.060s); without libfdt it takes 41.836s (averaged over 3 runs, stddev of
> > 0.065). That's a 3.242s absolute increase, or a 7.749% relative increase.
> >
> > I believe that the ease of use more than justifies the increased
> > compilation time.
> >
> > Testing
> > =======
> >
> > On x86 host, compiled for x86, MIPS, arm, arm64, powerpc, riscv. Booted an
> > x86 VM.
> >
> > On arm64 host, compiled for arm64 and arm. I tried to cross-compile for
> > other targets, but I couldn't find a cross-compiler toolchain for arm64
> > hosts that included a libc implementation. The arm cross-compiler was
> > downloaded from the developer.arm.com [2]. Ran kvm-unit-tests for arm64 and
> > arm (on the arm64 version of kvmtool), and also booted a kernel.
> >
> > *only for x86 hosts.
> >
> > [1] https://toolchains.bootlin.com/
> > [2] https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/downloads
> >
> > Alexandru Elisei (2):
> > Add libfdt library
> > INSTALL: Remove all mentions of libfdt
> >
> > INSTALL | 27 +-
> > Makefile | 53 +-
> > libfdt/.gitignore | 1 +
> > libfdt/Makefile.libfdt | 18 +
> > libfdt/TODO | 3 +
> > libfdt/fdt.c | 335 ++++++
> > libfdt/fdt.h | 66 ++
> > libfdt/fdt_addresses.c | 101 ++
> > libfdt/fdt_check.c | 93 ++
> > libfdt/fdt_empty_tree.c | 38 +
> > libfdt/fdt_overlay.c | 867 +++++++++++++++
> > libfdt/fdt_ro.c | 859 +++++++++++++++
> > libfdt/fdt_rw.c | 500 +++++++++
> > libfdt/fdt_strerror.c | 60 ++
> > libfdt/fdt_sw.c | 384 +++++++
> > libfdt/fdt_wip.c | 94 ++
> > libfdt/libfdt.h | 2147 ++++++++++++++++++++++++++++++++++++++
> > libfdt/libfdt_env.h | 96 ++
> > libfdt/libfdt_internal.h | 192 ++++
> > libfdt/meson.build | 55 +
> > libfdt/version.lds | 83 ++
> > 21 files changed, 6033 insertions(+), 39 deletions(-)
> > create mode 100644 libfdt/.gitignore
> > create mode 100644 libfdt/Makefile.libfdt
> > create mode 100644 libfdt/TODO
> > create mode 100644 libfdt/fdt.c
> > create mode 100644 libfdt/fdt.h
> > create mode 100644 libfdt/fdt_addresses.c
> > create mode 100644 libfdt/fdt_check.c
> > create mode 100644 libfdt/fdt_empty_tree.c
> > create mode 100644 libfdt/fdt_overlay.c
> > create mode 100644 libfdt/fdt_ro.c
> > create mode 100644 libfdt/fdt_rw.c
> > create mode 100644 libfdt/fdt_strerror.c
> > create mode 100644 libfdt/fdt_sw.c
> > create mode 100644 libfdt/fdt_wip.c
> > create mode 100644 libfdt/libfdt.h
> > create mode 100644 libfdt/libfdt_env.h
> > create mode 100644 libfdt/libfdt_internal.h
> > create mode 100644 libfdt/meson.build
> > create mode 100644 libfdt/version.lds
> >
>
_______________________________________________
kvmarm mailing list
kvmarm@xxxxxxxxxxxxxxxxxxxxx
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
