On Tue, Jul 19, 2022 at 11:44:31PM +0200, Willy Tarreau wrote: > Hi Paul, > > as previously promised, here comes the nolibc update which introduces the > minimal self-test infrastructure that aims at being reasonably easy to > expand further. > > It's based on your branch "dev.2022.06.30b" that contains the previous > minor fixes that aimed at addressing Linus' concerns about the build > process inconsistencies. > > The way it works tries to mimmick as much as possible the regular build > process, so that it reuses the same ARCH, CC, CROSS_COMPILE to build the > test program, that will be embedded into an initramfs and the kernel is > (re)built with that initramfs. Then you can decide to run that kernel > under QEMU for the supported archs, and the output of the tests appears > in an output text file in a format that's easily greppable and diffable. > A single target "run" does everything. > > By default it will reuse your existing .config (so that developers > continue to use their regular config handling), though it can also > create a known-to-work defconfig for each arch. The reason behind this > is that it took me a moment to figure certain defconfig + machine name > combinations and I found it better to put them there once for all. > > I've successfully tested it on arm, arm64, i386, x86_64. riscv64 works > except two syscalls which return unexpected errors, and mips segfaults > in sbrk(). I don't know why yet, but this proves that it's worth having > such a test. Excellent, thank you!!! As we often said during my misspent youth, "If it ain't tested, it don't work." But I do get "71 test(s) passed." when running on x86. I will let you decide whether that constitutes all being well or indicates a bug in the tests. ;-) > There are not that many tests yet (71), those that have to run can be > filtered either from the program's command line or from a NOLIBC_TEST > environment variable so that it's possible to skip broken ones or to > focus on a few ranges only. > > Tests are numerically numbered, and are conveniently handled in a > switch/case statement so that a relative line number assigns the number > to the test. That's convenient because the vast majority of syscall tests > are one-liners. This sometimes slightly upsets check-patch when lines get > moderately long but that significantly improves legibility. > > There are expectation for both successes and failures (e.g. -1 ENOTDIR). > I'm sure this can be improved later (and that's the goal). Right now it > covers two test families: > - syscalls > - stdlib (str* functions mostly) > > I suspect that over time we might want to split syscalls into different > parts (e.g. core, fs, etc maybe) but I could be wrong. This is a good start, and we can let experience drive any additional changes that might be required. > The program can automatically modulate QEMU's return value on x86 when > QEMU is run with the appropriate options, but for now I'm not using it > as I felt like it didn't bring much value, and the output is more useful. > That's debatable, and maybe some might want to use it in bisect scripts > for example. It's too early to say IMHO. For the moment, grepping the output works. And perhaps indefinitely. > Oh, I also arranged the code so that the test also builds with glibc. I > noticed that when adding a new test that fails, sometimes it's convenient > to see if it's the nolibc part that's broken or the test. I don't find > this critical but the required includes and ifdefs are there so that it > should be easy to maintain over time as well. If nothing else, the ability to run against glibc is a good way to test the test. > I'm obviously interested in comments, but really, I don't want to > overdesign something for a first step, it remains a very modest test > program and I'd like that it remains easy to hack on it and to contribute > new tests that are deemed useful. I am good with a simple starting point. > I'm CCing the few who already contributed some patches and/or expressed > interest, as well as Linus who had a first bad experience when trying to > test it, hoping this one will be better. I'm pasting below [1] a copy of > a test on x86_64 below, that's summed up as "71 test(s) passed" at the > end of the "run" target. > > If there's no objection, it would be nice to have this with your current > series, as it definitely helps spot and fix the bugs. In parallel I'll see > if I can figure the problems with the two tests that fail each on a > specific arch and I might possibly have a few extra fixes for the current > nolibc. This series is now on the -rcu tree's "dev" branch. I got two almost identical copies of patch 7, so I took the later of the two. Please let me know if I guessed wrong. Thanx, Paul > Thank you! > Willy > > [1] example output > ----8<---- > Running test 'syscall' > 0 getpid = 1 [OK] > 1 getppid = 0 [OK] > 5 getpgid_self = 0 [OK] > 6 getpgid_bad = -1 ESRCH [OK] > 7 kill_0 = 0 [OK] > 8 kill_CONT = 0 [OK] > 9 kill_BADPID = -1 ESRCH [OK] > 10 sbrk = 0 [OK] > 11 brk = 0 [OK] > 12 chdir_root = 0 [OK] > 13 chdir_dot = 0 [OK] > 14 chdir_blah = -1 ENOENT [OK] > 15 chmod_net = 0 [OK] > 16 chmod_self = -1 EPERM [OK] > 17 chown_self = -1 EPERM [OK] > 18 chroot_root = 0 [OK] > 19 chroot_blah = -1 ENOENT [OK] > 20 chroot_exe = -1 ENOTDIR [OK] > 21 close_m1 = -1 EBADF [OK] > 22 close_dup = 0 [OK] > 23 dup_0 = 3 [OK] > 24 dup_m1 = -1 EBADF [OK] > 25 dup2_0 = 100 [OK] > 26 dup2_m1 = -1 EBADF [OK] > 27 dup3_0 = 100 [OK] > 28 dup3_m1 = -1 EBADF [OK] > 29 execve_root = -1 EACCES [OK] > 30 getdents64_root = 120 [OK] > 31 getdents64_null = -1 ENOTDIR [OK] > 32 gettimeofday_null = 0 [OK] > 38 ioctl_tiocinq = 0 [OK] > 39 ioctl_tiocinq = 0 [OK] > 40 link_root1 = -1 EEXIST [OK] > 41 link_blah = -1 ENOENT [OK] > 42 link_dir = -1 EPERM [OK] > 43 link_cross = -1 EXDEV [OK] > 44 lseek_m1 = -1 EBADF [OK] > 45 lseek_0 = -1 ESPIPE [OK] > 46 mkdir_root = -1 EEXIST [OK] > 47 open_tty = 3 [OK] > 48 open_blah = -1 ENOENT [OK] > 49 poll_null = 0 [OK] > 50 poll_stdout = 1 [OK] > 51 poll_fault = -1 EFAULT [OK] > 52 read_badf = -1 EBADF [OK] > 53 sched_yield = 0 [OK] > 54 select_null = 0 [OK] > 55 select_stdout = 1 [OK] > 56 select_fault = -1 EFAULT [OK] > 57 stat_blah = -1 ENOENT [OK] > 58 stat_fault = -1 EFAULT [OK] > 59 symlink_root = -1 EEXIST [OK] > 60 unlink_root = -1 EISDIR [OK] > 61 unlink_blah = -1 ENOENT [OK] > 62 wait_child = -1 ECHILD [OK] > 63 waitpid_min = -1 ESRCH [OK] > 64 waitpid_child = -1 ECHILD [OK] > 65 write_badf = -1 EBADF [OK] > 66 write_zero = 0 [OK] > Errors during this test: 0 > > Running test 'stdlib' > 0 getenv_TERM = <linux> [OK] > 1 getenv_blah = <(null)> [OK] > 2 setcmp_blah_blah = 0 [OK] > 3 setcmp_blah_blah2 = -50 [OK] > 4 setncmp_blah_blah = 0 [OK] > 5 setncmp_blah_blah4 = 0 [OK] > 6 setncmp_blah_blah5 = -53 [OK] > 7 setncmp_blah_blah6 = -54 [OK] > 8 strchr_foobar_o = <oobar> [OK] > 9 strchr_foobar_z = <(null)> [OK] > 10 strrchr_foobar_o = <obar> [OK] > 11 strrchr_foobar_z = <(null)> [OK] > Errors during this test: 0 > > Total number of errors: 0 > ---->8---- > > > -- > > Willy Tarreau (17): > tools/nolibc: make argc 32-bit in riscv startup code > tools/nolibc: fix build warning in sys_mmap() when my_syscall6 is not > defined > tools/nolibc: make sys_mmap() automatically use the right __NR_mmap > definition > selftests/nolibc: add basic infrastructure to ease creation of nolibc > tests > selftests/nolibc: support a test definition format > selftests/nolibc: implement a few tests for various syscalls > selftests/nolibc: add a few tests for some stdlib functions > selftests/nolibc: exit with poweroff on success when getpid() == 1 > selftests/nolibc: on x86, support exiting with isa-debug-exit > selftests/nolibc: recreate and populate /dev and /proc if missing > selftests/nolibc: condition some tests on /proc existence > selftests/nolibc: support glibc as well > selftests/nolibc: add a "kernel" target to build the kernel with the > initramfs > selftests/nolibc: add a "defconfig" target > selftests/nolibc: add a "run" target to start the kernel in QEMU > selftests/nolibc: "sysroot" target installs a local copy of the > sysroot > selftests/nolibc: add a "help" target > > MAINTAINERS | 1 + > tools/include/nolibc/arch-riscv.h | 2 +- > tools/include/nolibc/sys.h | 4 +- > tools/testing/selftests/nolibc/Makefile | 135 ++++ > tools/testing/selftests/nolibc/nolibc-test.c | 757 +++++++++++++++++++ > 5 files changed, 896 insertions(+), 3 deletions(-) > create mode 100644 tools/testing/selftests/nolibc/Makefile > create mode 100644 tools/testing/selftests/nolibc/nolibc-test.c > > -- > 2.17.5 >
