On Sat, Apr 10, 2021 at 12:05 AM David Gow <davidgow@xxxxxxxxxx> wrote: > > The kernel now has a number of testing and debugging tools, and we've > seen a bit of confusion about what the differences between them are. > > Add a basic documentation outlining the testing tools, when to use each, > and how they interact. > > This is a pretty quick overview rather than the idealised "kernel > testing guide" that'd probably be optimal, but given the number of times > questions like "When do you use KUnit and when do you use Kselftest?" > are being asked, it seemed worth at least having something. Hopefully > this can form the basis for more detailed documentation later. > > Signed-off-by: David Gow <davidgow@xxxxxxxxxx> > --- > Documentation/dev-tools/index.rst | 3 + > Documentation/dev-tools/testing-overview.rst | 102 +++++++++++++++++++ > 2 files changed, 105 insertions(+) > create mode 100644 Documentation/dev-tools/testing-overview.rst > > diff --git a/Documentation/dev-tools/index.rst b/Documentation/dev-tools/index.rst > index 1b1cf4f5c9d9..f590e5860794 100644 > --- a/Documentation/dev-tools/index.rst > +++ b/Documentation/dev-tools/index.rst > @@ -7,6 +7,8 @@ be used to work on the kernel. For now, the documents have been pulled > together without any significant effort to integrate them into a coherent > whole; patches welcome! > > +A brief overview of testing-specific tools can be found in :doc:`testing-overview`. > + > .. class:: toc-title > > Table of contents > @@ -14,6 +16,7 @@ whole; patches welcome! > .. toctree:: > :maxdepth: 2 > > + testing-overview > coccinelle > sparse > kcov > diff --git a/Documentation/dev-tools/testing-overview.rst b/Documentation/dev-tools/testing-overview.rst > new file mode 100644 > index 000000000000..8452adcb8608 > --- /dev/null > +++ b/Documentation/dev-tools/testing-overview.rst > @@ -0,0 +1,102 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +==================== > +Kernel Testing Guide > +==================== > + > + > +There are a number of different tools for testing the Linux kernel, so knowing > +when to use each of them can be a challenge. This document provides a rough > +overview of their differences, and how they fit together. > + > + > +Writing and Running Tests > +========================= > + > +The bulk of kernel tests are written using either the :doc:`kselftest > +<kselftest>` or :doc:`KUnit <kunit/index>` frameworks. These both provide > +infrastructure to help make running tests and groups of tests easier, as well > +as providing helpers to aid in writing new tests. > + > +If you're looking to verify the behaviour of the Kernel — particularly specific > +parts of the kernel — then you'll want to use `KUnit` or `kselftest`. > + > + > +The Difference Between KUnit and kselftest > +------------------------------------------ This section does a good job, but on a pragmatic level, there are a few more reasons to pick one or the other. E.g. the edit/build/test cycle will likely always be faster in KUnit. I'd also initially drafted up a _very_ long list of reasons to prefer kselftest as well. But looking back at them, a lot will hopefully be mitigated soon, or naturally get better with more usage/time, and you touched on that it can be easier to set up state from userspace already down below. > + > +:doc:`KUnit <kunit/index>` is an entirely in-kernel system for "white box" > +testing: because test code is part of the kernel, it can access internal > +structures and functions which aren't exposed to userspace. > + > +`KUnit` tests therefore are best written against small, self-contained parts > +of the kernel, which can be tested in isolation. This aligns well with the > +concept of Unit testing. Nit: we have "Unit testing" here and "'system' or 'end-to-end' testing." Perhaps: 'unit' testing > + > +For example, a KUnit test might test an individual kernel function (or even a > +single codepath through a function, such as an error handling case), rather > +than a feature as a whole. > + > +There is a KUnit test style guide which may give further pointers Seems like this sentence got truncated? Hmm, I'm not sure what this would be referring to however. I'm not sure there's a doc that touches on what's amenable to being unit tested. > + > + > +:doc:`kselftest <kselftest>`, on the other hand, is largely implemented in > +userspace, and tests are normal userspace scripts or programs. > + > +This makes it easier to write more complicated tests, or tests which need to > +manipulate the overall system state more (e.g., spawning processes, etc.). > +However, it's not possible to call kernel functions directly unless they're Saying it's not possible to call kernel code before mentioning the use of kernel modules to call kernel code directly is a bit confusing. Perhaps instead: However, it's not possible to call kernel functions directly unless you write a companion kernel module for the test. If your test is mostly or entirely inside a kernel module, `KUnit` may be the better tool. > +exposed to userspace (by a syscall, device, filesystem, etc.) Some tests to > +also provide a kernel module which is loaded by the test, though for tests > +which run mostly or entirely within the kernel, `KUnit` may be the better tool. > + > +`kselftest` is therefore suited well to tests of whole features, as these will > +expose an interface to userspace, which can be tested, but not implementation > +details. This aligns well with 'system' or 'end-to-end' testing. > + > + > +Code Coverage Tools > +=================== > + > +The Linux Kernel supports two different code coverage mesurement tools. These *measurement > +can be used to verify that a test is executing particular functions or lines > +of code. This is useful for determining how much of the kernel is being tested, > +and for finding corner-cases which are not covered by the appropriate test. > + > +:doc:`kcov` is a feature which can be built in to the kernel to allow > +capturing coverage on a per-task level. It's therefore useful for fuzzing and > +other situations where information about code executed during, for example, a > +single syscall is useful. > + > +:doc:`gcov` is GCC's coverage testing tool, which can be used with the kernel > +to get global or per-module coverage. Unlike KCOV, it does not record per-task > +coverage. Coverage data can be read from debugfs, and interpreted using the > +usual gcov tooling. Nit: I think gcov is the one most people reading this doc are going to be interested in, so I'd mention it first. > + > + > +Sanitizers > +========== > + > +The kernel also supports a number of sanitizers, which attempt to detect > +classes of issues when the occur in a running kernel. These typically *they occur > +look for undefined behaviour of some kind, such as invalid memory accesses, > +concurrency issues such as data races, or other undefined behaviour like > +integer overflows. > + > +* :doc:`kmemleak` (Kmemleak) detects possible memory leaks. > +* :doc:`kasan` detects invalid memory accesses such as out-of-bounds and > + use-after-free errors. > +* :doc:`ubsan` detects behaviour that is undefined by the C standard, like > + integer overflows. > +* :doc:`kcsan` detects data races. > +* :doc:`kfence` is a low-overhead detector of memory issues, which is much > + faster than KASAN and can be used in production. Hmm, it lives elsewhere, but would also calling out lockdep here be useful? I've also not heard anyone call it a sanitizer before, but it fits the definition you've given. Now that I think about it, I've never looked for documentation on it, is this the best page? https://www.kernel.org/doc/html/latest/locking/lockdep-design.html > + > +These tools tend to test the kernel as a whole, and do not "pass" like > +kselftest or KUnit tests. They can be combined with KUnit or kselftest by > +running tests on a kernel with a sanitizer enabled: you can then be sure > +that none of these errors are occurring during the test. > + > +Some of these sanitizers integrate with KUnit or kselftest and will > +automatically fail tests if an issue is detected by a sanitizer. > + > -- > 2.31.1.295.g9ea45b61b8-goog >
