On Sat, Apr 10, 2021 at 12:05 AM David Gow <davidgow@xxxxxxxxxx> wrote: > > The kernel now has a number of testing and debugging tools, and we've > seen a bit of confusion about what the differences between them are. > > Add a basic documentation outlining the testing tools, when to use each, > and how they interact. > > This is a pretty quick overview rather than the idealised "kernel > testing guide" that'd probably be optimal, but given the number of times > questions like "When do you use KUnit and when do you use Kselftest?" > are being asked, it seemed worth at least having something. Hopefully > this can form the basis for more detailed documentation later. > > Signed-off-by: David Gow <davidgow@xxxxxxxxxx> With the exception of some minor nits, I think the below will make a great initial testing overview guide! Thanks for getting the ball rolling on this! > --- > Documentation/dev-tools/index.rst | 3 + > Documentation/dev-tools/testing-overview.rst | 102 +++++++++++++++++++ > 2 files changed, 105 insertions(+) > create mode 100644 Documentation/dev-tools/testing-overview.rst > > diff --git a/Documentation/dev-tools/index.rst b/Documentation/dev-tools/index.rst > index 1b1cf4f5c9d9..f590e5860794 100644 > --- a/Documentation/dev-tools/index.rst > +++ b/Documentation/dev-tools/index.rst > @@ -7,6 +7,8 @@ be used to work on the kernel. For now, the documents have been pulled > together without any significant effort to integrate them into a coherent > whole; patches welcome! > > +A brief overview of testing-specific tools can be found in :doc:`testing-overview`. > + I think I would like to make this a little more apparent. This index here is a bit bare bones and I think this testing-overview could be a good: "I am lost where do I start?" sort of doc. That being said, I am not sure what the best way to emphasize this might be. Maybe just have an intro paragraph here with some callout text like in a `note` or something like that. > .. class:: toc-title > > Table of contents > @@ -14,6 +16,7 @@ whole; patches welcome! > .. toctree:: > :maxdepth: 2 > > + testing-overview > coccinelle > sparse > kcov > diff --git a/Documentation/dev-tools/testing-overview.rst b/Documentation/dev-tools/testing-overview.rst > new file mode 100644 > index 000000000000..8452adcb8608 > --- /dev/null > +++ b/Documentation/dev-tools/testing-overview.rst > @@ -0,0 +1,102 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +==================== > +Kernel Testing Guide > +==================== > + > + > +There are a number of different tools for testing the Linux kernel, so knowing > +when to use each of them can be a challenge. This document provides a rough > +overview of their differences, and how they fit together. > + > + > +Writing and Running Tests > +========================= > + > +The bulk of kernel tests are written using either the :doc:`kselftest > +<kselftest>` or :doc:`KUnit <kunit/index>` frameworks. These both provide > +infrastructure to help make running tests and groups of tests easier, as well > +as providing helpers to aid in writing new tests. > + > +If you're looking to verify the behaviour of the Kernel — particularly specific > +parts of the kernel — then you'll want to use `KUnit` or `kselftest`. > + > + > +The Difference Between KUnit and kselftest > +------------------------------------------ > + > +:doc:`KUnit <kunit/index>` is an entirely in-kernel system for "white box" > +testing: because test code is part of the kernel, it can access internal > +structures and functions which aren't exposed to userspace. > + > +`KUnit` tests therefore are best written against small, self-contained parts > +of the kernel, which can be tested in isolation. This aligns well with the > +concept of Unit testing. I think I might have pushed the "unit testing" stuff too hard in the past, but I feel that if you are going to mention "the concept of unit testing" it might be a good idea to link to an authoritative source on what unit testing is. Maybe link to Martin Fowler or something like that? > +For example, a KUnit test might test an individual kernel function (or even a > +single codepath through a function, such as an error handling case), rather > +than a feature as a whole. > + > +There is a KUnit test style guide which may give further pointers I know you linked the index page for KUnit above, but I think you might want to link the KUnit style guide here since you mention it. > +:doc:`kselftest <kselftest>`, on the other hand, is largely implemented in > +userspace, and tests are normal userspace scripts or programs. > + > +This makes it easier to write more complicated tests, or tests which need to > +manipulate the overall system state more (e.g., spawning processes, etc.). > +However, it's not possible to call kernel functions directly unless they're > +exposed to userspace (by a syscall, device, filesystem, etc.) Some tests to > +also provide a kernel module which is loaded by the test, though for tests > +which run mostly or entirely within the kernel, `KUnit` may be the better tool. > + > +`kselftest` is therefore suited well to tests of whole features, as these will > +expose an interface to userspace, which can be tested, but not implementation > +details. This aligns well with 'system' or 'end-to-end' testing. Again, I think you might want to link to some sources that explain what "system" and "end-to-end" testing are. Also, I think maybe adding a section on some common examples of when to use Kselftest vs when to use KUnit would be helpful. For example: - If I add a new syscall, you might want to mention that the author is *required* to add accompanying Kselftests. - A new internal API - for example a new crypto API - is *strongly recommended* to have accompanying KUnit tests. - Many new features, that have a large in kernel API, but also have a user visible API should probably have both Kselftests as well as KUnit tests. Enumerating other examples is probably a good idea, but I think this offers a good flavor. > +Code Coverage Tools > +=================== > + > +The Linux Kernel supports two different code coverage mesurement tools. These > +can be used to verify that a test is executing particular functions or lines > +of code. This is useful for determining how much of the kernel is being tested, > +and for finding corner-cases which are not covered by the appropriate test. > + > +:doc:`kcov` is a feature which can be built in to the kernel to allow > +capturing coverage on a per-task level. It's therefore useful for fuzzing and > +other situations where information about code executed during, for example, a > +single syscall is useful. > + > +:doc:`gcov` is GCC's coverage testing tool, which can be used with the kernel > +to get global or per-module coverage. Unlike KCOV, it does not record per-task > +coverage. Coverage data can be read from debugfs, and interpreted using the > +usual gcov tooling. > + > + > +Sanitizers > +========== > + > +The kernel also supports a number of sanitizers, which attempt to detect > +classes of issues when the occur in a running kernel. These typically > +look for undefined behaviour of some kind, such as invalid memory accesses, > +concurrency issues such as data races, or other undefined behaviour like > +integer overflows. > + > +* :doc:`kmemleak` (Kmemleak) detects possible memory leaks. > +* :doc:`kasan` detects invalid memory accesses such as out-of-bounds and > + use-after-free errors. > +* :doc:`ubsan` detects behaviour that is undefined by the C standard, like > + integer overflows. > +* :doc:`kcsan` detects data races. > +* :doc:`kfence` is a low-overhead detector of memory issues, which is much > + faster than KASAN and can be used in production. > + > +These tools tend to test the kernel as a whole, and do not "pass" like > +kselftest or KUnit tests. They can be combined with KUnit or kselftest by > +running tests on a kernel with a sanitizer enabled: you can then be sure > +that none of these errors are occurring during the test. > + > +Some of these sanitizers integrate with KUnit or kselftest and will > +automatically fail tests if an issue is detected by a sanitizer. > + > -- > 2.31.1.295.g9ea45b61b8-goog >
