This patchset proposes roadtest, a device-driver testing framework. Drivers are tested under User Mode Linux (UML) and interact with mocked/modelled hardware. The tests and hardware models are written in Python, the former using Python's built-in unittest framework. Drivers are tested via their userspace interfaces. The hardware models allow tests to inject values into registers and assert that drivers control the hardware in the right way and react as expected to stimuli. Roadtest is meant to be used for relatively simple drivers, such as the ones part of the IIO, regulator and RTC subsystems. Questions and answers: = Why do we need this? There are a large amount of these kind of drivers in the kernel. Most of the hardware is not available in current CI systems so most drivers can only, at best, be build-tested there. Even basic soundness such as a driver successfully probing and binding to the devices it tries to be support cannot be tested. Drivers cannot be easily regression-tested to ensure that bugs fixed once do not get reintroduced. Many drivers support multiple related hardware variants, and far from all patch submitters have access to all the variants which the driver that they are patching supports, so there is no way for them to easily verify that they haven't broken something basic on a variant which they do not own. Furthermore, hardware can be used in many different configurations with drivers supporting many different devicetree properties, so even just having access to all the variants would be insufficient. On top of that, some of the chips measure environmental conditions such as temperature, so testing extreme cases may not be simple even if one has access to the hardware. All this makes development, modification, maintenance, and reviewing of these drivers harder than it necessarily needs to be. Roadtest hopes to make some of these things slightly easier by providing a framework to create hardware models/mocks and to write testcases which exercise drivers using these models. = Do you have some specific examples of the kind of code this could be used to test? Here is an example of a patch which can easily be regression-tested using roadtest (in fact, this series includes such a regression test) but is much harder to do so automatically with real hardware since it requires specific environmental conditions: iio: light: opt3001: Fixed timeout error when 0 lux https://lore.kernel.org/lkml/20210920125351.6569-1-valek@xxxxx/ Here is another example. This driver has code which correctly parses a documented devicetree property (amstaos,proximity-diodes) but which then fails to actually communicate this setting to the hardware in any way. Such code can be easily tested with roadtest since the framework integrates devicetree support and provides functions to assert that drivers writes expected registers with expected values: drivers/iio/light/tsl2772.c tsl2772_read_prox_diodes() (Both the above examples happen to be from the same subsystem but that should in no way be taken to imply that such issues are unique to that subsystem or that that subsystem has more of them.) = How does this relate to kselftests? Tests in kselftests also test kernel code using the userspace interfaces, but that's about what's common between the frameworks. kselftests has other goals and does not provide any kind of mechanism for hardware mocking. = How does this relate to kunit? Kunit is for unit testing of functions in kernel code, and is not meant for testing kernel code via userspace interfaces. It could in theory be used to test some of the simple drivers too, but that would require (1) a large amount of mocking code in various kernel frameworks, and, more importantly, (2) refactoring of the drivers to be tested. This can be contrasted with roadtest which works with mostly unmodified drivers and which mocks the hardware at the lowest level without having to change kernel frameworks. = How do I use it? See Documentation/dev-tools/roadtest.rst added by the documentation patch for more information about running and writing tests using this framework. = What's included in the patchset? The current framework allows developing tests for hardware which uses the I2C bus. Hardware models can also control GPIOs and use them to trigger interrupts. This series includes tests for some IIO, regulator and RTC drivers. The regulator and RTC tests depend on a few driver patches which are either in review or in linux-next. These are noted in the commit messages. The entire patch set, including the required dependencies, is also available in a git tree: https://github.com/vwax/linux/commits/roadtest/rfc-v1 Cc: linux-kernel@xxxxxxxxxxxxxxx Cc: devicetree@xxxxxxxxxxxxxxx Cc: linux-um@xxxxxxxxxxxxxxxxxxx Cc: shuah@xxxxxxxxxx Cc: brendanhiggins@xxxxxxxxxx Cc: linux-kselftest@xxxxxxxxxxxxxxx Cc: jic23@xxxxxxxxxx Cc: linux-iio@xxxxxxxxxxxxxxx Cc: lgirdwood@xxxxxxxxx Cc: broonie@xxxxxxxxxx Cc: a.zummo@xxxxxxxxxxxx Cc: alexandre.belloni@xxxxxxxxxxx Cc: linux-rtc@xxxxxxxxxxxxxxx Cc: corbet@xxxxxxx Cc: linux-doc@xxxxxxxxxxxxxxx Vincent Whitchurch (10): roadtest: import libvhost-user from QEMU roadtest: add C backend roadtest: add framework roadtest: add base config roadtest: add build files roadtest: add documentation iio: light: opt3001: add roadtest iio: light: vcnl4000: add roadtest regulator: tps62864: add roadtest rtc: pcf8563: add roadtest Documentation/dev-tools/index.rst | 1 + Documentation/dev-tools/roadtest.rst | 669 ++++ tools/testing/roadtest/.gitignore | 2 + tools/testing/roadtest/Dockerfile | 25 + tools/testing/roadtest/Makefile | 84 + tools/testing/roadtest/init.sh | 19 + tools/testing/roadtest/pyproject.toml | 10 + tools/testing/roadtest/requirements.txt | 4 + tools/testing/roadtest/roadtest/__init__.py | 2 + .../roadtest/roadtest/backend/__init__.py | 0 .../roadtest/roadtest/backend/backend.py | 32 + .../testing/roadtest/roadtest/backend/gpio.py | 111 + .../testing/roadtest/roadtest/backend/i2c.py | 123 + .../testing/roadtest/roadtest/backend/main.py | 13 + .../testing/roadtest/roadtest/backend/mock.py | 20 + .../roadtest/roadtest/backend/test_gpio.py | 98 + .../roadtest/roadtest/backend/test_i2c.py | 84 + .../testing/roadtest/roadtest/cmd/__init__.py | 0 tools/testing/roadtest/roadtest/cmd/main.py | 146 + tools/testing/roadtest/roadtest/cmd/remote.py | 48 + .../roadtest/roadtest/core/__init__.py | 0 .../testing/roadtest/roadtest/core/control.py | 52 + .../roadtest/roadtest/core/devicetree.py | 155 + .../roadtest/roadtest/core/hardware.py | 94 + tools/testing/roadtest/roadtest/core/log.py | 42 + .../testing/roadtest/roadtest/core/modules.py | 38 + .../testing/roadtest/roadtest/core/opslog.py | 35 + tools/testing/roadtest/roadtest/core/proxy.py | 48 + tools/testing/roadtest/roadtest/core/suite.py | 286 ++ tools/testing/roadtest/roadtest/core/sysfs.py | 77 + .../roadtest/roadtest/core/test_control.py | 35 + .../roadtest/roadtest/core/test_devicetree.py | 31 + .../roadtest/roadtest/core/test_hardware.py | 41 + .../roadtest/roadtest/core/test_log.py | 54 + .../roadtest/roadtest/core/test_opslog.py | 27 + .../roadtest/roadtest/tests/__init__.py | 0 .../roadtest/roadtest/tests/base/config | 84 + .../roadtest/roadtest/tests/iio/__init__.py | 0 .../roadtest/roadtest/tests/iio/config | 1 + .../roadtest/roadtest/tests/iio/iio.py | 112 + .../roadtest/tests/iio/light/__init__.py | 0 .../roadtest/roadtest/tests/iio/light/config | 2 + .../roadtest/tests/iio/light/test_opt3001.py | 95 + .../roadtest/tests/iio/light/test_vcnl4000.py | 132 + .../roadtest/tests/iio/light/test_vcnl4010.py | 282 ++ .../roadtest/tests/iio/light/test_vcnl4040.py | 104 + .../roadtest/tests/iio/light/test_vcnl4200.py | 96 + .../roadtest/tests/regulator/__init__.py | 0 .../roadtest/roadtest/tests/regulator/config | 4 + .../roadtest/tests/regulator/test_tps62864.py | 187 ++ .../roadtest/roadtest/tests/rtc/__init__.py | 0 .../roadtest/roadtest/tests/rtc/config | 1 + .../roadtest/roadtest/tests/rtc/rtc.py | 73 + .../roadtest/tests/rtc/test_pcf8563.py | 348 ++ tools/testing/roadtest/src/.gitignore | 1 + tools/testing/roadtest/src/backend.c | 884 +++++ .../src/libvhost-user/include/atomic.h | 310 ++ .../src/libvhost-user/libvhost-user.c | 2885 +++++++++++++++++ .../src/libvhost-user/libvhost-user.h | 691 ++++ 59 files changed, 8798 insertions(+) create mode 100644 Documentation/dev-tools/roadtest.rst create mode 100644 tools/testing/roadtest/.gitignore create mode 100644 tools/testing/roadtest/Dockerfile create mode 100644 tools/testing/roadtest/Makefile create mode 100755 tools/testing/roadtest/init.sh create mode 100644 tools/testing/roadtest/pyproject.toml create mode 100644 tools/testing/roadtest/requirements.txt create mode 100644 tools/testing/roadtest/roadtest/__init__.py create mode 100644 tools/testing/roadtest/roadtest/backend/__init__.py create mode 100644 tools/testing/roadtest/roadtest/backend/backend.py create mode 100644 tools/testing/roadtest/roadtest/backend/gpio.py create mode 100644 tools/testing/roadtest/roadtest/backend/i2c.py create mode 100644 tools/testing/roadtest/roadtest/backend/main.py create mode 100644 tools/testing/roadtest/roadtest/backend/mock.py create mode 100644 tools/testing/roadtest/roadtest/backend/test_gpio.py create mode 100644 tools/testing/roadtest/roadtest/backend/test_i2c.py create mode 100644 tools/testing/roadtest/roadtest/cmd/__init__.py create mode 100644 tools/testing/roadtest/roadtest/cmd/main.py create mode 100644 tools/testing/roadtest/roadtest/cmd/remote.py create mode 100644 tools/testing/roadtest/roadtest/core/__init__.py create mode 100644 tools/testing/roadtest/roadtest/core/control.py create mode 100644 tools/testing/roadtest/roadtest/core/devicetree.py create mode 100644 tools/testing/roadtest/roadtest/core/hardware.py create mode 100644 tools/testing/roadtest/roadtest/core/log.py create mode 100644 tools/testing/roadtest/roadtest/core/modules.py create mode 100644 tools/testing/roadtest/roadtest/core/opslog.py create mode 100644 tools/testing/roadtest/roadtest/core/proxy.py create mode 100644 tools/testing/roadtest/roadtest/core/suite.py create mode 100644 tools/testing/roadtest/roadtest/core/sysfs.py create mode 100644 tools/testing/roadtest/roadtest/core/test_control.py create mode 100644 tools/testing/roadtest/roadtest/core/test_devicetree.py create mode 100644 tools/testing/roadtest/roadtest/core/test_hardware.py create mode 100644 tools/testing/roadtest/roadtest/core/test_log.py create mode 100644 tools/testing/roadtest/roadtest/core/test_opslog.py create mode 100644 tools/testing/roadtest/roadtest/tests/__init__.py create mode 100644 tools/testing/roadtest/roadtest/tests/base/config create mode 100644 tools/testing/roadtest/roadtest/tests/iio/__init__.py create mode 100644 tools/testing/roadtest/roadtest/tests/iio/config create mode 100644 tools/testing/roadtest/roadtest/tests/iio/iio.py create mode 100644 tools/testing/roadtest/roadtest/tests/iio/light/__init__.py create mode 100644 tools/testing/roadtest/roadtest/tests/iio/light/config create mode 100644 tools/testing/roadtest/roadtest/tests/iio/light/test_opt3001.py create mode 100644 tools/testing/roadtest/roadtest/tests/iio/light/test_vcnl4000.py create mode 100644 tools/testing/roadtest/roadtest/tests/iio/light/test_vcnl4010.py create mode 100644 tools/testing/roadtest/roadtest/tests/iio/light/test_vcnl4040.py create mode 100644 tools/testing/roadtest/roadtest/tests/iio/light/test_vcnl4200.py create mode 100644 tools/testing/roadtest/roadtest/tests/regulator/__init__.py create mode 100644 tools/testing/roadtest/roadtest/tests/regulator/config create mode 100644 tools/testing/roadtest/roadtest/tests/regulator/test_tps62864.py create mode 100644 tools/testing/roadtest/roadtest/tests/rtc/__init__.py create mode 100644 tools/testing/roadtest/roadtest/tests/rtc/config create mode 100644 tools/testing/roadtest/roadtest/tests/rtc/rtc.py create mode 100644 tools/testing/roadtest/roadtest/tests/rtc/test_pcf8563.py create mode 100644 tools/testing/roadtest/src/.gitignore create mode 100644 tools/testing/roadtest/src/backend.c create mode 100644 tools/testing/roadtest/src/libvhost-user/include/atomic.h create mode 100644 tools/testing/roadtest/src/libvhost-user/libvhost-user.c create mode 100644 tools/testing/roadtest/src/libvhost-user/libvhost-user.h -- 2.34.1
