On Wed, Jul 31, 2024 at 05:19:45PM -0600, Shuah Khan wrote: > On 7/24/24 15:40, Nícolas F. R. A. Prado wrote: > > Introduce a new test to identify regressions causing devices to go > > missing on the system. > > > > For each bus and class on the system the test checks the number of > > devices present against a reference file, which needs to have been > > generated by the program at a previous point on a known-good kernel, and > > if there are missing devices they are reported. > > Can you elaborate on how to generate reference file? It isn't clear. Indeed, I'll make that information clearer in future versions. The reference file is generated by passing the --generate-reference flag to the test: ./exist.py --generate-reference It will be printed as standard output. > > > > > Signed-off-by: Nícolas F. R. A. Prado <nfraprado@xxxxxxxxxxxxx> > > --- > > Hi, > > > > Key points about this test: > > * Goal: Identify regressions causing devices to go missing on the system > > * Focus: > > * Ease of maintenance: the reference file is generated programatically > > * Minimum of false-positives: the script makes as few assumptions as possible > > about the stability of device identifiers to ensure renames/refactors don't > > trigger false-positives > > * How it works: For each bus and class on the system the test checks the number > > of devices present against a reference file, which needs to have been > > generated by the program at a previous point on a known-good kernel, and if > > there are missing devices they are reported. > > * Comparison to other tests: It might be possible(*) to replace the discoverable > > devices test [1] with this. The benefits of this test is that it's easier > > to setup and maintain and has wider coverage of devices. > > > > Additional detail: > > * Having more devices on the running system than the reference does not cause a > > failure, but a warning is printed in that case to suggest that the reference > > be updated. > > * Missing devices are detected per bus/class based on the number of devices. > > When the test fails, the known metadata for each of the expected and detected > > devices is printed and some simple similitarity comparison is done to suggest > > the devices that are the most likely to be missing. > > * The proposed place to store the generated reference files is the > > 'platform-test-parameters' repository in KernelCI [2]. > > How would a user run this on their systems - do they need to access > this repository in KernelCI? No, that repository would just be a place where people could find pre-generated reference files (which we'll be using when running this test in KernelCI), but anyone can always generate their own reference files and store them wherever they want. > > This is what I see when I run the test on my system: > > make -C tools/testing/selftests/devices/exist/ run_tests > make: Entering directory '/linux/linux_6.11/tools/testing/selftests/devices/exist' > TAP version 13 > 1..1 > # timeout set to 45 > # selftests: devices/exist: exist.py > # TAP version 13 > # # No matching reference file found (tried './LENOVO,20XH005JUS.yaml') First generate the reference file for your system like so: tools/testing/selftests/devices/exist/exist.py --generate-reference > tools/testing/selftests/devices/exist/LENOVO,20XH005JUS.yaml Then you can run the test and it should work. Thanks, Nícolas
