Hi Dan, Thank you for your reply! On 13/11/2024 18:08, Dan Carpenter wrote: > On Fri, Nov 08, 2024 at 07:21:59PM +0100, Matthieu Baerts wrote: >> KSelftests from the same version >> -------------------------------- >> >> According to the doc [2], kselftests should support all previous kernel >> versions. The LKFT CI is then using the kselftests from the last stable >> release to validate all stable versions. Even if there are good reasons >> to do that, we would like to ask for an opt-out for this policy for the >> networking tests: this is hard to maintain with the increased >> complexity, hard to validate on all stable kernels before applying >> patches, and hard to put in place in some situations. As a result, many >> tests are failing on older kernels, and it looks like it is a lot of >> work to support older kernels, and to maintain this. >> >> Many networking tests are validating the internal behaviour that is not >> exposed to the userspace. A typical example: some tests look at the raw >> packets being exchanged during a test, and this behaviour can change >> without modifying how the userspace is interacting with the kernel. The >> kernel could expose capabilities, but that's not something that seems >> natural to put in place for internal behaviours that are not exposed to >> end users. Maybe workarounds could be used, e.g. looking at kernel >> symbols, etc. Nut that doesn't always work, increase the complexity, and >> often "false positive" issue will be noticed only after a patch hits >> stable, and will cause a bunch of tests to be ignored. >> >> Regarding fixes, ideally they will come with a new or modified test that >> can also be backported. So the coverage can continue to grow in stable >> versions too. >> >> Do you think that from the kernel v6.12 (or before?), the LKFT CI could >> run the networking kselftests from the version that is being validated, >> and not from a newer one? So validating the selftests from v6.12.1 on a >> v6.12.1, and not the ones from a future v6.16.y on a v6.12.42. >> > > These kinds of decisions are something that Greg and Shuah need to decide on. Thank you, it makes sense. > You would still need some way to automatically detect that kselftest is running > on an old kernel and disable the networking checks. Otherwise when random > people on the internet try to run selftests they would run into issues. Indeed. I guess we can always add a warning when the kernel and selftests versions are different. I suppose the selftests are built using the same kernel version, then executed on older versions: we could then compare the kernel versions at build time and run time, no? Regarding the other questions from my previous email -- skipped tests (e.g. I think Netfilter tests are no longer validated), KVM, notifications -- do you know who at Linaro could eventually look at them? Cheers, Matt -- Sponsored by the NGI0 Core fund.
