On 9/24/24 08:56, Jani Nikula wrote:
On Tue, 24 Sep 2024, Guenter Roeck <linux@xxxxxxxxxxxx> wrote:
On Tue, Sep 24, 2024 at 12:06:28PM GMT, Simona Vetter wrote:
Yeah I think long-term we might want a kunit framework so that we can
catch dmesg warnings we expect and test for those, without those warnings
actually going to dmesg. Similar to how the lockdep tests also reroute
locking validation, so that the expected positive tests don't wreak
lockdep for real.
But until that exists, we can't have tests that splat in dmesg when they
work as intended.
FWIW, that is arguable. More and more tests are added which do add such splats,
and I don't see any hesitance by developers to adding more. So far I counted
two alone in this commit window, and that does not include new splats from
tests which I had already disabled. I simply disable those tests or don't
enable them in the first place if they are new. I did the same with the drm
unit tests due to the splats generated by the scaling unit tests, so any
additional drm unit test splats don't make a difference for me since the
tests are already disabled.
What's the point of having unit tests that CI systems routinely have to
filter out of test runs? Or filter warnings generated by the tests,
potentially missing new warnings. Who is going to run the tests if the
existing CI systems choose to ignore them?
Automation on a massive scale is key here, and making that harder is
counter-productive.
As I have said elsewhere, not being able to handle backtraces generated
on purpose is a limitation of my testbed. Other testbeds with active (and
paid) maintainers may not have that limitation. I deal with it by disabling
affected tests. Others may deal with it by maintaining exception lists.
Ultimately it is up to developers and testbed maintainers to decide the level
of test coverage they are looking for or able to support.
Thanks,
Guenter
