On Thu, Feb 4, 2021 at 2:54 PM Daniel Latypov <dlatypov@xxxxxxxxxx> wrote: > > When using `kunit.py run` to run tests, users must populate a > `kunitconfig` file to select the options the tests are hidden behind and > all their dependencies. > > The patch [1] to allow specifying a path to kunitconfig promises to make > this nicer as we can have checked in files corresponding to different > sets of tests. > > But it's still annoying > 1) when trying to run a subet of tests > 2) when you want to run tests that don't have such a pre-existing > kunitconfig and selecting all the necessary options is tricky. > > This patch series aims to alleviate both: > 1) `kunit.py run 'my-suite-*'` > I.e. use my current kunitconfig, but just run suites that match this glob > 2) `kunit.py run --alltests 'my-suite-*'` > I.e. use allyesconfig so I don't have to worry about writing a > kunitconfig at all. > > See the first commit message for more details and discussion about > future work. > > This patch series also includes a bugfix for a latent bug that can't be > triggered right now but has worse consequences as a result of the > changes needed to plumb in this suite name glob. > > [1] https://lore.kernel.org/linux-kselftest/20210201205514.3943096-1-dlatypov@xxxxxxxxxx/ I tried applying this series on top of several of your other patches over the past cycle and running the kunit_tool unit tests (kunit_tool_test.py) and I got several failures: ====================================================================== FAIL: test_exec_builddir (__main__.KUnitMainTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "tools/testing/kunit/kunit_tool_test.py", line 417, in test_exec_builddir self.linux_source_mock.run_kernel.assert_called_once_with(build_dir=build_dir, timeout=300) File "/usr/lib/python3.8/unittest/mock.py", line 925, in assert_called_once_with return self.assert_called_with(*args, **kwargs) File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with raise AssertionError(_error_message()) from cause AssertionError: expected call not found. Expected: run_kernel(build_dir='.kunit', timeout=300) Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit') ====================================================================== FAIL: test_exec_passes_args_pass (__main__.KUnitMainTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "tools/testing/kunit/kunit_tool_test.py", line 337, in test_exec_passes_args_pass self.linux_source_mock.run_kernel.assert_called_once_with(build_dir='.kunit', timeout=300) File "/usr/lib/python3.8/unittest/mock.py", line 925, in assert_called_once_with return self.assert_called_with(*args, **kwargs) File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with raise AssertionError(_error_message()) from cause AssertionError: expected call not found. Expected: run_kernel(build_dir='.kunit', timeout=300) Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit') ====================================================================== FAIL: test_exec_timeout (__main__.KUnitMainTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "tools/testing/kunit/kunit_tool_test.py", line 385, in test_exec_timeout self.linux_source_mock.run_kernel.assert_called_once_with(build_dir='.kunit', timeout=timeout) File "/usr/lib/python3.8/unittest/mock.py", line 925, in assert_called_once_with return self.assert_called_with(*args, **kwargs) File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with raise AssertionError(_error_message()) from cause AssertionError: expected call not found. Expected: run_kernel(build_dir='.kunit', timeout=3453) Actual: run_kernel(timeout=3453, filter_glob='', build_dir='.kunit') ====================================================================== FAIL: test_run_builddir (__main__.KUnitMainTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "tools/testing/kunit/kunit_tool_test.py", line 400, in test_run_builddir self.linux_source_mock.run_kernel.assert_called_once_with( File "/usr/lib/python3.8/unittest/mock.py", line 925, in assert_called_once_with return self.assert_called_with(*args, **kwargs) File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with raise AssertionError(_error_message()) from cause AssertionError: expected call not found. Expected: run_kernel(build_dir='.kunit', timeout=300) Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit') ====================================================================== FAIL: test_run_passes_args_pass (__main__.KUnitMainTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "tools/testing/kunit/kunit_tool_test.py", line 344, in test_run_passes_args_pass self.linux_source_mock.run_kernel.assert_called_once_with( File "/usr/lib/python3.8/unittest/mock.py", line 925, in assert_called_once_with return self.assert_called_with(*args, **kwargs) File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with raise AssertionError(_error_message()) from cause AssertionError: expected call not found. Expected: run_kernel(build_dir='.kunit', timeout=300) Actual: run_kernel(timeout=300, filter_glob='', build_dir='.kunit') ====================================================================== FAIL: test_run_timeout (__main__.KUnitMainTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "tools/testing/kunit/kunit_tool_test.py", line 392, in test_run_timeout self.linux_source_mock.run_kernel.assert_called_once_with( File "/usr/lib/python3.8/unittest/mock.py", line 925, in assert_called_once_with return self.assert_called_with(*args, **kwargs) File "/usr/lib/python3.8/unittest/mock.py", line 913, in assert_called_with raise AssertionError(_error_message()) from cause AssertionError: expected call not found. Expected: run_kernel(build_dir='.kunit', timeout=3453) Actual: run_kernel(timeout=3453, filter_glob='', build_dir='.kunit') ---------------------------------------------------------------------- Ran 37 tests in 0.082s FAILED (failures=6)
