On Tue, Apr 16, 2024 at 07:31:56PM +0900, Shin'ichiro Kawasaki wrote: > In the recent discussion for nvme test group [1], two pain points were mentioned > regarding the test case runs. > > 1) Several test cases in nvme test group do exactly the same test except the > NVME transport backend set up condition difference (device vs. file). This > results in duplicate test script codes. It is desired to unify the test cases > and run them repeatedly with the different conditions. > > 2) NVME transport types can be specified with nvme_trtype parameter so that the > same tests can be run for various transport types. However, some test cases > do not depend on the transport types. They are repeated in multiple runs for > the various transport types under the exact same conditions. It is desired to > repeat the test cases only when such repetition is required. > > [1] https://lore.kernel.org/linux-block/w2eaegjopbah5qbjsvpnrwln2t5dr7mv3v4n2e63m5tjqiochm@uonrjm2i2g72/ > > One idea to address these pain points is to add the test repeat feature to the > nvme test group. However, Daniel questioned if the feature could be implemented > in the blktests framework. Actually, a similar feature has already been > implemented to repeat some test cases for non-zoned block devices and zoned > block devices. However, this feature is implemented only for the zoned and non- > zoned device conditions. It can not fulfill the desires for nvme test group. > > This series proposes to generalize the feature in the blktests framework to > repeat the test cases with different conditions. Introduce a new function > set_conditions() that each test case can define and instruct the framework to > repeat the test case. The first four patches introduce the feature and apply it > to the repetition for non-zoned and zoned block devices. The following seven > patches apply the feature to nvme test group so that the test cases can be > repeated for NVME transport types and backend types in the ideal way. Two of the > seven patches are reused from the work by Daniel. The all patches are listed in > the order that does not lose the test coverage with the default set up. > > This series introduces new config parameters NVMET_TRTYPES and > NVMET_BLKDEV_TYPES, which can take multiple values with space separators. When > they are defined in the config file as follows, > > NVMET_TRTYPES="loop rdma tcp" > NVMET_BLKDEV_TYPES="device file" > > the test cases which depend on these parameters are repeated 3 x 2 = 6 times. > For example, nvme/006 is repeated as follows. > > nvme/006 (nvmet bd=device tr=loop) (create an NVMeOF target) [passed] > runtime 0.148s ... 0.165s > nvme/006 (nvmet bd=device tr=rdma) (create an NVMeOF target) [passed] > runtime 0.273s ... 0.235s > nvme/006 (nvmet bd=device tr=tcp) (create an NVMeOF target) [passed] > runtime 0.162s ... 0.164s > nvme/006 (nvmet bd=file tr=loop) (create an NVMeOF target) [passed] > runtime 0.138s ... 0.134s > nvme/006 (nvmet bd=file tr=rdma) (create an NVMeOF target) [passed] > runtime 0.216s ... 0.201s > nvme/006 (nvmet bd=file tr=tcp) (create an NVMeOF target) [passed] > runtime 0.154s ... 0.146s Looks all good to me. I also run the nvme part of blktests. The framework works fine but as expected we have a bunch of new errors reported because of the additional test coverage this change brings. Thanks a lot for your work! Reviewed-by: Daniel Wagner <dwagner@xxxxxxx>
