On Fri, Feb 22, 2019 at 12:00:43AM +0000, Chaitanya Kulkarni wrote: > On 02/21/2019 07:04 AM, Theodore Y. Ts'o wrote: > > On Wed, Feb 20, 2019 at 11:35:00AM -0800, Omar Sandoval wrote: > >>> It should fail on older cli versions, we want latest cli/tools to be > >>> used when running these testcases so that all the latest tools are > >>> also been tested for the compliance, else we might be running tests > >>> with any bugs with older version(s) which are fixed already. > >>> > >>> Actually I'd like to add quick nvme-cli version check at the start of > >>> each test that will certainly help/force user to have a latest version. > >> > >> Perfect, I was going to suggest adding a check for the minimum version > >> needed for each test. > > > > It would be great if we could document what is (a) the minimum version > > for things to work at all, > Actually the latest version or the HEAD of the git repo should work all > the time if it is breaking test or the tool or driver we need to add > a fix respectively. > > and (b) the ideal version to use so that > > tests aren't being skipped due to version constraints and so we are > > testing the kernel as much as possible. > > > Agree. > > Ideally, tests should either be designed primarily to test userspace, > > or to test the kernel, so that people who want to use blktests aren't > > having to build the latest versions of userspace. > > > At least with the current NVMe infrastructure we need latest tools > (git HEAD) to be compliant all the time with the tests, since it > guarantees the tools, driver and kernel compatibility. > > We can a little install script to build and install the > cli if desired version is not present. > > > (As an aside, Bart has told me that the failures I'm seeing with > > srp/002 and srp/011 are due to the fact that I'm using a version of > > multipathd which is too old. Apparently both the version of > > multipathd in Debian stable *and* Debian unstable are still too old, > > since they are still failing. Next stop, building multipathd from > > the git repo.... Grrr......) > This is exact reason why tools needs to be built and check with latest > HEAD. I don't think we should be _requiring_ that the tests are run with the latest tools from git. That just adds another annoying step to someone trying to get the tests up and running. Like I said, each test should check for the minimum version it needs, and we should avoid making the golden output too dependent on version. However, I do think it's a reasonable expectation that if you want maximum test coverage, you should build the tools from git.
