On Fri, Mar 11, 2022 at 12:52:41PM -0800, Luis Chamberlain wrote:
>
> The only way to move forward with enabling more automation for kernel
> code integration is through better and improved kernel test automation.
> And it is *exactly* why I've been working so hard on that problem.
I think we're on the same page here.
> Also let's recall that just because you have your own test framework
> it does not mean we could not benefit from others testing our
> filesystems on their own silly hardware at home as well. Yes tons
> of projects can be used which wrap fstests...
No argument from me! I'm strongly in favor of diversity in test
framework automation as well as test environments.
In particular, I think there are some valuable things we can learn
from each other, in terms of cross polination in terms of features and
as well as feedback about how easy it is to use a particular test
framework.
For example: README.md doesn't say anything about running make as root
when running "make" as kdevops. At least, I *think* this is why
running make as kdevops failed:
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/usr/sbin/apparmor_status", "--enabled"], "delta": "0:00:00.001426", "end": "2022-03-11 16:23:11.769658", "failed_when_result": true, "rc": 0, "start": "2022-03-11 16:23:11.768232", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
(I do have apparmor installed, but it's currently not enabled. I
haven't done more experimentation since I'm a bit scared of running
"make XXX" as root for any package I randomly download from the net,
so I haven't explored trying to use kdevops, at least not until I set
up a sandboxed VM. :-)
Including the Debian package names that should be installed would also
be helpful in kdevops/doc/requirements.md. That's not a problem for
the experienced Debian developer, but one of my personal goals for
kvm-xfstests and gce-xfstests is to allow a random graduate student
who has presented some research file system like Betrfs at the Usenix
FAST conference to be able to easily run fstests. And it sounds like
you have similar goals of "enabling the average user to also easily
run tests".
> but I never found one
> as easy to use as compiling the kernel and running a few make commands.
I've actually done a lot of work to optimize developer velocity using
my test framework. So for example:
kvm-xfstests install-kconfig # set up a kernel Kconfig suitable for kvm-xfstests and gce-xfstests
make
kvm-xfstests smoke # boot the test appliance VM, using the kernel that was just built
And a user can test a particular stable kernel using a single command
line (this will checkout a particular kernel, and build it on a build
VM, and then launch tests in parallel on a dozen or so VM's):
gce-xfstests ltm -c ext4/all -g auto --repo stable.git --commit v5.15.28
... or if we want to bisect a particular test failure, we might do
something like this:
gce-xfstests ltm -c ext4 generic/361 --bisect-good v5.15 --bisect-bad v5.16
... or I can establish a watcher that will automatically build a git
tree when a branch on a git tree changes:
gce-xfstests ltm -c ext4/4k -g auto --repo next.git --watch master
Granted, this only works on GCE --- but feel free to take these ideas
and integrate them into kdevops if you feel inspired to do so. :-)
> There is the concept of results too and a possible way to share things..
> but this is getting a bit off topic and I don't want to bore people more.
This would be worth chatting about, perhaps at LSF/MM. xfstests
already supports junit results files; we could convert it to TAP
format, but junit has more functionality, so perhaps the right
approach is to have tools that can support both TAP and junit? What
about some way to establish interchange of test artifacts? i.e.,
saving the kernel logs, and the generic/NNN.full and
generic/NNN.out.bad files?
I have a large library of these test results and test artifacts, and
perhaps others would find it useful if we had a way sharing test
results between developers, especially we have multiple test
infrastructures that might be running ext4, f2fs, and xfs tests?
Cheers,
- Ted
