A quick status update regarding creating test scripts for regression testing on kernelci. I'm also CC-ing Dmitry to give an update on the work fixing syzkaller bugs. On 11/6/18 9:37 AM, Hans Verkuil wrote: > Hi all, > > After the media summit (heavy on test discussions) and the V4L2 event regression > we just found it is clear we need to do a better job with testing. > > All the pieces are in place, so what is needed is to combine it and create a > script that anyone of us as core developers can run to check for regressions. > The same script can be run as part of the kernelci regression testing. > > We have four virtual drivers: vivid, vim2m, vimc and vicodec. The last one > is IMHO not quite good enough yet for testing: it is not fully compliant to the > upcoming stateful codec spec. Work for that is planned as part of an Outreachy > project. This Outreachy project started last week, so hopefully we'll have something that is usable by January. > > My idea is to create a script that is maintained as part of v4l-utils that > loads the drivers and runs v4l2-compliance and possibly other tests against > the virtual drivers. > > It should be simple to use and require very little in the way of dependencies. > Ideally no dependencies other than what is in v4l-utils so it can easily be run > on an embedded system as well. I have a simple script: modprobe vivid no_error_inj=1 n_devs=3 multiplanar=1,2,1 modprobe vim2m modprobe vimc v4l2-ctl -d0 -i3 -v width=3840,height=2160,pixelformat=XR24 v4l2-ctl -d1 -o1 -x width=3840,height=2160,pixelformat=XR24 v4l2-ctl -d2 -i3 -v width=3840,height=2160,pixelformat=XR24 v4l2-ctl -d3 -o1 -x width=3840,height=2160,pixelformat=XR24 v4l2-ctl -d4 -i3 -v width=3840,height=2160,pixelformat=XR24 v4l2-compliance -m0 -e4 -s10 v4l2-compliance -m1 -e4 -s10 v4l2-compliance -m3 -e4 -s10 v4l2-compliance -m4 -e4 -s10 But there are too many failures at the moment. Most are fixed, but several first need to be merged into the 4.20 mainline kernel before they can be backported to our master branch. Once they are backported, then there is still one outstanding vivid/vimc failure, fixed in this series: https://www.mail-archive.com/linux-media@xxxxxxxxxxxxxxx/msg136900.html That won't make 4.21, but hopefully it will make 4.22. This is probably the main blocker for the kernelci regression test. vimc fails at several places, but most are known API issues that we really need to nail down at some point. Finding time is the issue, but in the meantime I can probably downgrade the errors to warnings in v4l2-compliance. In addition, work is ongoing to make the topology configurable with configfs, and I think we can postpone regression testing with vimc until that is merged (I hope for 4.22). > For a 64-bit kernel it should run the tests both with 32-bit and 64-bit > applications. Haven't done that yet. > > It should also test with both single and multiplanar modes where available. That's done (see vivid multiplanar module option in the script). > > Since vivid emulates CEC as well, it should run CEC tests too. Not done yet. > > As core developers we should have an environment where we can easily test > our patches with this script (I use a VM for that). > > I think maintaining the script (or perhaps scripts) in v4l-utils is best since > that keeps it in sync with the latest kernel and v4l-utils developments. > > Comments? Ideas? Regarding fixing the open syzkaller issues: most is done, but there is one fix still pending: https://www.mail-archive.com/linux-media@xxxxxxxxxxxxxxx/msg136512.html It's more complex than I'd like, but I'm not sure if it can be improved. I will have to take another look when 4.22 opens. This patch series fixes issues when the filehandle is dupped and different fds can do actions in parallel that would otherwise not happen. Regards, Hans
