On 10/18/2018 09:23 PM, Ezequiel Garcia wrote: > Hi everyone, > > In Collabora, and as part of our kernelci work, we are doing > research on kernel functional testing with kernelci. > > For those new to kernelci, see https://github.com/kernelci/kernelci-doc/wiki/KernelCI > and https://kernelci.org/. > > The goal is to lay down the infrastructure required to make > automated test coverage an integral part of our feature > and bugfix development process. > > So, as a first attempt, we've decided to extend kernelci test > v4l2 plan support, leading the way to extending > other subsystems' test plans. > > Currently, kernelci looks for a list of branches every hour and > see if anything changed. For any branch that has changed, it triggers > builds, boots, tests and reports for each branch that had some changes > since last time it ran. > > For this pilot, we've decided to target just a few devices: > qemu with vivid, rk3399-gru-kevin and rk3288-veyron-jaq > with uvc. It's running v4l2-compliance, right? Looking at the test cases, they appear in the reverse order that v4l2-compliance performs them, that's a bit odd. And if we include uvc in the testing, then I need to prioritize the work I started for uvc to remove the last FAILs. Regards, Hans > > We'd like to get some early feedback on this, so we are sending > an example of how a kernelci report would look like, to trigger > some discussion around > the direction this should take. > > Thanks! > > === > Test results for: > Tree: gtucker > Branch: kernelci-media > Kernel: gtucker-kernelci-media-002-2-gaa27eb0392c7 > URL: https://gitlab.collabora.com/gtucker/linux.git > Commit: aa27eb0392c70adec713e911a9b5267a1d853624 > Test plans: v4l2 > > Summary > ------- > 3 test groups results > > 1 | v4l2 | rk3399-gru-kevin | arm64 | 49 total: 17 PASS 4 FAIL 28 SKIP > 2 | v4l2 | rk3288-veyron-jaq | arm | 49 total: 17 PASS 4 FAIL 28 SKIP > 3 | v4l2 | qemu | arm64 | 168 total: 102 PASS 0 FAIL 66 SKIP > > > Tests > ----- > > 1 | v4l2 | rk3399-gru-kevin | arm64 | 49 total: 17 PASS 4 FAIL 28 SKIP > > Config: defconfig > Lab Name: lab-collabora-dev > Date: 2018-10-18 18:48:52.426000 > TXT log: http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm64/defconfig/lab-collabora-dev > /v4l2-rk3399-gru-kevin.txt > HTML log: http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm64/defconfig/lab-collabora-dev > /v4l2-rk3399-gru-kevin.html > Rootfs: http://staging-storage.kernelci.org/images/rootfs/debian/stretchv4l2/20181017.1//arm64/rootfs.cpio.gz > Test Git: git://linuxtv.org/v4l-utils.git > Test Commit: e889b0d56757b9a74eb8c4c8cc2ebd5b18e228b2 > > > Test cases: > > * MMAP: FAIL > * blocking-wait: PASS > * read/write: SKIP > * read/write: SKIP > * read/write: SKIP > * VIDIOC_EXPBUF: PASS > * VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: PASS > * VIDIOC_TRY_DECODER_CMD: SKIP > * VIDIOC_G_ENC_INDEX: SKIP > * VIDIOC_TRY_ENCODER_CMD: SKIP > * Scaling: SKIP > * Composing: SKIP > * Cropping: SKIP > * VIDIOC_G_SLICED_VBI_CAP: SKIP > * VIDIOC_S_FMT: PASS > * VIDIOC_TRY_FMT: PASS > * VIDIOC_G_FMT: PASS > * VIDIOC_G_FBUF: SKIP > * VIDIOC_G/S_PARM: FAIL > * VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: PASS > * VIDIOC_G/S_JPEGCOMP: SKIP > * VIDIOC_UNSUBSCRIBE_EVENT/DQEVENT: PASS > * VIDIOC_G/S/TRY_EXT_CTRLS: FAIL > * VIDIOC_G/S_CTRL: PASS > * VIDIOC_QUERYCTRL: PASS > * VIDIOC_QUERY_EXT_CTRL/QUERYMENU: FAIL > * VIDIOC_G/S_EDID: SKIP > * VIDIOC_DV_TIMINGS_CAP: SKIP > * VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: SKIP > * VIDIOC_ENUM/G/S/QUERY_STD: SKIP > * VIDIOC_G/S_AUDOUT: SKIP > * VIDIOC_G/S/ENUMOUTPUT: SKIP > * VIDIOC_ENUMAUDOUT: SKIP > * VIDIOC_G/S_FREQUENCY: SKIP > * VIDIOC_G/S_MODULATOR: SKIP > * VIDIOC_G/S_AUDIO: SKIP > * VIDIOC_G/S/ENUMINPUT: PASS > * VIDIOC_ENUMAUDIO: SKIP > * VIDIOC_S_HW_FREQ_SEEK: SKIP > * VIDIOC_G/S_FREQUENCY: SKIP > * VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: SKIP > * VIDIOC_LOG_STATUS: SKIP > * VIDIOC_DBG_G/S_REGISTER: SKIP > * for-unlimited-opens: PASS > * VIDIOC_G/S_PRIORITY: PASS > * VIDIOC_QUERYCAP: PASS > * second-/dev/video0-open: PASS > * VIDIOC_QUERYCAP: PASS > * MC-information-see-Media-Driver-Info-above: PASS > > > > 2 | v4l2 | rk3288-veyron-jaq | arm | 49 total: 17 PASS 4 FAIL 28 SKIP > > Config: multi_v7_defconfig > Lab Name: lab-collabora-dev > Date: 2018-10-18 17:00:41.724000 > TXT log: http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm/multi_v7_defconfig/lab-collab > ora-dev/v4l2-rk3288-veyron-jaq.txt > HTML log: http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm/multi_v7_defconfig/lab-collab > ora-dev/v4l2-rk3288-veyron-jaq.html > Rootfs: http://staging-storage.kernelci.org/images/rootfs/debian/stretchv4l2/20181017.1//armhf/rootfs.cpio.gz > Test Git: git://linuxtv.org/v4l-utils.git > Test Commit: e889b0d56757b9a74eb8c4c8cc2ebd5b18e228b2 > > > Test cases: > > * MMAP: FAIL > * blocking-wait: PASS > * read/write: SKIP > * read/write: SKIP > * read/write: SKIP > * VIDIOC_EXPBUF: PASS > * VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: PASS > * VIDIOC_TRY_DECODER_CMD: SKIP > * VIDIOC_G_ENC_INDEX: SKIP > * VIDIOC_TRY_ENCODER_CMD: SKIP > * Scaling: SKIP > * Composing: SKIP > * Cropping: SKIP > * VIDIOC_G_SLICED_VBI_CAP: SKIP > * VIDIOC_S_FMT: PASS > * VIDIOC_TRY_FMT: PASS > * VIDIOC_G_FMT: PASS > * VIDIOC_G_FBUF: SKIP > * VIDIOC_G/S_PARM: FAIL > * VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: PASS > * VIDIOC_G/S_JPEGCOMP: SKIP > * VIDIOC_UNSUBSCRIBE_EVENT/DQEVENT: PASS > * VIDIOC_G/S/TRY_EXT_CTRLS: FAIL > * VIDIOC_G/S_CTRL: PASS > * VIDIOC_QUERYCTRL: PASS > * VIDIOC_QUERY_EXT_CTRL/QUERYMENU: FAIL > * VIDIOC_G/S_EDID: SKIP > * VIDIOC_DV_TIMINGS_CAP: SKIP > * VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: SKIP > * VIDIOC_ENUM/G/S/QUERY_STD: SKIP > * VIDIOC_G/S_AUDOUT: SKIP > * VIDIOC_G/S/ENUMOUTPUT: SKIP > * VIDIOC_ENUMAUDOUT: SKIP > * VIDIOC_G/S_FREQUENCY: SKIP > * VIDIOC_G/S_MODULATOR: SKIP > * VIDIOC_G/S_AUDIO: SKIP > * VIDIOC_G/S/ENUMINPUT: PASS > * VIDIOC_ENUMAUDIO: SKIP > * VIDIOC_S_HW_FREQ_SEEK: SKIP > * VIDIOC_G/S_FREQUENCY: SKIP > * VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: SKIP > * VIDIOC_LOG_STATUS: SKIP > * VIDIOC_DBG_G/S_REGISTER: SKIP > * for-unlimited-opens: PASS > * VIDIOC_G/S_PRIORITY: PASS > * VIDIOC_QUERYCAP: PASS > * second-/dev/video0-open: PASS > * VIDIOC_QUERYCAP: PASS > * MC-information-see-Media-Driver-Info-above: PASS > > > > 3 | v4l2 | qemu | arm64 | 168 total: 102 PASS 0 FAIL 66 SKIP > > Config: defconfig+virtualvideo > Lab Name: lab-collabora-dev > Date: 2018-10-18 18:51:19.917000 > TXT log: http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm64/defconfig+virtualvideo/lab- > collabora-dev/v4l2-qemu.txt > HTML log: http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm64/defconfig+virtualvideo/lab- > collabora-dev/v4l2-qemu.html > Rootfs: http://staging-storage.kernelci.org/images/rootfs/debian/stretchv4l2/20181017.1//arm64/rootfs.cpio.gz > Test Git: git://linuxtv.org/v4l-utils.git > Test Commit: e889b0d56757b9a74eb8c4c8cc2ebd5b18e228b2 > > > Test cases: > > * blocking-wait: PASS > * read/write: PASS > * VIDIOC_EXPBUF: PASS > * VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: PASS > * VIDIOC_TRY_DECODER_CMD: SKIP > * VIDIOC_G_ENC_INDEX: SKIP > * VIDIOC_TRY_ENCODER_CMD: SKIP > * Scaling: SKIP > * Composing: SKIP > * Cropping: SKIP > * VIDIOC_G_SLICED_VBI_CAP: SKIP > * VIDIOC_S_FMT: PASS > * VIDIOC_TRY_FMT: PASS > * VIDIOC_G_FMT: PASS > * VIDIOC_G_FBUF: PASS > * VIDIOC_G/S_PARM: PASS > * VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: PASS > * VIDIOC_G/S_JPEGCOMP: SKIP > * VIDIOC_UNSUBSCRIBE_EVENT/DQEVENT: PASS > * VIDIOC_G/S/TRY_EXT_CTRLS: PASS > * VIDIOC_G/S_CTRL: PASS > * VIDIOC_QUERYCTRL: PASS > * VIDIOC_QUERY_EXT_CTRL/QUERYMENU: PASS > * VIDIOC_EXPBUF: PASS > * VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: PASS > * VIDIOC_TRY_DECODER_CMD: SKIP > * VIDIOC_G_ENC_INDEX: SKIP > * VIDIOC_TRY_ENCODER_CMD: SKIP > * Scaling: SKIP > * Composing: SKIP > * Cropping: SKIP > * read/write: PASS > * VIDIOC_EXPBUF: PASS > * VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: PASS > * VIDIOC_TRY_DECODER_CMD: SKIP > * VIDIOC_G_ENC_INDEX: SKIP > * VIDIOC_TRY_ENCODER_CMD: SKIP > * Scaling: SKIP > * Composing: SKIP > * Cropping: SKIP > * VIDIOC_G_SLICED_VBI_CAP: SKIP > * VIDIOC_S_FMT: PASS > * VIDIOC_TRY_FMT: PASS > * VIDIOC_G_FMT: PASS > * VIDIOC_G_FBUF: PASS > * VIDIOC_G/S_PARM: PASS > * VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: PASS > * VIDIOC_G/S_JPEGCOMP: SKIP > * VIDIOC_UNSUBSCRIBE_EVENT/DQEVENT: PASS > * VIDIOC_G/S/TRY_EXT_CTRLS: PASS > * VIDIOC_G/S_CTRL: PASS > * VIDIOC_QUERYCTRL: PASS > * VIDIOC_QUERY_EXT_CTRL/QUERYMENU: PASS > * VIDIOC_EXPBUF: PASS > * VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: PASS > * VIDIOC_TRY_DECODER_CMD: SKIP > * VIDIOC_G_ENC_INDEX: SKIP > * VIDIOC_TRY_ENCODER_CMD: SKIP > * Scaling: SKIP > * Composing: SKIP > * Cropping: SKIP > * read/write: PASS > * VIDIOC_EXPBUF: PASS > * VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: PASS > * VIDIOC_TRY_DECODER_CMD: SKIP > * VIDIOC_G_ENC_INDEX: SKIP > * VIDIOC_TRY_ENCODER_CMD: SKIP > * Scaling: SKIP > * Composing: SKIP > * Cropping: SKIP > * VIDIOC_G_SLICED_VBI_CAP: SKIP > * VIDIOC_S_FMT: PASS > * VIDIOC_TRY_FMT: PASS > * VIDIOC_G_FMT: PASS > * VIDIOC_G_FBUF: PASS > * VIDIOC_G/S_PARM: PASS > * VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: PASS > * VIDIOC_G/S_JPEGCOMP: SKIP > * VIDIOC_UNSUBSCRIBE_EVENT/DQEVENT: PASS > * VIDIOC_G/S/TRY_EXT_CTRLS: PASS > * VIDIOC_G/S_CTRL: PASS > * VIDIOC_QUERYCTRL: PASS > * VIDIOC_QUERY_EXT_CTRL/QUERYMENU: PASS > * VIDIOC_EXPBUF: PASS > * VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: PASS > * VIDIOC_TRY_DECODER_CMD: SKIP > * VIDIOC_G_ENC_INDEX: SKIP > * VIDIOC_TRY_ENCODER_CMD: SKIP > * Scaling: SKIP > * Composing: SKIP > * Cropping: SKIP > * VIDIOC_G_SLICED_VBI_CAP: SKIP > * VIDIOC_S_FMT: PASS > * VIDIOC_TRY_FMT: PASS > * VIDIOC_G_FMT: PASS > * VIDIOC_G_FBUF: PASS > * VIDIOC_G/S_PARM: PASS > * VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: PASS > * VIDIOC_G/S_JPEGCOMP: SKIP > * VIDIOC_UNSUBSCRIBE_EVENT/DQEVENT: PASS > * VIDIOC_G/S/TRY_EXT_CTRLS: PASS > * VIDIOC_G/S_CTRL: PASS > * VIDIOC_QUERYCTRL: PASS > * VIDIOC_QUERY_EXT_CTRL/QUERYMENU: PASS > * VIDIOC_EXPBUF: PASS > * VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: PASS > * VIDIOC_TRY_DECODER_CMD: SKIP > * VIDIOC_G_ENC_INDEX: SKIP > * VIDIOC_TRY_ENCODER_CMD: SKIP > * Scaling: SKIP > * Composing: SKIP > * Cropping: SKIP > * VIDIOC_G_SLICED_VBI_CAP: SKIP > * VIDIOC_S_FMT: PASS > * VIDIOC_TRY_FMT: PASS > * VIDIOC_G_FMT: PASS > * VIDIOC_G_FBUF: PASS > * VIDIOC_G/S_PARM: PASS > * VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: PASS > * VIDIOC_G/S_JPEGCOMP: SKIP > * VIDIOC_UNSUBSCRIBE_EVENT/DQEVENT: PASS > * VIDIOC_G/S/TRY_EXT_CTRLS: PASS > * VIDIOC_G/S_CTRL: PASS > * VIDIOC_QUERYCTRL: PASS > * VIDIOC_QUERY_EXT_CTRL/QUERYMENU: PASS > * VIDIOC_EXPBUF: PASS > * VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: PASS > * VIDIOC_TRY_DECODER_CMD: SKIP > * VIDIOC_G_ENC_INDEX: SKIP > * VIDIOC_TRY_ENCODER_CMD: SKIP > * Scaling: SKIP > * Composing: SKIP > * Cropping: SKIP > * VIDIOC_G_SLICED_VBI_CAP: SKIP > * VIDIOC_S_FMT: PASS > * VIDIOC_TRY_FMT: PASS > * VIDIOC_G_FMT: PASS > * VIDIOC_G_FBUF: PASS > * VIDIOC_G/S_PARM: PASS > * VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: PASS > * VIDIOC_G/S_JPEGCOMP: SKIP > * VIDIOC_UNSUBSCRIBE_EVENT/DQEVENT: PASS > * VIDIOC_G/S/TRY_EXT_CTRLS: PASS > * VIDIOC_G/S_CTRL: PASS > * VIDIOC_QUERYCTRL: PASS > * VIDIOC_QUERY_EXT_CTRL/QUERYMENU: PASS > * VIDIOC_G/S_EDID: PASS > * VIDIOC_DV_TIMINGS_CAP: PASS > * VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: PASS > * VIDIOC_ENUM/G/S/QUERY_STD: PASS > * VIDIOC_G/S_AUDOUT: SKIP > * VIDIOC_G/S/ENUMOUTPUT: SKIP > * VIDIOC_ENUMAUDOUT: SKIP > * VIDIOC_G/S_FREQUENCY: PASS > * VIDIOC_G/S_MODULATOR: SKIP > * VIDIOC_G/S_AUDIO: PASS > * VIDIOC_G/S/ENUMINPUT: PASS > * VIDIOC_ENUMAUDIO: PASS > * VIDIOC_S_HW_FREQ_SEEK: SKIP > * VIDIOC_G/S_FREQUENCY: PASS > * VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: PASS > * VIDIOC_LOG_STATUS: PASS > * VIDIOC_DBG_G/S_REGISTER: SKIP > * for-unlimited-opens: PASS > * VIDIOC_G/S_PRIORITY: PASS > * VIDIOC_QUERYCAP: PASS > * second-/dev/video0-open: PASS > * VIDIOC_QUERYCAP: PASS > >
