Hi Helen, Great to see this coming to the DRM! Just wondering, any chance we could add a stage to perform tests on VKMS? The main way of validating VKMS is through IGT tests, so I feel it would be a perfect match to have VKMS as a stage on the CI. As a generic KMS driver, VKMS is also great to validate changes on DRM core. Another question, could we add V3D to the default arm and arm64 config? Best Regards, - Maíra On 7/20/23 12:27, Helen Koike wrote:
From: Tomeu Vizoso <tomeu.vizoso@xxxxxxxxxxxxx> Developers can easily execute several tests on different devices by just pushing their branch to their fork in a repository hosted on gitlab.freedesktop.org which has an infrastructure to run jobs in several runners and farms with different devices. There are also other automated tools that uprev dependencies, monitor the infra, and so on that are already used by the Mesa project, and we can reuse them too. Also, store expectations about what the DRM drivers are supposed to pass in the IGT test suite. By storing the test expectations along with the code, we can make sure both stay in sync with each other so we can know when a code change breaks those expectations. Also, include a configuration file that points to the out-of-tree CI scripts. This will allow all contributors to drm to reuse the infrastructure already in gitlab.freedesktop.org to test the driver on several generations of the hardware. Signed-off-by: Tomeu Vizoso <tomeu.vizoso@xxxxxxxxxxxxx> Signed-off-by: Helen Koike <helen.koike@xxxxxxxxxxxxx> --- Hello, I'm re-spining this patch sent originally by Tomeu. This is meant to be an auxiliary tool where developers and maintainers can just submit their code to fdo and see if tests passes, than they can decide if it is worthy merging it or not. This tool has proven its value on the Mesa community and it can bring a lot of value here too. Please review and let me know your thoughts. You can also see this patch on https://gitlab.freedesktop.org/helen.fornazier/linux/-/tree/drm-ci-tests Thanks! v2: - Fix names of result expectation files to match SoC - Don't execute tests that are going to skip on all boards v3: - Remove tracking of dmesg output during test execution v4: - Move up to drivers/gpu/drm - Add support for a bunch of other drivers - Explain how to incorporate fixes for CI from a ${TARGET_BRANCH}-external-fixes branch - Remove tests that pass from expected results file, to reduce the size of in-tree files - Add docs about how to deal with outages in automated testing labs - Specify the exact SHA of the CI scripts to be used v5: - Remove unneeded skips from Meson expectations file - Use a more advanced runner that detects flakes automatically - Use a more succint format for the expectations - Run many more tests (and use sharding to finish in time) - Use skip lists to avoid hanging machines - Add some build testing - Build IGT in each pipeline for faster uprevs - List failures in the GitLab UI v6: - Rebase on top of latest drm-next - Lower priority of LAVA jobs to not impact Mesa CI as much - Update docs v7: - Rebase on top of latest drm-next v8: - Move all files specific to testing the kernel into the kernel tree (thus I have dropped the r-bs I had collected so far) - Uprev Gitlab CI infrastructure scripts to the latest from Mesa - Add MAINTAINERS entry - Fix boot on MT8173 by adding some Kconfigs that are now needed - Link to the docs from index.rst and hard-wrap the file v9: - Only automatically run the pipelines for merge requests - Switch to zstd for the build artifacts to align with Mesa - Add Qcom USB PHYs to config as they are now =m in the defconfig v10: - Include ci yml files from mesa/mesa (where the development is current active) instead of a spin off project. - Uprev Gitlab CI infrastructure scripts to the latest from Mesa - Update MAINTAINERS entry - Uprev igt tool - add LAVA_JOB_PRIORITY: 30 - pipeline example: https://gitlab.freedesktop.org/helen.fornazier/linux/-/pipelines/940506 --- Documentation/gpu/automated_testing.rst | 144 + Documentation/gpu/index.rst | 1 + MAINTAINERS | 8 + drivers/gpu/drm/ci/arm.config | 69 + drivers/gpu/drm/ci/arm64.config | 199 ++ drivers/gpu/drm/ci/build-igt.sh | 35 + drivers/gpu/drm/ci/build.sh | 157 + drivers/gpu/drm/ci/build.yml | 110 + drivers/gpu/drm/ci/check-patch.py | 57 + drivers/gpu/drm/ci/container.yml | 61 + drivers/gpu/drm/ci/gitlab-ci.yml | 252 ++ drivers/gpu/drm/ci/igt_runner.sh | 77 + drivers/gpu/drm/ci/image-tags.yml | 15 + drivers/gpu/drm/ci/lava-submit.sh | 57 + drivers/gpu/drm/ci/static-checks.yml | 12 + drivers/gpu/drm/ci/test.yml | 335 ++ drivers/gpu/drm/ci/testlist.txt | 2912 +++++++++++++++++ drivers/gpu/drm/ci/x86_64.config | 111 + .../gpu/drm/ci/xfails/amdgpu-stoney-fails.txt | 22 + .../drm/ci/xfails/amdgpu-stoney-flakes.txt | 19 + .../gpu/drm/ci/xfails/amdgpu-stoney-skips.txt | 2 + drivers/gpu/drm/ci/xfails/i915-amly-fails.txt | 17 + .../gpu/drm/ci/xfails/i915-amly-flakes.txt | 32 + drivers/gpu/drm/ci/xfails/i915-amly-skips.txt | 4 + drivers/gpu/drm/ci/xfails/i915-apl-fails.txt | 57 + drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt | 1 + drivers/gpu/drm/ci/xfails/i915-apl-skips.txt | 4 + drivers/gpu/drm/ci/xfails/i915-cml-fails.txt | 18 + drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt | 37 + drivers/gpu/drm/ci/xfails/i915-cml-skips.txt | 2 + drivers/gpu/drm/ci/xfails/i915-glk-fails.txt | 18 + drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt | 41 + drivers/gpu/drm/ci/xfails/i915-glk-skips.txt | 5 + drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt | 26 + drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt | 25 + drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt | 5 + drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt | 37 + drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt | 5 + drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt | 11 + drivers/gpu/drm/ci/xfails/i915-whl-fails.txt | 47 + drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt | 1 + drivers/gpu/drm/ci/xfails/i915-whl-skips.txt | 2 + .../drm/ci/xfails/mediatek-mt8173-fails.txt | 29 + .../drm/ci/xfails/mediatek-mt8173-flakes.txt | 0 .../drm/ci/xfails/mediatek-mt8183-fails.txt | 10 + .../drm/ci/xfails/mediatek-mt8183-flakes.txt | 14 + .../gpu/drm/ci/xfails/meson-g12b-fails.txt | 12 + .../gpu/drm/ci/xfails/meson-g12b-flakes.txt | 4 + .../gpu/drm/ci/xfails/msm-apq8016-fails.txt | 15 + .../gpu/drm/ci/xfails/msm-apq8016-flakes.txt | 4 + .../gpu/drm/ci/xfails/msm-apq8096-fails.txt | 2 + .../gpu/drm/ci/xfails/msm-apq8096-flakes.txt | 4 + .../gpu/drm/ci/xfails/msm-apq8096-skips.txt | 2 + .../gpu/drm/ci/xfails/msm-sc7180-fails.txt | 25 + .../gpu/drm/ci/xfails/msm-sc7180-flakes.txt | 7 + .../gpu/drm/ci/xfails/msm-sc7180-skips.txt | 23 + .../gpu/drm/ci/xfails/msm-sdm845-fails.txt | 68 + .../gpu/drm/ci/xfails/msm-sdm845-flakes.txt | 11 + .../gpu/drm/ci/xfails/msm-sdm845-skips.txt | 2 + .../drm/ci/xfails/rockchip-rk3288-fails.txt | 49 + .../drm/ci/xfails/rockchip-rk3288-flakes.txt | 8 + .../drm/ci/xfails/rockchip-rk3288-skips.txt | 52 + .../drm/ci/xfails/rockchip-rk3399-fails.txt | 39 + .../drm/ci/xfails/rockchip-rk3399-flakes.txt | 23 + .../drm/ci/xfails/rockchip-rk3399-skips.txt | 5 + .../drm/ci/xfails/virtio_gpu-none-fails.txt | 38 + .../drm/ci/xfails/virtio_gpu-none-flakes.txt | 0 .../drm/ci/xfails/virtio_gpu-none-skips.txt | 6 + test | 0 69 files changed, 5502 insertions(+) create mode 100644 Documentation/gpu/automated_testing.rst create mode 100644 drivers/gpu/drm/ci/arm.config create mode 100644 drivers/gpu/drm/ci/arm64.config create mode 100644 drivers/gpu/drm/ci/build-igt.sh create mode 100644 drivers/gpu/drm/ci/build.sh create mode 100644 drivers/gpu/drm/ci/build.yml create mode 100755 drivers/gpu/drm/ci/check-patch.py create mode 100644 drivers/gpu/drm/ci/container.yml create mode 100644 drivers/gpu/drm/ci/gitlab-ci.yml create mode 100755 drivers/gpu/drm/ci/igt_runner.sh create mode 100644 drivers/gpu/drm/ci/image-tags.yml create mode 100755 drivers/gpu/drm/ci/lava-submit.sh create mode 100644 drivers/gpu/drm/ci/static-checks.yml create mode 100644 drivers/gpu/drm/ci/test.yml create mode 100644 drivers/gpu/drm/ci/testlist.txt create mode 100644 drivers/gpu/drm/ci/x86_64.config create mode 100644 drivers/gpu/drm/ci/xfails/amdgpu-stoney-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/amdgpu-stoney-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/amdgpu-stoney-skips.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-amly-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-amly-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-amly-skips.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-apl-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-apl-skips.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-cml-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-cml-skips.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-glk-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-glk-skips.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-whl-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/i915-whl-skips.txt create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8173-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8173-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8183-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8183-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/meson-g12b-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/meson-g12b-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8096-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8096-skips.txt create mode 100644 drivers/gpu/drm/ci/xfails/msm-sc7180-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/msm-sc7180-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/msm-sc7180-skips.txt create mode 100644 drivers/gpu/drm/ci/xfails/msm-sdm845-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/msm-sdm845-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/msm-sdm845-skips.txt create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3288-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3288-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3399-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3399-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3399-skips.txt create mode 100644 drivers/gpu/drm/ci/xfails/virtio_gpu-none-fails.txt create mode 100644 drivers/gpu/drm/ci/xfails/virtio_gpu-none-flakes.txt create mode 100644 drivers/gpu/drm/ci/xfails/virtio_gpu-none-skips.txt create mode 100644 test
