On Mon, Sep 12, 2022 at 12:29 AM Tomeu Vizoso
<tomeu.vizoso@xxxxxxxxxxxxx> wrote:
>
> And use it to store expectations about what the DRM drivers are
> supposed to pass in the IGT test suite.
>
> Also include a configuration file that points to the out-of-tree CI
> scripts.
>
> By storing the test expectations along the code we can make sure both
> stay in sync with each other, and so we can know when a code change
> breaks those expectations.
>
> This will allow all contributors to drm to reuse the infrastructure
> already in gitlab.freedesktop.org to test the driver on several
> generations of the hardware.
>
> v2:
> - Fix names of result expectation files to match SoC
> - Don't execute tests that are going to skip on all boards
>
> v3:
> - Remove tracking of dmesg output during test execution
>
> v4:
> - Move up to drivers/gpu/drm
> - Add support for a bunch of other drivers
> - Explain how to incorporate fixes for CI from a
> ${TARGET_BRANCH}-external-fixes branch
> - Remove tests that pass from expected results file, to reduce the
> size of in-tree files
> - Add docs about how to deal with outages in automated testing labs
> - Specify the exact SHA of the CI scripts to be used
>
> v5:
> - Remove unneeded skips from Meson expectations file
> - Use a more advanced runner that detects flakes automatically
> - Use a more succint format for the expectations
> - Run many more tests (and use sharding to finish in time)
> - Use skip lists to avoid hanging machines
> - Add some build testing
> - Build IGT in each pipeline for faster uprevs
> - List failures in the GitLab UI
>
> v6:
> - Rebase on top of latest drm-next
> - Lower priority of LAVA jobs to not impact Mesa CI as much
> - Update docs
>
> v7:
> - Rebase on top of latest drm-next
>
> v8:
> - Move all files specific to testing the kernel into the kernel tree
> (thus I have dropped the r-bs I had collected so far)
> - Uprev Gitlab CI infrastructure scripts to the latest from Mesa
> - Add MAINTAINERS entry
> - Fix boot on MT8173 by adding some Kconfigs that are now needed
> - Link to the docs from index.rst and hard-wrap the file
>
> v9:
> - Only automatically run the pipelines for merge requests
> - Switch to zstd for the build artifacts to align with Mesa
> - Add Qcom USB PHYs to config as they are now =m in the defconfig
>
> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@xxxxxxxxxxxxx>
Reviewed-by: Rob Clark <robdclark@xxxxxxxxx>
> ---
> Documentation/gpu/automated_testing.rst | 144 +
> Documentation/gpu/index.rst | 1 +
> MAINTAINERS | 8 +
> drivers/gpu/drm/ci/arm.config | 57 +
> drivers/gpu/drm/ci/arm64.config | 179 ++
> drivers/gpu/drm/ci/build-igt.sh | 43 +
> drivers/gpu/drm/ci/build.sh | 158 +
> drivers/gpu/drm/ci/build.yml | 110 +
> drivers/gpu/drm/ci/check-patch.py | 57 +
> drivers/gpu/drm/ci/container.yml | 54 +
> drivers/gpu/drm/ci/gitlab-ci.yml | 225 ++
> drivers/gpu/drm/ci/igt_runner.sh | 77 +
> drivers/gpu/drm/ci/image-tags.yml | 13 +
> drivers/gpu/drm/ci/lava-submit.sh | 53 +
> drivers/gpu/drm/ci/static-checks.yml | 12 +
> drivers/gpu/drm/ci/test.yml | 322 ++
> drivers/gpu/drm/ci/testlist.txt | 2763 +++++++++++++++++
> drivers/gpu/drm/ci/x86_64.config | 105 +
> .../gpu/drm/ci/xfails/amdgpu-stoney-fails.txt | 19 +
> .../drm/ci/xfails/amdgpu-stoney-flakes.txt | 15 +
> .../gpu/drm/ci/xfails/amdgpu-stoney-skips.txt | 2 +
> .../gpu/drm/ci/xfails/i915-amly-flakes.txt | 32 +
> drivers/gpu/drm/ci/xfails/i915-amly-skips.txt | 2 +
> drivers/gpu/drm/ci/xfails/i915-apl-fails.txt | 29 +
> drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt | 1 +
> drivers/gpu/drm/ci/xfails/i915-apl-skips.txt | 2 +
> drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt | 37 +
> drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt | 40 +
> drivers/gpu/drm/ci/xfails/i915-glk-skips.txt | 2 +
> drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt | 8 +
> drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt | 25 +
> drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt | 2 +
> drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt | 19 +
> drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt | 5 +
> drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt | 8 +
> drivers/gpu/drm/ci/xfails/i915-whl-fails.txt | 30 +
> drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt | 1 +
> .../drm/ci/xfails/mediatek-mt8173-fails.txt | 29 +
> .../drm/ci/xfails/mediatek-mt8183-fails.txt | 10 +
> .../drm/ci/xfails/mediatek-mt8183-flakes.txt | 14 +
> .../gpu/drm/ci/xfails/meson-g12b-fails.txt | 5 +
> .../gpu/drm/ci/xfails/meson-g12b-flakes.txt | 4 +
> .../gpu/drm/ci/xfails/msm-apq8016-fails.txt | 15 +
> .../gpu/drm/ci/xfails/msm-apq8016-flakes.txt | 4 +
> .../gpu/drm/ci/xfails/msm-apq8096-fails.txt | 2 +
> .../gpu/drm/ci/xfails/msm-apq8096-flakes.txt | 4 +
> .../gpu/drm/ci/xfails/msm-apq8096-skips.txt | 2 +
> .../gpu/drm/ci/xfails/msm-sc7180-fails.txt | 21 +
> .../gpu/drm/ci/xfails/msm-sc7180-flakes.txt | 6 +
> .../gpu/drm/ci/xfails/msm-sc7180-skips.txt | 23 +
> .../gpu/drm/ci/xfails/msm-sdm845-fails.txt | 43 +
> .../gpu/drm/ci/xfails/msm-sdm845-flakes.txt | 11 +
> .../gpu/drm/ci/xfails/msm-sdm845-skips.txt | 2 +
> .../drm/ci/xfails/rockchip-rk3288-fails.txt | 43 +
> .../drm/ci/xfails/rockchip-rk3288-flakes.txt | 2 +
> .../drm/ci/xfails/rockchip-rk3288-skips.txt | 49 +
> .../drm/ci/xfails/rockchip-rk3399-fails.txt | 29 +
> .../drm/ci/xfails/rockchip-rk3399-flakes.txt | 20 +
> .../drm/ci/xfails/rockchip-rk3399-skips.txt | 5 +
> .../drm/ci/xfails/virtio_gpu-none-fails.txt | 38 +
> .../drm/ci/xfails/virtio_gpu-none-flakes.txt | 0
> .../drm/ci/xfails/virtio_gpu-none-skips.txt | 6 +
> 62 files changed, 5047 insertions(+)
> create mode 100644 Documentation/gpu/automated_testing.rst
> create mode 100644 drivers/gpu/drm/ci/arm.config
> create mode 100644 drivers/gpu/drm/ci/arm64.config
> create mode 100644 drivers/gpu/drm/ci/build-igt.sh
> create mode 100644 drivers/gpu/drm/ci/build.sh
> create mode 100644 drivers/gpu/drm/ci/build.yml
> create mode 100755 drivers/gpu/drm/ci/check-patch.py
> create mode 100644 drivers/gpu/drm/ci/container.yml
> create mode 100644 drivers/gpu/drm/ci/gitlab-ci.yml
> create mode 100755 drivers/gpu/drm/ci/igt_runner.sh
> create mode 100644 drivers/gpu/drm/ci/image-tags.yml
> create mode 100755 drivers/gpu/drm/ci/lava-submit.sh
> create mode 100644 drivers/gpu/drm/ci/static-checks.yml
> create mode 100644 drivers/gpu/drm/ci/test.yml
> create mode 100644 drivers/gpu/drm/ci/testlist.txt
> create mode 100644 drivers/gpu/drm/ci/x86_64.config
> create mode 100644 drivers/gpu/drm/ci/xfails/amdgpu-stoney-fails.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/amdgpu-stoney-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/amdgpu-stoney-skips.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/i915-amly-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/i915-amly-skips.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/i915-apl-fails.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/i915-apl-skips.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/i915-glk-skips.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/i915-whl-fails.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8173-fails.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8183-fails.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8183-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/meson-g12b-fails.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/meson-g12b-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8096-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8096-skips.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/msm-sc7180-fails.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/msm-sc7180-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/msm-sc7180-skips.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/msm-sdm845-fails.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/msm-sdm845-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/msm-sdm845-skips.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3288-fails.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3288-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3399-fails.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3399-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3399-skips.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/virtio_gpu-none-fails.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/virtio_gpu-none-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/xfails/virtio_gpu-none-skips.txt
>
[snip]
