Re: [PATCH v10] drm: Add initial ci/ subdirectory

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Helen,

Great to see this coming to the DRM!

Just wondering, any chance we could add a stage to perform tests on
VKMS? The main way of validating VKMS is through IGT tests, so I feel
it would be a perfect match to have VKMS as a stage on the CI. As a
generic KMS driver, VKMS is also great to validate changes on DRM core.

Another question, could we add V3D to the default arm and arm64 config?

Best Regards,
- Maíra

On 7/20/23 12:27, Helen Koike wrote:
From: Tomeu Vizoso <tomeu.vizoso@xxxxxxxxxxxxx>

Developers can easily execute several tests on different devices
by just pushing their branch to their fork in a repository hosted
on gitlab.freedesktop.org which has an infrastructure to run jobs
in several runners and farms with different devices.

There are also other automated tools that uprev dependencies,
monitor the infra, and so on that are already used by the Mesa
project, and we can reuse them too.

Also, store expectations about what the DRM drivers are supposed
to pass in the IGT test suite. By storing the test expectations
along with the code, we can make sure both stay in sync with each
other so we can know when a code change breaks those expectations.

Also, include a configuration file that points to the out-of-tree
CI scripts.

This will allow all contributors to drm to reuse the infrastructure
already in gitlab.freedesktop.org to test the driver on several
generations of the hardware.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@xxxxxxxxxxxxx>
Signed-off-by: Helen Koike <helen.koike@xxxxxxxxxxxxx>

---

Hello,

I'm re-spining this patch sent originally by Tomeu.

This is meant to be an auxiliary tool where developers and
maintainers can just submit their code to fdo and see if
tests passes, than they can decide if it is worthy merging
it or not.

This tool has proven its value on the Mesa community
and it can bring a lot of value here too.

Please review and let me know your thoughts.

You can also see this patch on
https://gitlab.freedesktop.org/helen.fornazier/linux/-/tree/drm-ci-tests

Thanks!

v2:
   - Fix names of result expectation files to match SoC
   - Don't execute tests that are going to skip on all boards

v3:
   - Remove tracking of dmesg output during test execution

v4:
   - Move up to drivers/gpu/drm
   - Add support for a bunch of other drivers
   - Explain how to incorporate fixes for CI from a
     ${TARGET_BRANCH}-external-fixes branch
   - Remove tests that pass from expected results file, to reduce the
     size of in-tree files
   - Add docs about how to deal with outages in automated testing labs
   - Specify the exact SHA of the CI scripts to be used

v5:
   - Remove unneeded skips from Meson expectations file
   - Use a more advanced runner that detects flakes automatically
   - Use a more succint format for the expectations
   - Run many more tests (and use sharding to finish in time)
   - Use skip lists to avoid hanging machines
   - Add some build testing
   - Build IGT in each pipeline for faster uprevs
   - List failures in the GitLab UI

v6:
   - Rebase on top of latest drm-next
   - Lower priority of LAVA jobs to not impact Mesa CI as much
   - Update docs

v7:
   - Rebase on top of latest drm-next

v8:
   - Move all files specific to testing the kernel into the kernel tree
     (thus I have dropped the r-bs I had collected so far)
   - Uprev Gitlab CI infrastructure scripts to the latest from Mesa
   - Add MAINTAINERS entry
   - Fix boot on MT8173 by adding some Kconfigs that are now needed
   - Link to the docs from index.rst and hard-wrap the file

v9:
   - Only automatically run the pipelines for merge requests
   - Switch to zstd for the build artifacts to align with Mesa
   - Add Qcom USB PHYs to config as they are now =m in the defconfig

v10:
   - Include ci yml files from mesa/mesa (where the development is
     current active) instead of a spin off project.
   - Uprev Gitlab CI infrastructure scripts to the latest from Mesa
   - Update MAINTAINERS entry
   - Uprev igt tool
   - add LAVA_JOB_PRIORITY: 30
   - pipeline example:
   https://gitlab.freedesktop.org/helen.fornazier/linux/-/pipelines/940506
---
  Documentation/gpu/automated_testing.rst       |  144 +
  Documentation/gpu/index.rst                   |    1 +
  MAINTAINERS                                   |    8 +
  drivers/gpu/drm/ci/arm.config                 |   69 +
  drivers/gpu/drm/ci/arm64.config               |  199 ++
  drivers/gpu/drm/ci/build-igt.sh               |   35 +
  drivers/gpu/drm/ci/build.sh                   |  157 +
  drivers/gpu/drm/ci/build.yml                  |  110 +
  drivers/gpu/drm/ci/check-patch.py             |   57 +
  drivers/gpu/drm/ci/container.yml              |   61 +
  drivers/gpu/drm/ci/gitlab-ci.yml              |  252 ++
  drivers/gpu/drm/ci/igt_runner.sh              |   77 +
  drivers/gpu/drm/ci/image-tags.yml             |   15 +
  drivers/gpu/drm/ci/lava-submit.sh             |   57 +
  drivers/gpu/drm/ci/static-checks.yml          |   12 +
  drivers/gpu/drm/ci/test.yml                   |  335 ++
  drivers/gpu/drm/ci/testlist.txt               | 2912 +++++++++++++++++
  drivers/gpu/drm/ci/x86_64.config              |  111 +
  .../gpu/drm/ci/xfails/amdgpu-stoney-fails.txt |   22 +
  .../drm/ci/xfails/amdgpu-stoney-flakes.txt    |   19 +
  .../gpu/drm/ci/xfails/amdgpu-stoney-skips.txt |    2 +
  drivers/gpu/drm/ci/xfails/i915-amly-fails.txt |   17 +
  .../gpu/drm/ci/xfails/i915-amly-flakes.txt    |   32 +
  drivers/gpu/drm/ci/xfails/i915-amly-skips.txt |    4 +
  drivers/gpu/drm/ci/xfails/i915-apl-fails.txt  |   57 +
  drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt |    1 +
  drivers/gpu/drm/ci/xfails/i915-apl-skips.txt  |    4 +
  drivers/gpu/drm/ci/xfails/i915-cml-fails.txt  |   18 +
  drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt |   37 +
  drivers/gpu/drm/ci/xfails/i915-cml-skips.txt  |    2 +
  drivers/gpu/drm/ci/xfails/i915-glk-fails.txt  |   18 +
  drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt |   41 +
  drivers/gpu/drm/ci/xfails/i915-glk-skips.txt  |    5 +
  drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt  |   26 +
  drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt |   25 +
  drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt  |    5 +
  drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt  |   37 +
  drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt |    5 +
  drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt  |   11 +
  drivers/gpu/drm/ci/xfails/i915-whl-fails.txt  |   47 +
  drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt |    1 +
  drivers/gpu/drm/ci/xfails/i915-whl-skips.txt  |    2 +
  .../drm/ci/xfails/mediatek-mt8173-fails.txt   |   29 +
  .../drm/ci/xfails/mediatek-mt8173-flakes.txt  |    0
  .../drm/ci/xfails/mediatek-mt8183-fails.txt   |   10 +
  .../drm/ci/xfails/mediatek-mt8183-flakes.txt  |   14 +
  .../gpu/drm/ci/xfails/meson-g12b-fails.txt    |   12 +
  .../gpu/drm/ci/xfails/meson-g12b-flakes.txt   |    4 +
  .../gpu/drm/ci/xfails/msm-apq8016-fails.txt   |   15 +
  .../gpu/drm/ci/xfails/msm-apq8016-flakes.txt  |    4 +
  .../gpu/drm/ci/xfails/msm-apq8096-fails.txt   |    2 +
  .../gpu/drm/ci/xfails/msm-apq8096-flakes.txt  |    4 +
  .../gpu/drm/ci/xfails/msm-apq8096-skips.txt   |    2 +
  .../gpu/drm/ci/xfails/msm-sc7180-fails.txt    |   25 +
  .../gpu/drm/ci/xfails/msm-sc7180-flakes.txt   |    7 +
  .../gpu/drm/ci/xfails/msm-sc7180-skips.txt    |   23 +
  .../gpu/drm/ci/xfails/msm-sdm845-fails.txt    |   68 +
  .../gpu/drm/ci/xfails/msm-sdm845-flakes.txt   |   11 +
  .../gpu/drm/ci/xfails/msm-sdm845-skips.txt    |    2 +
  .../drm/ci/xfails/rockchip-rk3288-fails.txt   |   49 +
  .../drm/ci/xfails/rockchip-rk3288-flakes.txt  |    8 +
  .../drm/ci/xfails/rockchip-rk3288-skips.txt   |   52 +
  .../drm/ci/xfails/rockchip-rk3399-fails.txt   |   39 +
  .../drm/ci/xfails/rockchip-rk3399-flakes.txt  |   23 +
  .../drm/ci/xfails/rockchip-rk3399-skips.txt   |    5 +
  .../drm/ci/xfails/virtio_gpu-none-fails.txt   |   38 +
  .../drm/ci/xfails/virtio_gpu-none-flakes.txt  |    0
  .../drm/ci/xfails/virtio_gpu-none-skips.txt   |    6 +
  test                                          |    0
  69 files changed, 5502 insertions(+)
  create mode 100644 Documentation/gpu/automated_testing.rst
  create mode 100644 drivers/gpu/drm/ci/arm.config
  create mode 100644 drivers/gpu/drm/ci/arm64.config
  create mode 100644 drivers/gpu/drm/ci/build-igt.sh
  create mode 100644 drivers/gpu/drm/ci/build.sh
  create mode 100644 drivers/gpu/drm/ci/build.yml
  create mode 100755 drivers/gpu/drm/ci/check-patch.py
  create mode 100644 drivers/gpu/drm/ci/container.yml
  create mode 100644 drivers/gpu/drm/ci/gitlab-ci.yml
  create mode 100755 drivers/gpu/drm/ci/igt_runner.sh
  create mode 100644 drivers/gpu/drm/ci/image-tags.yml
  create mode 100755 drivers/gpu/drm/ci/lava-submit.sh
  create mode 100644 drivers/gpu/drm/ci/static-checks.yml
  create mode 100644 drivers/gpu/drm/ci/test.yml
  create mode 100644 drivers/gpu/drm/ci/testlist.txt
  create mode 100644 drivers/gpu/drm/ci/x86_64.config
  create mode 100644 drivers/gpu/drm/ci/xfails/amdgpu-stoney-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/amdgpu-stoney-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/amdgpu-stoney-skips.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-amly-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-amly-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-amly-skips.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-apl-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-apl-skips.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-cml-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-cml-skips.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-glk-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-glk-skips.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-whl-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/i915-whl-skips.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8173-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8173-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8183-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8183-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/meson-g12b-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/meson-g12b-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8096-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8096-skips.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/msm-sc7180-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/msm-sc7180-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/msm-sc7180-skips.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/msm-sdm845-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/msm-sdm845-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/msm-sdm845-skips.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3288-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3288-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3399-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3399-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3399-skips.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/virtio_gpu-none-fails.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/virtio_gpu-none-flakes.txt
  create mode 100644 drivers/gpu/drm/ci/xfails/virtio_gpu-none-skips.txt
  create mode 100644 test




[Index of Archives]     [Kernel Newbies]     [Security]     [Netfilter]     [Bugtraq]     [Linux FS]     [Yosemite Forum]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Samba]     [Video 4 Linux]     [Device Mapper]     [Linux Resources]

  Powered by Linux