Re: [PATCH] drm/doc: ci: Require more context for flaky tests

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 





On 19/10/2023 13:51, Helen Koike wrote:


On 19/10/2023 06:46, Maxime Ripard wrote:
Flaky tests can be very difficult to reproduce after the facts, which
will make it even harder to ever fix.

Let's document the metadata we agreed on to provide more context to
anyone trying to address these fixes.

Link: https://lore.kernel.org/dri-devel/CAPj87rPbJ1V1-R7WMTHkDat2A4nwSd61Df9mdGH2PR=ZzxaU=Q@xxxxxxxxxxxxxx/
Signed-off-by: Maxime Ripard <mripard@xxxxxxxxxx>
---
  Documentation/gpu/automated_testing.rst | 13 +++++++++++++
  1 file changed, 13 insertions(+)

diff --git a/Documentation/gpu/automated_testing.rst b/Documentation/gpu/automated_testing.rst
index 469b6fb65c30..2dd0e221c2c3 100644
--- a/Documentation/gpu/automated_testing.rst
+++ b/Documentation/gpu/automated_testing.rst
@@ -67,6 +67,19 @@ Lists the tests that for a given driver on a specific hardware revision are   known to behave unreliably. These tests won't cause a job to fail regardless of
  the result. They will still be run.
+Each new flake entry must be associated with a link to a bug report to

What do you mean by but report? Just a link to an email to the mailing list is enough?

Also, I had made a mistake to the first flakes lists, which I corrected with https://www.spinics.net/lists/kernel/msg4959629.html (there was a bug in my script which ended up erroneous adding a bunch of tests in the flake list, so I cleaned them up), I would like to kind request to let me add those documentation in a future patch to not block that patch series.

Thanks
Helen


+the author of the affected driver, the board name or Device Tree name of
+the board, the first kernel version affected, and an approximation of
+the failure rate.
+
+They should be provided under the following format::
+
+  # Bug Report: $LORE_OR_PATCHWORK_URL

I wonder if the commit adding the test into the flakes.txt file with and Acked-by from the device maintainer shouldn't be already considered the Bug Report.

+  # Board Name: broken-board.dtb

Maybe Board Name isn't required, since it is already in the name of the file.

+  # Version: 6.6-rc1
+  # Failure Rate: 100

Maybe also:

# Pipeline url: https://gitlab.freedesktop.org/helen.fornazier/linux/-/pipelines/1014435

All this info will complicated a bit the update-xfails.py script, but well, we can handle... (see https://patchwork.kernel.org/project/dri-devel/patch/20231020034124.136295-4-helen.koike@xxxxxxxxxxxxx/ )
We need to update that script to make life easier.
Vignesh sent a patch adding at least the pipeline url to the file
https://patchwork.kernel.org/project/linux-arm-msm/patch/20231019070650.61159-9-vignesh.raman@xxxxxxxxxxxxx/
but to meet this doc that needs to be updated too.

Regards,
Helen

+  flaky-test
+
  drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-skips.txt
  -----------------------------------------------------------



[Index of Archives]     [Linux DRI Users]     [Linux Intel Graphics]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [XFree86]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Linux Kernel]     [Linux SCSI]     [XFree86]
  Powered by Linux