Hello!
Thanks for that!
I have an orangepi 5b (not supported in mainline kernel yet) and an x1e laptop (ditto), but i do have an orangepi 5 plus on the way and should be here monday. So i should be able to do some testing on this.
On Wed, Sep 18, 2024, 11:06 a.m. Kamil Paral <kparal@xxxxxxxxxx> wrote:
Hello,
KDE on aarch64 became a blocking deliverable this cycle, at KDE team's
request [1]. The QA team sadly doesn't have the capacity to test it
regularly, and so it was agreed that it would be the KDE team's
responsibility. I talked to Neal Gompa today and he told me that the
QA processes are hard to discover and it's one of the reasons why no
test results were submitted to our release validation matrices, even
for F41 Beta. In this email, I'd like to provide guidance on how to
navigate QA matrices and hopefully make the process clearer and easier
for you to contribute :-)
I'll first showcase the test matrices on F41 Beta, and later I'll show
how to discover the "current" test matrix and figure out what needs
testing.
== How to provide test results ==
Specifically for KDE on aarch64, the most relevant sections of our
matrices are these:
1. https://fedoraproject.org/wiki/Test_Results:Fedora_41_Beta_1.2_Installation#AArch64_disk_images
2. https://fedoraproject.org/wiki/Test_Results:Fedora_41_Beta_1.2_Desktop#Release-blocking_desktops:_aarch64
3. https://fedoraproject.org/wiki/Test_Results:Fedora_41_Beta_1.2_Base#Release-blocking_environments_(aarch64)
You can see the KDE column (or a row) in all of these tables. The
Milestone column shows when exactly this feature is supposed to be
fully functional (when there are multiple milestones, usually you can
see links inside the test case to multiple release criteria related to
multiple milestones), the Test Case column leads you to test steps
description. (Note: the "coconut" results are provided by our test
automation framework).
For a Beta/Final release candidate images (RCs), all the included test
cases (not marked as Optional in the Milestone column) are supposed to
be tested and their results filled out. For interim nightly images,
we're supposed to perform some reasonable coverage in between, e.g.
test all Final test cases at least once before a Final RC actually
arrives, to identify bugs as early as possible. It's also a good idea
to re-test stuff after a big system update (e.g. a new KDE release,
etc).
At the beginning of each page, there are links to download composed
images, and a section describing how to edit the wiki page and provide
test results.
== How to know which test matrix to fill out ==
There are several ways on how to learn which compose to test and where
to fill out the results:
1. We announce all "important composes" (all RCs and some nightly
composes which we decided to nominate for collaborative testing) in
our test-announce mailing list. Search for "candidate" and "nightly
compose" in email subjects:
https://lists.fedoraproject.org/archives/list/test-announce@xxxxxxxxxxxxxxxxxxxxxxx/
2. All old (non-current) wiki pages have a link at the very top
leading to the latest test matrix page. See e.g. the Beta pages linked
above.
3. There's a handy category containing all Fedora 41 test matrices here:
https://fedoraproject.org/wiki/Category:Fedora_41_Test_Results
At the top, you can see links which always lead to the latest test
matrices (the Summary one displays all of them as a single page).
Below that, you can see subcategories for Nightly/Beta/Final (when
available) test matrices, which show all pages created during this
cycle.
== Which test cases to focus on? ==
For RCs, it's simple, everything should be tested and filled out (as
described above). As for nominated composes, we can't test them all
every time, and so there's an obvious question "What should I focus
on? When was this test case tested last, so that I can retest the
oldest results?". We have an ugly but functional tool for that called
testcase_stats:
https://openqa.fedoraproject.org/testcase_stats/
Here's an example, the F41 Desktop testing stats:
https://openqa.fedoraproject.org/testcase_stats/41/Desktop.html
See the explanatory key at the top.
However, it currently only shows stats per each testcase (row in a
matrix), but not also combined per each environment (column, e.g.
Workstation vs KDE). For this specific use case (looking just at KDE
results), it's not very handy. You can always click on Details for a
particular test case and see if KDE was tested or not and when, but it
gets time-consuming quickly. Here's an example of "Release-blocking
desktops: aarch64" -> "QA:Testcase_desktop_panel_basic" detailed stats
(scroll down):
https://openqa.fedoraproject.org/testcase_stats/41/Desktop/QA_Testcase_desktop_panel_basic_Release_blocking_desktops___lt_b_gt_aarch64_lt__b_gt_.html
We'll see if we can improve the UI to make it more useful to teams
with some narrow and not global test focus.
== Ask us ==
If there's anything unclear, we'll be happy to explain it better,
either here in this kde mailing list, or our test list [2], or on our
#quality:fedoraproject.org Matrix channel, please join it!
Also, if you have time to not just test KDE on aarch64, but also help
with KDE on x86_64 testing (see the matrices, they are clearly labeled
with x86_64), that would be very appreciated!
Cheers,
Kamil
Fedora QA
[1] https://pagure.io/fesco/issue/3232
[2] https://lists.fedoraproject.org/archives/list/test@xxxxxxxxxxxxxxxxxxxxxxx/
--
_______________________________________________
kde mailing list -- kde@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to kde-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/kde@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- _______________________________________________ kde mailing list -- kde@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to kde-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kde@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
