improving upgrade testcases templates

[Kamil Paral <kparal@xxxxxxxxxx>]

Hello,

I wanted to tweak a gnome-software upgrade test case [1] lately, and realized that... I can't. I got lost in the number of templates it's composed from (try it yourself). Even if you know wiki templates basics, it was too difficult to even find the page containing the source text that I saw rendered. And I was afraid that I'll break other pages elsewhere. So I decided I try to simplify that.

The current situation is that everything gets composed in Testcase upgrade [2]. It's very complex and uses a lot of parameters. It forwards these parameters to other templates, and includes them. It's a very clever system (kudos to Adam), but unfortunately it's very hard to read and edit.

So, I did the following:

1. I replaced anonymous numbered parameters with named ones. So {{{flavor}}} instead of {{{1}}}.

2. I split dnf steps from gnome-software steps. They are sufficiently different, merging them together just makes it more complex.

3. I created short separate templates for each test case section (description, setup, how to test, expected results). Separating it into chunks makes it more readable and easier to navigate.

4. I introduced "setup" section in testcases, because it allowed to separate generic steps from specific steps.

5. I moved some of the expected results into the "how to test" section. That allowed me to make "expected results" section generic, and as a bonus, the "how to test" section is now much more pleasant to read, because you don't need to constantly jump to expected results and search whether you missed something. Things that you should check directly after a particular step is now written down at that very step. Only generic things to check not related to any particular step are in "expected results" (that makes much more sense, imho).

6. I simplified the conditions a bit. The instructions don't need to be perfectly parametrized, that just makes them hard to edit. So sometimes the instructions include sentences like "If your disks are encrypted, do X." even when you're displaying a test case for a non-encrypted upgrade. We're humans, we can deal with an "If" clause well (we're using this approach everywhere else), and this makes a lot of stuff much simpler.

7. I removed some old workarounds (F24 ones, should no longer apply).

(I also improved some instructions quite a bit, but that's just a content change, not really a templating change).

I made all the adjustments on the staging wiki (I hope it's not getting purged over night). To compare the changes, click on the testcases from the production wiki:
https://fedoraproject.org/wiki/Template:Installation_test_matrix#Upgrade

and from the staging wiki:
https://stg.fedoraproject.org/wiki/Template:Installation_test_matrix#Upgrade

See how the templates are used.  The basic structure is so short and simple that I'll quote it here:

{{QA/Test_Case
|description={{upgrade_description|from=current|flavor=workstation|pkgset=workstation|live=1|tool=gnome-software}}
|setup={{upgrade_setup|from=current|flavor=workstation|pkgset=workstation|live=1}}
|actions={{gnome_software_upgrade_procedure}}
|results={{upgrade_results}}
}}

I think it's easy to see which template to look at if you want to adjust a particular section. Also it's quite obvious which parameters are passed to those templates and what effect it has.

All those test cases are composed from these templates:
https://stg.fedoraproject.org/wiki/Template:Upgrade_description
https://stg.fedoraproject.org/wiki/Template:Upgrade_setup
https://stg.fedoraproject.org/wiki/Template:Dnf_upgrade_procedure
https://stg.fedoraproject.org/wiki/Template:Gnome_software_upgrade_procedure
https://stg.fedoraproject.org/wiki/Template:Upgrade_results

Only the first one is somewhat complex (but short!). The rest is pretty simple, the last 3 are completely unparametrized.

With this setup, I believe we still hit the goal of avoid testcase content duplication (all the real content exists in a single place only, a change affects all testcases), and it's much easier for folks to adjust them now. The basic QA/Test_Case structure needs to be copied to all existing testcases, but it should be very stable and I don't think we need to fear desynchronization (I'll watch all those pages for changes anyway).

Thoughts? Objections to putting it to production?


[1] https://fedoraproject.org/wiki/QA:Testcase_upgrade_gnome-software_current_workstation

[2] https://fedoraproject.org/w/index.php?title=Template:Testcase_upgrade&action="">

_______________________________________________
test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx