Re: Updates to disk unmount test case.

[Kamil Paral <kparal@xxxxxxxxxx>]

On Tue, Nov 26, 2019 at 9:16 PM Adam Williamson <adamwill@xxxxxxxxxxxxxxxxx> wrote:

On Tue, 2019-11-26 at 13:59 -0500, pmkellly@xxxxxxxxxxxx wrote:
>
> I assure you that I wasn't being lazy when I created the web draft test
> case. It wasn't copied from e'mail, but from my original LibriWriter
> file where I composed it. The formatting with the results right under
> the test step was a demonstration of a format that has some advantages
> over the existing format. Just a little demo; I will change it.

I think it can be a good format indeed, and I've seen it used in other
test contexts, but all our other significant test cases use the format
intended by the template where all the test steps are together and all
the expected results are together after the test steps. So it'd
probably be best to stay consistent with that.

You can use little writing tricks to help people map each test step to
the correct expected result, like you can write in a test step "run
command X and then check the output" (not just "run command X"), and
then in the results you write something like "The output of command X
should be..." I find that's usually enough to make it clear for
testers.

And I'm going to heartily disagree with Adam here. If I hate something about certain our test cases, it's when you have to constantly jump forward and back and look for expected results somewhere else on the page. This makes the test case very hard to follow and very easy to unintentionally omit important actions. Sometimes it makes you repeat the test case just because you read an important instruction too late. I wanted to rewrite all of those bad test cases a long time ago, but never forced myself to do it.

The reasonable approach is to write steps one by one in the "How to test" section, and any time there is an expectation of something immediate, mention it at the very same step. It is not necessary to include obvious things. For example, this is not necessary:

1. Log in through virtual terminal. Login should be accepted.

because broken login would obviously prevent you from continuing your test case steps. So this is enough:

1. Log in through virtual terminal.

But in some cases you want to say something should happen right now that is not obvious and should be mentioned:

1. Create a disk layout without a swap partition. A warning dialog should appear, but it should not prevent you from confirming the layout.

or

1. Execute command "foobar". The command should successfully finish (with return code 0) and its output should contain "all ok".

The "Expected results" section should contain expectations that are relevant to the final state of the system (although these could be also written together with the last step in "How to test"), or to a condition that should be valid throughout the whole testing process. For example:

1. The final installed system must boot successfully and present a login screen.

or

1. Throughout the whole installation process, all mentions of the system version or the edition on all screens must identify the current system correctly (e.g. version is always shown as "31" and edition is always shown as "Server").

_______________________________________________
test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/test@xxxxxxxxxxxxxxxxxxxxxxx