On 2/4/20 06:20, Kamil Paral wrote:
> On Mon, Feb 3, 2020 at 3:48 PM pmkellly@xxxxxxxxxxxx <pmkellly@xxxxxxxxxxxx>
> wrote:
>
>>
>> This proposal is for changes to Base test case: "QA:Testcase package
>> install remove" The existing test case is here:
>>
>> https://fedoraproject.org/wiki/QA:Testcase_package_install_remove
>>
>> Though there is a requirement to install packages, there are no commands
>> provided to accomplish this.
>
>
> The process doesn't need to involve command. The purpose of the test case
> is to test any package manager, including graphical ones. So for
> Workstation, we test it with gnome-software. For KDE, we test it with both
> graphical managers they have in there (because KDE). On Server, we test it
> with dnf.
>
There are certainly advantages of being non-specific. You get several
package managers tested and likely with several ways of using the
package manager. I'm sure someone would install their favorite package
manager then run the test.
There has to be some misunderstanding here. The purpose is not to install your favorite package manager. This is a release validation test case that is present in the Base matrix and tested against major editions/spins. See here:
So for Workstation you use the default one, for KDE the default one(s), for Server the default one, ...
>> Also, there are no instructions concerning
>> things to look for during the installations.
>>
>> If we specified the packages to install, we could test an RPM package
>> and a Modular package. I don't think dnf can handle Flatpacks.
>>
>
> Modularity has a separate list of test cases, see e.g. here:
> https://fedoraproject.org/wiki/Test_Results:Fedora_32_Rawhide_20200131.n.0_Base#Modularity
>
Modularity is part of all of the Fedora composes; is it not? I'm curious
why these aren't in the Base cases.
I'm confused. The link I sent you above lists those test cases and they *are* part of the Base matrix, as you can see.
Though from what I've seen, there
aren't a lot of modules available yet. Is it just because modularity is
sort of just getting started. I admit modularity had been off my radar
until the Eclipse problem.
> We have no test cases for flatpaks, at the moment.
>
Another one that's just getting started. Though I do have two flatpacks
as part of my standard "as deployed" configuration.
>> This test case uses the command line: (rpm -q package1 package2) This is
>> done twice; once to remove the packages and once to verify the removal.
>>
>> I might be good to change the first one to: (dnf remove). Though we
>> could use this command again to verify the removal, it might be more
>> revealing to use (dnf info). The DNF info will show, among other things,
>> "Installed" or "Available".
>>
>
> If we want to test dnf, we need to verify the operation with something
> else, not dnf itself. Rpm is the lowest tool we can use for verification.
>
Your point is well made. However, since we are letting the testers use
their package manager of choice, I would suggest we remove the rpm
commands and just instruct them to remove the packages they installed
and verify they have been removed.
I wonder. Have you perhaps found the test case through Bodhi, because (I just noticed) the test case contains these categories?
Package PackageKit test casesPackage gnome-software test casesPackage appstream-data test casesPackage libappstream-glib test casesPackage librepo test casesPackage libhif test casesPackage hawkey test casesPackage dnf test casesPackage yum test casesPackage yumex test cases
In that case it the test case shows up in Bodhi for PackageKit/gnome-software/dnf/yum/libhif/hawkey/etc. And it might confuse somebody as to which package manager to use. We can try to clarify that. The obvious answer is the one that you see in Bodhi (packagekit, gnome-software, dnf, etc), but in the case of some of those libraries, it might not be obvious.
There is one last point I was thinking of. Though I have never dug into
a package manager. I imagine they have some basic test to see that they
have installed a package. I'm not sure this is good enough to say that
the package actually got installed so that it would work. Perhaps it
would be worthwhile to specify one of the two packages (something
simple) so the test could also start the application just to see that it
would start.
As for removing packages, That's problematic. I know a lot depends on
the actual package being removed, but there is always trash left behind.
Sometimes other things get broken. As in one of the blocker bugs
yesterday. I'm thinking that the only practical test for package removal
is that the system still boots and isn't otherwise broken.
I think that verifying the package manager operation (installation/removal) using `rpm` is completely satisfactory, honestly. Whether the app starts is not really relevant. And whether the system can still boot seems out of scope as well. Of course anything can happen when you run code as root, but the purpose is to check the basics. If the machine bursts into flames, we will notice very quickly and don't need to explicitly check for it in the test case.
A more systematic approach to installation/removal checking would be to check whether the package maintainer only removed the packages you explicitly requested and nothing else. But that quickly gets complex when the user selects packages some other packages depend on, or when the package manager performs some form of "no longer needed packages" checking. This is so complex that it's best automated and not intended as a manual exercise.
_______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@xxxxxxxxxxxxxxxxxxxxxxx
