Re: the sad state of installability tests

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Mon, Jul 15, 2024 at 10:39:37AM +0200, Cristian Le via devel wrote:
> Hi Zbyszek,
> 
> On 2024/07/14 20:04, Zbigniew Jędrzejewski-Szmek wrote:
> > I'm looking for a solution which doesn't just skip the installability
> > tests altogether.
> 
> On PRs with zuul or FedoraCI automation, the same instability tests that are
> done for Bodhi are performed. But what would help is to make these tests as
> required to pass unless they are manually waved. Manually that can be done
> by setting `gating.yaml`. There was some discussion on making some of these
> tests as gating by default.
> 
> Another issue specific to installability is that the issue is often further
> down the stream, particularly with the SELinux test. Definitely these need
> to be tracked down and fixed.

I fully agree. But a test that just does 'dnf install rpms-from-update/*.rpm'
will predicatably fail.

> > A second problem is that when the tests fail, it's just soooo hard to
> > find out why they failed.From the bodhi status page, one has to
> > go to the Jenkins status page, guess that it's useful to look at
> > Console Output, scroll over a few pages of incomrehensible JSON
> > gibberish, guess that it's worth clicking on Testing Farm Artifacts URL,
> > click that, scroll pages of output to see
> > "guest setup failed: Test environment installation failed: Install packages".
> 
> Weird, when the test is finished, you should have only the final
> testing-farm results page. Here's an example [1]. Maybe in your case it
> encountered an internal failure?
> 
> [1]: https://bodhi.fedoraproject.org/updates/FEDORA-2024-57f489c90d

Maybe I'm doing things wrong. I'd be happy to learn.

I do the following:
1. Look at the bodhi update page (https://bodhi.fedoraproject.org/updates/FEDORA-2024-3aafcac6a8)
2. Click on 'Automated Tests'
   (There seems to be no URL for the view. This means that it's always
    and extra click after every page load or reload.)
3. I click on one of the pinkish lines, e.g. the first one.
  (Another usability problem here is that the click open a new page in
  new tab/window. Why, oh why? I want to use left-click to open a link
  in the existing tab, and middle-click to open a new tab. The current
  UI breaks navigation.)
4. I switch to the new tab and see
   https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/398487/

   (BTW, I mentioned unrelated scary-looking warnings in my OP.
   Here:
     The following steps that have been detected may have insecure
     interpolation of sensitive variables (click here for an explanation):

     httpRequest: [TESTING_FARM_API_KEY]
   )
5. Click on 'Console Output'
6. Click on 'Testing Farm Artifacts URL: https://artifacts.dev.testing-farm.io/25316385-50d8-42b4-b4c1-3eff059034eb'
7. Click on 'build installation' (https://artifacts.dev.testing-farm.io/25316385-50d8-42b4-b4c1-3eff059034eb/guest-setup-09b7edc6-0b7e-431b-ae68-afac2527fbb1/artifact-installation-09b7edc6-0b7e-431b-ae68-afac2527fbb1)
8. Click on 60-Install-packages.txt

So 8 steps to get to the actual result…

Zbyszek

-- 
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux