On Sat, 2020-01-25 at 22:06 +0000, Bill Chatfield via devel wrote: > You need automated tests. You need a suit of manual tests. For every package. So to chime in on the QA aspects of this: the above is not realistically possible, and it's also an explicit non-goal of all QA and CI efforts. We have something like 10,000 source packages in Fedora. On a very optimistic count we might have 100 people who might plausibly be capable of doing something to do with the above (this is an over- estimate, but I'm just doing this as an illustration). There's no way each of those 100 people is going to contribute and maintain a sufficient test suite for 100 source packages. In practice the way we approach QA for Fedora is to try and define a subset of all possible QA activities which is actually manageable. So far as releases go, we define release criteria: https://fedoraproject.org/wiki/Fedora_Release_Criteria these are informed by other things, like the Edition PRDs and tech specs, e.g. https://fedoraproject.org/wiki/Workstation/Workstation_PRD and https://fedoraproject.org/wiki/Server/Product_Requirements_Documen t . We then *do* define test plans that cover the release criteria; we're at something like 95% coverage right now. Counting automation is a bit complex as there are fuzzy areas like what level of coverage across arches, editions and images we need for what test cases, but by my last count approximately 78% of those test cases are automated. If you read through the release criteria, you'll note that 'Eclipse must work' is not one of them. That's a choice the project has made, and that's why Eclipse not working didn't block the Fedora 31 release. (We require all applications *included in a default install of one of the release-blocking desktops* to work; Eclipse is not included in a default install of any release-blocking desktop). Of course these decisions can always be changed, but given the info Emmanuel and Fabio provided about the state of Java package maintenance, it doesn't seem likely that we'd be able to start blocking on Java packages at present. Beyond the release criteria and release validation process, we have a process and framework for testing updates to released packages. This is mostly an as-best-as-we-can arrangement for now. There is a mechanism by which manual test cases can be 'associated' with a given source package (via wiki categories), and when a package has test cases associated with it, they will be shown in the Bodhi - that's the update system, https://bodhi.fedoraproject.org/ - page for the update, so testers will be aware that those tests are available (and can provide specific feedback on them). We also have a couple of automated test systems that run on updates and/or package builds: the CI pipeline and openQA. The CI pipeline can run some basic checks and also per-package automated tests which are defined in a git repository for the package, if any have been written (more on this at https://docs.fedoraproject.org/en-US/ci/ ). openQA runs a subset of the automated release validation tests (which are distribution-level integration tests) on certain updates (all critical path updates, and updates containing some other whitelisted packages) - more on this at https://fedoraproject.org/wiki/OpenQA . The results from both systems are shown on the 'Automated Tests' tab in Bodhi, and there is an optional mechanism by which updates can be gated on the automated test results if the packager decides this makes sense. As I first explained, though, it's not realistically possible for us to provide comprehensive manual and/or automated tests for every package. That's just too much work. The mechanisms we have in place are, I think, pretty good and getting better; we are trying to extend test coverage as far as we can as well, but it is never going to be close to 100% of all packages in Fedora, and as a project we have to focus our efforts on the most critical packages and areas. I'm sorry you had a bad experience with Eclipse, it's definitely not what we want people to run into. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
