What is the long term plan for package automated testing? https://fedoraproject.org/wiki/Category:Test_Cases E.g. 18264 packages * 20 tests run continuously by humans is a lot work. The first question I expect many package maintainers to raise is that its a much better investment to maintain automated tests instead. As a user, I only trust that tests were run if I can see execution logs. I agree that upstream + package maintainers should be supplying per-package tests. and IMHO the source package is the most logical place to store and generate them from. SRPMs should be providing either: - include a standardized test section similar to DEB autopkgtest ("in-build" tests) - OR building produces a "-tests" package which puts a bunch of simple returncode test executables under somewhere standard. Example: /usr/lib/testing/<source pkg name>/bin/{test_foo,test_abc} and group them in plaintext lists /usr/lib/testing/<source pkg name>/testgroups/{quick,build,qa_intensive,smoke} etc. (Executing the tests could emit coverage information to some known location. (gcov, pytest etc)) The seperate "-tests" RPM has a few advantages over in-build tests, including: - Tests the final product: the built binary RPM installed on a working system (not the in-build environment) - Different test groups for different phases of package release process - Users can easily run test X seperately for troubleshooting - Makes the tests easily usable during new build contexts (rpm ostree, container build testing etc) - System-wide integration tests could simply be stored in their own package. -- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://lists.fedoraproject.org/admin/lists/test@xxxxxxxxxxxxxxxxxxxxxxx
