Wiki - https://fedoraproject.org/wiki/Changes/DeprecateSTI Discussion Thread - https://discussion.fedoraproject.org/t/f42-change-proposal-deprecation-of-sti-tests-self-contained/141622 This is a proposed Change for Fedora Linux. This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee. == Summary == Display a deprecation warning for Fedora 41 for all STI tests. Deprecate execution of STI tests in all CI pipelines for Fedora 42. * CI for bodhi updates * CI for dist-git pull requests All users of STI tests will need to [https://tmt.readthedocs.io/en/stable/questions.html#how-do-i-migrate-sti-tests-to-tmt migrate to the new tmt format]. '''The change will affect 281 components'''. The list of components affected can be found via [https://sourcegraph.com/search?q=context:global+repo:src.fedoraproject.org+file:tests/tests*%5C.yml%24+&patternType=keyword&sm=0 this sourcegraph query]. All references to STI will be removed from the Fedora CI documentation. == Owner == * Name: [[User:mvadkert| Miroslav Vadkerti]], [[User:lecris| Cristian Le]], * Email: mvadkert@xxxxxxxxxx, fedora@xxxxxxxxx == Detailed Description == For some time CI testing in Fedora can be defined using two different formats: * Standard Test Interface (STI) * Test Management Tool (tmt) As tmt has matured to a state it can fully replace the STI functionality, we are proposing to drop STI as the supported format. STI format is not actively developed and is causing us a maintenance burden. More and more STI test failures are starting to appear as the Fedora packages evolve out of sync with the test scripts. Tmt provides various advantages over STI that would make it easier to manage in the long term: * Better organization of `tests` and test environments (referred to as `plans` in tmt) * Local reproducible environment thanks to testing-farm reproducer script, allowing to thinker locally without pushing to dist-git * Tests can be defined in the local dist-git repo, inside the srpm archive, [https://src.fedoraproject.org/projects/tests/%2A `tests/*`] dist-git namespace, or in the upstream repo, allowing it to be reused more freely between packages and with upstream directly * Tmt tests are integrated with packit allowing it to be executed with upstream and better migrate to the new dist-git environment Hopefully this list can help inspire some better ways of reorganizing the tests during the migration. == Feedback == == Benefit to Fedora == Having two formats for executing the tests is an unnecessary duplication and causes confusion for the Fedora maintainers and community. STI tests have limited functionality and are harder to develop and maintain, when compared to tmt. == Scope == * Proposal owners: [[User:mvadkert| Miroslav Vadkerti]], [[User:lecris| Cristian Le]] * Other developers: * Release engineering: [https://pagure.io/releng/issues #Releng issue number] * Policies and guidelines: N/A (not needed for this Change) * Trademark approval: N/A (not needed for this Change) * Alignment with the Fedora Strategy: == Upgrade/compatibility impact == This is a change to the development experience, no changes to Fedora distribution are made. == How To Migrate == To find out if you package has STI tests, check if `tests/tests*.yml` files are present in your dist-git repository. Follow the [https://tmt.readthedocs.io/en/stable/questions.html#how-do-i-migrate-sti-tests-to-tmt sti to tmt migration guide]. For the majority of tests involve just executing the already provided script you only need to: 1. Initialize `tmt` structure using `tmt init` 2. Create a minimal plan <pre> # /plan.fmf # Boilerplate to indicate tmt to run any tests files it finds in the repo discover: how: fmf execute: how: tmt # Prepare test environment, e.g. install packages # Note: the current packages in the spec file will already be installed by default prepare: how: install packages: - foo - bar </pre> 3. Migrate individual tests <pre> # /tests/foo.fmf test: ./foo.sh </pre> For more complex needs feel free to discuss in the [https://matrix.to/#/#fedora-ci:fedoraproject.org `#fedora-ci`] or [https://matrix.to/#/#tmt:fedoraproject.org `#tmt`] matrix channels. You can also explore [https://sourcegraph.com/search?q=context:global+repo:src.fedoraproject.org/rpms+file:%5C.fmf/version%24+&patternType=keyword&sm=0 other project's tmt tests] (identified by having a `.fmf` folder) using `tmt tests show` and `tmt plans show` == User Experience == For Fedora 41 users will be presented with a deprecation warning in the Testing Farm artifacts. For Fedora 42 users who have setup gating on STI tests will be obligated to migrate to be able to push their packages to stable repositories. == Dependencies == N/A == Contingency Plan == * Contingency mechanism: N/A (not a System Wide Change) * Contingency deadline: N/A (not a System Wide Change) * Blocks release? No (not a System Wide Change) We will keep the STI support until all references to STI tests are gone in case all users have not migrated to tmt until Fedora 42. See the description for the list of the affected components. == Documentation == * [https://tmt.readthedocs.io/en/stable/questions.html#how-do-i-migrate-sti-tests-to-tmt How to migrate STI tests to tmt] * [https://src.fedoraproject.org/tests/python/tree/main Example test repo implemented in both STI and tmt format] (see for example the `smoke/venv.fmf` file and `smoke_default` test in `tests.yml` file) * [https://tmt.readthedocs.io/en/stable/spec.html Tmt specs] (reference for the available keys defining `plans` or `tests`) * Example project with shared [https://github.com/containers/aardvark-dns upstream]+[https://src.fedoraproject.org/rpms/aardvark-dns downstream] tests == Release Notes == -- Aoife Moloney Fedora Operations Architect Fedora Project Matrix: @amoloney:fedora.im IRC: amoloney -- _______________________________________________ devel-announce mailing list -- devel-announce@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-announce-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue -- _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
