On Tue, 2017-11-14 at 05:04 +0000, Nick Coghlan wrote: > > Dear Sumantro, > > > > Did you talk to DNF QE team? I suggest you coordinate your work with Karel Srot from DNF > > QE team. There is a tool to test DNF and I know Karel already has a good set of tests > > which covers Modular side of DNF [1], just missing some user stories. > > We're also trying to keep track of the places where automated testing > happens as part of the Fedora module developer's guide at https://doc > s.pagure.org/modularity/development/building-modules/testing.html > > I think the key item we have there that hasn't been mentioned in the > thread yet is your own automated tests at https://github.com/fedora-m > odularity/compose-tests > > Cheers, > Nick. > > P.S. It's entirely plausible that we're missing some aspects of the > automated testing setup for the modular server builds, so issue > reports and PRs to correct omissions at https://pagure.io/modularity > are most welcome. Note that there is a specific purpose to these test cases: they are for release validation testing. Release validation testing is about ensuring critical functionality works *as experienced by end users*. That is, for release validation testing, we want to test that the most common and critical interactions with modules that will be performed by admins of actual Fedora Server installations work, exactly as those admins would perform them. These tests need to run from clean installs of 'real' composes, and involve using the tools as real-world users will use them. So any tests that involve more artificial environments or interactions are not as useful for release validation purposes. Tests that test "too much" are also not as useful, because we don't immediately know whether a failure is something critical to the release process. So, we can certainly look at existing tests / tools and see if this testing can share resources with them, but it needs to take the above considerations into account. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
