On Wed, 2016-07-13 at 11:41 +0200, Vít Ondruch wrote: > > Dne 12.7.2016 v 18:49 Adam Williamson napsal(a): > > > > > > The idea is this: there could be a requirement for all packages to > > provide at least *some* kind of 'how to test' information. > > If the package should be tested by human, I'd expect that there will be > some additional value, i.e. the human can search for the "how to test" > information by him self on the internet, the human has some interest and > knowledge about the package, try something I was not thinking about when > writing the "how to test" script etc. Following some "how to test" > guidelines makes not much sense to me, since I do these tests by myself. People filing karma don't necessarily have 'interest and knowledge about the package'; fedora-easy-karma lists *every* package you currently have installed from updates-testing, which may include ones that were installed by default which the tester doesn't normally use, dependencies and so on. Not all packagers actually test their packages before submitting them. Most packagers probably at least run the code somehow, but often they use their development checkout rather than the actual package they produced, for instance, or they only verify the fix they're sending out rather than re-testing the entire thing. This is trivially demonstrable by the fact that we *do* still sometimes get completely broken updates showing up. Even if you *do* test your package properly, one thing Bodhi testing provides is broader testing, in several ways. Humans almost always find ways to do things differently; even if you provide very specific testing instructions you'll probably find there's *some* variance in exactly how people carry them out, and a tester may well find an issue you didn't just because they happen to perform the test a slightly different way. There's also testing environments. You probably test your package on your system, in the desktop environment you usually run. But what if it breaks on a different arch? Or you run GNOME, and your update doesn't work in KDE for some reason? Or it breaks when some config file setting is set the opposite way to the way you have it set? And so on and so on - the possibilities are endless. But Bodhi testing can at least help to get things checked in more configurations than just yours. > If I should provide "how to test" information, it should be probably > script which should be run by AutoQA or something. But TBH, I have no > idea how to provide such script. There is precisely zero information > about this stuff on places I would expect (this one [1] as an example). > There is also zero support in tools such as fedpkg and dist-git. If you're willing to do this, that's great, and we would definitely like to enable it. I know the Taskotron (it hasn't been AutoQA for a while) team is working on enabling generic dist-git integration (which is basically what you're asking for), but I don't know the current status of that...CCing tflink and kparal who may have more information. > I am pretty sure this would be more valuable then some "how to test" > document. I think it varies substantially by package, but I don't think we're realistically going to get sufficient automated testing for all packages, including ones that are important to users. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://lists.fedoraproject.org/admin/lists/devel@xxxxxxxxxxxxxxxxxxxxxxx
