On Wed, 2016-07-13 at 08:21 +0530, Siddhesh Poyarekar wrote: > On Tue, Jul 12, 2016 at 03:45:54PM -0700, Adam Williamson wrote: > > > > Bodhi works at the source package level, not binary package level. > > I think Jon's point was with respect to the scope of testing. With > glibc (or libstdc++ that Jon would be concerned with), an ideal set of > sanity tests would cover the library as well as its development files*. > It will be a lot of work for each tester though. We had talked about > using tunir or taskatron to automate some of it last year, but I don't > think there has been any progress on that front. > > Siddhesh > > * Well with glibc it would be library, devel, locales, networking and > boot, to begin to name a few things. It would not be 'a lot of work', it would be a gigantic, totally unsustainable burden. I honestly think you're shooting *way* too high here. Even with all the recent volunteers, we have like a couple dozen people who volunteer to spend *some* time testing updates. There is no way we are going to be running rigorous full functionality tests on every update, and honestly, it's not even reasonable to *expect* that, because it should be done far earlier in the process than 'distribution package updates'. What we can realistically do for distribution package updates is basic functionality tests and (if feasible) bug verification. We have, what, ~10k packages in the distro. We have at least a few hundred just on the *critical path*. Some of those are things like, oh say, *entire desktop environments*, which you'd be looking at a week of solid full time work to test *fully*. We have database stacks, whole things like fedmsg that you need a solid couple of days' reading to set up a dev environment of, stuff like FreeIPA which RH has entire teams dedicated to testing... we are not going to be doing comprehensive testing of all of this stuff, either manually or automated, at the distribution level. It's not feasible and it's not even *sensible*, because it'd essentially be a huge project of doing work in the wrong place. Comprehensive functionality testing of glibc does not belong in a distribution, it belongs in the glibc project. What makes sense to test at the distribution level is 'does this glibc, which we trust upstream to assert is at least vaguely functional in and of itself, have any unfortunate interactions with the rest of Fedora?', which boils down to, get a few people to install it and run with it for a day or two and see if anything looks wonky. That's the level of testing we realistically have the resources for - either doing manually, or writing automated tests for - at the distribution level. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://lists.fedoraproject.org/admin/lists/devel@xxxxxxxxxxxxxxxxxxxxxxx
