On Thu, 22 Nov 2012 12:00:20 +0100 Martin Krizek <mkrizek@xxxxxxxxxx> wrote: > Hi Tim, > > Is this "...default partitioning (no less than 500MB for /boot), > selecting the default package set." really necessary? I'd say that a > more valid use case would be upgrading with any partitioning and any > package set. I mean, who uses just a default package set? Would it > make sense to remove that part? Honestly, it depends on how you look at it. I agree that there needs to be more variation in the system to be upgraded in order to test upgrades more completely. It's a side effect of how we write test cases and the resources we have to do the testing. I think it really comes down to practicality. When you loosen up the test case like that, you have a combinatorial explosion of potential test cases that could cause problems. Since the fedup process uses yum at it's core - most of those bugs would come down to the individual packages which cause problems. If upgradepath was enforced, it would be less of a potential issue but that's not an option at the moment. If we look at real world use cases; using rpmfusion is common, using some arbitrary package set is almost definitely the case; do we test upgrades with all of those possibilities? The problem with our release validation process is where to draw the line between what we test and what we don't. Our test cases are pretty much human bash scripts and that seems to be what people expect. If you follow all the steps of a test case, you can mark it pass/fail and be done with it; no improvisation or filling in unwritten stuff should be required. I'm personally not a fan of this testing style but it is what it is and I choose my battles - test case writing style is not one that I'm choosing to fight right now. With the testing resources we have right now, I think that limiting the upgrade test case to the default package set is our best option. Does it cover everything? No. Does it cover the most common issues? Probably. It does make some assurances that the fedup process is at least working for a known base case. I'm not saying that people shouldn't test upgrades with something other than the base package set or shouldn't file bugs if they hit issues outside of the base package set. I encourage that kind of testing and I'd love to see more acceptance of reasonable improvisation during testing. I just don't think that the release validation test cases is the best place to be doing that right now. Tim
Attachment:
signature.asc
Description: PGP signature
-- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
