Hi, folks. So, our current set of storage validation tests is just a grab-bag of stuff we've held over from oldui, added and patched piecemeal; it's not very coherent or consistent and it doesn't come close to exercising all of the storage stuff in the installer. We wind up sort of inventing test plans particularly for custom partitioning as we go along, with the consequence that we're not sure what we're going to test, what's really important to test when, etc etc. I've made a few abortive tries at re-doing the storage tests and basically given up because it's just a hideous thing to try and cover, but I thought while I'm still on a momentum roll from F20 and remember some of the issues that came up during F20 validation, I'd take another cut at it. Here's what I came up with: https://fedoraproject.org/wiki/User:Adamwill/Draft_storage_matrix The proposal is to separate out storage testing from the 'installation testing' matrix as its own matrix, because I think we can get further with flexible columns, and it's such a big area the installation matrix gets pretty unwieldy with all the storage tests stuffed into it. remember this is *extremely* rough. It may be that we don't think my organizational approach here is right at all and we should tear it up and start over. But I thought I'd throw it out there for comments. Most of the tests, obviously, don't exist yet; they'd be fairly trivial to write, I think the hard part of the problem here is deciding what we want to test, and how to organize it. It's an extremely difficult area; there are so many different variables to storage configuration that it's utterly impractical to try and cover every possible permutation even assuming the user uses the interface perfectly. What I've tried to do is come up with something which would exercise most of the areas we really seem to want to care about, without being utterly unwieldy. It is still very large, that's probably the first thing you'd notice about it. I doubt we could run this on every build. What I think would be a nice complement is a somewhat improved version of testcase_stats - http://testdays.qa.fedoraproject.org/testcase_stats/ - which could track both dimensions of each matrix, so we could try to strategically ensure every spot on the table is covered at least once per milestone or once per release, say. Here's a quick key to the test names for tests I've made up which may not be self-explanatory, yell if you need explanations of any of the others: ----- guided_multi_empty - the 'multi' means multiple disks, this is checking we correctly autopart to multiple disks. in the custom tests, 'auto' means 'using the autopart mechanism inside custom partitioning', the blue text that lets you automatically create a set of volumes. existing_retain_home - this is the classic 'install over an existing Fedora install, retain the /home partition' trick. existing_precreated - this would be a test for running the installer with a set of empty volumes that you just wanted to assign mount points to. add_to_container - add a new volume to free space within an existing container. assign_* tests - these are for specifying that a given partition or container must be on a particular disk. help_text - just checks the 'help' screen works. small_disks - this would be a test where you check anaconda correctly refuses to install to a disk that's too small. cancel_encryption - this would be to check what happens if you cancel out of the 'enter a passphrase' dialog. shrink_maximum - try to shrink a partition to the largest possible size (right-hand end of the slider) shrink_minimum - shrink a partition as small as possible (left-hand end of the slider) shrink_no_size_change - set action for a partition to 'shrink' but don't actually change its size shrink_unusual_sizes - this would be for the bug we discovered in f20, test the shrinker handles partitions with 'weird' sizes shrink_change_action - this is just for changing the 'action' in shrink a bunch of times and making sure it doesn't explode multiple_trips - run through Installation Destination more than once custom_resize_no_size - just clear the 'desired size' field for a partition and hit 'update settings' custom_resize_invalid_unit - try entering something like '30 ZX' in the desired size field custom_resize_return - change the desired size and then change it back to the original value custom_invalid_mount_point - try entering a mount point name that is not allowed (invalid characters, spaces or something) custom_invalid_filesystem - try putting a system partition on vfat or bios boot or swap or something ---- Please, absolutely all comments, suggestions, alternative proposals, flames etc etc welcome. I'm sure we could improve this proposal or do better, but I think we have to try and do _something_ better than our current tests. And I think drawing up a table like this, if nothing else, illustrates what a hard task this is... -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net -- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
