Hey, folks. So, as per https://fedorahosted.org/fedora-qa/ticket/307 , I think we need to add a process for validating the multi-live and multi-install images that we've been building for the last few releases. I've put together a draft matrix template for this validation here: https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Compendium_image_validation_results_template It's similar to the other validation matrices, and the changes are mostly obvious. The key bit is the layout of the matrix and the test cases themselves. I haven't actually written any of the specific test cases yet, but I tried to think of the best way to organize things. We obviously need to do the basic sanity tests on each image, then we need to test that each image boots and the menu that lets you select which sub-image to boot works properly, and also that each sub-image boots and installs properly. I think that set of tests ought to pretty much cover the intended functionality of the compendium images. When actually writing the test cases, we might be able to come up with ways to take advantage of some of the existing install/base/desktop validation test cases, but I haven't quite thought that far ahead yet. I tried to organize the planned test cases as efficiently as possible; this is the best layout I could come up with, but someone might be able to suggest improvements. In case it's not obvious, here's roughly what each test case does: QA:Testcase Mediakit_ISO_Size and QA:Testcase_Mediakit_ISO_Checksums - these test cases already exist and can apply without modification to the compendium images, they simply check that they're within required size limits (dual-layer DVD size for these images) and the published checksums match the images. QA:Testcase_multilive_boot and QA:Testcase_multidvd_boot - these will simply check that the compendium images themselves boot, when written to a physical dual-layer DVD and booted as normal. We _may_ also want to test booting them when written to USB using one specific method. I don't think we need to test any other way of booting the images, as these images are specifically intended to be written to physical media and given out at events; we don't really intend people to download them and write/boot them via all possible methods themselves, though they can do that if they really want to. So we don't really need to cover VM booting of the image, writing to USB via all other possible methods and so on. QA:Testcase_multilive_menu and QA:Testcase_multidvd_menu - these will test that the top-level menus on each image, for picking which sub-image to boot, function correctly. I'm not sure at present if it'll be possible to just instead write a generic QA:Testcase_multi_menu test case which would apply to either image; if so, we can do just QA:Testcase_multi_menu and QA:Testcase_multi_boot test cases and have one result column for each image (as in the Image sanity table) instead of four test cases and a single result column. QA:Testcase_multi_sub_boot and QA:Testcase_multi_sub_install - these will test that each sub-image on each compendium image boots and installs correctly. It would probably be unusual for _just one_ sub-image to fail, so we could probably get away with just testing a single sub-image on each medium, but everything that can go wrong will go wrong, so we might as well cover everything if we have time/resources. We may be able to re-use an existing test case instead of writing new ones here, I'll have to check the exact wording of some of our existing cases. Thoughts, comments, objections, suggestions? Can everyone see where this is going, or am I zooming off up the wrong tree? Can you think of a better approach? Thanks! -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
