* Uri Lublin <uril@xxxxxxxxxx> [2009-03-01 13:10]: > Hello, > > KVM-autotest is a test framework for kvm, based on autotest > (http://autotest.kernel.org). > > Its purpose is to keep kvm stable. > For developers, we want to find regressions early. > For users, we want users to feel confident kvm runs well on their own > machine. > Also we would like to present results nicely, similar to > http://test.kernel.org. > > > Today we have quite a few guests (including linux, windows, unix) and some > tests (guest installation, boot, reboot and more) > > For guest installation, we need the guest iso images. One needs to get the > iso images before running guest installation tests (which is by default > needed for all other tests to run). > > Source code is located at > git://git.kernel.org/pub/scm/virt/kvm/kvm-autotest.git > > Interesting directories are > client/tests/kvm_runtest_2 (newer) > client/tests/kvm_runtest (older, almost obsolete, not as interesting) I've been digging into kvm_runtest_2 and have some feedback, but first to say, runtest_2 is huge cleanup and simplification from runtest, so thanks. Now for some comments: - kvm_tests.cfg has a decent learning curve to wrap your head around. It would have been useful to have some debugging that would dump out what rules were filtering out guests... the dependencies aren't always easy to find. My first experience with the dep hunt is just changing 'only qcow2' in fc8_quick to raw and getting no output from kvm_config.py. Turns out, that 'raw' has a smp2 requirement ... that sort of filtering could be displayed with debugging output making configuration changes easier. - documentation of keywords and structure would be nice, explaining what -variant , only and @ are doing for you, etc. - it seems like the definition and rules ought to be separate from the last section which defines which tests to run (the fc8_quick area), so adding something as simple as include support to kvm_config.py would be sufficient to support a common definition file but different testing rules. - kvm_runtest_2 as mentioned doesn't mess with your host networking and relies on -net user and redir, would be good to plumb through -net tap support that can be configured instead of always using -net user - make -vnc parameter config/optional I noticed the references to the setup isos for windows that presumbly install cygwin telnetd/sshd, are those available? if the isos themselves aren't, if the build instructions are, that would be very useful. - guest install wizard using md5sum region matching ... ouch. This is quite fickle. I've seen different kvms generate different md5sum for the same region a couple of times. I know distributing screenshots of certain OSes is a grey area, but it would be nice to plumb through screenshot comparison and make that configurable. FWIW, I'll probably look at pulling the screenshot comparison bits from kvmtest and getting that integrated in kvm_runtest_2. - kvm_runtest_2 looks a lot more like a regular autotest test, which is a Good Thing(TM). There are still some things that would prevent it going upstream autotest (which I assume is the long term goal) - a lot of the ssh and scp work to copy autotest client into a guest is already handled by autoserv - vm.py has a lot of infrastructure that should be integrated into autotest/server/kvm.py or possibly client-side common code to support the next comment - kvm_tests.py defines new tests as functions, each of these tests should be a separate client tests which sounds like a pain, but should allow for easier test composition and hopefully make it easier to add new tests that look like any other client side test with just the implementation.py and control file - this model moves toward eliminating kvm_runtest_2 and having a server-side generate a set of tests to run and spawning those on a target system. I do still like the idea of having a client-side test that can just run on a developer/user's system to produce results without having to configure all of the autotest server-side bits. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx ryanh@xxxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
