On Tue, 2011-04-05 at 16:29 +0100, Stefan Hajnoczi wrote: > Features that I think are important for a qemu.git kvm-autotest: > * Public results display (sounds like you're working on this) ^ Yes, we hope to get this setup soon. > * Public notifications of breakage, qemu.git/master failures to > qemu-devel mailing list. ^ The challenge is to get enough data to determine what is a new breakage from a known issue, mainly. More related to have historical data from test results than anything else, IMO. > * A one-time contributor can get their code tested. No requirement to > set up a server because contributors may not have the resources. Coming back to the point that many colleagues made: We need a sort of 'make test' on the qemu trees that would fetch autotest and could setup basic tests that people could run, maybe suggest test sets... The problem I see is, getting guests up and running using configs that actually matter is not trivial (there are things such as ensuring that all auxiliary utilities are installed in a distro agnostic fashion, having bridges and DHCP server setup on possibly a disconnected work laptop, and stuff). So, having a 'no brains involved at all' setup is quite a challenge, suggestions welcome. Also, downloading isos, waiting for guests to install and run thorough tests won't be fast. So J. Random Developer might not bother to run tests even if we can provide a fool proof, perfectly automated setup, because it'd take a long time at first to get the tests run. This is also a challenge. About the ideas of how a 'make tests' feature would work, please give me some feedback. I picture something like: ------------------------------------------------------------------- $ make test Checking out autotest... 1 - Verifying directories (check if the directory structure expected by the default test config is there) Creating /tmp/kvm_autotest_root/images Creating /tmp/kvm_autotest_root/isos Creating /tmp/kvm_autotest_root/steps_data Do you want to setup NFS mounts or copy isos to some of those dirs? (y/n) 2 - Creating config files from samples (copy the default config samples to actual config files) Creating config file /home/lmr/Code/autotest-git/client/tests/kvm/build.cfg from sample Creating config file /home/lmr/Code/autotest-git/client/tests/kvm/cdkeys.cfg from sample Creating config file /home/lmr/Code/autotest-git/client/tests/kvm/tests_base.cfg from sample Creating config file /home/lmr/Code/autotest-git/client/tests/kvm/tests.cfg from sample Creating config file /home/lmr/Code/autotest-git/client/tests/kvm/unittests.cfg from sample 3 - Do you want to test Linux guests? (y/n) (if y) Verifying iso (make sure we have the OS ISO needed for the default test set) Verifying iso Fedora-14-x86_64-DVD.iso /tmp/kvm_autotest_root/isos/linux/Fedora-14-x86_64-DVD.iso not found or corrupted Would you like to download it? (y/n) 4 - Do you want to test windows guests? (y/n) (if y) Verifying winutils.iso (make sure we have the utility ISO needed for Windows testing) In order to run the KVM autotests in Windows guests, we provide you an ISO that this script can download Verifying iso winutils.iso /tmp/kvm_autotest_root/isos/windows/winutils.iso not found or corrupted Would you like to download it? (y/n) 5 - Checking for the KVM module (make sure kvm is loaded to accelerate qemu) Running '/sbin/lsmod' KVM module loaded 6 - Verify needed packages to get started Please take a look at the online documentation http://www.linux-kvm.org/page/KVM-Autotest/Client_Install (session 'Install Prerequisite packages') You can also edit the test config files (see output of step 2 for a list). 7 - What are the areas you want to verify (enter all that apply) A) Unit tests (does not require to install guests) B) Sanity (just install and boot guests) C) Networking D) Block device E) Migration F) Time drift 8 - Did you check your config and are ready to start testing (y/n) Running tests... ------------------------------------------------------------------- If someone answered n to both 3) and 4) only the unit tests would be available to execute. > Perhaps kvm-autotest is a good platform for the automated testing of > ARM TCG. Paul is CCed, I recently saw the Jenkins qemu build and boot > tests he has set up. Lucas, do you have ideas on how these efforts > can work together to bring testing to upstream QEMU? > http://validation.linaro.org/jenkins/job/qemu-boot-images/ I heard about jenkins before and it is indeed a nice project. What they do here, from what I could assess browsing at the webpage you provided is: 1) Build qemu.git every time there are commits 2) Boot pre-made 'pristine' images, one is a lenny arm image and the other is a linaro arm image. It is possible to do the same with kvm autotest, just a matter of not performing guest install tests and executing only the boot tests with pre-made images. What jenkins does here is a even quicker and shorter version of our sanity jobs. About how we can work together, I thought about some possibilities: 1) Modify the jenkins test step to execute a kvm autotest job after the build, with the stripped down test set. We might gain some extra debug info, that the current test step does not seem to provide 2) Do the normal test step and if that succeeds, trigger a kvm autotest job that does more comprehensive testing, such as migration, time drift, block layer, etc The funny thing is that KVM autotest has infrastructure to do the same as jenkins does, but jenkins is highly streamlined for the buildbot use case (continuous build and integration), and I see that as a very nice advantage. So I'd rather keep use jenkins and have kvm autotest plugged into it conveniently. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
