On Wed, Mar 4, 2009 at 11:45 PM, Ryan Harper <ryanh@xxxxxxxxxx> wrote: > * Uri Lublin <uril@xxxxxxxxxx> [2009-03-04 02:59]: >> >> > - it seems like the definition and rules ought to be separate from the >> > last section which defines which tests to run (the fc8_quick area), so >> > adding something as simple as include support to kvm_config.py would >> > be sufficient to support a common definition file but different >> > testing rules. >> An include support is one way to do it. We thought of a different way, >> which is >> to add rules from the control file. So the control file would pick the >> test-list >> it wants to run. Your suggestion is simpler but you need both a config file >> and >> a control file to change the test-list. We need to change only the control >> file. > > OK, well, I viewed almost all of the test config file as static. The > rules about guests, features, config variants, can change over time, but > not that often which is what I was wanting an include of that mostly > static data and then something else that picked which guests and tests > to run. It does make sense for the control file to pick the set of > tests, so I think we're in agreement, though I do think adding include > support is still a good idea, but much lower on the priority list. > >> >> > >> >- kvm_runtest_2 as mentioned doesn't mess with your host networking and >> >relies on -net user and redir, would be good to plumb through -net tap >> >support that can be configured instead of always using -net user >> We want to add -net tap support, as that is what users usually use. >> kvm_runtest does exactly that (a part of kvm_host.cfg). The drawbacks of >> tap are >> (among others): >> - One must run tests as root, or play with sudo/chmod (True for /dev/kvm, >> but >> simpler) >> - You have to a have a dhcpd around. kvm_runtest by default runs a local >> dhcpd >> to serve kvm guests (part of setup/cleanup tests). >> - A bit more difficult to configure. >> > > I don't want to have the test *setup* tap and all of the infrastructure > like dhcp, or dnsmasq... I just want to be able to configure whether a > guest is launched with -net tap or -net user. kvm_runtest_2 punts on > building and install kvm as well as the networking, which is a great > idea, I just want to be flexible enough to run kvm_runtest_2 with -net > tap as well as -net user. > > >> > >> >I noticed the references to the setup isos for windows that presumbly >> >install cygwin telnetd/sshd, are those available? if the isos >> >themselves aren't, if the build instructions are, that would be very >> >useful. >> You are right. We do have an installation iso images for telnetd/sshd. >> I did not want to commit iso images. Also, I am not sure about licensing, >> and I prefer that we would generate them on the user machine. We'll add the >> build instructions to the wiki. > > Agreed. Sounds like a plan. > >> >- guest install wizard using md5sum region matching ... ouch. This is >> >quite fickle. I've seen different kvms generate different md5sum for >> >the same region a couple of times. I know distributing screenshots of >> >certain OSes is a grey area, but it would be nice to plumb through >> >screenshot comparison and make that configurable. FWIW, I'll probably >> >look at pulling the screenshot comparison bits from kvmtest and getting >> >that integrated in kvm_runtest_2. >> Creating a step file is not as easy as it seems, exactly for that reason. I agree here 100% as per my experience. >> One has to pick a part of the screenshot that only available when input is >> expected and that would be consistent. We were thinking of replacing the >> md5sum with a tiny compressed image of the part of the image that was >> picked. > > It isn't just that step file creation isn't easy is that even with a > good stepfile with smart region boxes, md5sum can *still* fail because > KVM renders one pixel in the region differently than the system where the > original was created; this creates false positives failures. I also have faced this issue. Even on the same system the step file may fail. I saw few(though not frequent) situations where the same md5sum passed in one time and failed in the other attempt. > > >> We had two other implementation for guest-wizard, one which only compares >> two/three consecutive screendumps (problems with e.g. clock ticking), and > > Right, I like the idea of the region selection in stepmaker, it lets us > avoid areas which have VM specific info, like the device path or clock. > I'd like to keep the current stepmaker and region code, what I'd like to > see instead of just md5sum compare of the cropped region, to be able to > plug in different comparisons. If a user does have screens from > stepmaker available, guest_wizard could attempt to do screen compare > like we do in kvmtests with match percentages. If no screens are > available, fallback on md5 or some other comparison algorithm. That sounds better. Yet none of my step files worked in one go. I remember running my test and stepmaker parallelly and continue taking screenshots untill one passes and putting that md5sum in the step file. So at the end I was fully meshed up with respect to the cropped images and I had no idea which cropped image corresponded to which md5sum. > Though the RHEL5.3 step files that I generated in the text mode installation were quite strong. >> one similar to kvmtest. The kvmtest way is to let the user create his/her >> own screendumps to be used later. I did not want to add so many screendumps >> images to the repository. Step-Maker keeps the images it uses, so we can >> compare them upon failure. Step-Editor lets the user to change a single >> barrier_2 step (or more) by looking at the original image and picking a >> different area. > > Agreed, I don't want to commit screens to the repo either, I just want > to be able to use screens if a user has them available. > I have two questions with respect to stepfiles. 1. The timeouts: Timeouts may fall short if a step file generated on a high end machine is to be used on a very low config machine or installing N virtual machines (say N ~ 50,100 etc) in parallel. 2. If there are changes in KVM display in future the md5sum will fail. So are we prepared for that? >> > - a lot of the ssh and scp work to copy autotest client into a guest >> > is already handled by autoserv >> That is true, but we want to be able to run it as client test too. That way >> a user does not have to install the server to run kvm tests on his/her >> machine. > > While true, we should try to use the existing server code for autotest > install. > >> > - vm.py has a lot of infrastructure that should be integrated into >> > autotest/server/kvm.py or possibly client-side common code to support >> > the next comment >> In the long term, there should be a client-server shared directory that >> deals with kvm guests (letting the host-client be the server for kvm-guests >> clients) > > I believe client/common_lib is imported into server-side as common code, > so moving kvm infrastructure bits there will allow server-side and any > other client test to manipulate VM/KVM objects. > >> >> > - kvm_tests.py defines new tests as functions, each of these tests >> > should be a separate client tests which sounds like a pain, but >> > should allow for easier test composition and hopefully make it easier >> > to add new tests that look like any other client side test with just >> > the implementation.py and control file >> > - this model moves toward eliminating kvm_runtest_2 and having a >> > server-side generate a set of tests to run and spawning those on a >> > target system. >> I am not sure that I follow. Why implementing a test as a function is a >> pain ? > > Test as a function of course isn't a pain. What I meant was that if > I've already have to guests and I just want to do a migration test, it > would be nice to just be able to: > > % cd client/tests/kvm/migration > % $AUTOTEST_BIN ./control > > I'd like to move away from tests eval-ing and exec-ing other tests; it > just feels a little hacky and stands out versus other autotest client > tests. > > We can probably table the discussion until we push patches at autotest > and see what that community thinks of kvm_runtest_2 at that time. > > >> The plan is to keep kvm_runtest_2 in the client side, but move the >> configuration file + parser to the server (if one wants to dispatch test >> from the server). The server can dispatch the client test (using >> kvm_runtest_2 "test" + dictionary + tag). We have dependencies and we can >> spread unrelated kvm tests among similar hosts (e.g install guest OS and >> run tests for Fedora-10 64 bit on one machine, and install guest OS and run >> tests for Windows-XP 32 bit on another). > > Yeah, that sounds reasonable. > >> We may have to add hosts into the configuration file, or add it while >> running (from the control file ?). We sure do not want to go back to the >> way kvm_runtest works (playing with symbolic links so that autotest would >> find and run the test), and we do not want too many kvm_test_NAME >> directories under client/tests/ . > > Agree with no symbolic links. If we move common kvm utils and objects > to client/common_lib that avoids any of that hackery. > > On the dir structure, I agree we don't have to pollute client/tests with > a ton of kvm_* tests. I'll look around upstream autotest and see if > there is an other client side tests to use an example. > >> > I do still like the idea of having a client-side test that can just >> > run on a developer/user's system to produce results without having to >> > configure all of the autotest server-side bits. >> Me too :-) >> >> Thanks for all the comments and suggestions, > > Sure > > > -- > Ryan Harper > Software Engineer; Linux Technology Center > IBM Corp., Austin, Tx > ryanh@xxxxxxxxxx > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Sudhir Kumar -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
