From: "Ryan Harper" <ryanh@xxxxxxxxxx> > Uri Lublin <uril@xxxxxxxxxx> [2009-03-04 02:59]: >> >> > - it seems like the definition and rules ought to be separate from the >> > last section which defines which tests to run (the fc8_quick area), so >> > adding something as simple as include support to kvm_config.py would >> > be sufficient to support a common definition file but different >> > testing rules. >> An include support is one way to do it. We thought of a different way, >> which is >> to add rules from the control file. So the control file would pick the >> test-list >> it wants to run. Your suggestion is simpler but you need both a config file >> and >> a control file to change the test-list. We need to change only the control >> file. > >OK, well, I viewed almost all of the test config file as static. The >rules about guests, features, config variants, can change over time, but >not that often which is what I was wanting an include of that mostly >static data and then something else that picked which guests and tests >to run. It does make sense for the control file to pick the set of >tests, so I think we're in agreement, though I do think adding include >support is still a good idea, but much lower on the priority list. OK. Shouldn't be too hard. >> >> >> > >> >- kvm_runtest_2 as mentioned doesn't mess with your host networking and >> >relies on -net user and redir, would be good to plumb through -net tap >> >support that can be configured instead of always using -net user >> We want to add -net tap support, as that is what users usually use. >> kvm_runtest does exactly that (a part of kvm_host.cfg). The drawbacks of >> tap are >> (among others): >> - One must run tests as root, or play with sudo/chmod (True for /dev/kvm, >> but >> simpler) >> - You have to a have a dhcpd around. kvm_runtest by default runs a local >> dhcpd >> to serve kvm guests (part of setup/cleanup tests). >> - A bit more difficult to configure. >> >I don't want to have the test *setup* tap and all of the infrastructure >like dhcp, or dnsmasq... I just want to be able to configure whether a >guest is launched with -net tap or -net user. kvm_runtest_2 punts on >building and install kvm as well as the networking, which is a great >idea, I just want to be flexible enough to run kvm_runtest_2 with -net >tap as well as -net user. I agree we want to enable -net tap. I've just mentioned the drawbacks of it. We want to add kvm_install to kvm_runtest_2. It's important to build kvm on the client. It would be nice to also use it to build kvm on guests. >> > >> >I noticed the references to the setup isos for windows that presumbly >> >install cygwin telnetd/sshd, are those available? if the isos >> >themselves aren't, if the build instructions are, that would be very >> >useful. >> You are right. We do have an installation iso images for telnetd/sshd. >> I did not want to commit iso images. Also, I am not sure about licensing, >> and I prefer that we would generate them on the user machine. We'll add the >> build instructions to the wiki. > >Agreed. Sounds like a plan. >> >- guest install wizard using md5sum region matching ... ouch. This is >> >quite fickle. I've seen different kvms generate different md5sum for >> >the same region a couple of times. I know distributing screenshots of >> >certain OSes is a grey area, but it would be nice to plumb through >> >screenshot comparison and make that configurable. FWIW, I'll probably >> >look at pulling the screenshot comparison bits from kvmtest and getting >> >that integrated in kvm_runtest_2. >> Creating a step file is not as easy as it seems, exactly for that reason. >> One has to pick a part of the screenshot that only available when input is >> expected and that would be consistent. We were thinking of replacing the >> md5sum with a tiny compressed image of the part of the image that was >> picked. >It isn't just that step file creation isn't easy is that even with a >good stepfile with smart region boxes, md5sum can *still* fail because >KVM renders one pixel in the region differently than the system where the >original was created; this creates false positives failures. We need something more "forgiving" than md5sum, that would still be a compact representation of the region box. We may be able to use an image compression algorithm (need to investigate on that). That's what I meant by "tiny compressed image". >> We had two other implementation for guest-wizard, one which only compares >> two/three consecutive screendumps (problems with e.g. clock ticking), and >Right, I like the idea of the region selection in stepmaker, it lets us >avoid areas which have VM specific info, like the device path or clock. >I'd like to keep the current stepmaker and region code, what I'd like to >see instead of just md5sum compare of the cropped region, to be able to >plug in different comparisons. If a user does have screens from >stepmaker available, guest_wizard could attempt to do screen compare >like we do in kvmtests with match percentages. If no screens are >available, fallback on md5 or some other comparison algorithm. As mentioned above, an image compression/reduction algorithm may be better than md5sum. With kvmtest setting the percentage is based on assumptions too. Setting the percentage too high (100%) may result in the same problem of md5sum, and setting it too low, may find a match before the guest is ready to receive input. I would not mind adding the kvmtest way, but I am not sure I want it to be the default. After some tuning, we do not have that many false positives. We can let the user choose. >>> one similar to kvmtest. The kvmtest way is to let the user create his/her >>> own screendumps to be used later. I did not want to add so many screendumps >>> images to the repository. Step-Maker keeps the images it uses, so we can >>> compare them upon failure. Step-Editor lets the user to change a single >>> barrier_2 step (or more) by looking at the original image and picking a >>> different area. >> >>Agreed, I don't want to commit screens to the repo either, I just want >>to be able to use screens if a user has them available. >> > - a lot of the ssh and scp work to copy autotest client into a guest >> > is already handled by autoserv >> That is true, but we want to be able to run it as client test too. That way >> a user does not have to install the server to run kvm tests on his/her >> machine. >While true, we should try to use the existing server code for autotest >install. OK. >> > - vm.py has a lot of infrastructure that should be integrated into >> > autotest/server/kvm.py or possibly client-side common code to support >> > the next comment >> In the long term, there should be a client-server shared directory that >> deals with kvm guests (letting the host-client be the server for kvm-guests >> clients) >I believe client/common_lib is imported into server-side as common code, >so moving kvm infrastructure bits there will allow server-side and any >other client test to manipulate VM/KVM objects. For kvm-install that can be done. Running autotest tests on kvm guests would not be as easy. >> >> > - kvm_tests.py defines new tests as functions, each of these tests >> > should be a separate client tests which sounds like a pain, but >> > should allow for easier test composition and hopefully make it easier >> > to add new tests that look like any other client side test with just >> > the implementation.py and control file >> > - this model moves toward eliminating kvm_runtest_2 and having a >> > server-side generate a set of tests to run and spawning those on a >> > target system. >> I am not sure that I follow. Why implementing a test as a function is a >> pain ? >Test as a function of course isn't a pain. What I meant was that if >I've already have to guests and I just want to do a migration test, it >would be nice to just be able to: > >% cd client/tests/kvm/migration >% $AUTOTEST_BIN ./control That would be nice. Actually, we used to keep it that way with kvm_runtest but there were too many dependencies so we've dropped it. It is good only for specific cases (e.g. migration of a Fedora-8 32 bit guest with 512 MB ...). One can easily write a control file that uses kvm_runtest_2 "test" to do that. A simple configuration file is another, not as simple, way (just 1-2 dictionaries in the list). >I'd like to move away from tests eval-ing and exec-ing other tests; it >just feels a little hacky and stands out versus other autotest client >tests. I think testing kvm is more complex than testing e.g. kernelbuild. Control files files don't have to be one liners. I think that having a config-file parser in the control file, and calling appropriate tests, reduces complexity and code duplication. Think of the guests x tests matrix we want to test. >We can probably table the discussion until we push patches at autotest >and see what that community thinks of kvm_runtest_2 at that time. OK. >> The plan is to keep kvm_runtest_2 in the client side, but move the >> configuration file + parser to the server (if one wants to dispatch test >> from the server). The server can dispatch the client test (using >> kvm_runtest_2 "test" + dictionary + tag). We have dependencies and we can >> spread unrelated kvm tests among similar hosts (e.g install guest OS and >> run tests for Fedora-10 64 bit on one machine, and install guest OS and run >> tests for Windows-XP 32 bit on another). >Yeah, that sounds reasonable. >> We may have to add hosts into the configuration file, or add it while >> running (from the control file ?). We sure do not want to go back to the >> way kvm_runtest works (playing with symbolic links so that autotest would >> find and run the test), and we do not want too many kvm_test_NAME >> directories under client/tests/ . >Agree with no symbolic links. If we move common kvm utils and objects >to client/common_lib that avoids any of that hackery. >On the dir structure, I agree we don't have to pollute client/tests with >a ton of kvm_* tests. I'll look around upstream autotest and see if >there is an other client side tests to use an example. OK. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
