From: "sudhir kumar" <smalikphy@xxxxxxxxx> >On Wed, Mar 4, 2009 at 11:45 PM, Ryan Harper <ryanh@xxxxxxxxxx> wrote: >>> * Uri Lublin <uril@xxxxxxxxxx> [2009-03-04 02:59]: >>> >- guest install wizard using md5sum region matching ... ouch. This is >>> >quite fickle. I've seen different kvms generate different md5sum for >>> >the same region a couple of times. I know distributing screenshots of >>> >certain OSes is a grey area, but it would be nice to plumb through >>> >screenshot comparison and make that configurable. FWIW, I'll probably >>> >look at pulling the screenshot comparison bits from kvmtest and getting >>> >that integrated in kvm_runtest_2. >>> Creating a step file is not as easy as it seems, exactly for that reason. >I agree here 100% as per my experience. >>> One has to pick a part of the screenshot that only available when input is >>> expected and that would be consistent. We were thinking of replacing the >>> md5sum with a tiny compressed image of the part of the image that was >>> picked. >> >> It isn't just that step file creation isn't easy is that even with a >> good stepfile with smart region boxes, md5sum can *still* fail because >> KVM renders one pixel in the region differently than the system where the >> original was created; this creates false positives failures. >I also have faced this issue. Even on the same system the step file >may fail. I saw few(though not frequent) situations where the same >md5sum passed in one time and failed in the other attempt. I think we need something more forgiving than md5sum, which would still be pretty small. Image compression/reduction is what we are thinking of (but have yet to prove it's working and select the best algorithm for our needs). >>> We had two other implementation for guest-wizard, one which only compares >>> two/three consecutive screendumps (problems with e.g. clock ticking), and >> >> Right, I like the idea of the region selection in stepmaker, it lets us >> avoid areas which have VM specific info, like the device path or clock. >> I'd like to keep the current stepmaker and region code, what I'd like to >> see instead of just md5sum compare of the cropped region, to be able to >> plug in different comparisons. If a user does have screens from >> stepmaker available, guest_wizard could attempt to do screen compare >> like we do in kvmtests with match percentages. If no screens are >> available, fallback on md5 or some other comparison algorithm. >That sounds better. Yet none of my step files worked in one go. I >remember running my test and stepmaker parallelly and continue taking >screenshots untill one passes and putting that md5sum in the step >file. That's an interesting way. I have'nt tried that one before. >So at the end I was fully meshed up with respect to the cropped >images and I had no idea which cropped image corresponded to which >md5sum. Step file writes the step number (which is the image number) as a comment for the step. I'm sure (and hope) that after you'd create a few step files, you'd pick the right subimage and the whole process would be much easier. >Though the RHEL5.3 step files that I generated in the text mode >installation were quite strong. That's nice to know. >>> one similar to kvmtest. The kvmtest way is to let the user create his/her >>> own screendumps to be used later. I did not want to add so many screendumps >>> images to the repository. Step-Maker keeps the images it uses, so we can >>> compare them upon failure. Step-Editor lets the user to change a single >>> barrier_2 step (or more) by looking at the original image and picking a >>> different area. >> >> Agreed, I don't want to commit screens to the repo either, I just want >> to be able to use screens if a user has them available. >> >I have two questions with respect to stepfiles. >1. The timeouts: Timeouts may fall short if a step file generated on a >high end machine is to be used on a very low config machine or >installing N virtual machines (say N ~ 50,100 etc) in parallel. For those reasons, among others, we set the timeout to 5 times what the step actually took. Since creating the step file is human driven (compared to running the installation) effectively the multiplication is even higher. That also creates a problem when guest installation fails. We thought of a very quick and simple benchmark. We would run it in step-maker and write the score in the step file. And we would run it before guest installation. We'd adjust the timeout accordingly. >2. If there are changes in KVM display in future the md5sum will fail. >So are we prepared for that? It happened to us when suspend capability was added. I do not think you can be prepared for that. The before/after screens are different. We have step-editor for that. Using step-editor we can change the step that failed due to the change (adjust it to the "after" screendump or choosing a different region), and hopefully guest installation would be successful again. Uri. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
