* Michael Goldish <mgoldish@xxxxxxxxxx> [2009-03-11 03:08]: > > > I'd like to comment on this. I don't doubt that some fuzzy matching > > > algorithm (such as calculating match percentages) would generally > > be > > > more robust. I do however doubt it would significantly lower the > > false > > > positive rate in our case (which is fairly low already). False > > > positive failures in step files are typically caused by: > > > > I've seen multiple failures during the windows guest installs which I > > assume are well tested stepfiles. For example, 2k8 installs and the > > fails to pass the barrier when trying to set the user password for > > the > > first time. The cropped region *looks* exactly like the the intended > > location on the screendump, but md5sums to something different. > > > > A recent run of 2k3 and 2k8 installs resulted in the following > > failures: > > > > Win2k3-32bit -- screenshot of "Windows Setup" and Setup is starting > > windows, cropped region is of "Setup is starting Windows" full screen > > dump matches this text from a human pov > > > > Win2k3-64-bit -- same as above > > > > Win2k8-32-bit -- screenshot of "The user's password must be changed > > before logging in the first time with OK and cancel buttons. - > > cropped > > region is of the text "The user's password must be changed before > > logging in the first time" - matching the full screen screendump fine > > from a human POV > > > > Win2k8-64-bit -- same as above > > > > We've also been creating stepfiles for Linux guests as well that > > aren't > > here, various SLES and RHEL installs -- and I've repeatedly seen the > > same issue where the cropped region *should* match but isn't, and it > > isn't a result of any of the very correct reasons you've listed below > > as > > to why the stepfiles might fail. > > The Windows failures you're describing sound like they could be caused > by a known KVM bug, which results in Windows installations sometimes > booting from CDROM, instead of the HDD, immediately following the > installation. No, but I have seen a bug that after an install and guest OS reboots, KVM fails to boot from the harddrive; exiting KVM and then booting from HD works fine. We're looking into that one right now. > > I assume you don't have the stepmaker data of those Windows stepfiles. These are the stepfiles that came with kvm-autotest so no *I* don't have stepmaker data, but whomever committed the windows stepfiles to kvm-autotest *should* have the data ... > In that case, the images left by the stepfile test, scrdump.ppm and > cropped_scrdump.ppm, are in fact the full screendump and a cropped > region in it. They should always match perfectly, because the cropped > one is generated from the full one at runtime. None of them reflects > the expected guest behavior; they reflect what the stepfile test > actually found. The only thing you have that reflects the expected > guest behavior is the md5sum found in the stepfile. > > If you happened to keep the "debug" dirs which contain the screendumps > and test logs, and could somehow send them to me or Uri, I'd be able > to tell you what went wrong with the test and whether it is indeed > that KVM bug or a stepfile error. We probably could also use the > stepfiles you were working with, because we might have changed ours > recently, though that is unlikely because we don't change old > stepfiles very often nowadays. I do have the debug dir data from these runs. Looking at the cropped ppm and screendump ppm is how I determined that there must be something wrong with how the image is rendered since the cropped ppm matches the screendump output, but with whatever subtle difference that generates a different md5sum. I'll see about figuring out how to get the debug output to you. > > Regarding the stepfiles you created for Linux -- I can't help much > with those since I don't have the data. I do believe that if I had the > data and the stepfiles I could quickly identify the problem, so if you > think those can be sent to us, I'd like to have them. OK > > I'm not sure exactly what version of kvm_runtest_2 you're using (are > you are using kvm_runtest_2?), but I think it should support Yep, kvm_runtest_2; but I've seen the same issue on kvm_runtest. > automatic comparison of the actual screendump with the expected > screendump. If you have a slightly older version than the current git > HEAD, then you should probably place your <stepfile>_data directory > right next to <stepfile>, and whenever a stepfile test fails you'll > get -- in addition to scrdump.ppm and cropped_scrdump.ppm -- > scrdump_reference.ppm and cropped_scrdump_reference.ppm, as well as a > nice green-red comparison image which colors all matching pixels green > and all mismatching ones red. That last image is very helpful when > stepfiles require fixing. If you have the latest git HEAD, you should > place all your <stepfile>_data dirs in a dir named "steps_data" which > should reside next to "steps" (which should contain the stepfiles > themselves). Very useful information, should be added to the wiki; we should write a section on using stepmaker/stepeditor and best practices on picking barriers/cropped regions. > > Well, either there is a *bug* right now that is triggering a higher > > rate > > of false positives, or using a better algorithm is a requirement; > > distributing stepfiles and md5sums that don't work isn't productive, > > so > > in the case that it is a bug I still suggest we pursue a more > > resilient > > algorithm. > > Do the Windows tests you mentioned fail consistently, or have you > witnessed any of them succeed in some of the runs? Consistently fail, no passes so far. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx ryanh@xxxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
