* Michael Goldish <mgoldish@xxxxxxxxxx> [2009-03-10 20:55]: > > ----- "Ryan Harper" <ryanh@xxxxxxxxxx> wrote: > > > > >- guest install wizard using md5sum region matching ... ouch. This > > is > > > >quite fickle. I've seen different kvms generate different md5sum > > for > > > >the same region a couple of times. I know distributing screenshots > > of > > > >certain OSes is a grey area, but it would be nice to plumb through > > > >screenshot comparison and make that configurable. FWIW, I'll > > probably > > > >look at pulling the screenshot comparison bits from kvmtest and > > getting > > > >that integrated in kvm_runtest_2. > > > Creating a step file is not as easy as it seems, exactly for that > > reason. > > > One has to pick a part of the screenshot that only available when > > input is > > > expected and that would be consistent. We were thinking of replacing > > the > > > md5sum with a tiny compressed image of the part of the image that > > was > > > picked. > > > > It isn't just that step file creation isn't easy is that even with a > > good stepfile with smart region boxes, md5sum can *still* fail > > because > > KVM renders one pixel in the region differently than the system where > > the > > original was created; this creates false positives failures. > > I'd like to comment on this. I don't doubt that some fuzzy matching > algorithm (such as calculating match percentages) would generally be > more robust. I do however doubt it would significantly lower the false > positive rate in our case (which is fairly low already). False > positive failures in step files are typically caused by: I've seen multiple failures during the windows guest installs which I assume are well tested stepfiles. For example, 2k8 installs and the fails to pass the barrier when trying to set the user password for the first time. The cropped region *looks* exactly like the the intended location on the screendump, but md5sums to something different. A recent run of 2k3 and 2k8 installs resulted in the following failures: Win2k3-32bit -- screenshot of "Windows Setup" and Setup is starting windows, cropped region is of "Setup is starting Windows" full screen dump matches this text from a human pov Win2k3-64-bit -- same as above Win2k8-32-bit -- screenshot of "The user's password must be changed before logging in the first time with OK and cancel buttons. - cropped region is of the text "The user's password must be changed before logging in the first time" - matching the full screen screendump fine from a human POV Win2k8-64-bit -- same as above We've also been creating stepfiles for Linux guests as well that aren't here, various SLES and RHEL installs -- and I've repeatedly seen the same issue where the cropped region *should* match but isn't, and it isn't a result of any of the very correct reasons you've listed below as to why the stepfiles might fail. > > - an unexpected popup window covering the test region > - a dialog which has a different position every time (and varies by > many pixels) > - a barrier that passes before the controls get input focus, which > causes the following keystrokes to have no effect > - in text mode, sometimes a line of text is printed unexpectedly and > causes the entire screen to scroll up > - addition/modification of a KVM feature which changes the course of > the installation > > I may have left something out. In any case, all these problems are > solved by picking better barrier regions, but none can be solved by > using a more forgiving comparison method. I have encountered a single > installation that rendered a single pixel in an indeterministic > fashion, and though this problem was easily solved by correcting the > barrier (using a stepfile editor), I do agree we might get a small > decrease in the false positive rate if we use a more forgiving > algorithm. Well, either there is a *bug* right now that is triggering a higher rate of false positives, or using a better algorithm is a requirement; distributing stepfiles and md5sums that don't work isn't productive, so in the case that it is a bug I still suggest we pursue a more resilient algorithm. > > However, there is also a risk: a more forgiving algorithm may forgive > KVM for rendering errors. It may also make it risky to pick barriers > that are meant to catch small text; I believe a button with a "Yes" > caption and a button with a "No" caption would have a very high match > percentage, especially if you have to pick the whole button, or maybe > even some of its surroundings (and you often do). Noted, though I think as you indicated above, smart selection of the cropped region goes a long way toward avoiding these sorts of collisions. > > I still believe it's a good idea to look into other methods (we're > already doing that) and start experimenting with them. Cool. Obviously without the original ppm files from the stepmaker run, we can't determine if a different algo would help so we're generating new stepfiles and ppm data and trying to reproduce the md5sum mismatch issues. If there is anything I can do to help with the algo work let me know. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx ryanh@xxxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
