It's been interesting to follow the discussion after Michal's original submission. Those readers who are familiar with the PROTOS style of testing and the results of that research will know how effective protocol-based invalid input testing has proven to be in catching at least a majority of common implementation flaws. The PROTOS approach calls this "robustness testing" as per the following IEEE definition: "robustness. The degree to which a system or component can function correctly in the presence of invalid inputs or stressful environmental conditions." (IEEE Standard Glossary of Software Engineering Terminology) Of course, the idea of automated testing with invalid inputs harkens back all the way to fuzz testing. Whereas fuzz testing typically employs completely or very nearly random inputs, protocol-based robustness testing focuses on modifying otherwise valid protocol data in ways that have been proven during the last 20-30 years to most likely trigger flaws in implementations. The common bugs that are found in this type of testing would inevitably be found later on by various auditors and guys with interesting-coloured hats, so this just helps in catching some of these vulns early on. Needless to say the idea can be applied besides network protocols to almost any kind of structured data, including file formats. In addition to HTML, image and audio/video formats lend themselves well to this kind of testing. File format implementations are still not usually implemented with malicious attacks in mind, which means that they are trivial to break in most cases. (As compared with network protocols, although the boundary between the two has steadily become hazier in the last 10 years due to obvious reasons.) Since invalid input space is quite large even for simple protocols and data formats, automated test generation is a must, as is careful human design for the initial parameters of the used invalid data. Experience of past software vulnerabilities and a knowledge of common programming errors certainly helps in this. Disclaimer: I work for Codenomicon, which develops and sells test tools for this kind of testing. Codenomicon is a spin-off from the PROTOS research group OUSPG, and we sponsor them by supplying them with our test generation framework. However, I am not trying to sell anything, simply lauding Michal's excellent foray into a topic I myself am passionate about. Automated robustness testing is hardly a panacea, but it does help in catching a lot of the trivial input-based errors. This is especially valuable if the testing can be done even before the software is released. Mozilla developers seemed to "get it", since I understood they very rapidly integrated these tests to their regular (regression) testing cycle. -- Heikki Kortti
