| You'd be surprised, John, at what people do that's unexpected, even with really simple protocols (DHCP being the most obvious example with which I am intimately familiar). The most common thing they do is to consider perfectly valid responses to be errors, because they don't match the implementor's mental model of what the protocol is doing, and the implementor was very careful to ignore anything outside of his or her understanding of the protocol. These misunderstandings often come from doing interop testing with a single server that has a particular profile of default behaviors. So having well thought out tests that test edge cases and that send things the other side shouldn't accept are very useful. And the process of coming up with a testing regime is actually really helpful for finding ambiguities in the specification. I think the point you are making about OSI, which was very nicely illustrated by Ned's comment, is a valid point, but the problem wasn't that there was conformance testing, and it wasn't even that there was a need for conformance testing. It was a problem with the process that produced the standard that was being tested. The process had failed, and no amount of testing was going to successfully paper over that failure. |