Well... I was thinking more of "we" as an industry. I sure remember
having to pass various validation tests when delivering systems, back in
the day. As well as regression tests when shipping updates. These
days, the world seems to have moved to release early & often, then fix
bugs when someone opens a ticket. (I miss working in perl - where the
norm is to include automatic regression testing in every makefile.)
Miles
John C Klensin wrote:
Andy,
That is consistent with my memory. And all of those would
qualify more as Applicability Statements, not actual conformance
or compliance tests where there is reference code and/or
scenario-generating code to test against. So, to repeat my
point, Miles's question about "how we got away from writing
compliance tests..." appears to be based on an incorrect premise
because we never did write such tests.
thanks,
john
--On Sunday, October 23, 2022 10:30 -0400 "Andrew G. Malis"
<agmalis@xxxxxxxxx> wrote:
John,
The closest we ever came were the various host and router
requirements RFCs (1122, 1123, 1812, etc.), plus a few others
here and there like RFC 7766.
Cheers,
Andy
On Sat, Oct 22, 2022 at 6:55 PM John C Klensin
<john-ietf@xxxxxxx> wrote:
--On Sunday, October 23, 2022 08:18 +1100 Lloyd W
<lloyd.wood=40yahoo.co.uk@xxxxxxxxxxxxxx> wrote:
On 22 Oct 2022, at 03:43, Miles Fidelman
<mfidelman@xxxxxxxxxxxxxxxx> wrote:
It does lead me to wonder about how we got away from
writing compliance tests for everything. And, a follow-up
thought that, perhaps, a compliance test might be a nice
thing to add to RFCs (at least those that proceed to
standards) - with IETF or ISOC or maybe ICANN maintaining
a reference server to test against.
Well, it's "rough consensus and running code" and not
"rigorous compatibility and RFC compliance".
last I checked, standards required two separate
interoperable implementations - so which one would be
favoured and chosen for your reference server?
Another reason is that there has been a great deal of
experience showing that, if an SDO produces compliance tests,
those tests, and not the written text, rapidly become the
standard. If the test does not cover absolutely everything,
that can be bad news. It gets particularly problematic for
IETF specs because it is hard to express "SHOULD" or "SHOULD
NOT". I'm also unsure about "got away from" -- as far as I
know, the IETF, as the IETF, has never specified compliance
or validation tests for its standards track documents.
john
--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra
Theory is when you know everything but nothing works.
Practice is when everything works but no one knows why.
In our lab, theory and practice are combined:
nothing works and no one knows why. ... unknown
