On Jun 28, 2013, at 6:54 PM, Hannes Tschofenig <Hannes.Tschofenig@xxxxxxx> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > [I posted this question a little while ago to the WG chairs mailing list and got no response. > Maybe my question is too trivial but I thought I should try it on the ietf@xxxxxxxx list as well to > get some feedback.] > > Hi all, > > when concerns about the lack of interoperability surfaced mid last year > in the OAuth working group we (Derek, and myself) tried to figure out > whether we should schedule a face-to-face interop and/or to develop an > online test suite. We got in touch with Lucy Lynch (ISOC) and she helped > us to find developers to work with us on the test software. > > Roland Hedberg, one of the guys working on the project for OAuth > testing, presented his ongoing work in the OAuth working group, see > http://www.ietf.org/proceedings/86/slides/slides-86-oauth-2.pdf > > OAuth is a bit more complex since it involves more than two parties and > we were looking for a test framework that could be re-used to develop > the desired results more quickly. To our surprise we couldn't find > a test framework that we could easily re-use since most test frameworks > really focus on different types of tests. Of course, we might > have looked in the wrong direction. > > Here is how it works at the moment: > * Imagine you have developed an OAuth-based identity management server > (that contains an OAuth 2.0 authorization server) and it runs somewhere > on the Internet (or in your lab). You don't need to have access to the > source code to execute the tests. > * You download the scripts that Roland & Co had developed and configure > them. Of course you will have to create an account at your IdP as well. > * You run the test scripts against the authorization server and the > script plays the other OAuth 2.0 parties in the exchange. The script contains a number > of test cases (around 60+ at the moment) and determines whether the IdP > responds correctly in the exchanges. > > I know that these ideas have come up in other working groups in the past > already (such as in SCIM, which also has a test server up and > running). > > It would be interesting to hear what others have been doing and what > worked for you or what didn't. So it sounds like you are doing some sort of conformance testing... For SCTP we did a number of interoperability tests, which were face to face meetings and the people who were developing stacks we there. This events were always very helpful not only for improving the stacks but also for improving the IETF documents. I also developed a test tool for conformance testing based on some test descriptions provided by ETSI. However, it would have made sense to specify also some tests within the IETF. That can also help to clarify some protocol aspects and to focus on "common mistakes". Providing tests is a very good thing in my experience. Unfortunately, at the point we did this for SCTP, the IETF position was that testing isn't an objective. Maybe it is time to change that... Best regards Michael > > Ciao > Hannes > > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG/MacGPG2 v2.0.19 (Darwin) > Comment: GPGTools - http://gpgtools.org > > iQEcBAEBCgAGBQJRzb+vAAoJEGhJURNOOiAtZBwIAISKHjD7gv8irkL4yaBR31K8 > KLZCr/1n0n1OcXl3rE9MFOyA85hYNplZFd1giJLLgEX3UyofYXg/L2QOOLqtP0lT > JgnW2CvR0WWKfIT1iKjAwAodCVLsHF8MdPE4tl0LBlCeqhA1waj/oCLkBzZrrhhq > oWnZzP0I9nFdlSxV9EAHQ62RAxLUVQmBEqgMxl7A+iC9fGD8IhWSNSqqsaF0WOaB > 6bHdwCFLYYAyqKhiuJAo/f6YFGEzIbPgpHPGjwBZzBIjwP/EGiFnAliyF8WATHzF > RM+OWg6QASh1cNwzc0dbMcrcr1L1ve7amATMc4uPN7sRjhv0s62iguWfGRhQhHw= > =YT5M > -----END PGP SIGNATURE----- >