On Sat, 19 Mar 2005 20:14:55 -0600, Chris Adams <cmadams@xxxxxxxxxx> wrote: > How else do you run regression tests on an FTP client (for example)? By including a FTP server, of course. ;) But more seriously, .. you don't.. Or rather you don't put such tests in the same class of tests with tens of thousands of other tests which can happily be run with in a more limited environment. The idea being is that if you have a thousand packages with a thousand different testing requirements none of them will be tested at all, because everyone will disable the tests because they fail in on completely sane systems and the effort of sorting out which is a real failure and which is something silly is too much, especially for a package you're not familiar with. If instead you split tests into two classes, an extensive class with arbratarly tall requirements, and a basic class which should never fail you will likely get a lot more useful testing... and those that really care about a packages extensive tests will still undertake the additional effort to execute them. > The perl package that does network connections during "make test" (I've > only seen one IIRC) does it because the modules are network client > modules; the only real way to test them is to connect to network > servers. Well if anyone really cared they could override the socket calls with a testing shim... a lot of work for not much gain.... but it's exactly the sort of thing that should eventually be done for important infrastructure components that don't change often (libc stuff for example). Waiting for real users to encounter bugs in production is not really acceptable when it can be avoided via testing... But with the number of packages in a distro, and the complexity of the overall system... the myriad of potential interactions, we miss a lot of potential issues when regression testing is limited to the core developer running on his single system before issuing a new release. > Since OpenSSH includes both a server and a client, it can test both > against each other. And it would be fantastic for it to do that...
