On Fri, Dec 20, 2013 at 2:28 PM, Martin Gregorie <martin@xxxxxxxxxxxx> wrote: > On Fri, 2013-12-20 at 02:22 +0100, Frédéric Delanoy wrote: > >> Checking for regressions for every commit, given the amount of work >> needed, isn't feasible: there are a lot of windows programs in >> existence, so regressions inevitably happen... you can't test them all >> for every commit, and it can be very difficult to automate, if >> possible at all. >> > Is the regression test suite made available to devs so that they can > check their code changes before submitting them? If not, maybe it should > be. There are regressions tests for most of the DLLs in Wine, and code doesn't get committed if tests fail. Now granted they appeared fairly late in the history of Wine development (2003), after the "alpha period" (wine-yyyy-mm-dd releases) [Wine project began in 1993] There's also http://newtestbot.winehq.org/ so programmers can test their tests on a variety of Windows OSes, and http://test.winehq.org/data/ Also sometimes the same code works differently between different Windows releases (ex: 9x, 2000, vista), but Wine obviously can't implement all behaviours (though you can set the Windows version Wine mimics through winecfg). Now remember Wine purpose is to run Windows applications, not mimic a particular Windows version. >> There are a lot of unit tests in the code though, which are run for >> every commit. And FYI non-compiling code isn't committed... >> And only the project leader has commit rights. >> > I was assuming that the regression test suite would mainly consist of > full coverage for each API implemented by Wine, i.e. would cover corner > cases and errors as well as calls that should work, together with some > tests for common or known troublesome API call sequences, rather than > running actual applications. That's how it's done. Cf. http://wiki.winehq.org/UnitTestSuites. Though there are some limited support for automated Windows apps tests, AppInstall (cf. previous link) >> Kicking developers out for every regression... If one would do what you >> suggest, I guess you would be the only Wine developer remaining?!? >> Good way to kill a project... >> > Well, I would expect devs to have done pretty solid testing on anything > they submit and to include any new tests aloing with the code so the > tests can be included in the regression test suite. If somebody doesn't > do that and persists in not doing it despite getting suggestions and > advice on how to improve his testing, would you really want them on the > project? If someone sends patchs which fails tests repeatedly, be sure their patches would soon be redirected to /dev/null The maintainer, Alexandre Julliard, is actually enforcing strict rules for code inclusion, and even seasoned developers get their patches rejected from time to time (passing tests is a necessary, not sufficient condition), especially when a stable version is approaching. Note though that tests can only show the presence of bugs, not their absence. > You're producing code that, like Fedora or any of the 'unstable' Linux > releases, feeds into the commercial Crossover product, so at least some > of the standards and practises used in commercial code development > should apply. FYI, many of the core Wine developers are actually Codeweavers employees. >> If you want to avoid regressions at all costs, stay with stable branch >> (1.6.x). Use development versions at your own risk. >> > Noted. My comments apply to past experience with the versions that have > been packaged and released as part of RedHat Fedora upgrades which would > therefore seem to be the stable branch since 1.6.1 is the most recent > version to have been released as part of Fedora. Not necessarily. Can't speak about Fedora, but distribution packagers sometimes include stuff not present in vanilla wine (e.g. winepulseaudio)/ To get latest development Wine packages, one mostly has to rely on stuff like PPAs, except maybe for rolling releases like Gentoo. >> > (2) publishing a version that fails regression or has incomplete changes >> > testing is a really good way to destroy its reputation. Gnome 3 anybody? >> >> IIRC the issues with Gnome 3 was released as a stable version with a >> lot of bugs. Wine development releases ARE NOT stable versions. >> > Same case. I was getting Gnome 3 as part of Fedora upgrades. As you say, > it was buggy, as were the final releases of Gnome 2 that preceded it. > However, there I had a solution: move to XFCE, which I did. Me too actually ;) Although I got to XFCE when Ubuntu broke stuff with their Unity crao. >> Nobody forces you to upgrade your Wine version. You can keep an old >> Wine version for some apps, and even distinct versions on the same >> system (as a developer, this shouldn't be difficult for you). >> > Not entirely true. AFAIK there's no way to tell yum to avoid upgrading > Wine or any other package in the main Fedora repositories. You can always compile your own wine versions in their respective directories, and e.g. start wine from there. It's of course not that easy for plain users. >> Moreover, if you find a regression, please report it on >> http://bugs.winehq.org/. Not everybody runs the same applications as >> you do, again checking every app in existence is just unrealistic. >> > Fair comment. A problem we have also is to get users to perform regression tests (http://wiki.winehq.org/RegressionTesting). Some also tend to do stuff like use unsupported third-parties, like PlayOnLinux, copy native dlls, ..., which sometimes makes troubleshooting/but triaging a PITA. >> They do exist, and are very much enforced. Look e.g. on the wiki for more >> information. >> > I'm glad to hear it. > >> Feel free to suggest improvements on Wine development >> standards/procedures on the wine-devel mailing list, or on IRC >> (http://wiki.winehq.org/IRC). >> > Noted, though my Wine use is fairly limited: I chiefly use it to run > SPINE, which pulls down and displays NOTAMs. SPINE is about the only > Wine app I run on a weekly or daily basis. You might want to check the AppDB (http://appdb.winehq.org/) to see if it contains an entry for your specific app, and add one if necessary. Of course, good bug reports are also appreciated (http://bugs.winehq.org/ ; http://wiki.winehq.org/Bugs) > In addition I use it every > few months to check out new versions of glider navigation software > (XCSoar, LK8000 and SeeYou). These are normally run on WinCE 5/6 based > PNAs, but all have Windows versions for training, evaluation and > checking new versions of the data files they depend on (maps, restricted > airspace, turnpoints, etc. > > As this is fairly minority stuff it probably doesn't fit will with > WineDB. A small problem can hide a bigger issue. Don't hesitate to report bugs for that. If a bug isn't known, it's difficult to fix. > Best regards, > Martin > Regards, Frédéric
