On Mon, Feb 10, 2025 at 01:57:51PM +0100, Peter Krempa wrote: > On Mon, Feb 10, 2025 at 12:58:06 +0100, Tim Wiederhake wrote: > > On Mon, 2025-02-10 at 12:44 +0100, Peter Krempa wrote: > > > On Mon, Feb 10, 2025 at 11:30:24 +0100, Peter Krempa wrote: > > > > On Mon, Feb 10, 2025 at 11:25:55 +0100, Tim Wiederhake wrote: > > > > > meson's "test()" function provides a "should_fail: bool" argument > > > > > that > > > > > checks for a command to exit with a non-zero exit code instead of > > > > > the usual > > > > > zero exit code to signal success. If the program under test does > > > > > so, it is > > > > > recorded as "EXPECTEDFAIL" instead of "OK". While there is an > > > > > argument to be > > > > > made that the program under test failed as expected, the test in > > > > > itself is > > > > > successful and should be recorded as such. > > > > > > > > Could you please elaborate what the problem is? The wording of the > > > > > > I've re-read this and it ineed seems that just the wording is the > > > issue. > > > > > > I'm not sold that we need a script to bypass the return value just > > > for > > > meson to say "OK". > > > > > > > Expected failures or "xfail" results were used as a tool to write a > > test for a bug before committing the fix. You'd write the test first, > > mark it as "xfail" to make the issue visible without making the entire > > test suite fail. The commit that fixes the issue would then remove the > > "xfail" marker and demonstrate that it actually fixes the issue, i.e. > > makes the test pass. > > > > Using expected-fail is not the right tool to handle a test case that is > > always expected to exit with a non-zero exit code. As I said, the test > > is "program exits non-zero", and that test passes. There is no failure > > in the test, expected or otherwise. > > Meson developers seem to think otherwise: > > https://github.com/mesonbuild/meson/pull/4466#issuecomment-435688173 > > > This is not actually true. Using expected failures to mark bugs is an > > antipattern and you should not do that. Tests that should fail are > > for things like trying to run an executable with known invalid input. > > It must return an error in this case rather than silently accept it. > > The documentation also simply states the fact that it's for tests > returning non-zero: > > should_fail bool > > when true the test is considered passed if the executable returns a non-zero return value (i.e. reports an error) > > Which is why I chose to do this. > > Based on the above I'm still not persuaded we want this. In case you > still want to go forward I'd sugest you revert e57ce7fb452 instead of > adding a workaround script. IMHO based on the meson PR comment above, the current code is working as Meson developers intended and so we don't need todo anything. The only issue I see here is that the meson docs should be improved to better explain their intentions around usage pattersn for the 'should_fail' flag, as it is ambiguous/open to differing interpretations. They should expand the docs per that PR comment explanation. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
