We have recently noticed that the "msvc-meson" test job in GitLab CI succeeds even if there are failures. This is somewhat puzzling because we use exactly the same command as we do on GitHub Actions, and there the jobs fail as exected. As it turns out, this is another weirdness of the GitLab CI hosted runner for Windows [1]: by default, even successful commands will not make the job fail. Interestingly though, this depends on what exactly the command is that you're running -- the MinGW-based job for example works alright and does fail as expected. The root cause here seems to be specific behaviour of PowerShell. The invocation of `ForEach-Object` does not bubble up any errors in case the invocation of `meson test` fails, and thus we don't notice the error. This is specific to executing the command in a loop: other build steps where we execute commands directly fail as expected. This is because the specific version of PowerShell that we use in the runner does not know about `PSNativeCommandUseErrorActionPreference` yet, which controls whether native commands like "meson.exe" honor the `ErrorActionPreference` variable. The preference has been introduced with PowerShell 7.3 and is default-enabled since PowerShell 7.4, but GitLab's hosted runners still seem to use PowerShell 5.1. Consequently, when tests fail, we won't bubble up the error at all from the loop and thus the job doesn't fail. This isn't an issue in other cases though where we execute native commands directly, as the GitLab runner knows to check the last error code after every command. The same thing doesn't seem to be an issue on GitHub Actions, most likely because it uses PowerShell 7.4. Curiously, the preference for `PSNativeCommandUseErrorActionPreference` is disabled there, but the jobs fail as expected regardless of that. It's puzzling, but I do not have enough PowerShell expertise to give a definitive answer as to why it works there. In any case, Meson 1.8 will likely get support for slicing tests [1], so we can eventually get rid of the whole PowerShell script. For now, work around the issue by explicitly exiting out of the loop with a non-zero error code if we see that Meson has failed. [1]: https://github.com/mesonbuild/meson/pull/14092 Signed-off-by: Patrick Steinhardt <ps@xxxxxx> --- Hi, this fix addresses the issue found and discussed in the thread at [1]. A failing test run can be seen at [2]. Thanks! Patrick [1]: <xmqqh64gg0pu.fsf@gitster.g> [2]: https://gitlab.com/gitlab-org/git/-/jobs/9262132042 --- .gitlab-ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 3f29181708f..a10bb8372d7 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -179,7 +179,7 @@ test:msvc-meson: - job: "build:msvc-meson" artifacts: true script: - - meson test -C build --list | Select-Object -Skip 1 | Select-String .* | Group-Object -Property { $_.LineNumber % $Env:CI_NODE_TOTAL + 1 } | Where-Object Name -EQ $Env:CI_NODE_INDEX | ForEach-Object { meson test -C build --no-rebuild --print-errorlogs $_.Group } + - meson test -C build --list | Select-Object -Skip 1 | Select-String .* | Group-Object -Property { $_.LineNumber % $Env:CI_NODE_TOTAL + 1 } | Where-Object Name -EQ $Env:CI_NODE_INDEX | ForEach-Object { meson test -C build --no-rebuild --print-errorlogs $_.Group; if (!$?) { exit $LASTEXITCODE } } parallel: 10 test:fuzz-smoke-tests: --- base-commit: 5a526e5e18ddb9a7dfc5a2967d21d6154df64a4f change-id: 20250226-b4-pks-gitlab-ci-fix-msvc-meson-tests-77a314732729
