In preceding commits the --github-workflow-markup output was made the default under GitHub CI. There's a few outstanding issues with that new output target however: A. As noted in [1] it's much slower in some common cases, i.e. the page rendering time (under Firefox & Chrome debugging) goes from ~20s to on the order of 60-80s (less pronounced when not debugging, but still in that ballpark). See also [2] for tests on an earlier iteration of the series. B. Preceding commits in this series to omit non-failing test output may have mitigated that somewhat, i.e. skipping "ok" and "skip" tests in the *.markup files (although [1] is a benchmark with that "ok" change). Per [3] there are some CI users that prefer to see only the full verbose output, i.e. we can't in the general case assume that we can emit only the trace output for the failing test, and that's going to give us the information we need to debug the test. It might be failing because of earlier setup, or from trace output outside of any test (i.e. in the main body of the file). Users such as [3] could still just consult the *.out output, but then we'd make them download the archive instead of viewing the output in the web UX. C. Per B above and [3] an earlier change in this series modified the *.out output to change the *.out output inadvertently while trying to modify the *.markup output. See the commit message of "ci(github): avoid printing test case preamble twice" earlier in this series. It thus make sense for now to make the "ci/print-test-failures.sh" output be the "raw" one. I.e. the *.out output we'd get before --github-workflow-markup was implemented. To make it use ci/print-test-failures.sh we can create a "ci/config/print-test-failures-output-type" script on the "ci-config" branch that does e.g.: #!/bin/sh echo github That script will (like the existing "check-ref") get the ref as an argument, so we can in the future turn this on experimentally for "seen", and flip the default in the future. The script can also emit nothing to get whatever our idea is of the default (currently "raw"). Note the corresponding change to ci/print-test-failures.sh, i.e. we'll do the right thing vis-a-vis what we "zoom in" to depending on the setting. If it's "raw" we'd like a failed "make test" to "zoom in" to that step, if it's "github" we'd like the to open the subsequent "ci/print-test-failures.sh" step instead. 1. https://lore.kernel.org/git/220523.86ee0kzjix.gmgdl@xxxxxxxxxxxxxxxxxxx/ 2. https://lore.kernel.org/git/220222.86tucr6kz5.gmgdl@xxxxxxxxxxxxxxxxxxx/ 3. https://lore.kernel.org/git/20210309175249.GE3590451@xxxxxxxxxx/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx> --- .github/workflows/main.yml | 24 ++++++++++++++++++++++++ ci/lib.sh | 14 +++++++++++++- ci/print-test-failures.sh | 7 +++++-- 3 files changed, 42 insertions(+), 3 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 6835e942280..4195f37c1ba 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -8,6 +8,7 @@ jobs: runs-on: ubuntu-latest outputs: enabled: ${{ steps.check-ref.outputs.enabled }}${{ steps.skip-if-redundant.outputs.enabled }} + print-test-failures-output-type: ${{ steps.print-test-failures-output-type.outputs.type }} steps: - name: try to clone ci-config branch run: | @@ -22,6 +23,21 @@ jobs: config-repo && cd config-repo && git checkout HEAD -- ci/config || : ignore + - id: print-test-failures-output-type + name: check what output type ci/print-test-failures.sh uses + run: | + type_default=raw + type=$type_default + + if test -x config-repo/ci/config/print-test-failures-output-type + then + type=$(config-repo/ci/config/print-test-failures-output-type '${{ github.ref }}') + if test -z "$type" + then + type=$type_default + fi + fi + echo "::set-output name=type::$type" - id: check-ref name: check whether CI is enabled for ref run: | @@ -124,6 +140,8 @@ jobs: - uses: git-for-windows/setup-git-for-windows-sdk@v1 - run: ci/lib.sh --test shell: bash + env: + GIT_CI_PTF_OUTPUT_TYPE: ${{needs.ci-config.outputs.print-test-failures-output-type}} - name: select tests run: . /etc/profile && ci/select-test-slice.sh ${{matrix.nr}} 10 shell: bash @@ -213,6 +231,8 @@ jobs: run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz - run: ci/lib.sh --test shell: bash + env: + GIT_CI_PTF_OUTPUT_TYPE: ${{needs.ci-config.outputs.print-test-failures-output-type}} - name: select tests run: . /etc/profile && ci/select-test-slice.sh ${{matrix.nr}} 10 shell: bash @@ -265,6 +285,8 @@ jobs: - run: ci/lib.sh --build - run: make - run: ci/lib.sh --test + env: + GIT_CI_PTF_OUTPUT_TYPE: ${{needs.ci-config.outputs.print-test-failures-output-type}} - run: make test if: success() - run: ci/print-test-failures.sh @@ -302,6 +324,8 @@ jobs: - run: make - run: ci/lib.sh --test if: success() && matrix.vector.skip-tests != 'yes' + env: + GIT_CI_PTF_OUTPUT_TYPE: ${{needs.ci-config.outputs.print-test-failures-output-type}} - run: make test if: success() && matrix.vector.skip-tests != 'yes' - run: ci/print-test-failures.sh diff --git a/ci/lib.sh b/ci/lib.sh index 422399b1305..b148d3f209f 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -95,6 +95,7 @@ if test -n "$GITHUB_ENV" then echo "CONFIG: GITHUB_ENV=$GITHUB_ENV" >&2 fi +echo "CONFIG: GIT_CI_PTF_OUTPUT_TYPE=$GIT_CI_PTF_OUTPUT_TYPE" >&2 echo >&2 # Helper functions @@ -189,7 +190,18 @@ MAKEFLAGS="$MAKEFLAGS SKIP_DASHED_BUILT_INS=$SKIP_DASHED_BUILT_INS" case "$CI_TYPE" in github-actions) setenv --test GIT_PROVE_OPTS "--timer --jobs $NPROC" - GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup" + GIT_TEST_OPTS="--verbose-log -x" + if test -n "$GIT_CI_PTF_OUTPUT_TYPE" + then + # For later use in ci/print-test-failures.sh + setenv --test GIT_CI_PTF_OUTPUT_TYPE "$GIT_CI_PTF_OUTPUT_TYPE" + + case "$GIT_CI_PTF_OUTPUT_TYPE" in + github) + GIT_TEST_OPTS="$GIT_TEST_OPTS --github-workflow-markup" + ;; + esac + fi test Windows != "$RUNNER_OS" || GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS" setenv --test GIT_TEST_OPTS "$GIT_TEST_OPTS" diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh index 72fae2b0cc4..1e0738e0058 100755 --- a/ci/print-test-failures.sh +++ b/ci/print-test-failures.sh @@ -9,10 +9,13 @@ set -e . ${0%/*}/lib-tput.sh github_workflow_markup=auto -case "$CI_TYPE" in -github-actions) +case "$GIT_CI_PTF_OUTPUT_TYPE" in +github) github_workflow_markup=t ;; +raw) + github_workflow_markup= + ;; esac while test $# != 0 -- 2.36.1.1045.gf356b5617dd
