This is a rebasing of v2 of js/ci-github-workflow-markup[1] on top of
my just-submitted v2 of my own CI changes[2] which make most of that
series's changes to ci/ unnecessary, as it'll get the split of "build"
and "test" output for free without the need for any special GitHub
workflow syntax.
That's because that series eliminates the monolithic:
- ci/run-build-and-tests.sh
In favor of:
- make
- make test
So what it did with markup to delineate those two is now done
automatically by focusing on the relevant "step" within a job that
failed.
A demo of a failing run for this series (with an intentionally broken
patch on top) can be seen at
https://github.com/avar/git/actions/runs/2041259003
As it shows the GitHub workflow syntax for test failures is retained,
e.g.:
https://github.com/avar/git/runs/5695803104?check_suite_focus=true
But for build failures the GitHub workflow syntax isn't needed
anymore, e.g. this job where we focus on the failing "make":
https://github.com/avar/git/runs/5695557199?check_suite_focus=true
Note that the UI of the latter is also better because we can see in
the "skipped" portion that we didn't run the "make test", since we're
now piggy-backing on the CI flow to show that, as opposed to in
js/ci-github-workflow-markup:
https://github.com/git-for-windows/git/runs/4822802185?check_suite_focus=true
The 5/6 change here (as seen in the range diff) the gets rid of a
limitation that js/ci-github-workflow-markup commented on being
unfortunate, i.e. we can now do more with the GitHub workflow syntax
as a result of not usign it for the "top-level" of "make" and "make
test".
1. https://lore.kernel.org/git/pull.1117.v2.git.1646130289.gitgitgadget@xxxxxxxxx/
2. https://lore.kernel.org/git/cover-v2-00.25-00000000000-20220325T182534Z-avarab@xxxxxxxxx
Johannes Schindelin (6):
ci: make it easier to find failed tests' logs in the GitHub workflow
tests: refactor --write-junit-xml code
test(junit): avoid line feeds in XML attributes
ci: optionally mark up output in the GitHub workflow
ci: use `--github-workflow-markup` in the GitHub workflow
ci: call `finalize_test_case_output` a little later
.github/workflows/main.yml | 20 +---
ci/lib.sh | 2 +-
ci/print-test-failures-github.sh | 35 +++++++
t/test-lib-functions.sh | 4 +-
t/test-lib-github-workflow-markup.sh | 50 ++++++++++
t/test-lib-junit.sh | 132 +++++++++++++++++++++++++++
t/test-lib.sh | 128 ++++----------------------
7 files changed, 244 insertions(+), 127 deletions(-)
create mode 100755 ci/print-test-failures-github.sh
create mode 100644 t/test-lib-github-workflow-markup.sh
create mode 100644 t/test-lib-junit.sh
Range-diff against v2:
1: db08b07c37a < -: ----------- ci: fix code style
2: 42ff3e170bf < -: ----------- ci/run-build-and-tests: take a more high-level view
3: bbbe1623257 ! 1: d88749c60c9 ci: make it easier to find failed tests' logs in the GitHub workflow
@@ Commit message
nested groupings in GitHub workflows' output).
Signed-off-by: Johannes Schindelin <johannes.schindelin@xxxxxx>
+ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx>
## .github/workflows/main.yml ##
@@ .github/workflows/main.yml: jobs:
+ shell: bash
- name: test
shell: bash
- run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10
+- run: . /etc/profile && make -C t -e
- - name: ci/print-test-failures.sh
- if: failure()
- shell: bash
- run: ci/print-test-failures.sh
++ run: . /etc/profile && make -C t -e || ci/print-test-failures-github.sh
- name: Upload failed tests' directories
if: failure() && env.FAILED_TEST_ARTIFACTS != ''
uses: actions/upload-artifact@v2
@@ .github/workflows/main.yml: jobs:
- env:
- NO_SVN_TESTS: 1
- run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10
+ shell: bash
+ - name: test
+ shell: bash
+- run: . /etc/profile && make -C t -e
- - name: ci/print-test-failures.sh
- if: failure()
- shell: bash
- run: ci/print-test-failures.sh
++ run: . /etc/profile && make -C t -e || ci/print-test-failures-github.sh
- name: Upload failed tests' directories
if: failure() && env.FAILED_TEST_ARTIFACTS != ''
uses: actions/upload-artifact@v2
@@ .github/workflows/main.yml: jobs:
- - uses: actions/checkout@v2
- - run: ci/install-dependencies.sh
- - run: ci/run-build-and-tests.sh
+ - run: ci/lib.sh --build
+ - run: make
+ - run: ci/lib.sh --test
+- - run: make test
++ - run: make test || ci/print-test-failures-github.sh
+ if: success()
- - run: ci/print-test-failures.sh
- if: failure()
- name: Upload failed tests' directories
if: failure() && env.FAILED_TEST_ARTIFACTS != ''
uses: actions/upload-artifact@v2
@@ .github/workflows/main.yml: jobs:
- - uses: actions/checkout@v1
- - run: ci/install-docker-dependencies.sh
- - run: ci/run-build-and-tests.sh
+ - run: ci/lib.sh --build
+ - run: make
+ - run: ci/lib.sh --test
+- - run: make test
++ - run: make test || ci/print-test-failures-github.sh
+ if: success() && matrix.vector.skip-tests != 'no'
- - run: ci/print-test-failures.sh
-- if: failure()
+- if: failure() && matrix.vector.skip-tests != 'no'
- name: Upload failed tests' directories
if: failure() && env.FAILED_TEST_ARTIFACTS != ''
uses: actions/upload-artifact@v1
- ## ci/lib.sh ##
-@@ ci/lib.sh: check_unignored_build_artifacts () {
- }
- }
-
-+handle_failed_tests () {
-+ return 1
-+}
+ ## ci/print-test-failures-github.sh (new) ##
+@@
++#!/bin/sh
+
- # GitHub Action doesn't set TERM, which is required by tput
- export TERM=${TERM:-dumb}
-
-@@ ci/lib.sh: then
- CI_JOB_ID="$GITHUB_RUN_ID"
- CC="${CC:-gcc}"
- DONT_SKIP_TAGS=t
++. ${0%/*}/lib-ci-type.sh
++
++set -e
++
++case "$CI_TYPE" in
++github-actions)
+ handle_failed_tests () {
+ mkdir -p t/failed-test-artifacts
+ echo "FAILED_TEST_ARTIFACTS=t/failed-test-artifacts" >>$GITHUB_ENV
@@ ci/lib.sh: then
+ done
+ return 1
+ }
-
- cache_dir="$HOME/none"
-
-
- ## ci/run-build-and-tests.sh ##
-@@ ci/run-build-and-tests.sh: esac
- make
- if test -n "$run_tests"
- then
-- make test
-+ make test ||
-+ handle_failed_tests
- fi
- check_unignored_build_artifacts
-
-
- ## ci/run-test-slice.sh ##
-@@ ci/run-test-slice.sh: esac
-
- make --quiet -C t T="$(cd t &&
- ./helper/test-tool path-utils slice-tests "$1" "$2" t[0-9]*.sh |
-- tr '\n' ' ')"
-+ tr '\n' ' ')" ||
++ ;;
++*)
++ echo "Unhandled CI type: $CI_TYPE" >&2
++ exit 1
++ ;;
++esac
++
+handle_failed_tests
-
- check_unignored_build_artifacts
4: f72254a9ac6 < -: ----------- ci/run-build-and-tests: add some structure to the GitHub workflow output
5: 9eda6574313 ! 2: ad1e1465a81 tests: refactor --write-junit-xml code
@@ Commit message
--color-moved-ws=allow-indentation-change <commit>`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@xxxxxx>
+ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx>
## t/test-lib-junit.sh (new) ##
@@
6: c8b240af749 ! 3: fc96e5b7296 test(junit): avoid line feeds in XML attributes
@@ Commit message
incorrect, so let's fix it.
Signed-off-by: Johannes Schindelin <johannes.schindelin@xxxxxx>
+ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx>
## t/test-lib-junit.sh ##
@@ t/test-lib-junit.sh: finalize_test_case_output () {
7: 15f199e810e ! 4: 429c256ac62 ci: optionally mark up output in the GitHub workflow
@@ Commit message
to test cases that were expected to fail but didn't.
Signed-off-by: Johannes Schindelin <johannes.schindelin@xxxxxx>
+ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx>
## t/test-lib-functions.sh ##
@@ t/test-lib-functions.sh: test_verify_prereq () {
8: 91ea54f36c5 ! 5: 72058db67b0 ci: use `--github-workflow-markup` in the GitHub workflow
@@ Metadata
## Commit message ##
ci: use `--github-workflow-markup` in the GitHub workflow
- This makes the output easier to digest.
+ [Ævar: due to re-structuring on top of my series the {begin,end}_group
+ in CI isn't needed at all to get "group" output for the test
+ suite. This commit includes the now-squashed "ci/run-build-and-tests:
+ add some structure to the GitHub workflow output":]
- Note: since workflow output currently cannot contain any nested groups
- (see https://github.com/actions/runner/issues/802 for details), we need
- to remove the explicit grouping that would span the entirety of each
- failed test script.
+ The current output of Git's GitHub workflow can be quite confusing,
+ especially for contributors new to the project.
+
+ To make it more helpful, let's introduce some collapsible grouping.
+ Initially, readers will see the high-level view of what actually
+ happened (did the build fail, or the test suite?). To drill down, the
+ respective group can be expanded.
+
+ Note: sadly, workflow output currently cannot contain any nested groups
+ (see https://github.com/actions/runner/issues/802 for details),
+ therefore we take pains to ensure to end any previous group before
+ starting a new one.
+
+ [Ævar: The above comment isn't true anymore, as that limitation has
+ been removed by basing this on my patches to run "make" and "make
+ test" directly from the top-level of main.yml.
+
+ Those are now effectively their own "group", effectively giving this
+ stage another group "level" to use. This means that the equivalent of
+ "make test" won't be on the same level as an individual test failure.
+
+ We no longer take any pains to ensure balanced group output as a
+ result (which was a caveat the previous ci/lib.sh implementation had
+ to deal with., We just need to "cat" the generated *.markup]
Signed-off-by: Johannes Schindelin <johannes.schindelin@xxxxxx>
+ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx>
## ci/lib.sh ##
-@@ ci/lib.sh: then
+@@ ci/lib.sh: MAKEFLAGS="DEVELOPER=$DEVELOPER SKIP_DASHED_BUILT_INS=$SKIP_DASHED_BUILT_INS"
+ case "$CI_TYPE" in
+ github-actions)
+ setenv --test GIT_PROVE_OPTS "--timer --jobs 10"
+- GIT_TEST_OPTS="--verbose-log -x"
++ GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
+ MAKEFLAGS="$MAKEFLAGS --jobs=10"
+ test Windows != "$RUNNER_OS" ||
+ GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
+
+ ## ci/print-test-failures-github.sh ##
+@@ ci/print-test-failures-github.sh: github-actions)
test_name="${test_exit%.exit}"
test_name="${test_name##*/}"
printf "\\e[33m\\e[1m=== Failed test: ${test_name} ===\\e[m\\n"
-- group "Failed test: $test_name" cat "t/test-results/$test_name.out"
+- cat "t/test-results/$test_name.out"
+ cat "t/test-results/$test_name.markup"
trash_dir="t/trash directory.$test_name"
cp "t/test-results/$test_name.out" t/failed-test-artifacts/
-@@ ci/lib.sh: then
- cache_dir="$HOME/none"
-
- export GIT_PROVE_OPTS="--timer --jobs 10"
-- export GIT_TEST_OPTS="--verbose-log -x"
-+ export GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
- MAKEFLAGS="$MAKEFLAGS --jobs=10"
- test windows != "$CI_OS_NAME" ||
- GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
9: be2a83f5da3 ! 6: 1d2b94436fc ci: call `finalize_test_case_output` a little later
@@ Commit message
test case.
Signed-off-by: Johannes Schindelin <johannes.schindelin@xxxxxx>
+ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx>
## t/test-lib.sh ##
@@ t/test-lib.sh: trap '{ code=$?; set +x; } 2>/dev/null; exit $code' INT TERM HUP
--
2.35.1.1517.g20a06c426a7
