Hi,
this is the third version of this patch series that adds GitLab CI
definitions to the Git project. Please refer to the cover letter for v1
of this series [1] for the motivation and intent -- I won't repeat it
here as it's a bit on the longer side.
Changes compared to v2:
- Patch 5: This is a new preparatory step to unify the setup of some
environment variables. It also fixes some smallish issues, like
e.g. the fact that some envvars were set _after_ the export.
- Patch 6: Another new preparatory step. It makes our infra around
certain helper functions that have the intent to interact with the
project repository, e.g. to cache good trees. We now detect the
case when there is no Git or when the project is not a Git repo
and bail out gracefully.
- Patch 7: The last new preparatory patch. Installs a bunch of
dependencies which are required for the test runtime in the Alpine
based job. This increases test coverage.
- Patch 8: Several smaller improvements:
- Added a note why we install sudo and shadow in linux-musl now.
- Fixed an issue where the HOME directory was part of the
project directory, and thus Git complained about newly added
untracked files.
- Added the "pedantic" job that does a pednatic compilation on
Fedora.
- Added more test time dependencies.
- Made the "echo -e" invocation portable by using printf
instead.
What I didn't address yet is a suggestion by Phillip, namely to unify
the logic that sets up unprivileged builds. I don't have the infra
available to test any such change that would ultimately also impact
GitHub Workflows and Azure Pipelines and thus do not feel comfortable
to refactor this. I agree with the suggestion though, so I propose to
rather handle it at a later point in time.
A test run of this patch series can be found at [2].
Thanks!
Patrick
[1]: <cover.1698305961.git.ps@xxxxxx>
[2]: https://gitlab.com/gitlab-org/git/-/pipelines/1054750795
Patrick Steinhardt (8):
ci: reorder definitions for grouping functions
ci: make grouping setup more generic
ci: group installation of Docker dependencies
ci: split out logic to set up failed test artifacts
ci: unify setup of some environment variables
ci: squelch warnings when testing with unusable Git repo
ci: install test dependencies for linux-musl
ci: add support for GitLab CI
.gitlab-ci.yml | 53 +++++++++
ci/install-docker-dependencies.sh | 22 +++-
ci/lib.sh | 187 +++++++++++++++++++++---------
ci/print-test-failures.sh | 6 +
t/lib-httpd.sh | 3 +-
5 files changed, 215 insertions(+), 56 deletions(-)
create mode 100644 .gitlab-ci.yml
Range-diff against v2:
1: 4eb9cfc816b = 1: ef44ed5c3b1 ci: reorder definitions for grouping functions
2: 85617ef8577 = 2: 77798fa7a7a ci: make grouping setup more generic
3: 57bbc50e3dc = 3: 4542bd38dc2 ci: group installation of Docker dependencies
4: 5ab11d5236d = 4: 5fdda7fd83f ci: split out logic to set up failed test artifacts
-: ----------- > 5: 6af0075fd87 ci: unify setup of some environment variables
-: ----------- > 6: 78d863bf24e ci: squelch warnings when testing with unusable Git repo
-: ----------- > 7: f150d61a1ce ci: install test dependencies for linux-musl
5: 37a507e9b25 ! 8: 5272d66d9f1 ci: add support for GitLab CI
@@ Commit message
This commit introduces the integration into our regular CI scripts so
that most of the setup continues to be shared across all of the CI
- solutions.
+ solutions. Note that as the builds on GitLab CI run as unprivileged
+ user, we need to pull in both sudo and shadow packages to our Alpine
+ based job to set this up.
[1]: https://gitlab.com/gitlab-org/git
@@ .gitlab-ci.yml (new)
+ before_script:
+ - ./ci/install-docker-dependencies.sh
+ script:
-+ - useradd builder --home-dir "${CI_PROJECT_DIR}"
++ - useradd builder --create-home
+ - chown -R builder "${CI_PROJECT_DIR}"
+ - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh
+ after_script:
@@ .gitlab-ci.yml (new)
+ - jobname: linux-asan-ubsan
+ image: ubuntu:latest
+ CC: clang
++ - jobname: pedantic
++ image: fedora:latest
+ - jobname: linux-musl
+ image: alpine:latest
+ artifacts:
@@ ci/install-docker-dependencies.sh: linux32)
;;
linux-musl)
- apk add --update build-base curl-dev openssl-dev expat-dev gettext \
-+ apk add --update git shadow sudo build-base curl-dev openssl-dev expat-dev gettext \
- pcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null
++ apk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \
+ pcre2-dev python3 musl-libintl perl-utils ncurses \
+ apache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null
;;
+linux-*)
+ apt update -q &&
-+ apt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}}
++ apt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \
++ libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \
++ perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \
++ libdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \
++ apache2 cvs cvsps gnupg libcgi-pm-perl subversion
+ ;;
pedantic)
dnf -yq update >/dev/null &&
@@ ci/lib.sh: then
+then
+ begin_group () {
+ need_to_end_group=t
-+ echo -e "\e[0Ksection_start:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K$1"
++ printf "\e[0Ksection_start:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K$1\n"
+ trap "end_group '$1'" EXIT
+ set -x
+ }
@@ ci/lib.sh: then
+ test -n "$need_to_end_group" || return 0
+ set +x
+ need_to_end_group=
-+ echo -e "\e[0Ksection_end:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K"
++ printf "\e[0Ksection_end:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K\n"
+ trap - EXIT
+ }
else
begin_group () { :; }
end_group () { :; }
@@ ci/lib.sh: then
- MAKEFLAGS="$MAKEFLAGS --jobs=10"
- test windows != "$CI_OS_NAME" ||
- GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
+ cache_dir="$HOME/none"
+
+ GIT_TEST_OPTS="--github-workflow-markup"
+elif test true = "$GITLAB_CI"
+then
+ CI_TYPE=gitlab-ci
@@ ci/lib.sh: then
+ case "$CI_JOB_IMAGE" in
+ macos-*)
+ CI_OS_NAME=osx;;
-+ alpine:*|ubuntu:*)
++ alpine:*|fedora:*|ubuntu:*)
+ CI_OS_NAME=linux;;
+ *)
+ echo "Could not identify OS image" >&2
@@ ci/lib.sh: then
+ cache_dir="$HOME/none"
+
+ runs_on_pool=$(echo "$CI_JOB_IMAGE" | tr : -)
-+
-+ export GIT_PROVE_OPTS="--timer --jobs $(nproc)"
-+ export GIT_TEST_OPTS="--verbose-log -x"
-+ MAKEFLAGS="$MAKEFLAGS --jobs=$(nproc)"
-+ test windows != "$CI_OS_NAME" ||
-+ GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
else
echo "Could not identify CI type" >&2
env >&2
--
2.42.0
Attachment:
signature.asc
Description: PGP signature
