On Mon, Jun 29, 2020 at 08:51:51PM +0200, Andrea Bolognani wrote:
> Instead of having static job definitions for FreeBSD and macOS,
> use a generic template for both and fill in the details that are
> actually different, such as the list of packages to install, in
> the GitLab CI job, right before calling cirrus-run.
>
> The target-specific information are provided by lcitool, so that
> keeping them up to date is just a matter of running the refresh
> script when necessary.
>
> Signed-off-by: Andrea Bolognani <abologna@xxxxxxxxxx>
> ---
> .gitlab-ci.yml | 36 ++++++++++++++-
> ci/cirrus/build.yml | 26 +++++++++++
> ci/cirrus/freebsd-12.yml.j2 | 73 -------------------------------
> ci/cirrus/libvirt-freebsd-12.vars | 7 +++
> ci/cirrus/libvirt-macos-1015.vars | 7 +++
> ci/cirrus/macos-1015.yml.j2 | 38 ----------------
> ci/cirrus/refresh | 22 ++++++++++
> 7 files changed, 97 insertions(+), 112 deletions(-)
> create mode 100644 ci/cirrus/build.yml
> delete mode 100644 ci/cirrus/freebsd-12.yml.j2
> create mode 100644 ci/cirrus/libvirt-freebsd-12.vars
> create mode 100644 ci/cirrus/libvirt-macos-1015.vars
> delete mode 100644 ci/cirrus/macos-1015.yml.j2
> create mode 100755 ci/cirrus/refresh
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index e6eb2f9905..5565750b7e 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -58,11 +58,35 @@ stages:
> # Jobs that we delegate to Cirrus CI because they require an operating
> # system other than Linux. These jobs will only run if the required
> # setup has been performed on the GitLab account (see ci/README.rst).
> +#
> +# The Cirrus CI configuration is generated by replacing target-specific
> +# variables in a generic template: some of these variables are provided
> +# when the GitLab CI job is defined, others are taken from a shell
> +# snippet generated using lcitool.
> +#
> +# Note that the $PATH environment variable has to be treated with
> +# special care, because we can't just override it at the GitLab CI job
> +# definition level or we risk breaking it completely.
Informative, thanks :). I just wish I didn't have to google what semantics the
various types of parameter expansion in Bash had (obviously not your fault).
> .cirrus_build_job_template: &cirrus_build_job_definition
> stage: builds
> image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
> script:
> - - cirrus-run ci/cirrus/$NAME.yml.j2
> + - source ci/cirrus/libvirt-$NAME.vars
> + - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
> + -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
> + -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
> + -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
> + -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
> + -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
> + -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
> + -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
> + -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
> + -e "s|[@]PKGS@|$PKGS|g"
> + -e "s|[@]MAKE@|$MAKE|g"
> + -e "s|[@]PYTHON@|$PYTHON|g"
> + <ci/cirrus/build.yml >ci/cirrus/$NAME.yml
> + - cat ci/cirrus/$NAME.yml
was ^this 'cat' just for debugging purposes or was that intended to be part of
the job output in production?
...
> diff --git a/ci/cirrus/refresh b/ci/cirrus/refresh
> new file mode 100755
> index 0000000000..b84910a645
> --- /dev/null
> +++ b/ci/cirrus/refresh
> @@ -0,0 +1,22 @@
> +#!/bin/sh
> +
> +if test -z "$1"
> +then
> + echo "syntax: $0 PATH-TO-LCITOOL"
> + exit 1
> +fi
> +
> +LCITOOL=$1
> +
> +if ! test -x "$LCITOOL"
> +then
> + echo "$LCITOOL is not executable"
> + exit 1
> +fi
> +
> +HOSTS=$($LCITOOL hosts | grep -E 'freebsd-12|macos')
> +
> +for host in $HOSTS
> +do
> + $LCITOOL dockerfile "$host" libvirt --variables >"$host.vars"
> +done
Overall, I prefer this to v1 because of the further level of abstraction this
introduces. However, the final format we're producing here is a YAML template
rather than a Dockerfile, so introducing just an option for the "dockerfile"
subcommand rather than a dedicated "cirrus-ci-file" (or similar) subcommand
doesn't feel completely right, especially from the long run perspective. At
some point we'd like to generate the whole gitlab-ci.yml file, so that one
would very likely get a dedicated subcommand as well - it should, as it's just
another "plugin" generator. If we expand beyond libvirt, say QEMU, we may need
to add a different CI provider plugin as well.
Regards,
Erik
