On Mon, Mar 23, 2020 at 12:01:49PM +0000, Daniel P. Berrangé wrote: > On Mon, Mar 23, 2020 at 12:50:45PM +0100, Erik Skultety wrote: > > On Fri, Mar 20, 2020 at 03:31:24PM +0000, Daniel P. Berrangé wrote: > > > On Fri, Mar 20, 2020 at 04:18:58PM +0100, Erik Skultety wrote: > > > > On Fri, Mar 20, 2020 at 02:59:36PM +0000, Daniel P. Berrangé wrote: > > > > > On Fri, Mar 20, 2020 at 03:52:15PM +0100, Erik Skultety wrote: > > > > > > On Tue, Mar 10, 2020 at 10:09:44AM +0000, Daniel P. Berrangé wrote: > > > > > > > With GitLab CI aiming to replace Jenkins and Travis for CI purposes, we > > > > > > > need to expand the coverage to include native builds. This patch adds > > > > > > > all the jobs currently run in Travis. Compared to Jenkins we obviously > > > > > > > miss the FreeBSD jobs, but also Debian 10 and Fedora 30, but we gain the > > > > > > > Ubuntu 1804 job as a substitute for Debian. > > > > > > > > > > > > > > Signed-off-by: Daniel P. Berrangé <berrange@xxxxxxxxxx> > > > > > > > --- > > > > > > > .gitlab-ci.yml | 41 +++++++++++++++++++++++++++++++++++++++++ > > > > > > > 1 file changed, 41 insertions(+) > > > > > > > > > > > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > > > > > > > index e28ec584ea..3e15d08d17 100644 > > > > > > > --- a/.gitlab-ci.yml > > > > > > > +++ b/.gitlab-ci.yml > > > > > > > @@ -1,7 +1,39 @@ > > > > > > > stages: > > > > > > > - website > > > > > > > + - native_build > > > > > > > - cross_build > > > > > > > > > > > > > > + > > > > > > > +.native_build_job_template: &native_build_job_definition > > > > > > > + stage: native_build > > > > > > > + script: > > > > > > > + - mkdir build > > > > > > > + - cd build > > > > > > > + - ../autogen.sh $CONFIGURE_OPTS || (cat config.log && exit 1) > > > > > > > + - make -j $(getconf _NPROCESSORS_ONLN) syntax-check > > > > > > > + - make -j $(getconf _NPROCESSORS_ONLN) distcheck > > > > > > > > > > > > I think ^this should more closely follow what we have in the lcitool playbooks, > > > > > > e.g. start with: > > > > > > - rm -rf build > > > > > > > > > > The source tree is already pristine because this is always executed in > > > > > a fresh container environment, so there's nothing that will need deleting. > > > > > > > > Right, but my point was that if e.g. we introduce a FreeBSD builder, we'd want > > > > to reference the same job template in which case the directory will exist. > > > > > > I've not looked in great detail about how gitlab runners work, but something > > > on the runner needs to be responsible for checking out the git repo before > > > any of this gitlab-ci.yml scrpit runs. > > > > > > IMHO that can ensure it is "git clean -xdf" so that its state matches what > > > we see with the built-in shared runners build environment. > > > > This is achieved by the GIT_CLEAN_FLAGS which, if unspecified, defaults to > > -ffdx, so we're covered there. > > > > One more thing that crossed my mind and may speedup the builds a bit - some > > time ago I proposed a patch against libvirt-jenkins-ci to enable shallow > > cloning which was rejected because of compelling reasons. However, for gitlab > > purposes, we can't really do any debugging on the shared runners, so I believe > > we may want to apply shallow cloning there, especially since libvirt is quite a > > large repo. Gitlab claims [1] the default strategy to be 'fetch' if the > > worktree can be found on the disk (this is when used with the 'docker' executor > > which should be the case with shared runners). What I couldn't find is how long > > are these worktrees cached which could therefore be a factor and we might want > > to favour the shallow clones instead. > > > > [1] https://docs.gitlab.com/ee/ci/large_repositories/#git-strategy > > I wonder if GitLab has the same problem with shallow cloning as we hit > before. I can't remember if it was with Travis or Jenkins now, but there > was a race where it was schedule the job, but before the git repo was > checked out someone would push another series. The shallow clone would > thus get the commit of the newly pushed series, and would not have enough > history to get to the commit that actually needed testing. > > I guess we could get a win even if we set the clone depth to something > fairly large like 100 commits of history. Google didn't return anything relevant in this regard, except of a few resources: [1] https://docs.gitlab.com/ce/ci/pipelines/settings.html#git-shallow-clone Although ^this doesn't apply to libvirt [2] https://gitlab.com/gitlab-org/gitlab-runner/issues/3460 ^This I believe would be the long term solution, but I wouldn't hold my breath I'm okay with anything shorter than the full history, branches and tags.
