> On 27 Dec 2017, at 17:49, SZEDER Gábor <szeder.dev@xxxxxxxxx> wrote:
>
> Travis CI dutifully builds and tests each new branch tip, even if its
> tree has previously been successfully built and tested. This happens
> often enough in contributors' workflows, when a work-in-progress
> branch is rebased changing e.g. only commit messages or the order or
> number of commits while leaving the resulting code intact, and is then
> pushed to a Travis CI-enabled GitHub fork.
>
> This is wasting Travis CI's resources and is sometimes scary-annoying
> when the new tip commit with a tree identical to the previous,
> successfully tested one is suddenly reported in red, because one of
> the OSX build jobs happened to exceed the time limit yet again.
>
> So extend our Travis CI build scripts to skip building commits whose
> trees have previously been successfully built and tested. Use the
> Travis CI cache feature to keep a record of the object names of trees
> that tested successfully, in a plain and simple flat text file, one
> line per tree object name. Append the current tree's object name at
> the end of every successful build job to this file, along with a bit
> of additional info about the build job (commit object name, Travis CI
> job number and id). Check, using a simple grep invocation, in each
> build job whether the current commit's tree is already in there, and
> skip the build if it is. Include a message in the skipped build job's
> trace log, containing the URL to the build job successfully testing
> that tree for the first time and instructions on how to force a
> re-build. Catch the case when a build job, which successfully built
> and tested a particular tree for the first time, is restarted and omit
> the URL of the previous build job's trace log, as in this case it's
> the same build job and the trace log has just been overwritten.
>
> Using an ever-growing flat text file might seem like asking for
> trouble on the long run, but it's perfectly adequate for this purpose.
> Contributors' topic branches are short-lived in general, so this file
> won't grow large enough to cause any issues. Grepping through several
> tens of thousands such lines is sufficiently fast, so not even
> git/git's forever living integration branches will cause scalability
> issues with the current rate of ~1 push/day for a couple of decades.
> And even if we reach the point that this file grows too big, the
> caches can be deleted on Travis CI's web interface.
One more thing:
Maybe we could delete "$HOME/travis-cache/good-trees" if the file
has more than 1000 lines *before* we add a new tree?
Or we use something like this to cap the file:
echo "$(tail -1000 $HOME/travis-cache/good-trees)" > $HOME/travis-cache/good-trees
I agree that the "always growing problem" is not a big one
but an approach like the one above would avoid any discussion
for sure.
Thanks,
Lars
>
> Note: this won't kick in if two identical trees are on two different
> branches, because Travis CI caches are not shared between build jobs
> of different branches.
>
> Signed-off-by: SZEDER Gábor <szeder.dev@xxxxxxxxx>
> ---
> ci/lib-travisci.sh | 43 +++++++++++++++++++++++++++++++++++++++++++
> ci/run-linux32-docker.sh | 2 ++
> ci/run-static-analysis.sh | 2 ++
> ci/run-tests.sh | 2 ++
> ci/run-windows-build.sh | 2 ++
> ci/test-documentation.sh | 2 ++
> 6 files changed, 53 insertions(+)
>
> diff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh
> index 348fe3c3c..05e73123f 100755
> --- a/ci/lib-travisci.sh
> +++ b/ci/lib-travisci.sh
> @@ -21,6 +21,48 @@ skip_branch_tip_with_tag () {
> fi
> }
>
> +good_trees_file="$HOME/travis-cache/good-trees"
> +
> +# Save some info about the current commit's tree, so we can skip the build
> +# job if we encounter the same tree again and can provide a useful info
> +# message.
> +save_good_tree () {
> + echo "$(git rev-parse $TRAVIS_COMMIT^{tree}) $TRAVIS_COMMIT $TRAVIS_JOB_NUMBER $TRAVIS_JOB_ID" >>"$good_trees_file"
> +}
> +
> +# Skip the build job if the same tree has already been built and tested
> +# successfully before (e.g. because the branch got rebased, changing only
> +# the commit messages).
> +skip_good_tree () {
> + if ! good_tree_info="$(grep "^$(git rev-parse $TRAVIS_COMMIT^{tree}) " "$good_trees_file")"
> + then
> + # haven't seen this tree yet; continue the build job
> + return
> + fi
> +
> + echo "$good_tree_info" | {
> + read tree prev_good_commit prev_good_job_number prev_good_job_id
> +
> + if test "$TRAVIS_JOB_ID" = "$prev_good_job_id"
> + then
> + cat <<-EOF
> + Skipping build job for commit $TRAVIS_COMMIT.
> + This commit has already been built and tested successfully by this build job.
> + To force a re-build delete the branch's cache and then hit 'Restart job'.
> + EOF
> + else
> + cat <<-EOF
> + Skipping build job for commit $TRAVIS_COMMIT.
> + This commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.
> + The log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id
> + To force a re-build delete the branch's cache and then hit 'Restart job'.
> + EOF
> + fi
> + }
> +
> + exit 0
> +}
> +
> # Set 'exit on error' for all CI scripts to let the caller know that
> # something went wrong.
> # Set tracing executed commands, primarily setting environment variables
> @@ -28,6 +70,7 @@ skip_branch_tip_with_tag () {
> set -ex
>
> skip_branch_tip_with_tag
> +skip_good_tree
>
> if test -z "$jobname"
> then
> diff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh
> index 3a8b2ba42..870a41246 100755
> --- a/ci/run-linux32-docker.sh
> +++ b/ci/run-linux32-docker.sh
> @@ -22,3 +22,5 @@ docker run \
> --volume "${HOME}/travis-cache:/tmp/travis-cache" \
> daald/ubuntu32:xenial \
> /usr/src/git/ci/run-linux32-build.sh $(id -u $USER)
> +
> +save_good_tree
> diff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh
> index 68dd0f080..fe4ee4e06 100755
> --- a/ci/run-static-analysis.sh
> +++ b/ci/run-static-analysis.sh
> @@ -6,3 +6,5 @@
> . ${0%/*}/lib-travisci.sh
>
> make coccicheck
> +
> +save_good_tree
> diff --git a/ci/run-tests.sh b/ci/run-tests.sh
> index ccdfc2b9d..eb5ba4058 100755
> --- a/ci/run-tests.sh
> +++ b/ci/run-tests.sh
> @@ -7,3 +7,5 @@
>
> ln -s $HOME/travis-cache/.prove t/.prove
> make --quiet test
> +
> +save_good_tree
> diff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh
> index 86999268a..d99a180e5 100755
> --- a/ci/run-windows-build.sh
> +++ b/ci/run-windows-build.sh
> @@ -99,3 +99,5 @@ gfwci "action=log&buildId=$BUILD_ID" | cut -c 30-
>
> # Set exit code for TravisCI
> test "$RESULT" = "success"
> +
> +save_good_tree
> diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh
> index 7a0a848e8..3d62e6c95 100755
> --- a/ci/test-documentation.sh
> +++ b/ci/test-documentation.sh
> @@ -25,3 +25,5 @@ sed '/^GIT_VERSION = / d' stderr.log
> ! test -s stderr.log
> test -s Documentation/git.html
> grep '<meta name="generator" content="Asciidoctor ' Documentation/git.html
> +
> +save_good_tree
> --
> 2.15.1.500.g54ea76cc4
>
