On Fri, Jul 10, 2020 at 4:10 AM Nicolas Iooss <nicolas.iooss@xxxxxxx> wrote:
>
> On Thu, Jul 9, 2020 at 8:00 PM Paul Moore <paul@xxxxxxxxxxxxxx> wrote:
> >
> > On Thu, Jul 9, 2020 at 10:33 AM William Roberts
> > <bill.c.roberts@xxxxxxxxx> wrote:
> > > So Nicolas initially created our travis script in commit c9adfe2d2653
> > > and has -k, or keep going, on the make commands. This causes make to
> > > plow ahead and bury the errors in the logs. Stephen noticed this the
> > > other day, and we have been chatting about it out of band and wanted
> > > to pull in the community.
> > >
> > > Are their compelling reasons for keeping this behavior? I am also
> > > concerned that we could get false positives on travis success results.
> >
> > In my opinion the whole point of automated testing is to catch
> > failures early and often. For that reason I would want the test to
> > fail and stop, both because I find it easier to identify the failure
> > that way and also because I'm not sure I would trust much of the
> > testing that occurred after an error condition.
> >
> Hi,
> There seems to be some confusion:
>
> * "make -k" does not stop the "make" command at the first error and
> allows seeing all the errors when there are several ones. In my humble
> opinion, it makes sense when compiling ("make all") and not when
> running tests ("make test"), and this is actually what is right now in
> Travis-CI. "make -k" returns a failure exit code when an error
> happens.
Ahh I thought it returned to 0. Not sure why I assumed that.
>
> * Travis-CI does not stop the job as soon as a sub-command fails. If I
Depends on the stage:
https://docs.travis-ci.com/user/job-lifecycle/
If before_install, install or before_script returns a non-zero exit
code, the build is errored and stops immediately.
If script returns a non-zero exit code, the build is failed, but
continues to run before being marked as failed.
I put a false command in the script section and it kept plowing ahead
as you foretold.
> understand correctly, this is what really bothers William, and I agree
> this is a behavior that can be improved. According to
> https://github.com/travis-ci/travis-ci/issues/1066, a possible
> solution could be to use "set -e", which could have unexpected
> side-effects in launched commands. It is possible to "emulate set -e"
> by adding exit statements, such as :
>
> - make install $EXPLICIT_MAKE_VARS -k || exit $?
> - make install-pywrap $EXPLICIT_MAKE_VARS -k || exit $?
> - make install-rubywrap $EXPLICIT_MAKE_VARS -k || exit $?
> # ...
> - make test $EXPLICIT_MAKE_VARS || exit $?
>
> I have not tested whether this works on Travis-CI, but if it does, it
> would be a nice improvement. I will take a look this week-end.
I think the scripts are more maintainable outside of travis yaml files
as separate build scripts,
for two reasons:
1. One can just execute the script locally, you can't, AFAIK, do that
with a travis yaml file.
2. The issue can be avoided as they afford more control. Some other
projects I am a part of we only
use script and after_failure. The bash scripts are set -e. I also
used this approach for the KVM
selinux test run.
script:
- ./.ci/travis.run
after_failure:
- cat build/test-suite.log
We could adopt like what's above...
>
> Cheers,
> Nicolas
>
