On Thu, Oct 20, 2016 at 11:10:39PM -0700, Stefan Beller wrote:
> > on TravisCI I see these weird "Tests out of sequence" errors with prove
> > and they seem to not go away. I assume the reason that they not go away
> > is that the ".prove" file is carried over from on build to another (but I can't
> > look into this file on TravisCI).
> >
> > Has anyone an idea where these errors might come from?
> >
> > ------------------------------------------------------------------------
> > t5547-push-quarantine.sh (Wstat: 0 Tests: 5 Failed: 0)
>
> push quarantine is a new thing made by Jeff 2 weeks ago, IIRC.
Yes, but I do not see how it can trigger this:
> > Parse errors: Tests out of sequence. Found (2) but expected (3)
> > Tests out of sequence. Found (3) but expected (4)
> > Tests out of sequence. Found (4) but expected (5)
> > Bad plan. You planned 4 tests but ran 5.
The TAP output from one of our tests should look something like:
ok 1 - subject one
ok 2 - subject two
ok 3 - subject three
ok 4 - subject four
# passed all 4 test(s)
1..4
the "plan" is the bit at the end. That looks like $test_count
accidentally got incremented by one and we generated something like:
ok 1 - subject one
ok 3 - subject two
ok 4 - subject three
ok 5 - subject four
1..4
which would explain the "out of sequence" errors as well as the "planned
4 but ran 5".
But I do not see how the test script could screw that up. The counting
is handled entirely by the harness in test-lib.sh.
Nor do I see how a stale .prove file could matter. It does not store
information about the test plan at all. E.g., here is the entry from
mine for t5547:
t5547-push-quarantine.sh:
elapsed: 0.0762169361114502
gen: 1
last_pass_time: 1477037708.741
last_result: 0
last_run_time: 1477037708.741
last_todo: 0
seq: 437
total_passes: 1
Puzzling.
-Peff