On Tue, Jul 07, 2020 at 12:50:33PM -0700, Junio C Hamano wrote:
> "Han-Wen Nienhuys via GitGitGadget" <gitgitgadget@xxxxxxxxx> writes:
>
> > From: Han-Wen Nienhuys <hanwen@xxxxxxxxxx>
> >
> > Adds test_tick to t5801-remote-helpers.sh and t3203-branch-output.sh
>
> That can be read from the patch. Also the subject tells us a half
> of what you want to achieve with this change (by the way, your
> subject is malformatted and lacks the <area>: prefix; perhaps
> "[PATCH] tests: make commit object names reproducible" or something),
> but the readers are left hanging without knowing what motivated the
> change. Do any test pieces in these scripts change their behaviour
> based on what exact object names are assigned to them, making them
> flaky and hard to test, and if so which one and in what way?
I agree that more discussion would be nice.
But I kind of wonder if we should be aiming for more determinism in
general, just to make debugging and reproduction simpler.
I.e., rather than pointing to _these_ tests, I think we could make an
argument for setting up a known timestamp in the test environment.
test_tick would continue to tick forward as usual, but for any tests
that don't use it, they'd by default get a deterministic outcome.
Something like this:
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 618a7c8d5b..d8adf5a199 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -441,15 +441,18 @@ TEST_AUTHOR_LOCALNAME=author
TEST_AUTHOR_DOMAIN=example.com
GIT_AUTHOR_EMAIL=${TEST_AUTHOR_LOCALNAME}@${TEST_AUTHOR_DOMAIN}
GIT_AUTHOR_NAME='A U Thor'
+GIT_AUTHOR_DATE='1112911993 -0700'
TEST_COMMITTER_LOCALNAME=committer
TEST_COMMITTER_DOMAIN=example.com
GIT_COMMITTER_EMAIL=${TEST_COMMITTER_LOCALNAME}@${TEST_COMMITTER_DOMAIN}
GIT_COMMITTER_NAME='C O Mitter'
+GIT_COMMITTER_DATE='1112911993 -0700'
GIT_MERGE_VERBOSITY=5
GIT_MERGE_AUTOEDIT=no
export GIT_MERGE_VERBOSITY GIT_MERGE_AUTOEDIT
export GIT_AUTHOR_EMAIL GIT_AUTHOR_NAME
export GIT_COMMITTER_EMAIL GIT_COMMITTER_NAME
+export GIT_COMMITTER_DATE GIT_AUTHOR_DATE
export EDITOR
# Tests using GIT_TRACE typically don't want <timestamp> <file>:<line> output
That's using the same start point as test_tick, though really it could
be anything. I've intentionally _not_ called test_tick at the beginning
of each script, because that would throw off all of the scripts that do
use it by one tick (whereas the first test_tick will overwrite these
values).
Trying to devil's advocate against this line of reasoning:
- using the current timestamp introduces more randomness into the test
suite, which could uncover problems. I'm somewhat skeptical, as the
usual outcome I see here is that we realize a test's expected output
is simply racy, and we remove the raciness by using test_tick
- using the current timestamp could alert us to problems that occur
only as the clock ticks forward (e.g., if we had a Y2021 bug, we'd
notice when the clock rolled forward).
- some tests may rely on having a "recent" timestamp in commits (e.g.,
when looking at relative date handling). I think all of the
relative-time tests already use a specific date, though, because
otherwise we have too many problems with raciness.
Note that the patch above does seem to cause two tests to fail. One of
them I _suspect_ is a raciness problem (order of commits output changes,
which implies the original was expecting the time to increment between
two commits without running test_tick). And the other looks like some
weird interaction with the perl test harness. I'd be happy to dig into
both if this direction seems sane.
-Peff
