On Tue, May 05, 2020 at 12:48:43PM +0200, Alban Gruin wrote:
> The old scripted `git stash' used to create a second index to save
> modified and untracked files, and restore untracked files, without
> affecting the main index. This behaviour was carried on when it was
> rewritten in C, and here, most operations performed on the second index
> are done by forked commands (ie. `read-tree' instead of reset_tree(),
> etc.). This works most of the time, except in some edge case with the
> split-index when the split file has expired and is deleted by a forked
> command: the main index may still contain a reference to the now-deleted
> file, and subsequent operations on the index will fail [0].
>
> The goal of this series is to modernise (a bit) builtin/stash.c, and to
> fix the aforementionned edge case.
While this patch series does fix this edge case, it doesn't fix the
root cause, which is not specific to 'git stash', but is more general
issue when using split index with an alternate index. More on this in
the notes attached to the patch below.
Having said that, and without actually looking at your patches, I
think removing the second index from the builtin stash is a good idea
nonetheless.
> I have to admit that I don't really know how to test this.
> GIT_TEST_SPLIT_INDEX failed on me (gdb showed me that it does not enable
> the split-index at all, at least in `git stash' and its forks)
You are right, as it turns out GIT_TEST_SPLIT_INDEX=1 has been broken
for a while now. The patch below fixes this, but there is a bit of a
fallout of test failures, which it doesn't fix, because I'm not quite
sure how to fix one of them. Again, more on this in the attached
notes.
--- >8 ---
Subject: [PATCH] [RFH] read-cache: fix GIT_TEST_SPLIT_INDEX
Setting GIT_TEST_SPLIT_INDEX=1 is supposed to turn on the split-index
feature in the test suite and trigger index splitting (mostly)
randomly. Alas, this has been broken since 6e37c8ed3c (read-cache.c:
fix writing "link" index ext with null base oid, 2019-02-13), and no
index splitting has been performed at all since then.
There are two places where we check the value of GIT_TEST_SPLIT_INDEX,
and before 6e37c8ed3c they worked like this:
1) In the lower-level do_write_index(), where, if
GIT_TEST_SPLIT_INDEX is enabled, we call init_split_index().
This call merely allocates and zero-initializes
'istate->split_index', but does nothing else (i.e. doesn't fill
the base/shared index with cache entries, doesn't actually write
a shared index file, etc.). Pertinent to this issue, the hash of
the base index remains all zeroed out.
2) In the higher-level write_locked_index(), where, if
GIT_TEST_SPLIT_INDEX is enabled _and_ 'istate->split_index' has
already been initialized, it randomly sets the flag that triggers
index splitting later in this function. This randomness comes
from the first byte of the hash of the base index via an 'if
((first_byte & 15) < 6)' condition.
However, if 'istate->split_index' hasn't been initialized (i.e.
still NULL), then it calls do_write_locked_index(), which
internally calls do_write_index() mentioned above, and then a
'goto out' skips right over this second GIT_TEST_SPLIT_INDEX
check.
This means that while GIT_TEST_SPLIT_INDEX=1 usually triggered index
splitting randomly, the first two index writes were always
deterministic, though I suspect this was unintentional (this is still
before 6e37c8ed3c):
- The initial index write never splits the index.
During the first index write write_locked_index() is called with
'istate->split_index' still uninitialized, so the check in 2) is
not executed. It still calls do_write_index(), though, which then
executes the check in 1). The resulting all zero base index hash
then leads to the 'link' extension being written to '.git/index',
though a shared index file is not written:
$ rm .git/index
$ GIT_TEST_SPLIT_INDEX=1 git update-index --add file
$ test-tool dump-split-index .git/index
own c6ef71168597caec8553c83d9d0048f1ef416170
base 0000000000000000000000000000000000000000
100644 d00491fd7e5bb6fa28c517a0bb32b8b506539d4d 0 file
replacements:
deletions:
$ ls -l .git/sharedindex.*
ls: cannot access '.git/sharedindex.*': No such file or directory
- The second index write always splits the index.
When the index written in the previous point is read,
'istate->split_index' is initialized because of the presence of
the 'link' extension. So during the second write
write_locked_index() does run the check in 2), and the first byte
of the all zero base index hash always fulfills the randomness
condition, which in turn always triggers the index splitting.
- Subsequent index writes will find the 'link' extension with a real
non-zero base index hash, so from then on the check in 2) is
executed and the first byte of the base index hash is as random as
it gets (coming from the SHA-1 of index data including timestamps
and inodes...).
All this worked until 6e37c8ed3c came along, and stopped writing the
'link' extension if the hash of the base index was all zero:
$ rm .git/index
$ GIT_TEST_SPLIT_INDEX=1 git update-index --add file
$ test-tool dump-split-index .git/index
own abbd6f6458d5dee73ae8e210ca15a68a390c6fd7
not a split index
$ ls -l .git/sharedindex.*
ls: cannot access '.git/sharedindex.*': No such file or directory
Since there is no 'link' extension, in the second index write
'istate->split_index' remains uninitialized, and the check in 2) is
not executed, and ultimately the index is never split.
Fix this by modifying write_locked_index() to make sure to check
GIT_TEST_SPLIT_INDEX even if 'istate->split_index' is still
uninitialized, and initialize it if necessary. The check for
GIT_TEST_SPLIT_INDEX and separate init_split_index() call in
do_write_index() thus becomes unnecessary, so remove it. Furthermore,
add a test to 't1700-split-index.sh' to make sure that
GIT_TEST_SPLIT_INDEX=1 will keep working (though only check the
index splitting on the first index write, because after that it will
be random).
Note that this change does not restore the pre-6e37c8ed3c behaviour,
as it will deterministically split the index already on the first
index write. Since GIT_TEST_SPLIT_INDEX is purely a developer aid,
there is no backwards compatibility issue here. The new behaviour
does trigger test failures in 't0003-attributes.sh' and
't1600-index.sh', though, which should be fixed in preparatory patches
first.
---
Notes:
The failures in 't1600-index.sh' are easy: those tests check how bogus
values in $GIT_INDEX_VERSION or 'index.version' are handled, and all
failures look like this:
--- expect.err 2020-06-14 17:28:21.043527044 +0000
+++ actual.err 2020-06-14 17:28:21.043527044 +0000
@@ -1,2 +1,4 @@
warning: index.version set, but the value is invalid.
Using version
+warning: index.version set, but the value is invalid.
+Using version
We get one such warning when writing any index file, and since we now
split the index on the first write, we get one warning when writing
the shared index and a second warning when writing '.git/index'. A
simple 'sane_unset GIT_TEST_SPLIT_INDEX' fixes these failures, and
since this test script is about some intricacies of the index, I think
this is the right thing to do in this case.
The failure in 't0003-attributes.sh' is tricky, because it does this:
# The previous tests wrote the split and shared shared index files
# in '$TRASH_DIRECTORY/.git'.
(
cd bare.git &&
GIT_INDEX_FILE=../.git/index \
git check-attr --cached ....
)
which fails with:
fatal: ./sharedindex.3e9b03a8e3e1533d2bb2e22c77d4784ce2e7b108: index file open failed: No such file or directory
Unsetting GIT_TEST_SPLIT_INDEX in the test script fixes this failure
as well, of course, though I don't think this is the right thing to do
in this case. Worse, this failure raises some serious questions:
- Where to look for and where to write the shared index file when
using an alternate index location?
Currently it's read from / written to gitdir. This might be fine
as long as both cwd and the alternate index are in the same
repository while writing and reading. However, as this failing
test shows, if the shared index was written in one repository,
then an attempt to read it while in another repository using
GIT_INDEX_FILE will fail.
I think it would be more sensible to write the shared index next
to the (alternate) index file in the same directory, wherever that
might be. But what about backwards compatibility? IOW can there
be a sensible use case that relies on GIT_INDEX_FILE pointing to
somewhere, but its shared index file is still being written to the
current repository?
- What about expiring shared index files while using an alternate
index?
We must be careful not to expire any sharedindex.<sha1> files that
are used by '.git/index' or any existing alternate index files.
Unfortunately, we are not that careful, and I think this is the
root cause of the issue that Alban is trying to fix. The failing
test below demonstrates this using basically only 'git
update-index', without all the complexity of 'git stash'.
test_expect_failure 'split index expiration vs. alternate index' '
>file1 &&
>file2 &&
git update-index --split-index --add file1 &&
# debug: this should print .git/index and .git/sharedindex.<sha1>.
echo .git/*index* &&
GIT_INDEX_FILE=.git/otherindex \
git -c splitIndex.sharedIndexExpire=now \
update-index --split-index --add file2 &&
# debug: this should print .git/index, .git/otherindex and _two_
# .git/sharedindex.<sha1> files, but, alas, it prints only one
# shared index, because the other got expired.
echo .git/*index* &&
git diff --cached --name-only >actual1 &&
echo file1 >expect1 &&
test_cmp expect1 actual1 &&
GIT_INDEX_FILE=.git/otherindex git diff --cached --name-only >actual2 &&
echo file2 >expect2 &&
test_cmp expect2 actual2
'
Now, not expiring the shared index used by '.git/index' while
writing to an alternate split index is easy, because we do know
about '.git/index', of course, and thus can check it to exclude
its shared index. But how could a git process expiring shared
indexes know whether a shared index file belongs to an alternate
index?! I'm afraid it can't, which leads to the next question...
- Should we even allow writing a split index when using an alternate
index file?
If we don't do that, then there is no risk of expiring the shared
index of an alternate index, because there will be no such shared
index to begin with. Furthermore, if we don't write a shared
index while using an alternate index, then we won't trigger
expiration while using an alternate index, so we won't
accidentally expire the shared index used by '.git/index'.
Note, however, that we should still support _reading_ the shared
index of an alternate index, as that alternate index might be the
"real" '.git/index' of a different repository, like in the failing
test in t0003.
- Should we even allow 'splitIndex.sharedIndexExpire=now'?
I believe, though haven't confirmed, that it can cause trouble
even without using an alternate index. Consider the following
sequence of events:
- Git process A reads '.git/index', finds the 'link' extension,
and reads the SHA1 recorded there that determines the filename
of its shared index.
- The scheduler steps in, and puts process A to sleep.
- Git process B updates the index, decides that it's time to
write a new shared index, does so, and then because of
'splitIndex.sharedIndexExpire=now' it removes all other shared
index files.
- The scheduler wakes process A, which now tries to open the
shared index file it just learned about, but fails because
that file has just been removed by process B.
This is similar to the issue we have with 'git gc --prune=now',
except that 'git gc's documentation explicitly warns about the
risks of using '--prune=now', while the description of
'splitIndex.sharedIndexExpire' doesn't have any such warning.
I think that 'splitIndex.sharedIndexExpire=now' should be allowed,
for those who hopefully know what they are doing, just as we allow
'git gc --prune=now', but the documentation should clearly warn
against its potential pitfalls.
read-cache.c | 22 +++++++++++++---------
t/t1700-split-index.sh | 10 ++++++++++
2 files changed, 23 insertions(+), 9 deletions(-)
diff --git a/read-cache.c b/read-cache.c
index aa427c5c17..b888c5df44 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -2801,11 +2801,8 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
}
}
- if (!istate->version) {
+ if (!istate->version)
istate->version = get_index_format_default(the_repository);
- if (git_env_bool("GIT_TEST_SPLIT_INDEX", 0))
- init_split_index(istate);
- }
/* demote version 3 to version 2 when the latter suffices */
if (istate->version == 3 || istate->version == 2)
@@ -3222,7 +3219,7 @@ static int too_many_not_shared_entries(struct index_state *istate)
int write_locked_index(struct index_state *istate, struct lock_file *lock,
unsigned flags)
{
- int new_shared_index, ret;
+ int new_shared_index, ret, test_split_index_env;
struct split_index *si = istate->split_index;
if (git_env_bool("GIT_TEST_CHECK_CACHE_TREE", 0))
@@ -3237,7 +3234,9 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,
if (istate->fsmonitor_last_update)
fill_fsmonitor_bitmap(istate);
- if (!si || alternate_index_output ||
+ test_split_index_env = git_env_bool("GIT_TEST_SPLIT_INDEX", 0);
+
+ if ((!si && !test_split_index_env) || alternate_index_output ||
(istate->cache_changed & ~EXTMASK)) {
if (si)
oidclr(&si->base_oid);
@@ -3245,10 +3244,15 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,
goto out;
}
- if (git_env_bool("GIT_TEST_SPLIT_INDEX", 0)) {
- int v = si->base_oid.hash[0];
- if ((v & 15) < 6)
+ if (test_split_index_env) {
+ if (!si) {
+ si = init_split_index(istate);
istate->cache_changed |= SPLIT_INDEX_ORDERED;
+ } else {
+ int v = si->base_oid.hash[0];
+ if ((v & 15) < 6)
+ istate->cache_changed |= SPLIT_INDEX_ORDERED;
+ }
}
if (too_many_not_shared_entries(istate))
istate->cache_changed |= SPLIT_INDEX_ORDERED;
diff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh
index 12a5568844..e0c9e16a8a 100755
--- a/t/t1700-split-index.sh
+++ b/t/t1700-split-index.sh
@@ -507,4 +507,14 @@ test_expect_success 'do not refresh null base index' '
)
'
+test_expect_success 'GIT_TEST_SPLIT_INDEX works' '
+ test_create_repo git-test-split-index &&
+ (
+ cd git-test-split-index &&
+ >file &&
+ GIT_TEST_SPLIT_INDEX=1 git update-index --add file &&
+ verbose test $(ls -l .git/sharedindex.* |wc -l) = 1
+ )
+'
+
test_done
--
2.27.0.278.g0bed8b425d
