Re: Store refreshed stat info in a separate file?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Sat, Apr 19, 2014 at 12:43 AM, Junio C Hamano <gitster@xxxxxxxxx> wrote:
> Having said that, I do not think there is a fundamental reason why
> the stat data has to live inside the same index file.  A separate
> file is just fine, as long as you can reliably detect that they went
> out of sync for whatever reason (e.g. "the index proper updated, a
> stale stat file left beind"), and storing the trailer checksum from
> the corresponding index in this new file is an obvious and good
> solution.

I've gone further and store index updates (including entry removals
and additions) to the second index file so that index I/O cost is now
proportional to the number of changed entries, not the work tree size
(sort of). Which makes it scale much better when the work tree is
huge. There is one flaw though. I'm expecting many "yuck" responses
from people. So let's try to settle it now, or drop the idea.

The idea is we can support another mode, where index content is stored
in two files, the small $GIT_DIR/index and large $GIT_DIR/index.base.
"index" contains changes that should be applied to "index.base".
Whenever you do something to the index, "index" records those actions.
Git reads both index.base and index, then replay the action to have
the final index in memory. "index.base" contains full worktree data
and remains unchanged until "index" becomes too big/slow that changes
should be merged back to "index.base". This works great (my prototype
passed the test suite), and even greater than index v5 because v5
still rewrites the whole index file when an entry is added or removed.

But there is a problem with atomic update. The good old rename() does
not work well with 2 files. This is not a problem with the C part, I
can still make atomic update work. Scripts, on the other hand, may
rely on "mv" or similar commands/functions to prepare a temp index and
move it to $GIT_DIR/index. The workaround is merge back two files into
a single index file so that scripts can "mv $temp_index" as before and
pay the whole-index I/O penalty. An alternative is store two files in
one, the one index file actually consists two subfiles. We avoid the
atomic update problem, but we pay I/O cost for writing 10MB every time
an index is updated (but not hashing 10MB file) and introduce a new
index format. This is even yuckier in my opinion.

Should I continue, or drop it?
-- 
Duy
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]