On Mon, May 12, 2014 at 5:56 AM, David Turner <dturner@xxxxxxxxxxxxxxxx> wrote:
>> So without watchman I got
>>
>> 299.871ms read_index_from:1538 if (verify_hdr(hdr, mmap_size) < 0) go
>> 498.205ms cmd_status:1300 refresh_index(&the_index, REFRESH_QUIE
>> 796.050ms wt_status_collect:622 wt_status_collect_untracked(s)
>>
>> and with watchman ("git status" ran several times to make sure it's cached)
>>
>> 301.950ms read_index_from:1538 if (verify_hdr(hdr, mmap_size) < 0) go
>> 34.918ms read_fs_cache:347 if (verify_hdr(hdr, mmap_size) < 0) go
>> 1564.096ms watchman_load_fs_cache:628 update_fs_cache(istate, result);
>> 161.930ms cmd_status:1300 refresh_index(&the_index, REFRESH_QUIE
>> 251.614ms wt_status_collect:622 wt_status_collect_untracked(s)
>>
>> Given the total time of "git status" without watchman is 1.9s,,
>> update_fs_cache() nearly matches that number alone. All that is spent
>> in the exclude update code in the function, but if you do
>> last_excluding_matching() anyway, why cache it?
>
> My numbers are different (for my test repository):
>
> ---
> 30.031ms read_index:1386 r = read_index_from(istate, get_index_
> 71.625ms cmd_status:1302 refresh_index(&the_index, REFRESH_QUIE
> 259.712ms wt_status_collect:622 wt_status_collect_untracked(s)
> ----
> 41.110ms read_index:1386 r = read_index_from(istate, get_index_
> 9.294ms read_fs_cache:347 if (verify_hdr(hdr, mmap_size) < 0) go
> 0.173ms watchman_load_fs_cache:628 update_fs_cache(istate, result)
> 41.901ms read_index:1386 r = read_index_from(istate, get_index_
> 18.355ms cmd_status:1302 refresh_index(&the_index, REFRESH_QUIE
> 50.911ms wt_status_collect:622 wt_status_collect_untracked(s)
> ---
>
> I think something must be going wrong with update_fs_cache on your
> machine. I have a few hypotheses:
>
> 1. Maybe watchman isn't fully started up when you run your tests.
> 2. Maybe there is a bug.
It's probably me doing something wrong (I ran it more than a couple
times so watchman must have loaded the whole thing). I got small
numbers in update_fs_cache() now.
>> A bit surprised about wt_status_collect_untracked number. I verified
>> that last_excluding_matching() still runs (on the same number of
>> entries like in no-watchman case). Replacing fs_cache_open() in
>> add_excludes_from_file_to_list() to plain open() does not change the
>> number, so we probably won't need that (unless your worktree is filled
>> with .gitignore, which I doubt it's a norm).
>
> My test repo has a couple hundred of them. Maybe that's unusual? A
> repo with a lot of projects will tend to have lots of gitignore files,
> because each project will want to maintain them independently.
I tried the worst case, every directory had an empty .gitignore. The
numbers did not change much. And I found out that because
add_excludes.. were called twice, not on every .gitignore because of
the condition "!(dir->flags & DIR_EXCLUDE_CMDL_ONLY)". So the number
of .gitignore does not matter (yet).
This is your quote from above, moved down a bit:
> update_fs_cache should only have to update based on what it has learned
> from watchman. So if no .gitignore has been changed, it should not have
> to do very much work.
>
> I could take the fe_excluded check and move it above the
> last_exclude_matching check in fs_cache_is_excluded; it causes t7300 to
> fail when run under watchman but presumably that's fixable
So you exclude files early and make the real read_directory() pass do
pretty much nothing. This is probably not a good idea. Assume that I
touch $TOP/.gitignore then do something other than "git status" (or
"git add") then I have to pay read_directory() cost.
Back to the open vs fs_cache_open and the number of .gitignore files
above. I touch $TOP/.gitignore then do "git status" to make it read
all .gitignore files (6k of them) and change between open and
fs_cache_open. I think the numbers still do not make any visible
difference (~1620-1630ms).
--
Duy
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html