Hi Derrick,
Thank you for your reply. Please see my reply inline:
On Thu, Jul 14, 2022 at 11:06 PM Derrick Stolee
<derrickstolee@xxxxxxxxxx> wrote:
>
> On 7/12/2022 9:58 AM, Dian Xu wrote:
>
> > Reproduction steps:
> > 1. Local repo setup:
> > git init
> > git remote add origin <repo_url>
> > echo <alternative_objects_store> >! .git/objects/info/alternates
>
> Is this alternative object store on the local disk? Or is it
> across a network connection?
The alternate object store I used is across network connection.
>
> > git config --unset-all remote.origin.fetch
> > git config --unset-all remote.origin.tagopt
> > git config --add remote.origin.tagopt --no-tags
> > git config --add remote.origin.fetch
> > +refs/heads/main:refs/remotes/origin/main
> > git config core.sparsecheckout true
> > git config core.sparsecheckoutcone true
> > git fetch
>
> This is interesting that you are fetching without having
> any existing refs, only the alternate store. Are you using
> something like the core.alternateRefsCommand config option?
>
> Otherwise, I would expect this "git fetch" to act as if it
> was a normal clone (unless of course the alternate already
> has the current tip of 'main'). Maybe fetch negotiation helps
> find some common commits that exist in the alternate, but the
> client is useless in that discovery without refs.
I am not sure I understand your statement fully here:
- All these config setup here is to let 'git fetch' only fetches
e.g. 'refs/heads/main', nothing else
- The HEAD of the AOS (alternate object store) is indeed at refs/heads/main
- I can remove all these 4 specific refs config settings (now
fetching all refs instead), and the issue can still be reproduced
>
> > 2. Sparse checkout <dir1>:
> > git sparse-checkout set <dir1>
> > git checkout -b tmp1 origin/main
> >
> > 3. Add another <dir2> via sparse-checkout, which hangs:
> > git sparse-checkout add <dir2>
> > 09:39:11.169116 read-cache.c:2451 performance:
> > 6.326754679 s: read cache .git/index
>
> Could you re-run this with GIT_TRACE2_PERF=1 so we can see
> even more details about how this works?
Please see the output with GIT_TRACE2_PERF at the end of the email,
which hung at:
15:17:28.216829 cache-tree.c:477 | d0 | main
| region_enter | r1 | 85.183554 | | cache_tree
| ..label:update
>
> > Repo and dir info:
> > <repo_url> is a large repo, containing large number of directories
> > and 2 million files;
> > Local repo is using <alternative_objects_store>, which is working
> > by verifying: git count-objects -vH;
> > <dir1> used here is one level deep, <dir2> used here is 5 level deep
> >
> > Observations:
> > 1. Even 'sparse-checkout add' hangs, scm code from <dir2> was
> > actually downloaded pretty fast
>
> I'm interested to learn what you mean by "downloaded" here.
After the 'git sparse-checkout add' hung, I ctr-C then cd to inspect
the source code, they are on disk
>
> > 2. If waited out 'sparse-checkout add', git seems start to
> > download large number of objects from alternative objects store into
> > local .git/objects
>
> Hm. Adding the objects into .git/objects is strange. It should
> recognize that the objects are in the alternate and not write
> them to the local repo (this is independent of the sparse-checkout
> builtin).
Yep, seems like git is downloading objects from AOS to local, which is
a main concern. We choose to use AOS to save space/time in the first
place.
>
> Thanks,
> -Stolee
Please let me know if you need more details.
Thanks,
Dian Xu
Mathworks, Inc
1 Lakeside Campus Drive, Natick, MA 01760
508-647-3583
----------GIT_TRACE2_PERF----------
% git sparse-checkout add matlab/config
15:16:03.038943 common-main.c:49 | d0 | main
| version | | | | |
2.37.0
15:16:03.038997 common-main.c:50 | d0 | main
| start | | 0.005727 | | |
git sparse-checkout add matlab/config
15:16:03.039144 compat/linux/procinfo.c:170 | d0 | main
| cmd_ancestry | | | | |
ancestry:[tcsh mate-terminal mate-panel mate-session lightdm lightdm
systemd]
15:16:03.044327 repository.c:136 | d0 | main
| def_repo | r1 | | | |
worktree:/mathworks/devel/sandbox/dianxu/workspaces/tmp_git/cone
15:16:03.045148 git.c:461 | d0 | main
| cmd_name | | | | |
sparse-checkout (sparse-checkout)
15:16:03.045589 read-cache.c:2447 | d0 | main
| region_enter | r1 | 0.012321 | | index |
label:do_read_index .git/index
15:16:07.665184 cache-tree.c:628 | d0 | main
| region_enter | r1 | 4.631908 | | cache_tree |
..label:read
15:16:07.821302 cache-tree.c:630 | d0 | main
| region_leave | r1 | 4.788028 | 0.156120 | cache_tree |
..label:read
15:16:07.834318 read-cache.c:2393 | d0 | main
| data | r1 | 4.801047 | 4.788726 | index |
..read/version:3
15:16:07.834364 read-cache.c:2395 | d0 | main
| data | r1 | 4.801097 | 4.788776 | index |
..read/cache_nr:2314721
15:16:07.834375 read-cache.c:2451 performance: 4.788756364 s:
read cache .git/index
15:16:07.834379 read-cache.c:2452 | d0 | main
| region_leave | r1 | 4.801112 | 4.788791 | index |
label:do_read_index .git/index
15:16:07.950067 progress.c:268 | d0 | main
| region_enter | r1 | 4.916794 | | progress |
label:Updating index flags
15:16:07.950130 unpack-trees.c:1671 | d0 | main
| region_enter | r1 | 4.916863 | | unpack_trees |
..label:clear_ce_flags(0x00000000,0x02000000)
15:16:08.083835 unpack-trees.c:1678 | d0 | main
| region_leave | r1 | 5.050562 | 0.133699 | unpack_trees |
..label:clear_ce_flags(0x00000000,0x02000000)
15:16:08.083902 progress.c:339 | d0 | main
| data | r1 | 5.050634 | 0.133840 | progress |
..total_objects:2314721
15:16:08.083910 progress.c:346 | d0 | main
| region_leave | r1 | 5.050643 | 0.133849 | progress |
label:Updating index flags
15:16:08.161023 progress.c:268 | d0 | main
| region_enter | r1 | 5.127750 | | progress |
label:Updating files
Updating files: 100% (21877/21877), done.
15:17:11.071782 progress.c:339 | d0 | main
| data | r1 | 68.038510 | 62.910760 | progress |
..total_objects:21877
15:17:11.071825 progress.c:346 | d0 | main
| region_leave | r1 | 68.038557 | 62.910807 | progress |
label:Updating files
15:17:11.071859 progress.c:268 | d0 | main
| region_enter | r1 | 68.038591 | | progress |
label:Filtering content
15:17:11.071911 progress.c:339 | d0 | main
| data | r1 | 68.038643 | 0.000052 | progress |
..total_objects:0
15:17:11.071930 progress.c:346 | d0 | main
| region_leave | r1 | 68.038662 | 0.000071 | progress |
label:Filtering content
15:17:11.071960 unpack-trees.c:496 performance: 62.934445487 s:
check_updates
15:17:11.071974 unpack-trees.c:2062 performance: 63.187061820 s:
update_sparsity
15:17:11.071998 read-cache.c:3179 | d0 | main
| region_enter | r1 | 68.038728 | | index |
label:do_write_index
/mathworks/devel/sandbox/dianxu/workspaces/tmp_git/cone/.git/index.lock
15:17:12.387925 cache-tree.c:532 | d0 | main
| region_enter | r1 | 69.354652 | | cache_tree |
..label:write
15:17:12.426139 cache-tree.c:534 | d0 | main
| region_leave | r1 | 69.392868 | 0.038216 | cache_tree |
..label:write
15:17:16.696737 read-cache.c:3135 performance: 5.624723676 s:
write index, changed mask = 2
15:17:16.696828 read-cache.c:3141 | d0 | main
| data | r1 | 73.663556 | 5.624828 | index |
..write/version:3
15:17:16.696897 read-cache.c:3143 | d0 | main
| data | r1 | 73.663629 | 5.624901 | index |
..write/cache_nr:2314721
15:17:16.696953 read-cache.c:3182 | d0 | main
| region_leave | r1 | 73.663682 | 5.624954 | index |
label:do_write_index
/mathworks/devel/sandbox/dianxu/workspaces/tmp_git/cone/.git/index.lock
15:17:17.089520 sparse-index.c:209 | d0 | main
| region_enter | r1 | 74.056246 | | index |
label:convert_to_sparse
15:17:17.146799 cache-tree.c:477 | d0 | main
| region_enter | r1 | 74.113525 | | cache_tree |
..label:update
15:17:24.864900 cache-tree.c:482 | d0 | main
| region_leave | r1 | 81.831626 | 7.718101 | cache_tree |
..label:update
15:17:24.864975 cache-tree.c:483 performance: 7.718183242 s:
cache_tree_update
15:17:24.864989 sparse-index.c:223 | d0 | main
| region_leave | r1 | 81.831721 | 7.775475 | index |
label:convert_to_sparse
15:17:24.911366 cache-tree.c:477 | d0 | main
| region_enter | r1 | 81.878090 | | cache_tree |
label:update
15:17:25.890481 cache-tree.c:482 | d0 | main
| region_leave | r1 | 82.857209 | 0.979119 | cache_tree |
label:update
15:17:25.890528 cache-tree.c:483 performance: 0.979171895 s:
cache_tree_update
15:17:25.890536 sparse-index.c:340 | d0 | main
| region_enter | r1 | 82.857269 | | index |
label:ensure_full_index
15:17:28.216829 cache-tree.c:477 | d0 | main
| region_enter | r1 | 85.183554 | | cache_tree |
..label:update
