git fetch recursion problem

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I have some git repositories in a state where `git fetch` recursively
spawns subprocesses until all system resources are used up and the OS
locks up.
This has been reproduced on debian with git 2.39.2, and on macos with
git 2.40.1.

The repositories being tracked are mirrors of github repos, within a
secure corporate environment.

During the problem, the `ps aux` output looks like this:
    fhanson     3187  0.0  0.0  11400  5092 ?        S    12:58   0:00
             \_ git fetch --recurse-submodules --tags --force
--progress
    fhanson     3195  0.0  0.0   9932  3372 ?        S    12:58   0:00
             |   \_ /usr/lib/git-core/git remote-https origin
https://repomirror.eng.corporate.com/github-neovim/lewis6991/gitsigns.nvim.git
    fhanson     3207  0.3  0.0 101608 16004 ?        S    12:58   0:00
             |       \_ /usr/lib/git-core/git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/lewis6991/gitsigns.nvim.git
    fhanson     3555  0.0  0.0  10292  4788 ?        S    12:58   0:00
             |           \_ /usr/lib/git-core/git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
    fhanson     3583  0.0  0.0   9932  3956 ?        S    12:58   0:00
             |               \_ /usr/lib/git-core/git remote-https
origin https://repomirror.eng.corporate.com/github-neovim/lewis6991/gitsigns.nvim.git
    fhanson     3592  0.3  0.0 101600 16156 ?        S    12:58   0:00
             |                   \_ /usr/lib/git-core/git-remote-https
origin https://repomirror.eng.corporate.com/github-neovim/lewis6991/gitsigns.nvim.git
    ... repeating forever!

The git config for these repositories looks like this:

    $ cat .git/config
    [core]
        repositoryformatversion = 1
        filemode = true
        bare = false
        logallrefupdates = true
    [submodule]
        active = .
    [remote "origin"]
        url = https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
        fetch = +refs/heads/*:refs/remotes/origin/*
        promisor = true
        partialclonefilter = blob:none
    [branch "master"]
        remote = origin
        merge = refs/heads/master

The git trace output looks like this (some server names redacted):
    $ export GIT_TRACE=1
    $ git fetch
    07:41:19.325652 git.c:439               trace: built-in: git fetch
    07:41:19.330118 run-command.c:655       trace: run_command:
GIT_DIR=.git git remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:19.337765 git.c:725               trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:19.338220 run-command.c:655       trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:20.664527 run-command.c:655       trace: run_command: git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
    07:41:20.679814 git.c:439               trace: built-in: git fetch
origin --no-tags --no-write-fetch-head --recurse-submodules=no
--filter=blob:none --stdin
    07:41:20.681735 run-command.c:655       trace: run_command: git
remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:20.692494 git.c:725               trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:20.693132 run-command.c:655       trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:21.695677 run-command.c:655       trace: run_command: git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
    07:41:21.715621 git.c:439               trace: built-in: git fetch
origin --no-tags --no-write-fetch-head --recurse-submodules=no
--filter=blob:none --stdin
    07:41:21.717230 run-command.c:655       trace: run_command: git
remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:21.727616 git.c:725               trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:21.728337 run-command.c:655       trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:22.801117 run-command.c:655       trace: run_command: git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
    07:41:22.816026 git.c:439               trace: built-in: git fetch
origin --no-tags --no-write-fetch-head --recurse-submodules=no
--filter=blob:none --stdin
    07:41:22.817972 run-command.c:655       trace: run_command: git
remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:22.828436 git.c:725               trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:22.829219 run-command.c:655       trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:24.242220 run-command.c:655       trace: run_command: git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
    07:41:24.257211 git.c:439               trace: built-in: git fetch
origin --no-tags --no-write-fetch-head --recurse-submodules=no
--filter=blob:none --stdin
    07:41:24.259264 run-command.c:655       trace: run_command: git
remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:24.269718 git.c:725               trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:24.270612 run-command.c:655       trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:25.306129 run-command.c:655       trace: run_command: git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
    07:41:25.320842 git.c:439               trace: built-in: git fetch
origin --no-tags --no-write-fetch-head --recurse-submodules=no
--filter=blob:none --stdin
    07:41:25.322801 run-command.c:655       trace: run_command: git
remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:25.332775 git.c:725               trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:25.333522 run-command.c:655       trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:26.332580 run-command.c:655       trace: run_command: git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
    07:41:26.348349 git.c:439               trace: built-in: git fetch
origin --no-tags --no-write-fetch-head --recurse-submodules=no
--filter=blob:none --stdin
    07:41:26.350225 run-command.c:655       trace: run_command: git
remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:26.361171 git.c:725               trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:26.361969 run-command.c:655       trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:27.585641 run-command.c:655       trace: run_command: git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
    07:41:27.600316 git.c:439               trace: built-in: git fetch
origin --no-tags --no-write-fetch-head --recurse-submodules=no
--filter=blob:none --stdin
    07:41:27.602370 run-command.c:655       trace: run_command: git
remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:27.611823 git.c:725               trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:27.612607 run-command.c:655       trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:28.666477 run-command.c:655       trace: run_command: git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
    07:41:28.678956 git.c:439               trace: built-in: git fetch
origin --no-tags --no-write-fetch-head --recurse-submodules=no
--filter=blob:none --stdin
    07:41:28.680665 run-command.c:655       trace: run_command: git
remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:28.690534 git.c:725               trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    07:41:28.691277 run-command.c:655       trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
    ^C


Random clues and experiments:

A `git clone` from our corporate mirrors always succeeds.  Immediately
after a `git clone`, I am always able to `git fetch`, so far.
Something else must cause the repository to enter a broken state,
possibly the addition of upstream commits.  I can't verify that since
I can't complete `git fetch.`

I can fix a broken repository by deleting these settings from `.git/config`:
        promisor = true
        partialclonefilter = blob:none
After this, `git fetch` succeeds.
Deleting just one of these settings is not sufficient, it must be both.

I copied one such broken repository out of our secure environment onto
a mac laptop that has access to both our mirror sites and to the
internet.
I verified that `git fetch` from our mirror is still broken on the mac.
Then I modified .git/config to point to the https://github.com/ url
instead of our mirror site, leaving the 'promisor' and
'partialclonefilter' settings in place.
Running `git fetch` in this state succeeded.

This suggests that our internal mirror site is returning something
different from github itself.


Here is the trace from successfully fetching from github.com:
    $ git fetch
    07:43:30.901275 git.c:439               trace: built-in: git fetch
    07:43:30.904188 run-command.c:655       trace: run_command:
GIT_DIR=.git git remote-https origin
https://github.com/nvim-treesitter/nvim-treesitter-textobjects.git
    07:43:30.917345 git.c:725               trace: exec:
git-remote-https origin
https://github.com/nvim-treesitter/nvim-treesitter-textobjects.git
    07:43:30.918560 run-command.c:655       trace: run_command:
git-remote-https origin
https://github.com/nvim-treesitter/nvim-treesitter-textobjects.git
    remote: Enumerating objects: 8, done.
    remote: Counting objects: 100% (8/8), done.
    remote: Compressing objects: 100% (5/5), done.
    07:43:31.714337 run-command.c:655       trace: run_command: git
index-pack --stdin -v --fix-thin --promisor --pack_header=2,8
    remote: Total 8 (delta 2), reused 5 (delta 1), pack-reused 0
    07:43:31.729793 git.c:439               trace: built-in: git
index-pack --stdin -v --fix-thin --promisor --pack_header=2,8
    Receiving objects: 100% (8/8), 3.36 KiB | 3.36 MiB/s, done.
    Resolving deltas: 100% (2/2), completed with 1 local object.
    07:43:31.738815 run-command.c:655       trace: run_command: git
rev-list --objects --stdin --exclude-promisor-objects --not --all
--quiet --alternate-refs
    07:43:31.748286 git.c:439               trace: built-in: git
rev-list --objects --stdin --exclude-promisor-objects --not --all
--quiet --alternate-refs
    From https://github.com/nvim-treesitter/nvim-treesitter-textobjects
       9c5237b..95b76b9  master        -> origin/master
       9c5237b..11ae4c7  update-readme -> origin/update-readme
    07:43:31.757504 run-command.c:1524      run_processes_parallel:
preparing to run up to 1 tasks
    07:43:31.757515 run-command.c:1551      run_processes_parallel: done
    07:43:31.757522 run-command.c:655       trace: run_command: git
maintenance run --auto --no-quiet
    07:43:31.765282 git.c:439               trace: built-in: git
maintenance run --auto --no-quiet

The difference starts just a few lines in, my failed fetches never see this:
    remote: Counting objects: 100% (8/8), done.
    remote: Compressing objects: 100% (5/

Next I enabled both GIT_TRACE and GIT_CURL_VERBOSE to identify
differences in the HTTP conversation between a failed fetch versus a
successful one.
The successful run was made by deleting the `promisor` and
`partialclonefilter` settings from .git/config.
I'm having some some trouble attaching these to this email, I'll have
to include them in a follow up.

Nothing jumped out at me here but I'm no expert.


Is this a git bug, or a server misconfiguration, or both?



[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]

  Powered by Linux