Issue: Git pull hang occasionally, and when git pull start hanging, need manually "kill -9" to stop hanging
Environment:
Server side:
Git version: 2.11.0
OS: ubuntu 12.04
Nginx: 1.9.7.4
fcgiwrap: 1.1.0
Git repo: None bare, small size (less than 5 MB including .git folder), small file number (less than 100 files)
Nginx config for git:
location ~* /git(\/.*) {
root /var/git;
fastcgi_buffers 256 8k;
fastcgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend;
fastcgi_param GIT_HTTP_EXPORT_ALL true;
fastcgi_param GIT_PROJECT_ROOT /var/git;
fastcgi_param PATH_INFO $1;
fastcgi_pass unix:/var/run/fcgiwrap.socket;
include /opt/openresty/nginx/conf/fastcgi_params;
}
Client side:
Git version: 2.11.0
OS: ubuntu 12.04
All git operations go through http only
End to end work flow:
Keep committing small files to non-bare git repo on server side (twice per second), message will be sent to client side for every update, once client receives message, do git pull
Hanging frequency:
Around 4 times a day
Hanging command stack:
root 32640 23228 0 20:51 ? 00:00:00 git pull -v remote_name master --allow-unrelated-histories
root 32641 32640 0 20:51 ? 00:00:00 git fetch --update-head-ok -v remote_name master
root 32642 32641 0 20:51 ? 00:00:00 git-remote-http remote_name http://server:80/git/repo_name/.git
root 32651 32642 0 20:51 ? 00:00:00 git fetch-pack --stateless-rpc --stdin --lock-pack --thin --no-progress http://server:80/git/repo_name/.git/
Access log for hanging git pull:
10.1.0.10 - - [20/Dec/2016:20:38:10 +0000] "GET /git/repo_name/.git/info/refs?service=git-upload-pack HTTP/1.1" 200 363 "-" "git/2.11.0" "-"
10.1.0.10 - - [20/Dec/2016:20:38:10 +0000] "POST /git/repo_name/.git/git-upload-pack HTTP/1.1" 200 5 "-" "git/2.11.0" "-"
Error log for hanging git pull:
2016/12/20 20:38:10 [error] 9957#0: *687703 FastCGI sent in stderr: "fatal: 'HEAD' is a symref but it is not?" while reading response header from upstream, client: 10.1.0.11, server: server, request: "POST /git/repo_name/.git/git-upload-pack HTTP/1.1", upstream: "fastcgi://unix:/var/run/fcgiwrap.socket:", host: "server"
Some observation:
1. When hanging happen, same repository could be cloned or pulled by another process on the same client.
2. After killing hanging git pull, during retry, same repository can be sync up successfully.
3. Git pull has been executed twice per second. But hanging only happens around 4 times a day.
4. When "fatal: 'HEAD' is a symref but it is not?" happen for POST on server side, client side always start to hang. And when hanging happen on client side, this log for POST always appears. But, if "fatal: 'HEAD' is a symref but it is not?" happen for GET request on server side, client side never hang. For example:
2016/12/20 20:36:53 [error] 9954#0: *685174 FastCGI sent in stderr: "fatal: 'HEAD' is a symref but it is not?" while reading response header from upstream, client: 10.1.0.11, server: server, request: "GET /git/repo_name/.git/info/refs?service=git-upload-pack HTTP/1.1", upstream: "fastcgi://unix:/var/run/fcgiwrap.socket:", host: "server"
will not trigger hanging on client side. And this log "fatal: 'HEAD' is a symref but it is not?" is happening very rare (less than 10 times a day).
It seems a error handling issue on client side. Any help or pointer on where to look will be appreciated.
Regards
Kai
PS. I am not subscribed to the mailing list, please keep me in Cc