I'm trying to assist a Bitbucket Server customer who is seeing some "git-upload-pack" processes "hang" on their server. While investigating, we had them connect gdb to their processes (which are 2.10.0 built from source using a simple unzip-and-run-make approach) and get the backtraces for them. The output that they're seeing makes no sense to me, though, so I'm throwing this out to the list just to see if anyone has any idea how the processes could end up like this. One "family" of "hung" processes looks like this in ps: 32432 7811 atlbitb+ Mar22 /bin/git upload-pack /path/to/repository 32433 32432 atlbitb+ Mar22 git-upload-pack /path/to/repository Looks normal enough. The backtrace for process 32432 also looks like I'd expect: #0 0x00007fa39842fc9c in __libc_waitpid (pid=pid@entry=32433, stat_loc=stat_loc@entry=0x7fffb42a4e4c, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:31 #1 0x00000000005098f2 in wait_or_whine (pid=32433, argv0=0xd6e0c0 "git-upload-pack", in_signal=in_signal@entry=0) at run-command.c:229 #2 0x000000000050a949 in finish_command (cmd=cmd@entry=0x7fffb42a4ea0) at run-command.c:537 #3 0x000000000050a9c1 in run_command (cmd=cmd@entry=0x7fffb42a4ea0) at run-command.c:558 #4 0x000000000050aa5e in run_command_v_opt_cd_env (argv=argv@entry=0x7fffb42a5140, opt=opt@entry=40, dir=dir@entry=0x0, env=env@entry=0x0) at run-command.c:578 #5 0x000000000050aa79 in run_command_v_opt (argv=argv@entry=0x7fffb42a5140, opt=opt@entry=40) at run-command.c:563 #6 0x0000000000405ad0 in execv_dashed_external (argv=0x7fffb42a5140) at git.c:569 #7 run_argv (argv=0x7fffb42a4f10, argcp=0x7fffb42a4f1c) at git.c:596 #8 cmd_main (argc=2, argc@entry=3, argv=0x7fffb42a5140, argv@entry=0x7fffb42a5138) at git.c:665 #9 0x0000000000404d9d in main (argc=3, argv=0x7fffb42a5138) at common-main.c:40 It's in "wait_or_whine" for "git-upload-pack". When they attached to 32433 and printed its backtrace, though, things go a little sideways: (gdb) attach 32433 Attaching to program: /usr/bin/git, process 32433 Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/usr/lib64/ld-2.17.so.debug...done. done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 0x00007f79d1aca240 in ?? () (gdb) bt #0 0x00007f79d1aca240 in ?? () #1 0x000000000045179e in mktree_line (allow_missing=4, nul_term_line=0, len=<optimized out>, buf=<optimized out>) at builtin/mktree.c:103 #2 cmd_mktree (ac=<optimized out>, av=<optimized out>, prefix=<optimized out>) at builtin/mktree.c:173 #3 0x0000000000000000 in ?? () So ps shows "git-upload-pack", and the parent process shows it's waiting for "git-upload-pack", but gdb shows it's attaching to "git", not "git-upload-pack", and the stack trace on the child shows it's running "git mktree". I've searched through the source code and I don't see any path through upload-pack.c that could result in it essentially exec'ing "git mktree" over the top of itself, or even spawning a "git mktree" of its own. Has anyone seen anything like this before? Any thoughts on how what should be a "git-upload-pack" could possibly end up being a "git mktree" instead? Best regards, Bryan
