Hei,
Story begins from trying to add weekly builds to one of our jobs in GitHub. It has run earlier just fine, but with something getting updated - it now started failing. The job just hangs when doing a git checkout. The test job was related to address sanitizer (asan) unit tests and turns out that if I do:
export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libasan.so.5"
or
export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libasan.so.8"
and then try to clone essentially any repo (all I've tried so far) it will hang in GitHub runners.
When trying locally - it does not hang, but seems we do have memory leaks according to asan.
- locally: Ubuntu 22.04 machine on Intel x68 (Intel Xeon, Lenovo P720).
- uname -a: Linux jannek-P720 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
Are these genuine findings or false reports? These are reporting both with 2.34.1 and also the latest I could get with with the git ppa, 2.54.2.
$ ldd /usr/bin/git
linux-vdso.so.1 (0x00007fff87d62000)
libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007e0abd828000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007e0abd3e4000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007e0abd000000)
/lib64/ld-linux-x86-64.so.2 (0x00007e0abd8da000)
$ git version
git version 2.45.2
$ export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libasan.so.8"
$ git clone git@xxxxxxxxxx:git/sha1collisiondetection.git
Cloning into 'sha1collisiondetection'...
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
=================================================================
==1119515==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 9 byte(s) in 1 object(s) allocated from:
#0 0x7b4d8fc73668 in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cpp:439
#1 0x5db29368e7b9 (/usr/bin/git+0x1737b9)
Direct leak of 9 byte(s) in 1 object(s) allocated from:
#0 0x7b4d8fc73668 in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cpp:439
#1 0x5db29368e7b9 (/usr/bin/git+0x1737b9)
#2 0x7ffcaa6c5847 ([stack]+0x20847)
SUMMARY: AddressSanitizer: 18 byte(s) leaked in 2 allocation(s).
You get slightly different ones with asan5.
$ export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libasan.so.5"
$ git clone git@xxxxxxxxxx:git/sha1collisiondetection.git
Cloning into 'sha1collisiondetection'...
remote: Enumerating objects: 418, done.
remote: Total 418 (delta 0), reused 0 (delta 0), pack-reused 418
Receiving objects: 100% (418/418), 497.09 KiB | 1.14 MiB/s, done.
Resolving deltas: 100% (244/244), done.
=================================================================
==1115811==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 2168 byte(s) in 1 object(s) allocated from:
#0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
#1 0x5797e91bb9e7 (/usr/bin/ssh+0x4c9e7)
Direct leak of 501 byte(s) in 1 object(s) allocated from:
#0 0x7400ef50d808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
#1 0x5797e91af4e6 (/usr/bin/ssh+0x404e6)
#2 0x6020000167ef (<unknown module>)
Direct leak of 464 byte(s) in 1 object(s) allocated from:
#0 0x7400ef50dc3e in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:163
#1 0x7400eea88114 (/lib/x86_64-linux-gnu/libc.so.6+0x88114)
Direct leak of 306 byte(s) in 1 object(s) allocated from:
#0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
#1 0x5797e91cd6c7 (/usr/bin/ssh+0x5e6c7)
Direct leak of 48 byte(s) in 1 object(s) allocated from:
#0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
#1 0x5797e91be54e (/usr/bin/ssh+0x4f54e)
Direct leak of 11 byte(s) in 1 object(s) allocated from:
#0 0x7400ef50d808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
#1 0x5797e91bc5d1 (/usr/bin/ssh+0x4d5d1)
Indirect leak of 184 byte(s) in 1 object(s) allocated from:
#0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
#1 0x5797e917c58e (/usr/bin/ssh+0xd58e)
Indirect leak of 184 byte(s) in 1 object(s) allocated from:
#0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
#1 0x5797e917c573 (/usr/bin/ssh+0xd573)
Indirect leak of 160 byte(s) in 1 object(s) allocated from:
#0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
#1 0x5797e917c532 (/usr/bin/ssh+0xd532)
Indirect leak of 51 byte(s) in 5 object(s) allocated from:
#0 0x7400ef50d808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
#1 0x5797e91bc5d1 (/usr/bin/ssh+0x4d5d1)
Indirect leak of 1 byte(s) in 1 object(s) allocated from:
#0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
#1 0x5797e91eb7da (/usr/bin/ssh+0x7c7da)
SUMMARY: AddressSanitizer: 4078 byte(s) leaked in 15 allocation(s).
fatal: remote transport reported error
=================================================================
==1115810==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 18 byte(s) in 2 object(s) allocated from:
#0 0x7dcc516963ed in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cc:445
#1 0x5afffb42d7b9 (/usr/bin/git+0x1737b9)
Direct leak of 9 byte(s) in 1 object(s) allocated from:
#0 0x7dcc516963ed in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cc:445
#1 0x5afffb42d7b9 (/usr/bin/git+0x1737b9)
#2 0x7fff76e70847 ([stack]+0x20847)
Direct leak of 5 byte(s) in 1 object(s) allocated from:
#0 0x7dcc516963ed in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cc:445
#1 0x5afffb433040 (/usr/bin/git+0x179040)
SUMMARY: AddressSanitizer: 32 byte(s) leaked in 4 allocation(s).
Seems to me there's a few potential issues:
- GitHub runners hanging (it should not hang, it should exit with error).
- Git itself having potential memory leaks.
- I am misusing the LD_PRELOAD and it can not or does not as I expect?
(i.e. it should just slow down the execution, but not report any failures if there are no leaks/other issues).
I have checked the git repo itself and you do have already some address sanitizing test jobs in there so I would assume you do not have leaks.
Workaround for now has been to introduce the actual export as late as possible.
Best Regards,
Janne Kiiskilä
