Just came across what seems to be an (old) similar issue, where no resolution is mentioned (last reply 4 years ago): deadlock in fio.. client/server: https://www.spinics.net/lists/fio/msg03325.html \Rob On 08/03/2018, 17:13, "Sitsofe Wheeler" <sitsofe@xxxxxxxxx> wrote: OK wow that was big. The threads from 3 - 101 seem to be stuck with a backtrace that looks similar to this: Thread 3 (Thread 3784.0xd38): #0 0x00007ff845d06124 in ntdll!ZwWaitForSingleObject () from C:\Windows\SYSTEM32\ntdll.dll #1 0x00007ff842633acf in WaitForSingleObjectEx () from C:\Windows\System32\KernelBase.dll #2 0x000000000046c0df in pthread_mutex_lock_intern (timeout=4294967295, m=0x1e1880) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:167 #3 pthread_mutex_lock (m=m@entry=0x1b0000) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/mutex.c:188 #4 0x0000000000441213 in fio_mutex_down (mutex=0x1b0000) at mutex.c:220 #5 0x0000000000446eae in sfree_pool (ptr=0x2bc6060, pool=0x55c980 <mp>) at smalloc.c:312 #6 sfree (ptr=0x2bc6070) at smalloc.c:336 #7 0x000000000042df59 in close_and_free_files (td=td@entry=0xb0d2960) at filesetup.c:1358 #8 0x000000000045992c in thread_main (data=<optimized out>) at backend.c:1901 #9 0x000000000046df74 in pthread_create_wrapper (args=0x0) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492 #10 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll #11 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll #12 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll #13 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll #14 0x0000000000000000 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) From thread 102 - 201 things look like this: Thread 102 (Thread 3784.0x8d4): #0 0x00007ff845d061c4 in ntdll!ZwRemoveIoCompletion () from C:\Windows\SYSTEM32\ntdll.dll #1 0x00007ff84264ac2f in KERNELBASE!GetQueuedCompletionStatus () from C:\Windows\System32\KernelBase.dll #2 0x00000000004686b0 in IoCompletionRoutine (lpParameter=0xb001bd0) at engines/windowsaio.c:427 #3 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll #4 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll #5 0x0000000000000000 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) For completeness here's Thread 1 and 2: Thread 2 (Thread 3784.0x1574): #0 0x00007ff845d06bf4 in ntdll!ZwWaitForMultipleObjects () from C:\Windows\SYSTEM32\ntdll.dll #1 0x00007ff84264796f in WaitForMultipleObjectsEx () from C:\Windows\System32\KernelBase.dll #2 0x00007ff84264786e in WaitForMultipleObjects () from C:\Windows\System32\KernelBase.dll #3 0x000000000046b31f in do_sema_b_wait_intern (sema=sema@entry=0x8cc, nointerrupt=nointerrupt@entry=0, timeout=timeout@entry=500) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:625 #4 0x000000000046b60e in do_sema_b_wait (sema=0x8cc, nointerrupt=0, timeout=500, cs=0xafc9a60, val=0xafc9a88) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:584 #5 0x000000000046be8e in pthread_cond_timedwait_impl (c=c@entry=0x2c2d0f8, external_mutex=external_mutex@entry=0x2c2d0f0, t=t@entry=0x1ce9fe30, rel=rel@entry=0) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:504 #6 0x000000000046bea8 in pthread_cond_timedwait (c=c@entry=0x2c2d0f8, m=m@entry=0x2c2d0f0, t=t@entry=0x1ce9fe30) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/cond.c:515 #7 0x0000000000466978 in helper_thread_main (data=0x2c2d0d0) at helper_thread.c:95 #8 0x000000000046df74 in pthread_create_wrapper (args=0x0) at /usr/src/debug/mingw64-x86_64-winpthreads-5.0.3-1/src/thread.c:1492 #9 0x00007ff845a9b2ba in msvcrt!_beginthreadex () from C:\Windows\System32\msvcrt.dll #10 0x00007ff845a9b38c in msvcrt!_endthreadex () from C:\Windows\System32\msvcrt.dll #11 0x00007ff8448d8364 in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll #12 0x00007ff845cc7091 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll #13 0x0000000000000000 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 1 (Thread 3784.0x16fc): #0 0x00007ff845d06724 in ntdll!ZwDelayExecution () from C:\Windows\SYSTEM32\ntdll.dll #1 0x00007ff842647b47 in SleepEx () from C:\Windows\System32\KernelBase.dll #2 0x0000000000469fa8 in usleep (useconds=<optimized out>, useconds@entry=10000) at os/windows/posix.c:572 #3 0x000000000045d03a in do_usleep (usecs=10000) at backend.c:2129 #4 run_threads (sk_out=sk_out@entry=0x0) at backend.c:2444 #5 0x000000000045d3da in fio_backend (sk_out=sk_out@entry=0x0) at backend.c:2493 #6 0x0000000000480329 in main (argc=4, argv=0x1e13c0, envp=<optimized out>) at fio.c:65 This could take a while to figure out! On 8 March 2018 at 14:35, Rob Scheepens <rob.scheepens@xxxxxxxxxxx> wrote: > Sitsofe, > > See attached for the fio threads' backtraces. > > \Rob > > > On 08/03/2018, 14:39, "Rob Scheepens" <rob.scheepens@xxxxxxxxxxx> wrote: > > I'll see if I can get gdb setup like that. > > Below output shows spec 176 was the last job that ran ok, so you could try if it reproduces with 177. However, the job number exhibiting the hang usually is different every time. -- Sitsofe | https://urldefense.proofpoint.com/v2/url?u=http-3A__sucs.org_-7Esits_&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=OMged-t_5I_fmfpUaT3vaA06lgLL_alYnDQJxHmXz64&m=klmdlx61L5-QFKKNFgbuuwgQKtwsioCr4bZcTounUQM&s=fqSnEQXZz0hNXqRMlaIEDCOfmc15dbI0YFxXxr5aHrg&e= ��.n��������+%������w��{.n�������^n�r������&��z�ޗ�zf���h���~����������_��+v���)ߣ�
