Hi Yan,
ceph-fuse version: ceph-fuse-0.94.1 with some backports
fuse version:
fuse-libs-2.9.2-5.el7.x86_64
fuse-2.9.2-5.el7.x86_64
kernel version: 3.10.90-1
Please see the gdb backtrace file attached.
Regards,
Zhi Zhang (David)
Contact: zhang.david2011@xxxxxxxxx
zhangz.david@xxxxxxxxxxx
On Wed, Feb 3, 2016 at 2:07 PM, Yan, Zheng <ukernel@xxxxxxxxx> wrote:
> Which version kernel/ceph-fuse are you using? Besides, please use gdb
> to get backtrace of all threads in ceph-fuse and send the backtrace to
> us.
>
> Regards
> Yan, Zheng
>
> On Wed, Feb 3, 2016 at 12:54 PM, Zhi Zhang <zhang.david2011@xxxxxxxxx> wrote:
>> Hi ceph-devel,
>>
>> I know this mail might be better to send to fuse-devel, but I haven't
>> been approved by fuse-devel yet since I subscribed it.
>>
>> Occasionally we met ceph-fuse hanging issue when mapping ceph-fuse
>> directory into docker container using aufs.
>>
>> Here is the dmesg:
>>
>> [809401.613923] aufs au_opts_verify:1602:docker[36838]: dirperm1
>> breaks the protection by the permission bits on the lower branch
>> [825359.968412] aufs au_opts_verify:1602:docker[32013]: dirperm1
>> breaks the protection by the permission bits on the lower branch
>> [825359.970719] aufs au_opts_verify:1602:docker[32013]: dirperm1
>> breaks the protection by the permission bits on the lower branch
>> [825359.973689] aufs au_opts_verify:1602:docker[44954]: dirperm1
>> breaks the protection by the permission bits on the lower branch
>> [836447.630952] INFO: task df:30614 blocked for more than 120 seconds.
>> [836447.630955] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [836447.630957] df D ffff88203f851e00 0 30614 1 0x00000004
>> [836447.630960] ffff880a27b6fd08 0000000000000002 ffff880a27b6ffd8
>> 0000000000011e00
>> [836447.630964] ffff880a27b6ffd8 0000000000011e00 ffff880fe93f4b60
>> ffff880009f98000
>> [836447.630967] ffff881074050320 ffff881fd6333000 ffff880a27b6fd30
>> ffff881074050400
>> [836447.630970] Call Trace:
>> [836447.630977] [<ffffffff81acb689>] schedule+0x29/0x70
>> [836447.630983] [<ffffffff812be96d>] __fuse_request_send+0xdd/0x290
>> [836447.630987] [<ffffffff81066150>] ? wake_up_bit+0x30/0x30
>> [836447.630989] [<ffffffff812beb32>] fuse_request_send+0x12/0x20
>> [836447.630992] [<ffffffff812c3859>] fuse_do_getattr+0x109/0x2a0
>> [836447.630995] [<ffffffff812c4cd5>] fuse_update_attributes+0x75/0x80
>> [836447.630997] [<ffffffff812c4d23>] fuse_getattr+0x43/0x50
>> [836447.631001] [<ffffffff81168249>] vfs_getattr+0x29/0x40
>> [836447.631002] [<ffffffff811683b2>] vfs_fstatat+0x62/0xa0
>> [836447.631004] [<ffffffff8116859f>] SYSC_newstat+0x1f/0x40
>> [836447.631009] [<ffffffff8100e4f8>] ? syscall_trace_enter+0x18/0x210
>> [836447.631012] [<ffffffff81ad50bc>] ? tracesys+0x7e/0xe2
>> [836447.631014] [<ffffffff811689ee>] SyS_newstat+0xe/0x10
>> [836447.631016] [<ffffffff81ad511b>] tracesys+0xdd/0xe2
>>
>>
>> From above msg, it seems to be hung at __fuse_request_send, which will
>> queue request and wait for reply from ceph-fuse. When this happens,
>> 'ls' ceph-fuse directory or 'df' outside docker container will also
>> get hang.
>>
>> I generated ceph-fuse's core dump and found client_lock was not held
>> by any thread. So I wonder if something wrong with ceph-fuse that
>> can't get the request from the fuse's queue? Or something else related
>> to fuse itself?
>>
>> Thanks.
>>
>> Regards,
>> Zhi Zhang (David)
>> Contact: zhang.david2011@xxxxxxxxx
>> zhangz.david@xxxxxxxxxxx
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
Reading symbols from /usr/bin/ceph-fuse...Reading symbols from /data/usr/lib/debug/usr/bin/ceph-fuse.debug...done.
done.
[New LWP 13409]
[New LWP 13428]
[New LWP 13429]
[New LWP 13432]
[New LWP 13434]
[New LWP 13435]
[New LWP 13436]
[New LWP 13437]
[New LWP 13438]
[New LWP 13439]
[New LWP 13446]
[New LWP 13447]
[New LWP 13465]
[New LWP 13466]
[New LWP 13468]
[New LWP 13490]
[New LWP 13407]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/bin/ceph-fuse'.
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
warning: Source file is more recent than executable.
185 62: movl (%rsp), %edi
Missing separate debuginfos, use: debuginfo-install boost-system-1.53.0-18.el7.x86_64 boost-thread-1.53.0-18.el7.x86_64 fuse-libs-2.9.2-5.el7.x86_64 libgcc-4.8.2-16.2.el7_0.x86_64 libstdc++-4.8.2-16.2.el7_0.x86_64 libuuid-2.23.2-16.el7.x86_64 nspr-4.10.6-1.el7_0.x86_64 nss-3.16.2.3-2.el7_0.x86_64 nss-softokn-3.16.2.3-1.el7_0.x86_64 nss-softokn-freebl-3.16.2.3-1.el7_0.x86_64 nss-util-3.16.2.3-1.el7_0.x86_64 sqlite-3.7.17-4.el7.x86_64
(gdb) info thread
Id Target Id Frame
17 Thread 0x7fbfeb7d3780 (LWP 13407) 0x00007fbfea913f37 in pthread_join (threadid=140461847070464, thread_return=0x0) at pthread_join.c:92
16 Thread 0x7fbfd27fc700 (LWP 13490) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
15 Thread 0x7fbfd8ff9700 (LWP 13468) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
14 Thread 0x7fbfd97fa700 (LWP 13466) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
13 Thread 0x7fbfd9ffb700 (LWP 13465) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
12 Thread 0x7fbfda7fc700 (LWP 13447) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
11 Thread 0x7fbfdaffd700 (LWP 13446) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
10 Thread 0x7fbfdb7fe700 (LWP 13439) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
9 Thread 0x7fbfdbfff700 (LWP 13438) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
8 Thread 0x7fbfe0ff9700 (LWP 13437) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
7 Thread 0x7fbfe17fa700 (LWP 13436) 0x00007fbfe9a14793 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
6 Thread 0x7fbfe1ffb700 (LWP 13435) 0x00007fbfe9a14793 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
5 Thread 0x7fbfe27fc700 (LWP 13434) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
4 Thread 0x7fbfe2ffd700 (LWP 13432) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
3 Thread 0x7fbfe37fe700 (LWP 13429) 0x00007fbfe9a09bed in poll () at ../sysdeps/unix/syscall-template.S:81
2 Thread 0x7fbfe3fff700 (LWP 13428) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
* 1 Thread 0x7fbfe90e4700 (LWP 13409) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x000000000064d9cb in ceph::log::Log::entry (this=0x2b0a1b0) at log/Log.cc:349
#2 0x00007fbfea912df3 in start_thread (arg=0x7fbfe90e4700) at pthread_create.c:308
#3 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 2
[Switching to thread 2 (Thread 0x7fbfe3fff700 (LWP 13428))]
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
warning: Source file is more recent than executable.
238 62: movq %rax, %r14
(gdb) bt
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x000000000072b0aa in WaitUntil (when=..., mutex=..., this=0x2b245a8) at common/Cond.h:71
#2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x2b245a8) at common/Cond.h:79
#3 CephContextServiceThread::entry (this=0x2b24530) at common/ceph_context.cc:60
#4 0x00007fbfea912df3 in start_thread (arg=0x7fbfe3fff700) at pthread_create.c:308
#5 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 3
[Switching to thread 3 (Thread 0x7fbfe37fe700 (LWP 13429))]
#0 0x00007fbfe9a09bed in poll () at ../sysdeps/unix/syscall-template.S:81
warning: Source file is more recent than executable.
81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
#0 0x00007fbfe9a09bed in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00000000007140d3 in poll (__timeout=-1, __nfds=2, __fds=0x7fbfe37fdd70) at /usr/include/bits/poll2.h:46
#2 AdminSocket::entry (this=0x2b0a870) at common/admin_socket.cc:239
#3 0x00007fbfea912df3 in start_thread (arg=0x7fbfe37fe700) at pthread_create.c:308
#4 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 4
[Switching to thread 4 (Thread 0x7fbfe2ffd700 (LWP 13432))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 62: movl (%rsp), %edi
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x000000000063c5ac in Wait (mutex=..., this=0x2b4dd98) at common/Cond.h:55
#2 Finisher::finisher_thread_entry (this=0x2b4dd38) at common/Finisher.cc:81
#3 0x00007fbfea912df3 in start_thread (arg=0x7fbfe2ffd700) at pthread_create.c:308
#4 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 5
[Switching to thread 5 (Thread 0x7fbfe27fc700 (LWP 13434))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 62: movl (%rsp), %edi
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x000000000063c5ac in Wait (mutex=..., this=0x2b4ac20) at common/Cond.h:55
#2 Finisher::finisher_thread_entry (this=0x2b4abc0) at common/Finisher.cc:81
#3 0x00007fbfea912df3 in start_thread (arg=0x7fbfe27fc700) at pthread_create.c:308
#4 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 6
[Switching to thread 6 (Thread 0x7fbfe1ffb700 (LWP 13435))]
#0 0x00007fbfe9a14793 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
warning: Source file is more recent than executable.
81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
#0 0x00007fbfe9a14793 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1 0x0000000000833672 in EpollDriver::event_wait (this=0x2b24f60, fired_events=std::vector of length 0, capacity 0, tvp=<optimized out>) at msg/async/EventEpoll.cc:115
#2 0x000000000082f553 in EventCenter::process_events (this=this@entry=0x2b26520, timeout_microseconds=timeout_microseconds@entry=30000000) at msg/async/Event.cc:369
#3 0x00000000007f11e0 in Worker::entry (this=0x2b264e0) at msg/async/AsyncMessenger.cc:304
#4 0x00007fbfea912df3 in start_thread (arg=0x7fbfe1ffb700) at pthread_create.c:308
#5 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 7
[Switching to thread 7 (Thread 0x7fbfe17fa700 (LWP 13436))]
#0 0x00007fbfe9a14793 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
#0 0x00007fbfe9a14793 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1 0x0000000000833672 in EpollDriver::event_wait (this=0x2b35a20, fired_events=std::vector of length 0, capacity 0, tvp=<optimized out>) at msg/async/EventEpoll.cc:115
#2 0x000000000082f553 in EventCenter::process_events (this=this@entry=0x2b355d0, timeout_microseconds=timeout_microseconds@entry=30000000) at msg/async/Event.cc:369
#3 0x00000000007f11e0 in Worker::entry (this=0x2b35590) at msg/async/AsyncMessenger.cc:304
#4 0x00007fbfea912df3 in start_thread (arg=0x7fbfe17fa700) at pthread_create.c:308
#5 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 8
[Switching to thread 8 (Thread 0x7fbfe0ff9700 (LWP 13437))]
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
238 62: movq %rax, %r14
(gdb) bt
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x000000000071065f in WaitUntil (when=..., mutex=..., this=0x2b4a5d0) at common/Cond.h:71
#2 SafeTimer::timer_thread (this=0x2b4a5c0) at common/Timer.cc:118
#3 0x0000000000710f9d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4 0x00007fbfea912df3 in start_thread (arg=0x7fbfe0ff9700) at pthread_create.c:308
#5 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 9
[Switching to thread 9 (Thread 0x7fbfdbfff700 (LWP 13438))]
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
238 62: movq %rax, %r14
(gdb) bt
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x00000000005dc232 in WaitUntil (when=..., mutex=..., this=0x2b4dcc8) at common/Cond.h:71
#2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x2b4dcc8) at common/Cond.h:79
#3 ObjectCacher::flusher_entry (this=0x2b4dae0) at osdc/ObjectCacher.cc:1546
#4 0x00000000005ecd0d in ObjectCacher::FlusherThread::entry (this=<optimized out>) at osdc/ObjectCacher.h:374
#5 0x00007fbfea912df3 in start_thread (arg=0x7fbfdbfff700) at pthread_create.c:308
#6 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 10
[Switching to thread 10 (Thread 0x7fbfdb7fe700 (LWP 13439))]
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
238 62: movq %rax, %r14
(gdb) bt
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x000000000071065f in WaitUntil (when=..., mutex=..., this=0x2b4b858) at common/Cond.h:71
#2 SafeTimer::timer_thread (this=0x2b4b848) at common/Timer.cc:118
#3 0x0000000000710f9d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4 0x00007fbfea912df3 in start_thread (arg=0x7fbfdb7fe700) at pthread_create.c:308
#5 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 11
[Switching to thread 11 (Thread 0x7fbfdaffd700 (LWP 13446))]
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
238 62: movq %rax, %r14
(gdb) bt
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x000000000071065f in WaitUntil (when=..., mutex=..., this=0x2b254b8) at common/Cond.h:71
#2 SafeTimer::timer_thread (this=0x2b254a8) at common/Timer.cc:118
#3 0x0000000000710f9d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4 0x00007fbfea912df3 in start_thread (arg=0x7fbfdaffd700) at pthread_create.c:308
#5 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 12
[Switching to thread 12 (Thread 0x7fbfda7fc700 (LWP 13447))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 62: movl (%rsp), %edi
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x000000000063c5ac in Wait (mutex=..., this=0x2b255d0) at common/Cond.h:55
#2 Finisher::finisher_thread_entry (this=0x2b25570) at common/Finisher.cc:81
#3 0x00007fbfea912df3 in start_thread (arg=0x7fbfda7fc700) at pthread_create.c:308
#4 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 13
[Switching to thread 13 (Thread 0x7fbfd9ffb700 (LWP 13465))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 62: movl (%rsp), %edi
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x000000000063c5ac in Wait (mutex=..., this=0x2b4a860) at common/Cond.h:55
#2 Finisher::finisher_thread_entry (this=0x2b4a800) at common/Finisher.cc:81
#3 0x00007fbfea912df3 in start_thread (arg=0x7fbfd9ffb700) at pthread_create.c:308
#4 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 14
[Switching to thread 14 (Thread 0x7fbfd97fa700 (LWP 13466))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 62: movl (%rsp), %edi
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x000000000063c5ac in Wait (mutex=..., this=0x2b4a9a0) at common/Cond.h:55
#2 Finisher::finisher_thread_entry (this=0x2b4a940) at common/Finisher.cc:81
#3 0x00007fbfea912df3 in start_thread (arg=0x7fbfd97fa700) at pthread_create.c:308
#4 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 15
[Switching to thread 15 (Thread 0x7fbfd8ff9700 (LWP 13468))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 62: movl (%rsp), %edi
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x000000000063c5ac in Wait (mutex=..., this=0x2b4aae0) at common/Cond.h:55
#2 Finisher::finisher_thread_entry (this=0x2b4aa80) at common/Finisher.cc:81
#3 0x00007fbfea912df3 in start_thread (arg=0x7fbfd8ff9700) at pthread_create.c:308
#4 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 16
[Switching to thread 16 (Thread 0x7fbfd27fc700 (LWP 13490))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 62: movl (%rsp), %edi
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x000000000054ab5a in Wait (mutex=..., this=0x7fbfd27fb6b0) at common/Cond.h:55
Python Exception <type 'exceptions.IndexError'> list index out of range:
#2 Client::wait_on_list (this=this@entry=0x2b4a590, ls=std::list) at client/Client.cc:3086
#3 0x0000000000572dcc in Client::get_caps (this=this@entry=0x2b4a590, in=in@entry=0x7fbfcc00a6f0, need=need@entry=2048, want=want@entry=1024,
phave=phave@entry=0x7fbfd27fb9a8, endoff=endoff@entry=-1) at client/Client.cc:2719
#4 0x0000000000591e5a in Client::_read (this=this@entry=0x2b4a590, f=f@entry=0x7fbfc800f3b0, offset=offset@entry=0, size=size@entry=4096, bl=bl@entry=0x7fbfd27fbc10)
at client/Client.cc:6946
#5 0x0000000000592822 in Client::ll_read (this=0x2b4a590, fh=fh@entry=0x7fbfc800f3b0, off=off@entry=0, len=len@entry=4096, bl=bl@entry=0x7fbfd27fbc10)
at client/Client.cc:9789
#6 0x00000000005462eb in fuse_ll_read (req=0x7fbfc40008c0, ino=<optimized out>, size=4096, off=0, fi=<optimized out>) at client/fuse_ll.cc:447
#7 0x00007fbfeb4bb14e in do_read () from /lib64/libfuse.so.2
#8 0x00007fbfeb4bbbeb in fuse_ll_process_buf () from /lib64/libfuse.so.2
#9 0x00007fbfeb4b8481 in fuse_do_work () from /lib64/libfuse.so.2
#10 0x00007fbfea912df3 in start_thread (arg=0x7fbfd27fc700) at pthread_create.c:308
#11 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) thread 17
[Switching to thread 17 (Thread 0x7fbfeb7d3780 (LWP 13407))]
#0 0x00007fbfea913f37 in pthread_join (threadid=140461847070464, thread_return=0x0) at pthread_join.c:92
warning: Source file is more recent than executable.
92 lll_wait_tid (pd->tid);
(gdb) bt
#0 0x00007fbfea913f37 in pthread_join (threadid=140461847070464, thread_return=0x0) at pthread_join.c:92
#1 0x00007fbfeb4b86ab in fuse_session_loop_mt () from /lib64/libfuse.so.2
#2 0x00000000005416ff in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at ceph_fuse.cc:218
(gdb) thread 16
[Switching to thread 16 (Thread 0x7fbfd27fc700 (LWP 13490))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 62: movl (%rsp), %edi
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x000000000054ab5a in Wait (mutex=..., this=0x7fbfd27fb6b0) at common/Cond.h:55
Python Exception <type 'exceptions.IndexError'> list index out of range:
#2 Client::wait_on_list (this=this@entry=0x2b4a590, ls=std::list) at client/Client.cc:3086
#3 0x0000000000572dcc in Client::get_caps (this=this@entry=0x2b4a590, in=in@entry=0x7fbfcc00a6f0, need=need@entry=2048, want=want@entry=1024,
phave=phave@entry=0x7fbfd27fb9a8, endoff=endoff@entry=-1) at client/Client.cc:2719
#4 0x0000000000591e5a in Client::_read (this=this@entry=0x2b4a590, f=f@entry=0x7fbfc800f3b0, offset=offset@entry=0, size=size@entry=4096, bl=bl@entry=0x7fbfd27fbc10)
at client/Client.cc:6946
#5 0x0000000000592822 in Client::ll_read (this=0x2b4a590, fh=fh@entry=0x7fbfc800f3b0, off=off@entry=0, len=len@entry=4096, bl=bl@entry=0x7fbfd27fbc10)
at client/Client.cc:9789
#6 0x00000000005462eb in fuse_ll_read (req=0x7fbfc40008c0, ino=<optimized out>, size=4096, off=0, fi=<optimized out>) at client/fuse_ll.cc:447
#7 0x00007fbfeb4bb14e in do_read () from /lib64/libfuse.so.2
#8 0x00007fbfeb4bbbeb in fuse_ll_process_buf () from /lib64/libfuse.so.2
#9 0x00007fbfeb4b8481 in fuse_do_work () from /lib64/libfuse.so.2
#10 0x00007fbfea912df3 in start_thread (arg=0x7fbfd27fc700) at pthread_create.c:308
#11 0x00007fbfe9a141bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) frame 3
#3 0x0000000000572dcc in Client::get_caps (this=this@entry=0x2b4a590, in=in@entry=0x7fbfcc00a6f0, need=need@entry=2048, want=want@entry=1024,
phave=phave@entry=0x7fbfd27fb9a8, endoff=endoff@entry=-1) at client/Client.cc:2719
2719 wait_on_list(in->waitfor_caps);
(gdb) p *in
$2 = {cct = 0x2b07420, ino = {val = 1099516325291}, snapid = {val = 18446744073709551614}, rdev = 0, ctime = {tv = {tv_sec = 1452970201, tv_nsec = 757494730}},
mode = 33188, uid = 0, gid = 0, nlink = 0, dir_layout = {dl_dir_hash = 0 '\000', dl_unused1 = 0 '\000', dl_unused2 = 0, dl_unused3 = 0}, layout = {fl_stripe_unit = {
v = 4194304}, fl_stripe_count = {v = 1}, fl_object_size = {v = 4194304}, fl_cas_hash = {v = 0}, fl_object_stripe_unit = {v = 0}, fl_unused = {v = 4294967295},
fl_pg_pool = {v = 1}}, size = 15, truncate_seq = 2, truncate_size = 0, mtime = {tv = {tv_sec = 1452970201, tv_nsec = 762918432}}, atime = {tv = {tv_sec = 1452970201,
tv_nsec = 633606048}}, time_warp_seq = 0, max_size = 4194304, dirstat = {<scatter_info_t> = {version = 0}, mtime = {tv = {tv_sec = 0, tv_nsec = 0}}, nfiles = 0,
nsubdirs = 0}, rstat = {<scatter_info_t> = {version = 0}, rctime = {tv = {tv_sec = 0, tv_nsec = 0}}, rbytes = 15, rfiles = 1, rsubdirs = 0, rsnaprealms = 0},
version = 47096608, xattr_version = 1, inline_version = 18446744073709551615, inline_data = {_buffers = empty std::list, _len = 0, _memcopy_count = 0, append_buffer = {
_raw = 0x0, _off = 0, _len = 0}, last_p = {bl = 0x7fbfcc00a7e8, ls = 0x7fbfcc00a7e8, off = 0, p = {_raw = , _off = 0, _len = 0}, p_off = 0},
static CLAIM_DEFAULT = 0, static CLAIM_ALLOW_NONSHAREABLE = 1}, flags = 0, quota = {max_bytes = 0, max_files = 0}, qtree = 0x0,
dir_contacts = std::set with 0 elements, dir_hashed = false, dir_replicated = false, caps = std::map with 1 elements = {
[{<boost::totally_ordered1<mds_rank_t, boost::totally_ordered2<mds_rank_t, int, boost::detail::empty_base<mds_rank_t> > >> = {<boost::less_than_comparable1<mds_rank_t, boost::equality_comparable1<mds_rank_t, boost::totally_ordered2<mds_rank_t, int, boost::detail::empty_base<mds_rank_t> > > >> = {<boost::equality_comparable1<mds_rank_t, boost::totally_ordered2<mds_rank_t, int, boost::detail::empty_base<mds_rank_t> > >> = {<boost::totally_ordered2<mds_rank_t, int, boost::detail::empty_base<mds_rank_t> >> = {<boost::less_than_comparable2<mds_rank_t, int, boost::equality_comparable2<mds_rank_t, int, boost::detail::empty_base<mds_rank_t> > >> = {<boost::equality_comparable2<mds_rank_t, int, boost::detail::empty_base<mds_rank_t> >> = {<boost::detail::empty_base<mds_rank_t>> = {<No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, t = 0}] = 0x7fbfcc0031e0}, auth_cap = 0x7fbfcc0031e0, dirty_caps = 0, flushing_caps = 0, flushing_cap_seq = 48,
flushing_cap_tid = {0 <repeats 12 times>, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0}, shared_gen = 1, cache_gen = 2, snap_caps = 0, snap_cap_refs = 0, hold_caps_until = {tv = {
tv_sec = 0, tv_nsec = 0}}, cap_item = {_item = 0x7fbfcc00a6f0, _prev = 0x0, _next = 0x7fbfcc002bc0, _list = 0x2b4aef8}, flushing_cap_item = {_item = 0x7fbfcc00a6f0,
_prev = 0x0, _next = 0x0, _list = 0x0}, last_flush_tid = 1, snaprealm = 0x7fbfcc002e40, snaprealm_item = {_item = 0x7fbfcc00a6f0, _prev = 0x7fbfcc002c10, _next = 0x0,
_list = 0x7fbfcc002ef8}, snapdir_parent = 0x0, cap_snaps = std::map with 0 elements, open_by_mode = std::map with 2 elements = {[1] = 1, [2] = 0},
cap_refs = std::map with 3 elements = {[1024] = 0, [4096] = 0, [8192] = 0}, oset = {parent = 0x7fbfcc00a6f0, ino = {val = 1099516325291}, truncate_seq = 2,
truncate_size = 0, poolid = 1, objects = {_front = 0x0, _back = 0x0, _size = 0}, dirty_or_tx = 0, return_enoent = false}, reported_size = 15, wanted_max_size = 15,
requested_max_size = 15, _ref = 2, ll_ref = 3, dir = 0x0, dn_set = std::set with 0 elements, symlink = "", dirfragtree = {_splits = std::map with 0 elements},
xattrs = std::map with 0 elements, fragmap = std::map with 0 elements, waitfor_caps = std::list = {[0] = 0x7fbfd27fb6b0}, waitfor_commit = empty std::list,
fcntl_locks = 0x0, flock_locks = 0x0, async_err = 0}
(gdb) p *(Cap *)0x7fbfcc0031e0
$3 = {session = 0x2b58a40, inode = 0x7fbfcc00a6f0, cap_item = {_item = 0x7fbfcc0031e0, _prev = 0x0, _next = 0x7fbfcc002f30, _list = 0x2b58b30}, cap_id = 5981977,
issued = 9301, implemented = 9301, wanted = 3329, seq = 19, issue_seq = 19, mseq = 0, gen = 0}
(gdb)
