CCing in ceph-users:
That is a pretty old version of fio and I know a couple rbd-related
bugs / crashes have been fixed since fio 2.2.8. Can you retry using a
more up-to-date version of fio?
On Tue, Dec 6, 2016 at 2:40 AM, <yan.jun8@xxxxxxxxxx> wrote:
> Hello Jason,
>
> I'm from ZTE corporation, and we are using ceph as the backend storage of
> OpenStack.
>
> Sorry to bother you, but we stuck in the following exception when I run fio
> tool to test rbd image.
>
> Could you help to take a look? thanks.
>
>
> ---------------
>
> [Thread debugging using libthread_db enabled]
>
> Using host libthread_db library "/lib64/libthread_db.so.1".
>
> Core was generated by `fio fio.conf6'.
>
> Program terminated with signal 11, Segmentation fault.
>
> #0 0x00007ff841d6981e in Mutex::Lock (this=0x7ff818060800,
> no_lockdep=＜optimized out＞) at common/Mutex.cc:97
>
> 97 if (logger && cct && cct-＞_conf-＞mutex_perf_counter) {
>
> Missing separate debuginfos, use: debuginfo-install fio-2.2.8-2.el7.x86_64
>
> (gdb) bt
>
> #0 0x00007ff841d6981e in Mutex::Lock (this=0x7ff818060800,
> no_lockdep=＜optimized out＞) at common/Mutex.cc:97
>
> #1 0x00007ff841b981f0 in librbd::AioCompletion::get_return_value
> (this=0x7ff818060800) at librbd/AioCompletion.cc:217
>
> #2 0x0000000000452174 in _fio_rbd_finish_aiocb ()
>
> #3 0x00007ff841b97b1e in librbd::AioCompletion::complete
> (this=this@entry=0x7ff818079570) at librbd/AioCompletion.cc:107
>
> #4 0x00007ff841b99027 in librbd::AioCompletion::complete_request
> (this=0x7ff818079570, r=＜optimized out＞)
>
> at librbd/AioCompletion.cc:193
>
> #5 0x00007ff841b962c9 in Context::complete (this=0x7ff7f1375130,
> r=＜optimized out＞) at include/Context.h:64
>
> #6 0x00007ff841be23e7 in delete_me (this=0x7ff7f379d870) at
> include/Context.h:271
>
> #7 C_GatherBase＜Context, Context＞::sub_finish (this=0x7ff7f379d870,
> sub=0x7ff7f01cdb10, r=0) at include/Context.h:266
>
> #8 0x00007ff841be2502 in C_GatherBase＜Context,
> Context＞::C_GatherSub::finish (this=0x7ff7f01cdb10, r=＜optimized out＞)
>
> at include/Context.h:289
>
> #9 0x00007ff841bd29f9 in complete (r=＜optimized out＞, this=0x7ff7f01cdb10)
> at include/Context.h:64
>
> #10 C_GatherBase＜Context, Context＞::C_GatherSub::complete
> (this=0x7ff7f01cdb10, r=＜optimized out＞) at include/Context.h:286
>
> Python Exception ＜type 'exceptions.IndexError'＞ list index out of range:
>
> #11 0x00007ff8420193cc in finish_contexts＜Context＞ (cct=0x7ff818006d50,
> finished=empty std::list, result=result@entry=0)
>
> at include/Context.h:120
>
> #12 0x00007ff84200d6be in ObjectCacher::bh_write_commit
> (this=0x7ff7ec0010d0, poolid=＜optimized out＞, oid=...,
>
> ranges=std::vector of length 1, capacity 1 = {...}, tid=tid@entry=59669,
> r=r@entry=0) at osdc/ObjectCacher.cc:1117
>
> #13 0x00007ff84201ba68 in ObjectCacher::C_WriteCommit::finish
> (this=0x7ff7f13de770, r=0) at osdc/ObjectCacher.h:613
>
> #14 0x00007ff841b962c9 in Context::complete (this=0x7ff7f13de770,
> r=＜optimized out＞) at include/Context.h:64
>
> #15 0x00007ff841c2065e in librbd::LibrbdWriteback::complete_writes
> (this=0x7ff7ec0077f0, oid=...) at librbd/LibrbdWriteback.cc:351
>
> #16 0x00007ff841c222f4 in librbd::C_OrderedWrite::finish
> (this=0x7ff7f00011f0, r=0) at librbd/LibrbdWriteback.cc:87
>
> #17 0x00007ff841b962c9 in Context::complete (this=0x7ff7f00011f0,
> r=＜optimized out＞) at include/Context.h:64
>
> #18 0x00007ff841ba87f7 in
> librbd::AioObjectRequest＜librbd::ImageCtx＞::complete (this=0x7ff7f3e149e0,
> r=0)
>
> at librbd/AioObjectRequest.cc:97
>
> #19 0x00007ff838439d8d in librados::C_AioSafe::finish (this=0x7ff7f4003340,
> r=＜optimized out＞) at librados/AioCompletionImpl.h:199
>
> #20 0x00007ff838421c89 in Context::complete (this=0x7ff7f4003340,
> r=＜optimized out＞) at include/Context.h:64
>
> #21 0x00007ff838513996 in Finisher::finisher_thread_entry
> (this=0x7ff81800e6f0) at common/Finisher.cc:68
>
> #22 0x00007ff837650df3 in start_thread () from /lib64/libpthread.so.0
>
> #23 0x00007ff83717a54d in clone () from /lib64/libc.so.6
>
> (gdb) p cct
>
> $1 = (CephContext *) 0x1a5869000
>
> (gdb) p cct-＞_conf
>
> Cannot access memory at address 0x1a5869008
>
> (gdb)
>
> ----------------
>
>
> the ceph version is Jewel 10.2.3.1 and my fio.conf6 is as below:
>
> ----------------
>
> [global]
>
> direct=1
>
> time_based
>
> thread
>
> group_reporting
>
>
> [fiorbd]
>
> rw=randwrite #read、write、randread、randwrite
>
> bs=4k #4k、512k
>
> iodepth=32
>
> #ramp_time=30
>
> runtime=300
>
> ioengine=rbd
>
> clientname=admin
>
> pool=volumes
>
> rbdname=img6
>
> numjobs=1
>
> size=10240M
>
> -----------------
--
Jason
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
