On Mon, Nov 3, 2014 at 9:31 AM, <duan.xufeng@xxxxxxxxxx> wrote:
>
> root@CONTROLLER-4F:~# rbd -p volumes flatten
> f3e81ea3-1d5b-487a-a55e-53efff604d54_disk
> *** Caught signal (Segmentation fault) **
> in thread 7fe99984f700
> ceph version 0.79 (4c2d73a5095f527c3a2168deb5fa54b3c8991a6e)
> 1: (()+0x22a4f) [0x7fe9a1745a4f]
> 2: (()+0x10340) [0x7fe9a00f2340]
> 3: (librbd::aio_read(librbd::ImageCtx*, std::vector<std::pair<unsigned
> long, unsigned long>, std::allocator<std::pair<unsigned long, unsigned long>
>> > const&, char*, ceph::buffer::list*, librbd::AioCompletion*)+0x24)
> [0x7fe9a125daf4]
> 4: (librbd::AioRequest::read_from_parent(std::vector<std::pair<unsigned
> long, unsigned long>, std::allocator<std::pair<unsigned long, unsigned long>
>> >&)+0x85) [0x7fe9a1242745]
> 5: (librbd::AioRead::should_complete(int)+0x352) [0x7fe9a1242ca2]
> 6: (librbd::rados_req_cb(void*, void*)+0x1b) [0x7fe9a124cd7b]
> 7: (librados::C_AioComplete::finish(int)+0x1d) [0x7fe9a04a355d]
> 8: (Context::complete(int)+0x9) [0x7fe9a0480579]
> 9: (Finisher::finisher_thread_entry()+0x1b8) [0x7fe9a0531758]
> 10: (()+0x8182) [0x7fe9a00ea182]
> 11: (clone()+0x6d) [0x7fe99f2ce30d]
> 2014-11-03 14:21:02.413259 7fe99984f700 -1 *** Caught signal (Segmentation
> fault) **
> in thread 7fe99984f700
>
> ceph version 0.79 (4c2d73a5095f527c3a2168deb5fa54b3c8991a6e)
> 1: (()+0x22a4f) [0x7fe9a1745a4f]
> 2: (()+0x10340) [0x7fe9a00f2340]
> 3: (librbd::aio_read(librbd::ImageCtx*, std::vector<std::pair<unsigned
> long, unsigned long>, std::allocator<std::pair<unsigned long, unsigned long>
>> > const&, char*, ceph::buffer::list*, librbd::AioCompletion*)+0x24)
> [0x7fe9a125daf4]
> 4: (librbd::AioRequest::read_from_parent(std::vector<std::pair<unsigned
> long, unsigned long>, std::allocator<std::pair<unsigned long, unsigned long>
>> >&)+0x85) [0x7fe9a1242745]
> 5: (librbd::AioRead::should_complete(int)+0x352) [0x7fe9a1242ca2]
> 6: (librbd::rados_req_cb(void*, void*)+0x1b) [0x7fe9a124cd7b]
> 7: (librados::C_AioComplete::finish(int)+0x1d) [0x7fe9a04a355d]
> 8: (Context::complete(int)+0x9) [0x7fe9a0480579]
> 9: (Finisher::finisher_thread_entry()+0x1b8) [0x7fe9a0531758]
> 10: (()+0x8182) [0x7fe9a00ea182]
> 11: (clone()+0x6d) [0x7fe99f2ce30d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
> interpret this.
>
> --- begin dump of recent events ---
> -113> 2014-11-03 14:21:01.948799 7fe9a170c7c0 5 asok(0x7fe9a33483f0)
> register_command perfcounters_dump hook 0x7fe9a3349ee0
> -112> 2014-11-03 14:21:01.948850 7fe9a170c7c0 5 asok(0x7fe9a33483f0)
> register_command 1 hook 0x7fe9a3349ee0
> -111> 2014-11-03 14:21:01.948856 7fe9a170c7c0 5 asok(0x7fe9a33483f0)
> register_command perf dump hook 0x7fe9a3349ee0
> -110> 2014-11-03 14:21:01.948894 7fe9a170c7c0 5 asok(0x7fe9a33483f0)
> register_command perfcounters_schema hook 0x7fe9a3349ee0
> -109> 2014-11-03 14:21:01.948906 7fe9a170c7c0 5 asok(0x7fe9a33483f0)
> register_command 2 hook 0x7fe9a3349ee0
> -108> 2014-11-03 14:21:01.948915 7fe9a170c7c0 5 asok(0x7fe9a33483f0)
> register_command perf schema hook 0x7fe9a3349ee0
> -107> 2014-11-03 14:21:01.948919 7fe9a170c7c0 5 asok(0x7fe9a33483f0)
> register_command config show hook 0x7fe9a3349ee0
> -106> 2014-11-03 14:21:01.948931 7fe9a170c7c0 5 asok(0x7fe9a33483f0)
> register_command config set hook 0x7fe9a3349ee0
> -105> 2014-11-03 14:21:01.948936 7fe9a170c7c0 5 asok(0x7fe9a33483f0)
> register_command config get hook 0x7fe9a3349ee0
> -104> 2014-11-03 14:21:01.948944 7fe9a170c7c0 5 asok(0x7fe9a33483f0)
> register_command log flush hook 0x7fe9a3349ee0
> -103> 2014-11-03 14:21:01.948947 7fe9a170c7c0 5 asok(0x7fe9a33483f0)
> register_command log dump hook 0x7fe9a3349ee0
> -102> 2014-11-03 14:21:01.948954 7fe9a170c7c0 5 asok(0x7fe9a33483f0)
> register_command log reopen hook 0x7fe9a3349ee0
> -101> 2014-11-03 14:21:01.955080 7fe9a170c7c0 10 monclient(hunting):
> build_initial_monmap
> -100> 2014-11-03 14:21:01.955154 7fe9a170c7c0 1 librados: starting msgr
> at :/0
> -99> 2014-11-03 14:21:01.955169 7fe9a170c7c0 1 librados: starting
> objecter
> -98> 2014-11-03 14:21:01.955227 7fe9a170c7c0 1 -- :/0 messenger.start
> -97> 2014-11-03 14:21:01.955271 7fe9a170c7c0 1 librados: setting wanted
> keys
> -96> 2014-11-03 14:21:01.955279 7fe9a170c7c0 1 librados: calling
> monclient init
> -95> 2014-11-03 14:21:01.955280 7fe9a170c7c0 10 monclient(hunting): init
> -94> 2014-11-03 14:21:01.955295 7fe9a170c7c0 5 adding auth protocol:
> cephx
> -93> 2014-11-03 14:21:01.955304 7fe9a170c7c0 10 monclient(hunting):
> auth_supported 2 method cephx
> -92> 2014-11-03 14:21:01.955521 7fe9a170c7c0 2 auth: KeyRing::load:
> loaded key file /etc/ceph/ceph.client.admin.keyring
> -91> 2014-11-03 14:21:01.955627 7fe9a170c7c0 10 monclient(hunting):
> _reopen_session rank -1 name
> -90> 2014-11-03 14:21:01.955718 7fe9a170c7c0 10 monclient(hunting):
> picked mon.noname-a con 0x7fe9a336b660 addr 192.129.0.230:6789/0
> -89> 2014-11-03 14:21:01.955769 7fe9a170c7c0 10 monclient(hunting):
> _send_mon_message to mon.noname-a at 192.129.0.230:6789/0
> -88> 2014-11-03 14:21:01.955781 7fe9a170c7c0 1 -- :/1025623 -->
> 192.129.0.230:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0
> 0x7fe9a336bab0 con 0x7fe9a336b660
> -87> 2014-11-03 14:21:01.955797 7fe9a170c7c0 10 monclient(hunting):
> renew_subs
> -86> 2014-11-03 14:21:01.955805 7fe9a170c7c0 10 monclient(hunting):
> authenticate will time out at 2014-11-03 14:26:01.955805
> -85> 2014-11-03 14:21:01.956454 7fe9a1704700 1 --
> 192.129.0.201:0/1025623 learned my addr 192.129.0.201:0/1025623
> -84> 2014-11-03 14:21:01.957432 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== mon.0 192.129.0.230:6789/0 1 ==== mon_map v1
> ==== 494+0+0 (1741557205 0 0) 0x7fe98c000b90 con 0x7fe9a336b660
> -83> 2014-11-03 14:21:01.957485 7fe99c155700 10 monclient(hunting):
> handle_monmap mon_map v1
> -82> 2014-11-03 14:21:01.957513 7fe99c155700 10 monclient(hunting): got
> monmap 2, mon.noname-a is now rank -1
> -81> 2014-11-03 14:21:01.957520 7fe99c155700 10 monclient(hunting): dump:
> epoch 2
> fsid d0766792-176e-4ffe-a911-6a3b373c28e1
> last_changed 2014-10-13 09:48:17.907147
> created 0.000000
> 0: 192.129.0.230:6789/0 mon.SCU-4F-1
> 1: 192.129.0.231:6789/0 mon.SCU-4F-2
> 2: 192.129.0.232:6789/0 mon.SCU-4F-3
>
> -80> 2014-11-03 14:21:01.957565 7fe99c155700 1 monclient(hunting): found
> mon.SCU-4F-1
> -79> 2014-11-03 14:21:01.957579 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== mon.0 192.129.0.230:6789/0 2 ====
> auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (392146148 0 0) 0x7fe98c000ff0
> con 0x7fe9a336b660
> -78> 2014-11-03 14:21:01.957619 7fe99c155700 10 monclient: my global_id
> is 353985
> -77> 2014-11-03 14:21:01.957844 7fe99c155700 10 monclient:
> _send_mon_message to mon.SCU-4F-1 at 192.129.0.230:6789/0
> -76> 2014-11-03 14:21:01.957854 7fe99c155700 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.230:6789/0 -- auth(proto 2 32 bytes
> epoch 0) v1 -- ?+0 0x7fe990001920 con 0x7fe9a336b660
> -75> 2014-11-03 14:21:01.958778 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== mon.0 192.129.0.230:6789/0 3 ====
> auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (2853291314 0 0)
> 0x7fe98c000920 con 0x7fe9a336b660
> -74> 2014-11-03 14:21:01.958900 7fe99c155700 10 monclient:
> _send_mon_message to mon.SCU-4F-1 at 192.129.0.230:6789/0
> -73> 2014-11-03 14:21:01.958910 7fe99c155700 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.230:6789/0 -- auth(proto 2 165 bytes
> epoch 0) v1 -- ?+0 0x7fe990002140 con 0x7fe9a336b660
> -72> 2014-11-03 14:21:01.959724 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== mon.0 192.129.0.230:6789/0 4 ====
> auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (3509675157 0 0)
> 0x7fe98c000920 con 0x7fe9a336b660
> -71> 2014-11-03 14:21:01.959799 7fe99c155700 10 monclient:
> _send_mon_message to mon.SCU-4F-1 at 192.129.0.230:6789/0
> -70> 2014-11-03 14:21:01.959808 7fe99c155700 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.230:6789/0 --
> mon_subscribe({monmap=0+}) v2 -- ?+0 0x7fe9a3367b80 con 0x7fe9a336b660
> -69> 2014-11-03 14:21:01.959842 7fe9a170c7c0 5 monclient: authenticate
> success, global_id 353985
> -68> 2014-11-03 14:21:01.959874 7fe9a170c7c0 5 asok(0x7fe9a33483f0)
> register_command objecter_requests hook 0x7fe9a3367d90
> -67> 2014-11-03 14:21:01.959936 7fe9a170c7c0 10 monclient: renew_subs
> -66> 2014-11-03 14:21:01.959945 7fe9a170c7c0 10 monclient:
> _send_mon_message to mon.SCU-4F-1 at 192.129.0.230:6789/0
> -65> 2014-11-03 14:21:01.959953 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.230:6789/0 --
> mon_subscribe({monmap=3+,osdmap=0}) v2 -- ?+0 0x7fe9a336bb80 con
> 0x7fe9a336b660
> -64> 2014-11-03 14:21:01.959973 7fe9a170c7c0 10 monclient: renew_subs
> -63> 2014-11-03 14:21:01.959979 7fe9a170c7c0 10 monclient:
> _send_mon_message to mon.SCU-4F-1 at 192.129.0.230:6789/0
> -62> 2014-11-03 14:21:01.959983 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.230:6789/0 --
> mon_subscribe({monmap=3+,osdmap=0}) v2 -- ?+0 0x7fe9a336ca20 con
> 0x7fe9a336b660
> -61> 2014-11-03 14:21:01.960038 7fe9a170c7c0 1 librados: init done
> -60> 2014-11-03 14:21:01.960274 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== mon.0 192.129.0.230:6789/0 5 ==== mon_map v1
> ==== 494+0+0 (1741557205 0 0) 0x7fe98c001260 con 0x7fe9a336b660
> -59> 2014-11-03 14:21:01.960291 7fe99c155700 10 monclient: handle_monmap
> mon_map v1
> -58> 2014-11-03 14:21:01.960300 7fe99c155700 10 monclient: got monmap 2,
> mon.SCU-4F-1 is now rank 0
> -57> 2014-11-03 14:21:01.960307 7fe99c155700 10 monclient: dump:
> epoch 2
> fsid d0766792-176e-4ffe-a911-6a3b373c28e1
> last_changed 2014-10-13 09:48:17.907147
> created 0.000000
> 0: 192.129.0.230:6789/0 mon.SCU-4F-1
> 1: 192.129.0.231:6789/0 mon.SCU-4F-2
> 2: 192.129.0.232:6789/0 mon.SCU-4F-3
>
> -56> 2014-11-03 14:21:01.960332 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== mon.0 192.129.0.230:6789/0 6 ====
> mon_subscribe_ack(300s) v1 ==== 20+0+0 (2113563967 0 0) 0x7fe98c001450 con
> 0x7fe9a336b660
> -55> 2014-11-03 14:21:01.960348 7fe99c155700 10 monclient:
> handle_subscribe_ack sent 2014-11-03 14:21:01.955803 renew after 2014-11-03
> 14:23:31.955803
> -54> 2014-11-03 14:21:01.961062 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== mon.0 192.129.0.230:6789/0 7 ====
> osd_map(1901..1901 src has 1301..1901) v3 ==== 32893+0+0 (3667157083 0 0)
> 0x7fe98c000920 con 0x7fe9a336b660
> -53> 2014-11-03 14:21:01.961349 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== mon.0 192.129.0.230:6789/0 8 ====
> mon_subscribe_ack(300s) v1 ==== 20+0+0 (2113563967 0 0) 0x7fe98c0092f0 con
> 0x7fe9a336b660
> -52> 2014-11-03 14:21:01.961363 7fe99c155700 10 monclient:
> handle_subscribe_ack sent 0.000000, ignoring
> -51> 2014-11-03 14:21:01.961368 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== mon.0 192.129.0.230:6789/0 9 ====
> osd_map(1901..1901 src has 1301..1901) v3 ==== 32893+0+0 (3667157083 0 0)
> 0x7fe98c011640 con 0x7fe9a336b660
> -50> 2014-11-03 14:21:01.961385 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== mon.0 192.129.0.230:6789/0 10 ====
> mon_subscribe_ack(300s) v1 ==== 20+0+0 (2113563967 0 0) 0x7fe98c011ab0 con
> 0x7fe9a336b660
> -49> 2014-11-03 14:21:01.961397 7fe99c155700 10 monclient:
> handle_subscribe_ack sent 0.000000, ignoring
> -48> 2014-11-03 14:21:01.961521 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6860/359 --
> osd_op(client.353985.0:1 f3e81ea3-1d5b-487a-a55e-53efff604d54_disk.rbd
> [stat] 3.461461a0 ack+read e1901) v4 -- ?+0 0x7fe9a33695b0 con
> 0x7fe9a3369270
> -47> 2014-11-03 14:21:01.963525 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.43 192.129.0.232:6860/359 1 ====
> osd_op_reply(1 f3e81ea3-1d5b-487a-a55e-53efff604d54_disk.rbd [stat] v0'0 uv0
> ack = -2 (No such file or directory)) v6 ==== 212+0+0 (1681152760 0 0)
> 0x7fe988000a70 con 0x7fe9a3369270
> -46> 2014-11-03 14:21:01.963697 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.231:6810/1899 --
> osd_op(client.353985.0:2 rbd_id.f3e81ea3-1d5b-487a-a55e-53efff604d54_disk
> [stat] 3.652023ee ack+read e1901) v4 -- ?+0 0x7fe9a3369e40 con
> 0x7fe9a3369b00
> -45> 2014-11-03 14:21:01.966169 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.7 192.129.0.231:6810/1899 1 ====
> osd_op_reply(2 rbd_id.f3e81ea3-1d5b-487a-a55e-53efff604d54_disk [stat] v0'0
> uv3114101 ondisk = 0) v6 ==== 215+0+16 (3718540203 0 1748566453)
> 0x7fe980000b90 con 0x7fe9a3369b00
> -44> 2014-11-03 14:21:01.966255 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.231:6810/1899 --
> osd_op(client.353985.0:3 rbd_id.f3e81ea3-1d5b-487a-a55e-53efff604d54_disk
> [call rbd.get_id] 3.652023ee ack+read e1901) v4 -- ?+0 0x7fe9a3369e40 con
> 0x7fe9a3369b00
> -43> 2014-11-03 14:21:01.967877 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.7 192.129.0.231:6810/1899 2 ====
> osd_op_reply(3 rbd_id.f3e81ea3-1d5b-487a-a55e-53efff604d54_disk [call] v0'0
> uv3114101 ondisk = 0) v6 ==== 215+0+17 (2225607386 0 3190742893)
> 0x7fe980000b90 con 0x7fe9a3369b00
> -42> 2014-11-03 14:21:01.968022 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6845/30466 --
> osd_op(client.353985.0:4 rbd_header.4f431109cf92e [call rbd.get_size,call
> rbd.get_object_prefix] 3.a969d0d4 ack+read e1901) v4 -- ?+0 0x7fe9a336a950
> con 0x7fe9a336a5a0
> -41> 2014-11-03 14:21:01.978986 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.40 192.129.0.232:6845/30466 1 ====
> osd_op_reply(4 rbd_header.4f431109cf92e [call,call] v0'0 uv2132261 ondisk =
> 0) v6 ==== 233+0+35 (1897419089 0 3994759186) 0x7fe978000bc0 con
> 0x7fe9a336a5a0
> -40> 2014-11-03 14:21:01.979068 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6845/30466 --
> osd_op(client.353985.0:5 rbd_header.4f431109cf92e [call
> rbd.get_stripe_unit_count] 3.a969d0d4 ack+read e1901) v4 -- ?+0
> 0x7fe9a336a950 con 0x7fe9a336a5a0
> -39> 2014-11-03 14:21:01.980528 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.40 192.129.0.232:6845/30466 2 ====
> osd_op_reply(5 rbd_header.4f431109cf92e [call] v0'0 uv0 ondisk = -8 (Exec
> format error)) v6 ==== 191+0+0 (2140300419 0 0) 0x7fe978000bc0 con
> 0x7fe9a336a5a0
> -38> 2014-11-03 14:21:01.980629 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6845/30466 --
> osd_op(client.353985.0:6 rbd_header.4f431109cf92e [watch add cookie 1 ver 0]
> 3.a969d0d4 ondisk+read+write e1901) v4 -- ?+0 0x7fe9a336d090 con
> 0x7fe9a336a5a0
> -37> 2014-11-03 14:21:02.394870 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.40 192.129.0.232:6845/30466 3 ====
> osd_op_reply(6 rbd_header.4f431109cf92e [watch add cookie 1 ver 0]
> v1901'2465634 uv2132261 ondisk = 0) v6 ==== 191+0+0 (1085298969 0 0)
> 0x7fe978000bc0 con 0x7fe9a336a5a0
> -36> 2014-11-03 14:21:02.395099 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6845/30466 --
> osd_op(client.353985.0:7 rbd_header.4f431109cf92e [call rbd.get_size,call
> rbd.get_features,call rbd.get_snapcontext,call rbd.get_parent,call
> lock.get_info] 3.a969d0d4 ack+read e1901) v4 -- ?+0 0x7fe9a336a980 con
> 0x7fe9a336a5a0
> -35> 2014-11-03 14:21:02.396724 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.40 192.129.0.232:6845/30466 4 ====
> osd_op_reply(7 rbd_header.4f431109cf92e [call,call,call,call,call] v0'0
> uv2132261 ondisk = 0) v6 ==== 359+0+93 (3953812703 0 3278963491)
> 0x7fe9780010b0 con 0x7fe9a336a5a0
> -34> 2014-11-03 14:21:02.396924 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6840/29661 --
> osd_op(client.353985.0:8 rbd_header.12aa41190cde7 [call rbd.get_size,call
> rbd.get_object_prefix] 4.7f4a998d ack+read e1901) v4 -- ?+0 0x7fe9a336dd60
> con 0x7fe9a336d9f0
> -33> 2014-11-03 14:21:02.400351 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.39 192.129.0.232:6840/29661 1 ====
> osd_op_reply(8 rbd_header.12aa41190cde7 [call,call] v0'0 uv353 ondisk = 0)
> v6 ==== 233+0+35 (1921475055 0 3273630005) 0x7fe970000bc0 con 0x7fe9a336d9f0
> -32> 2014-11-03 14:21:02.400406 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6840/29661 --
> osd_op(client.353985.0:9 rbd_header.12aa41190cde7 [call
> rbd.get_stripe_unit_count] 4.7f4a998d ack+read e1901) v4 -- ?+0
> 0x7fe9a336f3f0 con 0x7fe9a336d9f0
> -31> 2014-11-03 14:21:02.402937 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.39 192.129.0.232:6840/29661 2 ====
> osd_op_reply(9 rbd_header.12aa41190cde7 [call] v0'0 uv0 ondisk = -8 (Exec
> format error)) v6 ==== 191+0+0 (1327880786 0 0) 0x7fe970000bc0 con
> 0x7fe9a336d9f0
> -30> 2014-11-03 14:21:02.403026 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6840/29661 --
> osd_op(client.353985.0:10 rbd_header.12aa41190cde7 [call rbd.get_size,call
> rbd.get_features,call rbd.get_snapcontext,call rbd.get_parent,call
> lock.get_info] 4.7f4a998d ack+read e1901) v4 -- ?+0 0x7fe9a336f3f0 con
> 0x7fe9a336d9f0
> -29> 2014-11-03 14:21:02.404781 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.39 192.129.0.232:6840/29661 3 ====
> osd_op_reply(10 rbd_header.12aa41190cde7 [call,call,call,call,call] v0'0
> uv353 ondisk = 0) v6 ==== 359+0+88 (1691764099 0 3345369574) 0x7fe970000f00
> con 0x7fe9a336d9f0
> -28> 2014-11-03 14:21:02.404890 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6840/29661 --
> osd_op(client.353985.0:11 rbd_header.12aa41190cde7 [call
> rbd.get_snapshot_name,call rbd.get_size,call rbd.get_features,call
> rbd.get_parent,call rbd.get_protection_status] 4.7f4a998d ack+read e1901) v4
> -- ?+0 0x7fe9a336f3f0 con 0x7fe9a336d9f0
> -27> 2014-11-03 14:21:02.406466 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.39 192.129.0.232:6840/29661 4 ====
> osd_op_reply(11 rbd_header.12aa41190cde7 [call,call,call,call,call] v0'0
> uv353 ondisk = 0) v6 ==== 359+0+74 (438505924 0 3867595636) 0x7fe970000ee0
> con 0x7fe9a336d9f0
> -26> 2014-11-03 14:21:02.406638 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.231:6870/2751 --
> osd_op(client.353985.0:12 rbd_data.4f431109cf92e.0000000000000000
> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 0~0]
> 3.a0c472b0 ack+ondisk+write e1901) v4 -- ?+0 0x7fe9a336e550 con
> 0x7fe9a336e210
> -25> 2014-11-03 14:21:02.406729 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6875/3290 --
> osd_op(client.353985.0:13 rbd_data.4f431109cf92e.0000000000000001
> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 0~0]
> 3.ef1e57c8 ack+ondisk+write e1901) v4 -- ?+0 0x7fe9a3370880 con
> 0x7fe9a33704d0
> -24> 2014-11-03 14:21:02.406856 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.230:6820/12563 --
> osd_op(client.353985.0:14 rbd_data.4f431109cf92e.0000000000000002
> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 0~0]
> 3.1e2f00b1 ack+ondisk+write e1901) v4 -- ?+0 0x7fe9a3372270 con
> 0x7fe9a3371ec0
> -23> 2014-11-03 14:21:02.406970 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6810/24988 --
> osd_op(client.353985.0:15 rbd_data.4f431109cf92e.0000000000000003
> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 0~0]
> 3.889f424f ack+ondisk+write e1901) v4 -- ?+0 0x7fe9a3374980 con
> 0x7fe9a3374670
> -22> 2014-11-03 14:21:02.407091 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.230:6805/12381 --
> osd_op(client.353985.0:16 rbd_data.4f431109cf92e.0000000000000004
> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 0~0]
> 3.6014bc72 ack+ondisk+write e1901) v4 -- ?+0 0x7fe9a3376340 con
> 0x7fe9a3375fc0
> -21> 2014-11-03 14:21:02.407207 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6835/28502 --
> osd_op(client.353985.0:17 rbd_data.4f431109cf92e.0000000000000005
> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 0~0]
> 3.65c2e40a ack+ondisk+write e1901) v4 -- ?+0 0x7fe9a3377ec0 con
> 0x7fe9a3377d60
> -20> 2014-11-03 14:21:02.407684 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6805/24350 --
> osd_op(client.353985.0:18 rbd_data.4f431109cf92e.0000000000000006
> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 0~0]
> 3.ceefbfd9 ack+ondisk+write e1901) v4 -- ?+0 0x7fe9a3379600 con
> 0x7fe9a3379180
> -19> 2014-11-03 14:21:02.408804 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6800/17543 --
> osd_op(client.353985.0:19 rbd_data.4f431109cf92e.0000000000000007
> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 0~0]
> 3.9f8727d1 ack+ondisk+write e1901) v4 -- ?+0 0x7fe9a3376340 con
> 0x7fe9a336e7b0
> -18> 2014-11-03 14:21:02.408863 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.230:6805/12381 --
> osd_op(client.353985.0:20 rbd_data.4f431109cf92e.0000000000000008
> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 0~0]
> 3.6536834e ack+ondisk+write e1901) v4 -- ?+0 0x7fe9a337afb0 con
> 0x7fe9a3375fc0
> -17> 2014-11-03 14:21:02.409297 7fe9a170c7c0 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.230:6860/13065 --
> osd_op(client.353985.0:21 rbd_data.4f431109cf92e.0000000000000009
> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 0~0]
> 3.3c58e27f ack+ondisk+write e1901) v4 -- ?+0 0x7fe9a337b9c0 con
> 0x7fe9a3378250
> -16> 2014-11-03 14:21:02.410136 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.46 192.129.0.232:6875/3290 1 ====
> osd_op_reply(13 rbd_data.4f431109cf92e.0000000000000001 [stat,set-alloc-hint
> object_size 4194304 write_size 4194304,write 0~0] v0'0 uv3314720 ondisk = -2
> (No such file or directory)) v6 ==== 290+0+0 (2859704282 0 0) 0x7fe944000ac0
> con 0x7fe9a33704d0
> -15> 2014-11-03 14:21:02.410424 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.32 192.129.0.232:6805/24350 1 ====
> osd_op_reply(18 rbd_data.4f431109cf92e.0000000000000006 [stat,set-alloc-hint
> object_size 4194304 write_size 4194304,write 0~0] v0'0 uv4341095 ondisk = -2
> (No such file or directory)) v6 ==== 290+0+0 (3454566580 0 0) 0x7fe938000ac0
> con 0x7fe9a3379180
> -14> 2014-11-03 14:21:02.410871 7fe99984f700 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.231:6805/1805 --
> osd_op(client.353985.0:22 rbd_data.12aa41190cde7.0000000000000001@8f
> [sparse-read 0~4194304] 4.20a6c6d3 ack+read+localize_reads e1901) v4 -- ?+0
> 0x7fe934002970 con 0x7fe934002630
> -13> 2014-11-03 14:21:02.410885 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.33 192.129.0.232:6810/24988 1 ====
> osd_op_reply(15 rbd_data.4f431109cf92e.0000000000000003 [stat,set-alloc-hint
> object_size 4194304 write_size 4194304,write 0~0] v0'0 uv3125373 ondisk = -2
> (No such file or directory)) v6 ==== 290+0+0 (961650793 0 0) 0x7fe930000ac0
> con 0x7fe9a3374670
> -12> 2014-11-03 14:21:02.411050 7fe99984f700 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.230:6875/13275 --
> osd_op(client.353985.0:23 rbd_data.12aa41190cde7.0000000000000006@8f
> [sparse-read 0~4194304] 4.63d13870 ack+read+localize_reads e1901) v4 -- ?+0
> 0x7fe934003d80 con 0x7fe934003a40
> -11> 2014-11-03 14:21:02.411113 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.30 192.129.0.230:6805/12381 1 ====
> osd_op_reply(16 rbd_data.4f431109cf92e.0000000000000004 [stat,set-alloc-hint
> object_size 4194304 write_size 4194304,write 0~0] v0'0 uv4820637 ondisk = -2
> (No such file or directory)) v6 ==== 290+0+0 (2608701185 0 0) 0x7fe928000ac0
> con 0x7fe9a3375fc0
> -10> 2014-11-03 14:21:02.411149 7fe99984f700 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.230:6860/13065 --
> osd_op(client.353985.0:24 rbd_data.12aa41190cde7.0000000000000003@8f
> [sparse-read 0~4194304] 4.9dcccf87 ack+read+localize_reads e1901) v4 -- ?+0
> 0x7fe934004b90 con 0x7fe9a3378250
> -9> 2014-11-03 14:21:02.411276 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.38 192.129.0.232:6835/28502 1 ====
> osd_op_reply(17 rbd_data.4f431109cf92e.0000000000000005 [stat,set-alloc-hint
> object_size 4194304 write_size 4194304,write 0~0] v0'0 uv3969777 ondisk = -2
> (No such file or directory)) v6 ==== 290+0+0 (3782309098 0 0) 0x7fe92c000ac0
> con 0x7fe9a3377d60
> -8> 2014-11-03 14:21:02.411317 7fe99984f700 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6845/30466 --
> osd_op(client.353985.0:25 rbd_data.12aa41190cde7.0000000000000004@8f
> [sparse-read 0~4194304] 4.169b49fe ack+read+localize_reads e1901) v4 -- ?+0
> 0x7fe9340055a0 con 0x7fe9a336a5a0
> -7> 2014-11-03 14:21:02.411416 7fe99984f700 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6805/24350 --
> osd_op(client.353985.0:26 rbd_data.12aa41190cde7.0000000000000005@8f
> [sparse-read 0~4194304] 4.92ce333c ack+read+localize_reads e1901) v4 -- ?+0
> 0x7fe9340055a0 con 0x7fe9a3379180
> -6> 2014-11-03 14:21:02.411583 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.31 192.129.0.232:6800/17543 1 ====
> osd_op_reply(19 rbd_data.4f431109cf92e.0000000000000007 [stat,set-alloc-hint
> object_size 4194304 write_size 4194304,write 0~0] v0'0 uv4510031 ondisk = -2
> (No such file or directory)) v6 ==== 290+0+0 (2898314773 0 0) 0x7fe924000ac0
> con 0x7fe9a336e7b0
> -5> 2014-11-03 14:21:02.411663 7fe99984f700 1 --
> 192.129.0.201:0/1025623 --> 192.129.0.232:6860/359 --
> osd_op(client.353985.0:27 rbd_data.12aa41190cde7.0000000000000007@8f
> [sparse-read 0~4194304] 4.bfbca25b ack+read+localize_reads e1901) v4 -- ?+0
> 0x7fe9340069c0 con 0x7fe9a3369270
> -4> 2014-11-03 14:21:02.412135 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.40 192.129.0.232:6845/30466 5 ====
> osd_op_reply(25 rbd_data.12aa41190cde7.0000000000000004 [sparse-read
> 0~4194304] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 206+0+0
> (1523082640 0 0) 0x7fe978001090 con 0x7fe9a336a5a0
> -3> 2014-11-03 14:21:02.412311 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.32 192.129.0.232:6805/24350 2 ====
> osd_op_reply(26 rbd_data.12aa41190cde7.0000000000000005 [sparse-read
> 0~4194304] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 206+0+0
> (2320465840 0 0) 0x7fe938000ac0 con 0x7fe9a3379180
> -2> 2014-11-03 14:21:02.412569 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.23 192.129.0.230:6820/12563 1 ====
> osd_op_reply(14 rbd_data.4f431109cf92e.0000000000000002 [stat,set-alloc-hint
> object_size 4194304 write_size 4194304,write 0~0] v0'0 uv4001079 ondisk = -2
> (No such file or directory)) v6 ==== 290+0+0 (3414600131 0 0) 0x7fe91c000ac0
> con 0x7fe9a3371ec0
> -1> 2014-11-03 14:21:02.412656 7fe99c155700 1 --
> 192.129.0.201:0/1025623 <== osd.43 192.129.0.232:6860/359 2 ====
> osd_op_reply(27 rbd_data.12aa41190cde7.0000000000000007 [sparse-read
> 0~4194304] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 206+0+0
> (1607725410 0 0) 0x7fe988000a70 con 0x7fe9a3369270
> 0> 2014-11-03 14:21:02.413259 7fe99984f700 -1 *** Caught signal
> (Segmentation fault) **
> in thread 7fe99984f700
>
> ceph version 0.79 (4c2d73a5095f527c3a2168deb5fa54b3c8991a6e)
> 1: (()+0x22a4f) [0x7fe9a1745a4f]
> 2: (()+0x10340) [0x7fe9a00f2340]
> 3: (librbd::aio_read(librbd::ImageCtx*, std::vector<std::pair<unsigned
> long, unsigned long>, std::allocator<std::pair<unsigned long, unsigned long>
>> > const&, char*, ceph::buffer::list*, librbd::AioCompletion*)+0x24)
> [0x7fe9a125daf4]
> 4: (librbd::AioRequest::read_from_parent(std::vector<std::pair<unsigned
> long, unsigned long>, std::allocator<std::pair<unsigned long, unsigned long>
>> >&)+0x85) [0x7fe9a1242745]
> 5: (librbd::AioRead::should_complete(int)+0x352) [0x7fe9a1242ca2]
> 6: (librbd::rados_req_cb(void*, void*)+0x1b) [0x7fe9a124cd7b]
> 7: (librados::C_AioComplete::finish(int)+0x1d) [0x7fe9a04a355d]
> 8: (Context::complete(int)+0x9) [0x7fe9a0480579]
> 9: (Finisher::finisher_thread_entry()+0x1b8) [0x7fe9a0531758]
> 10: (()+0x8182) [0x7fe9a00ea182]
> 11: (clone()+0x6d) [0x7fe99f2ce30d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
> interpret this.
>
> --- logging levels ---
> 0/ 5 none
> 0/ 1 lockdep
> 0/ 1 context
> 1/ 1 crush
> 1/ 5 mds
> 1/ 5 mds_balancer
> 1/ 5 mds_locker
> 1/ 5 mds_log
> 1/ 5 mds_log_expire
> 1/ 5 mds_migrator
> 0/ 1 buffer
> 0/ 1 timer
> 0/ 1 filer
> 0/ 1 striper
> 0/ 1 objecter
> 0/ 5 rados
> 0/ 5 rbd
> 0/ 5 journaler
> 0/ 5 objectcacher
> 0/ 5 client
> 0/ 5 osd
> 0/ 5 optracker
> 0/ 5 objclass
> 1/ 3 filestore
> 1/ 3 keyvaluestore
> 1/ 3 journal
> 0/ 5 ms
> 1/ 5 mon
> 0/10 monc
> 1/ 5 paxos
> 0/ 5 tp
> 1/ 5 auth
> 1/ 5 crypto
> 1/ 1 finisher
> 1/ 5 heartbeatmap
> 1/ 5 perfcounter
> 1/ 5 rgw
> 1/ 5 javaclient
> 1/ 5 asok
> 1/ 1 throttle
> -2/-2 (syslog threshold)
> 99/99 (stderr threshold)
> max_recent 500
> max_new 1000
> log_file
> --- end dump of recent events ---
> Segmentation fault (core dumped)
Looks like http://tracker.ceph.com/issues/8845. It's been fixed in
v0.85 I think (commit 2545e80d274b) and was backported to firefly
(v0.80) and dumpling (v0.67).
Thanks,
Ilya
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
