Re: Issues with fresh 0.93 OSD adding to existing cluster

Joao Eduardo Luis <joao@xxxxxxxxxx> · Thu, 12 Mar 2015 14:01:48 +0000

On 03/12/2015 05:16 AM, Malcolm Haak wrote:
Sorry about all the unrelated grep issues..

So I've rebuilt and reinstalled and it's still broken.

On the working node, even with the new packages, everything works.
On the new broken node, I've added a mon and it works. But I still cannot start an OSD on the new node.

What else do you need from me? I'll get logs run any number of tests.

I've got data in this cluster already, and it's full so I need to expand it, I've already got hardware.

Thanks in advance for even having a look

Sam mentioned to me on IRC that the next step would be to grab the 
offending osdmaps.  Easiest way for that will be to stop a monitor and 
run 'ceph-monstore-tool' in order to obtain the full maps, and then use 
'ceph-kvstore-tool' to obtain incrementals.

Given the osd is crashing on version 2759, the following would be best:

(Assuming you have stopped a given monitor with id FOO, whose store is 
sitting at default path /var/lib/ceph/mon/ceph-FOO)

ceph-monstore-tool /var/lib/ceph/mon/ceph-FOO get osdmap -- --version 
2758 --out /tmp/osdmap.full.2758

ceph-monstore-tool /var/lib/ceph/mon/ceph-FOO get osdmap -- --version 
2759 --out /tmp/osdmap.full.2759

(please note the '--' between 'osdmap' and '--version', as that is 
required for the tool to do its thing)

and then

ceph-kvstore-tool /var/lib/ceph/mon/ceph-FOO/store.db get osdmap 2758 
out /tmp/osdmap.inc.2758

ceph-kvstore-tool /var/lib/ceph/mon/ceph-FOO/store.db get osdmap 2759 
out /tmp/osdmap.inc.2759

Cheers!

  -Joao

-----Original Message-----
From: Samuel Just [mailto:sjust@xxxxxxxxxx]
Sent: Wednesday, 11 March 2015 1:41 AM
To: Malcolm Haak; jluis@xxxxxxxxxx
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re:  Issues with fresh 0.93 OSD adding to existing cluster

Joao, it looks like map 2759 is causing trouble, how would he get the
full and incremental maps for that out of the mons?
-Sam

On Tue, 2015-03-10 at 14:12 +0000, Malcolm Haak wrote:
Hi Samuel,

The sha1? I'm going to admit ignorance as to what you are looking for. They are all running the same release if that is what you are asking.
Same tarball built into rpms using rpmbuild on both nodes...
Only difference being that the other node has been upgraded and the problem node is fresh.

added the requested config here is the command line output

microserver-1:/etc # /etc/init.d/ceph start osd.3
=== osd.3 ===
Mounting xfs on microserver-1:/var/lib/ceph/osd/ceph-3
2015-03-11 01:00:13.492279 7f05b2f72700  1 -- :/0 messenger.start
2015-03-11 01:00:13.492823 7f05b2f72700  1 -- :/1002795 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x7f05ac0290b0 con 0x7f05ac027c40
2015-03-11 01:00:13.510814 7f05b07ef700  1 -- 192.168.0.250:0/1002795 learned my addr 192.168.0.250:0/1002795
2015-03-11 01:00:13.527653 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x7f05a0000ab0 con 0x7f05ac027c40
2015-03-11 01:00:13.527899 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (3859410672 0 0) 0x7f05a0000e70 con 0x7f05ac027c40
2015-03-11 01:00:13.527973 7f05abfff700  1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x7f05ac029730 con 0x7f05ac027c40
2015-03-11 01:00:13.528124 7f05b2f72700  1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x7f05ac029a50 con 0x7f05ac027c40
2015-03-11 01:00:13.528265 7f05b2f72700  1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x7f05ac029f20 con 0x7f05ac027c40
2015-03-11 01:00:13.530359 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 3 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x7f05a0000ab0 con 0x7f05ac027c40
2015-03-11 01:00:13.530548 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 4 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3648139960 0 0) 0x7f05a0000fb0 con 0x7f05ac027c40
2015-03-11 01:00:13.531114 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x7f05a0002800 con 0x7f05ac027c40
2015-03-11 01:00:13.531772 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3648139960 0 0) 0x7f05a0000fb0 con 0x7f05ac027c40
2015-03-11 01:00:13.532186 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 7 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x7f05a0001250 con 0x7f05ac027c40
2015-03-11 01:00:13.532260 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3648139960 0 0) 0x7f05a0000fb0 con 0x7f05ac027c40
2015-03-11 01:00:13.556748 7f05b2f72700  1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- ?+0 0x7f05ac016ac0 con 0x7f05ac027c40
2015-03-11 01:00:13.564968 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0  v0) v1 ==== 72+0+34995 (1092875540 0 1727986498) 0x7f05a0000a70 con 0x7f05ac027c40
2015-03-11 01:00:13.770122 7f05b2f72700  1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_command({"prefix": "osd crush create-or-move", "args": ["host=microserver-1", "root=default"], "id": 3, "weight": 1.81} v 0) v1 -- ?+0 0x7f05ac016ac0 con 0x7f05ac027c40
2015-03-11 01:00:13.772299 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "args": ["host=microserver-1", "root=default"], "id": 3, "weight": 1.81}]=0 create-or-move updated item name 'osd.3' weight 1.81 at location {host=microserver-1,root=default} to crush map v3277) v1 ==== 256+0+0 (1191546821 0 0) 0x7f05a0001000 con 0x7f05ac027c40
create-or-move updated item name 'osd.3' weight 1.81 at location {host=microserver-1,root=default} to crush map
2015-03-11 01:00:13.776891 7f05b2f72700  1 -- 192.168.0.250:0/1002795 mark_down 0x7f05ac027c40 -- 0x7f05ac0239a0
2015-03-11 01:00:13.777212 7f05b2f72700  1 -- 192.168.0.250:0/1002795 mark_down_all
2015-03-11 01:00:13.778120 7f05b2f72700  1 -- 192.168.0.250:0/1002795 shutdown complete.
Starting Ceph osd.3 on microserver-1...
microserver-1:/etc #

Log file

2015-03-11 01:00:13.876152 7f41a1ba4880  0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 2840
2015-03-11 01:00:13.877059 7f41a1ba4880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/2840 need_addr=1
2015-03-11 01:00:13.877111 7f41a1ba4880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/2840 need_addr=1
2015-03-11 01:00:13.877140 7f41a1ba4880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/2840 need_addr=1
2015-03-11 01:00:13.877169 7f41a1ba4880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/2840 need_addr=1
2015-03-11 01:00:13.880323 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) dump_stop
2015-03-11 01:00:13.890355 7f41a1ba4880 10 ErasureCodePluginSelectJerasure: load: jerasure_generic
2015-03-11 01:00:13.897843 7f41a1ba4880 10 load: jerasure load: lrc load: isa
2015-03-11 01:00:13.898172 7f41a1ba4880  5 filestore(/var/lib/ceph/osd/ceph-3) test_mount basedir /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal
2015-03-11 01:00:13.898290 7f41a1ba4880  1 -- 0.0.0.0:6800/2840 messenger.start
2015-03-11 01:00:13.898362 7f41a1ba4880  1 -- :/0 messenger.start
2015-03-11 01:00:13.898450 7f41a1ba4880  1 -- 0.0.0.0:6803/2840 messenger.start
2015-03-11 01:00:13.898527 7f41a1ba4880  1 -- 0.0.0.0:6802/2840 messenger.start
2015-03-11 01:00:13.898609 7f41a1ba4880  1 -- 0.0.0.0:6801/2840 messenger.start
2015-03-11 01:00:13.898672 7f41a1ba4880  1 -- :/0 messenger.start
2015-03-11 01:00:13.898792 7f41a1ba4880  2 osd.3 0 mounting /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
2015-03-11 01:00:13.898826 7f41a1ba4880  5 filestore(/var/lib/ceph/osd/ceph-3) basedir /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal
2015-03-11 01:00:13.898868 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) mount fsid is 05fb481f-68b3-4442-b7de-af4c867e18d5
2015-03-11 01:00:13.898937 7f41a1ba4880  0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342)
2015-03-11 01:00:13.941605 7f41a1ba4880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work
2015-03-11 01:00:13.941665 7f41a1ba4880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2015-03-11 01:00:13.970056 7f41a1ba4880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2015-03-11 01:00:13.970442 7f41a1ba4880  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5
2015-03-11 01:00:14.001199 7f41a1ba4880  5 filestore(/var/lib/ceph/osd/ceph-3) mount op_seq is 2
2015-03-11 01:00:14.122464 7f41a1ba4880 20 filestore (init)dbobjectmap: seq is 1
2015-03-11 01:00:14.122580 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) open_journal at /var/lib/ceph/osd/ceph-3/journal
2015-03-11 01:00:14.122674 7f41a1ba4880  0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2015-03-11 01:00:14.122682 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) list_collections
2015-03-11 01:00:14.123024 7f41a1ba4880 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2015-03-11 01:00:14.123040 7f41a1ba4880  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
2015-03-11 01:00:14.123003 7f4196c1b700 20 filestore(/var/lib/ceph/osd/ceph-3) sync_entry waiting for max_interval 5.000000
2015-03-11 01:00:14.123724 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x4b9b6c0
2015-03-11 01:00:14.123837 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
2015-03-11 01:00:14.123971 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
2015-03-11 01:00:14.124020 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) touch meta/a468ec03/snapmapper/0//-1
2015-03-11 01:00:14.124103 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) touch meta/a468ec03/snapmapper/0//-1 = 0
2015-03-11 01:00:14.124247 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x4b9b6c0
2015-03-11 01:00:14.124282 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
2015-03-11 01:00:14.124332 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
2015-03-11 01:00:14.124348 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) write meta/a143428/osdmap.2757/0//-1 0~3076
2015-03-11 01:00:14.124420 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) write meta/a143428/osdmap.2757/0//-1 0~3076 = 3076
2015-03-11 01:00:14.124461 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
2015-03-11 01:00:14.124547 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
2015-03-11 01:00:14.124557 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) write meta/23c2fcde/osd_superblock/0//-1 0~413
2015-03-11 01:00:14.124615 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) write meta/23c2fcde/osd_superblock/0//-1 0~413 = 413
2015-03-11 01:00:14.124716 7f41a1ba4880  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
2015-03-11 01:00:14.125247 7f41a1ba4880  2 osd.3 0 boot
2015-03-11 01:00:14.125270 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) read meta/23c2fcde/osd_superblock/0//-1 0~0
2015-03-11 01:00:14.125343 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) FileStore::read meta/23c2fcde/osd_superblock/0//-1 0~413/413
2015-03-11 01:00:14.125393 7f41a1ba4880 10 osd.3 0 read_superblock sb(3bf19e54-97e2-4a5f-b811-7c3780cd5e57 osd.3 05fb481f-68b3-4442-b7de-af4c867e18d5 e2757 [2757,2757] lci=[0,0])
2015-03-11 01:00:14.125521 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) stat meta/a468ec03/snapmapper/0//-1 = 0 (size 0)
2015-03-11 01:00:14.125540 7f41a1ba4880 10 open_all_classes
2015-03-11 01:00:14.125586 7f41a1ba4880 10 open_all_classes found hello
2015-03-11 01:00:14.125623 7f41a1ba4880 10 _get_class adding new class name hello 0x4c9e028
2015-03-11 01:00:14.125630 7f41a1ba4880 10 _load_class hello from /usr/lib64/rados-classes/libcls_hello.so
2015-03-11 01:00:14.126058 7f41a1ba4880  0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
2015-03-11 01:00:14.126080 7f41a1ba4880 10 register_class hello status 3
2015-03-11 01:00:14.126084 7f41a1ba4880 10 register_cxx_method hello.say_hello flags 1 0x7f4192a10350
2015-03-11 01:00:14.126092 7f41a1ba4880 10 register_cxx_method hello.record_hello flags 2 0x7f4192a113c0
2015-03-11 01:00:14.126098 7f41a1ba4880 10 register_cxx_method hello.writes_dont_return_data flags 2 0x7f4192a10650
2015-03-11 01:00:14.126103 7f41a1ba4880 10 register_cxx_method hello.replay flags 1 0x7f4192a10320
2015-03-11 01:00:14.126109 7f41a1ba4880 10 register_cxx_method hello.turn_it_to_11 flags 3 0x7f4192a10920
2015-03-11 01:00:14.126114 7f41a1ba4880 10 register_cxx_method hello.bad_reader flags 2 0x7f4192a10310
2015-03-11 01:00:14.126119 7f41a1ba4880 10 register_cxx_method hello.bad_writer flags 1 0x7f4192a10300
2015-03-11 01:00:14.126127 7f41a1ba4880 10 _load_class hello success
2015-03-11 01:00:14.126132 7f41a1ba4880 10 open_all_classes found kvs
2015-03-11 01:00:14.126137 7f41a1ba4880 10 _get_class adding new class name kvs 0x4c9e108
2015-03-11 01:00:14.126142 7f41a1ba4880 10 _load_class kvs from /usr/lib64/rados-classes/libcls_kvs.so
2015-03-11 01:00:14.126655 7f41a1ba4880 10 register_class kvs status 3
2015-03-11 01:00:14.126665 7f41a1ba4880 10 register_cxx_method kvs.get_idata_from_key flags 1 0x7f41927ff060
2015-03-11 01:00:14.126669 7f41a1ba4880 10 register_cxx_method kvs.get_next_idata flags 1 0x7f4192801050
2015-03-11 01:00:14.126672 7f41a1ba4880 10 register_cxx_method kvs.get_prev_idata flags 1 0x7f41927fd570
2015-03-11 01:00:14.126675 7f41a1ba4880 10 register_cxx_method kvs.read_many flags 1 0x7f41927f8f30
2015-03-11 01:00:14.126678 7f41a1ba4880 10 register_cxx_method kvs.check_writable flags 3 0x7f41927f8ac0
2015-03-11 01:00:14.126681 7f41a1ba4880 10 register_cxx_method kvs.assert_size_in_bound flags 2 0x7f41927f8d40
2015-03-11 01:00:14.126684 7f41a1ba4880 10 register_cxx_method kvs.omap_insert flags 2 0x7f41927fcb50
2015-03-11 01:00:14.126688 7f41a1ba4880 10 register_cxx_method kvs.create_with_omap flags 2 0x7f41927fc0e0
2015-03-11 01:00:14.126691 7f41a1ba4880 10 register_cxx_method kvs.omap_remove flags 2 0x7f41927f9e70
2015-03-11 01:00:14.126696 7f41a1ba4880 10 register_cxx_method kvs.maybe_read_for_balance flags 1 0x7f41927fab70
2015-03-11 01:00:14.126735 7f41a1ba4880 10 _load_class kvs success
2015-03-11 01:00:14.126739 7f41a1ba4880 10 open_all_classes found lock
2015-03-11 01:00:14.126742 7f41a1ba4880 10 _get_class adding new class name lock 0x4c9e1e8
2015-03-11 01:00:14.126745 7f41a1ba4880 10 _load_class lock from /usr/lib64/rados-classes/libcls_lock.so
2015-03-11 01:00:14.126972 7f41a1ba4880 10 register_class lock status 3
2015-03-11 01:00:14.126977 7f41a1ba4880 10 register_cxx_method lock.lock flags 3 0x7f41925ee800
2015-03-11 01:00:14.126981 7f41a1ba4880 10 register_cxx_method lock.unlock flags 3 0x7f41925ed210
2015-03-11 01:00:14.126984 7f41a1ba4880 10 register_cxx_method lock.break_lock flags 3 0x7f41925ece60
2015-03-11 01:00:14.126986 7f41a1ba4880 10 register_cxx_method lock.get_info flags 1 0x7f41925ed610
2015-03-11 01:00:14.126990 7f41a1ba4880 10 register_cxx_method lock.list_locks flags 1 0x7f41925eace0
2015-03-11 01:00:14.126993 7f41a1ba4880 10 register_cxx_method lock.assert_locked flags 1 0x7f41925ec330
2015-03-11 01:00:14.126996 7f41a1ba4880 10 _load_class lock success
2015-03-11 01:00:14.126999 7f41a1ba4880 10 open_all_classes found log
2015-03-11 01:00:14.127002 7f41a1ba4880 10 _get_class adding new class name log 0x4c9e2c8
2015-03-11 01:00:14.127005 7f41a1ba4880 10 _load_class log from /usr/lib64/rados-classes/libcls_log.so
2015-03-11 01:00:14.127223 7f41a1ba4880 10 register_class log status 3
2015-03-11 01:00:14.127225 7f41a1ba4880 10 register_cxx_method log.add flags 3 0x7f41923e1560
2015-03-11 01:00:14.127228 7f41a1ba4880 10 register_cxx_method log.list flags 1 0x7f41923e2740
2015-03-11 01:00:14.127231 7f41a1ba4880 10 register_cxx_method log.trim flags 3 0x7f41923e1f90
2015-03-11 01:00:14.127234 7f41a1ba4880 10 register_cxx_method log.info flags 1 0x7f41923e1090
2015-03-11 01:00:14.127238 7f41a1ba4880 10 _load_class log success
2015-03-11 01:00:14.127241 7f41a1ba4880 10 open_all_classes found rbd
2015-03-11 01:00:14.127244 7f41a1ba4880 10 _get_class adding new class name rbd 0x4c9e3a8
2015-03-11 01:00:14.127247 7f41a1ba4880 10 _load_class rbd from /usr/lib64/rados-classes/libcls_rbd.so
2015-03-11 01:00:14.127735 7f41a1ba4880 10 register_class rbd status 3
2015-03-11 01:00:14.127745 7f41a1ba4880 10 register_cxx_method rbd.create flags 3 0x7f41921cc5c0
2015-03-11 01:00:14.127748 7f41a1ba4880 10 register_cxx_method rbd.get_features flags 1 0x7f41921d3190
2015-03-11 01:00:14.127751 7f41a1ba4880 10 register_cxx_method rbd.get_size flags 1 0x7f41921d2de0
2015-03-11 01:00:14.127754 7f41a1ba4880 10 register_cxx_method rbd.set_size flags 3 0x7f41921ca7e0
2015-03-11 01:00:14.127758 7f41a1ba4880 10 register_cxx_method rbd.get_snapcontext flags 1 0x7f41921cdb40
2015-03-11 01:00:14.127762 7f41a1ba4880 10 register_cxx_method rbd.get_object_prefix flags 1 0x7f41921c9770
2015-03-11 01:00:14.127766 7f41a1ba4880 10 register_cxx_method rbd.get_snapshot_name flags 1 0x7f41921d1dd0
2015-03-11 01:00:14.127770 7f41a1ba4880 10 register_cxx_method rbd.snapshot_add flags 3 0x7f41921d0830
2015-03-11 01:00:14.127773 7f41a1ba4880 10 register_cxx_method rbd.snapshot_remove flags 3 0x7f41921d1b40
2015-03-11 01:00:14.127776 7f41a1ba4880 10 register_cxx_method rbd.get_all_features flags 1 0x7f41921c78c0
2015-03-11 01:00:14.127779 7f41a1ba4880 10 register_cxx_method rbd.copyup flags 3 0x7f41921c7940
2015-03-11 01:00:14.127782 7f41a1ba4880 10 register_cxx_method rbd.get_parent flags 1 0x7f41921d2080
2015-03-11 01:00:14.127786 7f41a1ba4880 10 register_cxx_method rbd.set_parent flags 3 0x7f41921ccce0
2015-03-11 01:00:14.127789 7f41a1ba4880 10 register_cxx_method rbd.remove_parent flags 3 0x7f41921c9140
2015-03-11 01:00:14.127792 7f41a1ba4880 10 register_cxx_method rbd.set_protection_status flags 3 0x7f41921d2770
2015-03-11 01:00:14.127815 7f41a1ba4880 10 register_cxx_method rbd.get_protection_status flags 1 0x7f41921d2b60
2015-03-11 01:00:14.127818 7f41a1ba4880 10 register_cxx_method rbd.get_stripe_unit_count flags 1 0x7f41921c9330
2015-03-11 01:00:14.127822 7f41a1ba4880 10 register_cxx_method rbd.set_stripe_unit_count flags 3 0x7f41921cc070
2015-03-11 01:00:14.127825 7f41a1ba4880 10 register_cxx_method rbd.get_flags flags 1 0x7f41921d2470
2015-03-11 01:00:14.127828 7f41a1ba4880 10 register_cxx_method rbd.set_flags flags 3 0x7f41921cb370
2015-03-11 01:00:14.127831 7f41a1ba4880 10 register_cxx_method rbd.add_child flags 3 0x7f41921cd790
2015-03-11 01:00:14.127834 7f41a1ba4880 10 register_cxx_method rbd.remove_child flags 3 0x7f41921cdf50
2015-03-11 01:00:14.127838 7f41a1ba4880 10 register_cxx_method rbd.get_children flags 1 0x7f41921cd4e0
2015-03-11 01:00:14.127841 7f41a1ba4880 10 register_cxx_method rbd.get_id flags 1 0x7f41921ca580
2015-03-11 01:00:14.127844 7f41a1ba4880 10 register_cxx_method rbd.set_id flags 3 0x7f41921cb6c0
2015-03-11 01:00:14.127848 7f41a1ba4880 10 register_cxx_method rbd.dir_get_id flags 1 0x7f41921c9f40
2015-03-11 01:00:14.127851 7f41a1ba4880 10 register_cxx_method rbd.dir_get_name flags 1 0x7f41921ca210
2015-03-11 01:00:14.127855 7f41a1ba4880 10 register_cxx_method rbd.dir_list flags 1 0x7f41921cf9f0
2015-03-11 01:00:14.127858 7f41a1ba4880 10 register_cxx_method rbd.dir_add_image flags 3 0x7f41921cf840
2015-03-11 01:00:14.127862 7f41a1ba4880 10 register_cxx_method rbd.dir_remove_image flags 3 0x7f41921c9d60
2015-03-11 01:00:14.127865 7f41a1ba4880 10 register_cxx_method rbd.dir_rename_image flags 3 0x7f41921cf660
2015-03-11 01:00:14.127868 7f41a1ba4880 10 register_cxx_method rbd.object_map_load flags 1 0x7f41921ce5f0
2015-03-11 01:00:14.127872 7f41a1ba4880 10 register_cxx_method rbd.object_map_resize flags 3 0x7f41921ceec0
2015-03-11 01:00:14.127875 7f41a1ba4880 10 register_cxx_method rbd.object_map_update flags 3 0x7f41921ce720
2015-03-11 01:00:14.127879 7f41a1ba4880 10 register_cxx_method rbd.snap_list flags 1 0x7f41921c8540
2015-03-11 01:00:14.127882 7f41a1ba4880 10 register_cxx_method rbd.snap_add flags 3 0x7f41921cacc0
2015-03-11 01:00:14.127885 7f41a1ba4880 10 register_cxx_method rbd.snap_remove flags 3 0x7f41921cb910
2015-03-11 01:00:14.127891 7f41a1ba4880 10 _load_class rbd success
2015-03-11 01:00:14.127894 7f41a1ba4880 10 open_all_classes found refcount
2015-03-11 01:00:14.127897 7f41a1ba4880 10 _get_class adding new class name refcount 0x4c9e488
2015-03-11 01:00:14.127900 7f41a1ba4880 10 _load_class refcount from /usr/lib64/rados-classes/libcls_refcount.so
2015-03-11 01:00:14.132942 7f41a1ba4880 10 register_class refcount status 3
2015-03-11 01:00:14.132946 7f41a1ba4880 10 register_cxx_method refcount.get flags 3 0x7f4191f32be0
2015-03-11 01:00:14.132957 7f41a1ba4880 10 register_cxx_method refcount.put flags 3 0x7f4191f31b30
2015-03-11 01:00:14.132960 7f41a1ba4880 10 register_cxx_method refcount.set flags 3 0x7f4191f32510
2015-03-11 01:00:14.132963 7f41a1ba4880 10 register_cxx_method refcount.read flags 1 0x7f4191f31fb0
2015-03-11 01:00:14.132970 7f41a1ba4880 10 _load_class refcount success
2015-03-11 01:00:14.132977 7f41a1ba4880 10 open_all_classes found replica_log
2015-03-11 01:00:14.132984 7f41a1ba4880 10 _get_class adding new class name replica_log 0x4c9e568
2015-03-11 01:00:14.132987 7f41a1ba4880 10 _load_class replica_log from /usr/lib64/rados-classes/libcls_replica_log.so
2015-03-11 01:00:14.133269 7f41a1ba4880 10 register_class replica_log status 3
2015-03-11 01:00:14.133271 7f41a1ba4880 10 register_cxx_method replica_log.set flags 3 0x7f4191c7f920
2015-03-11 01:00:14.133276 7f41a1ba4880 10 register_cxx_method replica_log.get flags 1 0x7f4191c809f0
2015-03-11 01:00:14.133279 7f41a1ba4880 10 register_cxx_method replica_log.delete flags 3 0x7f4191c80400
2015-03-11 01:00:14.133283 7f41a1ba4880 10 _load_class replica_log success
2015-03-11 01:00:14.133285 7f41a1ba4880 10 open_all_classes found rgw
2015-03-11 01:00:14.133288 7f41a1ba4880 10 _get_class adding new class name rgw 0x4c9e728
2015-03-11 01:00:14.133291 7f41a1ba4880 10 _load_class rgw from /usr/lib64/rados-classes/libcls_rgw.so
2015-03-11 01:00:14.138712 7f41a1ba4880 10 register_class rgw status 3
2015-03-11 01:00:14.138717 7f41a1ba4880 10 register_cxx_method rgw.bucket_init_index flags 3 0x7f41919c0860
2015-03-11 01:00:14.138725 7f41a1ba4880 10 register_cxx_method rgw.bucket_set_tag_timeout flags 3 0x7f41919c4a40
2015-03-11 01:00:14.138728 7f41a1ba4880 10 register_cxx_method rgw.bucket_list flags 1 0x7f41919c8160
2015-03-11 01:00:14.138732 7f41a1ba4880 10 register_cxx_method rgw.bucket_check_index flags 1 0x7f41919ca300
2015-03-11 01:00:14.138735 7f41a1ba4880 10 register_cxx_method rgw.bucket_rebuild_index flags 3 0x7f41919ca1a0
2015-03-11 01:00:14.138740 7f41a1ba4880 10 register_cxx_method rgw.bucket_prepare_op flags 3 0x7f41919c5180
2015-03-11 01:00:14.138745 7f41a1ba4880 10 register_cxx_method rgw.bucket_complete_op flags 3 0x7f41919c5980
2015-03-11 01:00:14.138748 7f41a1ba4880 10 register_cxx_method rgw.bucket_link_olh flags 3 0x7f41919cc980
2015-03-11 01:00:14.138751 7f41a1ba4880 10 register_cxx_method rgw.bucket_unlink_instance flags 3 0x7f41919cbb80
2015-03-11 01:00:14.138754 7f41a1ba4880 10 register_cxx_method rgw.bucket_read_olh_log flags 1 0x7f41919cde40
2015-03-11 01:00:14.138757 7f41a1ba4880 10 register_cxx_method rgw.bucket_trim_olh_log flags 3 0x7f41919cb2d0
2015-03-11 01:00:14.138760 7f41a1ba4880 10 register_cxx_method rgw.bucket_clear_olh flags 3 0x7f41919cacd0
2015-03-11 01:00:14.138763 7f41a1ba4880 10 register_cxx_method rgw.obj_remove flags 3 0x7f41919c3ff0
2015-03-11 01:00:14.138771 7f41a1ba4880 10 register_cxx_method rgw.obj_check_attrs_prefix flags 1 0x7f41919c0b60
2015-03-11 01:00:14.138774 7f41a1ba4880 10 register_cxx_method rgw.bi_get flags 1 0x7f41919bf050
2015-03-11 01:00:14.138777 7f41a1ba4880 10 register_cxx_method rgw.bi_put flags 3 0x7f41919be700
2015-03-11 01:00:14.138780 7f41a1ba4880 10 register_cxx_method rgw.bi_list flags 1 0x7f41919c6790
2015-03-11 01:00:14.138784 7f41a1ba4880 10 register_cxx_method rgw.bi_log_list flags 1 0x7f41919c2180
2015-03-11 01:00:14.138787 7f41a1ba4880 10 register_cxx_method rgw.bi_log_trim flags 3 0x7f41919c26e0
2015-03-11 01:00:14.138791 7f41a1ba4880 10 register_cxx_method rgw.dir_suggest_changes flags 3 0x7f41919c9260
2015-03-11 01:00:14.138794 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_add flags 3 0x7f41919ce6a0
2015-03-11 01:00:14.138797 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_read flags 1 0x7f41919c31c0
2015-03-11 01:00:14.138799 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_trim flags 3 0x7f41919c3950
2015-03-11 01:00:14.138802 7f41a1ba4880 10 register_cxx_method rgw.gc_set_entry flags 3 0x7f41919c01b0
2015-03-11 01:00:14.138806 7f41a1ba4880 10 register_cxx_method rgw.gc_defer_entry flags 3 0x7f41919bfe90
2015-03-11 01:00:14.138809 7f41a1ba4880 10 register_cxx_method rgw.gc_list flags 1 0x7f41919c0f80
2015-03-11 01:00:14.138812 7f41a1ba4880 10 register_cxx_method rgw.gc_remove flags 3 0x7f41919c0420
2015-03-11 01:00:14.138815 7f41a1ba4880 10 _load_class rgw success
2015-03-11 01:00:14.138820 7f41a1ba4880 10 open_all_classes found statelog
2015-03-11 01:00:14.138826 7f41a1ba4880 10 _get_class adding new class name statelog 0x4c9e808
2015-03-11 01:00:14.138830 7f41a1ba4880 10 _load_class statelog from /usr/lib64/rados-classes/libcls_statelog.so
2015-03-11 01:00:14.139141 7f41a1ba4880 10 register_class statelog status 3
2015-03-11 01:00:14.139143 7f41a1ba4880 10 register_cxx_method statelog.add flags 3 0x7f4191700bc0
2015-03-11 01:00:14.139146 7f41a1ba4880 10 register_cxx_method statelog.list flags 1 0x7f4191700f40
2015-03-11 01:00:14.139150 7f41a1ba4880 10 register_cxx_method statelog.remove flags 3 0x7f41916ffa80
2015-03-11 01:00:14.139153 7f41a1ba4880 10 register_cxx_method statelog.check_state flags 1 0x7f41917002d0
2015-03-11 01:00:14.139175 7f41a1ba4880 10 _load_class statelog success
2015-03-11 01:00:14.139178 7f41a1ba4880 10 open_all_classes found user
2015-03-11 01:00:14.139182 7f41a1ba4880 10 _get_class adding new class name user 0x4c9e8e8
2015-03-11 01:00:14.139185 7f41a1ba4880 10 _load_class user from /usr/lib64/rados-classes/libcls_user.so
2015-03-11 01:00:14.139386 7f41a1ba4880 10 register_class user status 3
2015-03-11 01:00:14.139388 7f41a1ba4880 10 register_cxx_method user.set_buckets_info flags 3 0x7f41914f5180
2015-03-11 01:00:14.139392 7f41a1ba4880 10 register_cxx_method user.complete_stats_sync flags 3 0x7f41914f4aa0
2015-03-11 01:00:14.139394 7f41a1ba4880 10 register_cxx_method user.remove_bucket flags 3 0x7f41914f3d30
2015-03-11 01:00:14.139397 7f41a1ba4880 10 register_cxx_method user.list_buckets flags 1 0x7f41914f64c0
2015-03-11 01:00:14.139400 7f41a1ba4880 10 register_cxx_method user.get_header flags 1 0x7f41914f2d50
2015-03-11 01:00:14.139403 7f41a1ba4880 10 _load_class user success
2015-03-11 01:00:14.139406 7f41a1ba4880 10 open_all_classes found version
2015-03-11 01:00:14.139409 7f41a1ba4880 10 _get_class adding new class name version 0x4c9e9c8
2015-03-11 01:00:14.139412 7f41a1ba4880 10 _load_class version from /usr/lib64/rados-classes/libcls_version.so
2015-03-11 01:00:14.139608 7f41a1ba4880 10 register_class version status 3
2015-03-11 01:00:14.139611 7f41a1ba4880 10 register_cxx_method version.set flags 3 0x7f41912ebe60
2015-03-11 01:00:14.139615 7f41a1ba4880 10 register_cxx_method version.inc flags 3 0x7f41912ecc80
2015-03-11 01:00:14.139618 7f41a1ba4880 10 register_cxx_method version.inc_conds flags 3 0x7f41912ecc80
2015-03-11 01:00:14.139621 7f41a1ba4880 10 register_cxx_method version.read flags 1 0x7f41912ebb60
2015-03-11 01:00:14.139624 7f41a1ba4880 10 register_cxx_method version.check_conds flags 1 0x7f41912ec260
2015-03-11 01:00:14.139627 7f41a1ba4880 10 _load_class version success
2015-03-11 01:00:14.139682 7f41a1ba4880 20 osd.3 0 get_map 2757 - loading and decoding 0x4b4cb40
2015-03-11 01:00:14.139693 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) read meta/a143428/osdmap.2757/0//-1 0~0
2015-03-11 01:00:14.139804 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) FileStore::read meta/a143428/osdmap.2757/0//-1 0~3076/3076
2015-03-11 01:00:14.139818 7f41a1ba4880 10 osd.3 0 add_map_bl 2757 3076 bytes
2015-03-11 01:00:14.139908 7f41a1ba4880  0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients
2015-03-11 01:00:14.139914 7f41a1ba4880  0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons
2015-03-11 01:00:14.139919 7f41a1ba4880  0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds
2015-03-11 01:00:14.139922 7f41a1ba4880 10 osd.3 2757 create_recoverystate_perf
2015-03-11 01:00:14.139939 7f41a1ba4880  0 osd.3 2757 load_pgs
2015-03-11 01:00:14.139942 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) list_collections
2015-03-11 01:00:14.139981 7f41a1ba4880 10 osd.3 2757 load_pgs ignoring unrecognized meta
2015-03-11 01:00:14.139985 7f41a1ba4880  0 osd.3 2757 load_pgs opened 0 pgs
2015-03-11 01:00:14.139989 7f41a1ba4880 10 osd.3 2757 build_past_intervals_parallel nothing to build
2015-03-11 01:00:14.139992 7f41a1ba4880  2 osd.3 2757 superblock: i am osd.3
2015-03-11 01:00:14.139995 7f41a1ba4880 10 osd.3 2757 create_logger
2015-03-11 01:00:14.140126 7f41a1ba4880  1 accepter.accepter.start
2015-03-11 01:00:14.140269 7f41a1ba4880  1 accepter.accepter.start
2015-03-11 01:00:14.140434 7f41a1ba4880  1 accepter.accepter.start
2015-03-11 01:00:14.140575 7f41a1ba4880  1 accepter.accepter.start
2015-03-11 01:00:14.140813 7f41a1ba4880 -1 osd.3 2757 log_to_monitors {default=true}
2015-03-11 01:00:14.141325 7f41a1ba4880 10 osd.3 2757 set_disk_tp_priority class  priority -1
2015-03-11 01:00:14.141599 7f41a1ba4880  1 -- 0.0.0.0:6800/2840 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x4b4cd80 con 0x4cbe3c0
2015-03-11 01:00:14.142020 7f4180ac7700 20 osd.3 2757 update_osd_stat osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist [])
2015-03-11 01:00:14.142048 7f4180ac7700  5 osd.3 2757 heartbeat: osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist [])
2015-03-11 01:00:14.142221 7f417e2c2700 10 osd.3 2757 agent_entry start
2015-03-11 01:00:14.142241 7f417e2c2700 20 osd.3 2757 agent_entry empty queue
2015-03-11 01:00:14.158932 7f41a1ba0700  1 -- 192.168.0.250:6800/2840 learned my addr 192.168.0.250:6800/2840
2015-03-11 01:00:14.159058 7f41a1ba0700 10 osd.3 2757  new session (outgoing) 0x4b92280 con=0x4b602c0 addr=192.168.0.250:6800/2840
2015-03-11 01:00:14.159166 7f41a1ba0700 10 osd.3 2757 OSD::ms_get_authorizer type=mon
2015-03-11 01:00:14.173274 7f41912e8700 10 osd.3 2757 ms_handle_connect on mon
2015-03-11 01:00:14.173332 7f41912e8700 20 osd.3 2757 send_pg_stats
2015-03-11 01:00:14.175783 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x4b4f840 con 0x4cbe3c0
2015-03-11 01:00:14.175940 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (335930126 0 0) 0x4b4fa80 con 0x4cbe3c0
2015-03-11 01:00:14.175997 7f41912e8700  1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x4b31a00 con 0x4cbe3c0
2015-03-11 01:00:14.176059 7f41912e8700  1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0,osdmap=2757}) v2 -- ?+0 0x4b32800 con 0x4cbe3c0
2015-03-11 01:00:14.176147 7f41a1ba4880 10 osd.3 2757 ensuring pgs have consumed prior maps
2015-03-11 01:00:14.176170 7f41a1ba4880  7 osd.3 2757 consume_map version 2757
2015-03-11 01:00:14.176187 7f41a1ba4880  0 osd.3 2757 done with init, starting boot process
2015-03-11 01:00:14.176191 7f41a1ba4880 10 osd.3 2757 start_boot - have maps 2757..2757
2015-03-11 01:00:14.176221 7f41a1ba4880  1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x4bc41e0 con 0x4cbe3c0
2015-03-11 01:00:14.178539 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 3 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x4b4fcc0 con 0x4cbe3c0
2015-03-11 01:00:14.178650 7f41912e8700 10 osd.3 2757 do_waiters -- start
2015-03-11 01:00:14.178658 7f41912e8700 10 osd.3 2757 do_waiters -- finish
2015-03-11 01:00:14.178661 7f41912e8700 20 osd.3 2757 _dispatch 0x4b4fcc0 osd_map(3277..3277 src has 2757..3277) v3
2015-03-11 01:00:14.178888 7f41912e8700  3 osd.3 2757 handle_osd_map epochs [3277,3277], i have 2757, src has [2757,3277]
2015-03-11 01:00:14.178906 7f41912e8700 10 osd.3 2757 handle_osd_map message skips epochs 2758..3276
2015-03-11 01:00:14.178954 7f41912e8700  1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=2758}) v2 -- ?+0 0x4b31a00 con 0x4cbe3c0
2015-03-11 01:00:14.178991 7f41912e8700 10 osd.3 2757 do_waiters -- start
2015-03-11 01:00:14.178996 7f41912e8700 10 osd.3 2757 do_waiters -- finish
2015-03-11 01:00:14.179012 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 4 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x4b4f840 con 0x4cbe3c0
2015-03-11 01:00:14.184757 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(2757..2857 src has 2757..3277) v3 ==== 37648+0+0 (2648067016 0 0) 0x4b4fa80 con 0x4cbe3c0
2015-03-11 01:00:14.184831 7f41912e8700 10 osd.3 2757 do_waiters -- start
2015-03-11 01:00:14.184838 7f41912e8700 10 osd.3 2757 do_waiters -- finish
2015-03-11 01:00:14.184841 7f41912e8700 20 osd.3 2757 _dispatch 0x4b4fa80 osd_map(2757..2857 src has 2757..3277) v3
2015-03-11 01:00:14.184872 7f41912e8700  3 osd.3 2757 handle_osd_map epochs [2757,2857], i have 2757, src has [2757,3277]
2015-03-11 01:00:14.184884 7f41912e8700 10 osd.3 2757 handle_osd_map  got inc map for epoch 2758
2015-03-11 01:00:14.185229 7f41912e8700 10 osd.3 2757 handle_osd_map  got inc map for epoch 2759
2015-03-11 01:00:14.195165 7f41912e8700 -1 *** Caught signal (Aborted) **
  in thread 7f41912e8700

  ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
  1: /usr/bin/ceph-osd() [0xac7cea]
  2: (()+0x10050) [0x7f41a0aaf050]
  3: (gsignal()+0x37) [0x7f419ef820f7]
  4: (abort()+0x13a) [0x7f419ef834ca]
  5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f419f867fe5]
  6: (()+0x63186) [0x7f419f866186]
  7: (()+0x631b3) [0x7f419f8661b3]
  8: (()+0x633d2) [0x7f419f8663d2]
  9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
  10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
  11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
  12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
  13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
  14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
  15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
  16: (DispatchQueue::entry()+0x649) [0xc6fe09]
  17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
  18: (()+0x83a4) [0x7f41a0aa73a4]
  19: (clone()+0x6d) [0x7f419f031a4d]
  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -363> 2015-03-11 01:00:13.871379 7f41a1ba4880  5 asok(0x4b50000) register_command perfcounters_dump hook 0x4b04030
   -362> 2015-03-11 01:00:13.871462 7f41a1ba4880  5 asok(0x4b50000) register_command 1 hook 0x4b04030
   -361> 2015-03-11 01:00:13.871471 7f41a1ba4880  5 asok(0x4b50000) register_command perf dump hook 0x4b04030
   -360> 2015-03-11 01:00:13.871484 7f41a1ba4880  5 asok(0x4b50000) register_command perfcounters_schema hook 0x4b04030
   -359> 2015-03-11 01:00:13.871507 7f41a1ba4880  5 asok(0x4b50000) register_command 2 hook 0x4b04030
   -358> 2015-03-11 01:00:13.871512 7f41a1ba4880  5 asok(0x4b50000) register_command perf schema hook 0x4b04030
   -357> 2015-03-11 01:00:13.871545 7f41a1ba4880  5 asok(0x4b50000) register_command perf reset hook 0x4b04030
   -356> 2015-03-11 01:00:13.871555 7f41a1ba4880  5 asok(0x4b50000) register_command config show hook 0x4b04030
   -355> 2015-03-11 01:00:13.871562 7f41a1ba4880  5 asok(0x4b50000) register_command config set hook 0x4b04030
   -354> 2015-03-11 01:00:13.871571 7f41a1ba4880  5 asok(0x4b50000) register_command config get hook 0x4b04030
   -353> 2015-03-11 01:00:13.871576 7f41a1ba4880  5 asok(0x4b50000) register_command config diff hook 0x4b04030
   -352> 2015-03-11 01:00:13.871585 7f41a1ba4880  5 asok(0x4b50000) register_command log flush hook 0x4b04030
   -351> 2015-03-11 01:00:13.871590 7f41a1ba4880  5 asok(0x4b50000) register_command log dump hook 0x4b04030
   -350> 2015-03-11 01:00:13.871600 7f41a1ba4880  5 asok(0x4b50000) register_command log reopen hook 0x4b04030
   -349> 2015-03-11 01:00:13.876152 7f41a1ba4880  0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 2840
   -348> 2015-03-11 01:00:13.877059 7f41a1ba4880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/2840 need_addr=1
   -347> 2015-03-11 01:00:13.877111 7f41a1ba4880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/2840 need_addr=1
   -346> 2015-03-11 01:00:13.877140 7f41a1ba4880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/2840 need_addr=1
   -345> 2015-03-11 01:00:13.877169 7f41a1ba4880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/2840 need_addr=1
   -344> 2015-03-11 01:00:13.880323 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) dump_stop
   -343> 2015-03-11 01:00:13.880362 7f41a1ba4880  5 asok(0x4b50000) init /var/run/ceph/ceph-osd.3.asok
   -342> 2015-03-11 01:00:13.880382 7f41a1ba4880  5 asok(0x4b50000) bind_and_listen /var/run/ceph/ceph-osd.3.asok
   -341> 2015-03-11 01:00:13.880604 7f41a1ba4880  5 asok(0x4b50000) register_command 0 hook 0x4b000c0
   -340> 2015-03-11 01:00:13.880620 7f41a1ba4880  5 asok(0x4b50000) register_command version hook 0x4b000c0
   -339> 2015-03-11 01:00:13.880626 7f41a1ba4880  5 asok(0x4b50000) register_command git_version hook 0x4b000c0
   -338> 2015-03-11 01:00:13.880633 7f41a1ba4880  5 asok(0x4b50000) register_command help hook 0x4b04110
   -337> 2015-03-11 01:00:13.880639 7f41a1ba4880  5 asok(0x4b50000) register_command get_command_descriptions hook 0x4b04120
   -336> 2015-03-11 01:00:13.880703 7f41a1ba4880 10 monclient(hunting): build_initial_monmap
   -335> 2015-03-11 01:00:13.880697 7f419cf7d700  5 asok(0x4b50000) entry start
   -334> 2015-03-11 01:00:13.890355 7f41a1ba4880 10 ErasureCodePluginSelectJerasure: load: jerasure_generic
   -333> 2015-03-11 01:00:13.897843 7f41a1ba4880 10 load: jerasure load: lrc load: isa
   -332> 2015-03-11 01:00:13.897900 7f41a1ba4880  5 adding auth protocol: none
   -331> 2015-03-11 01:00:13.897905 7f41a1ba4880  5 adding auth protocol: none
   -330> 2015-03-11 01:00:13.898086 7f41a1ba4880  5 asok(0x4b50000) register_command objecter_requests hook 0x4b041b0
   -329> 2015-03-11 01:00:13.898172 7f41a1ba4880  5 filestore(/var/lib/ceph/osd/ceph-3) test_mount basedir /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal
   -328> 2015-03-11 01:00:13.898290 7f41a1ba4880  1 -- 0.0.0.0:6800/2840 messenger.start
   -327> 2015-03-11 01:00:13.898362 7f41a1ba4880  1 -- :/0 messenger.start
   -326> 2015-03-11 01:00:13.898450 7f41a1ba4880  1 -- 0.0.0.0:6803/2840 messenger.start
   -325> 2015-03-11 01:00:13.898527 7f41a1ba4880  1 -- 0.0.0.0:6802/2840 messenger.start
   -324> 2015-03-11 01:00:13.898609 7f41a1ba4880  1 -- 0.0.0.0:6801/2840 messenger.start
   -323> 2015-03-11 01:00:13.898672 7f41a1ba4880  1 -- :/0 messenger.start
   -322> 2015-03-11 01:00:13.898792 7f41a1ba4880  2 osd.3 0 mounting /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
   -321> 2015-03-11 01:00:13.898826 7f41a1ba4880  5 filestore(/var/lib/ceph/osd/ceph-3) basedir /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal
   -320> 2015-03-11 01:00:13.898868 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) mount fsid is 05fb481f-68b3-4442-b7de-af4c867e18d5
   -319> 2015-03-11 01:00:13.898937 7f41a1ba4880  0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342)
   -318> 2015-03-11 01:00:13.941605 7f41a1ba4880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work
   -317> 2015-03-11 01:00:13.941665 7f41a1ba4880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
   -316> 2015-03-11 01:00:13.970056 7f41a1ba4880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
   -315> 2015-03-11 01:00:13.970442 7f41a1ba4880  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5
   -314> 2015-03-11 01:00:14.001199 7f41a1ba4880  5 filestore(/var/lib/ceph/osd/ceph-3) mount op_seq is 2
   -313> 2015-03-11 01:00:14.122464 7f41a1ba4880 20 filestore (init)dbobjectmap: seq is 1
   -312> 2015-03-11 01:00:14.122580 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) open_journal at /var/lib/ceph/osd/ceph-3/journal
   -311> 2015-03-11 01:00:14.122674 7f41a1ba4880  0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
   -310> 2015-03-11 01:00:14.122682 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) list_collections
   -309> 2015-03-11 01:00:14.122987 7f41a1ba4880  2 journal open /var/lib/ceph/osd/ceph-3/journal fsid 05fb481f-68b3-4442-b7de-af4c867e18d5 fs_op_seq 2
   -308> 2015-03-11 01:00:14.123024 7f41a1ba4880 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
   -307> 2015-03-11 01:00:14.123040 7f41a1ba4880  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
   -306> 2015-03-11 01:00:14.123003 7f4196c1b700 20 filestore(/var/lib/ceph/osd/ceph-3) sync_entry waiting for max_interval 5.000000
   -305> 2015-03-11 01:00:14.123590 7f41a1ba4880  2 journal read_entry 8192 : seq 3 193 bytes
   -304> 2015-03-11 01:00:14.123639 7f41a1ba4880  2 journal read_entry 8192 : seq 3 193 bytes
   -303> 2015-03-11 01:00:14.123661 7f41a1ba4880  3 journal journal_replay: applying op seq 3
   -302> 2015-03-11 01:00:14.123724 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x4b9b6c0
   -301> 2015-03-11 01:00:14.123837 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
   -300> 2015-03-11 01:00:14.123971 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
   -299> 2015-03-11 01:00:14.124020 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) touch meta/a468ec03/snapmapper/0//-1
   -298> 2015-03-11 01:00:14.124103 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) touch meta/a468ec03/snapmapper/0//-1 = 0
   -297> 2015-03-11 01:00:14.124163 7f41a1ba4880  3 journal journal_replay: r = 0, op_seq now 3
   -296> 2015-03-11 01:00:14.124220 7f41a1ba4880  2 journal read_entry 12288 : seq 4 3829 bytes
   -295> 2015-03-11 01:00:14.124231 7f41a1ba4880  3 journal journal_replay: applying op seq 4
   -294> 2015-03-11 01:00:14.124247 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x4b9b6c0
   -293> 2015-03-11 01:00:14.124282 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
   -292> 2015-03-11 01:00:14.124332 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
   -291> 2015-03-11 01:00:14.124348 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) write meta/a143428/osdmap.2757/0//-1 0~3076
   -290> 2015-03-11 01:00:14.124420 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) write meta/a143428/osdmap.2757/0//-1 0~3076 = 3076
   -289> 2015-03-11 01:00:14.124461 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
   -288> 2015-03-11 01:00:14.124547 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
   -287> 2015-03-11 01:00:14.124557 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) write meta/23c2fcde/osd_superblock/0//-1 0~413
   -286> 2015-03-11 01:00:14.124615 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) write meta/23c2fcde/osd_superblock/0//-1 0~413 = 413
   -285> 2015-03-11 01:00:14.124676 7f41a1ba4880  3 journal journal_replay: r = 0, op_seq now 4
   -284> 2015-03-11 01:00:14.124697 7f41a1ba4880  2 journal No further valid entries found, journal is most likely valid
   -283> 2015-03-11 01:00:14.124700 7f41a1ba4880  3 journal journal_replay: end of journal, done.
   -282> 2015-03-11 01:00:14.124716 7f41a1ba4880  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
   -281> 2015-03-11 01:00:14.125247 7f41a1ba4880  2 osd.3 0 boot
   -280> 2015-03-11 01:00:14.125270 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) read meta/23c2fcde/osd_superblock/0//-1 0~0
   -279> 2015-03-11 01:00:14.125343 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) FileStore::read meta/23c2fcde/osd_superblock/0//-1 0~413/413
   -278> 2015-03-11 01:00:14.125393 7f41a1ba4880 10 osd.3 0 read_superblock sb(3bf19e54-97e2-4a5f-b811-7c3780cd5e57 osd.3 05fb481f-68b3-4442-b7de-af4c867e18d5 e2757 [2757,2757] lci=[0,0])
   -277> 2015-03-11 01:00:14.125521 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) stat meta/a468ec03/snapmapper/0//-1 = 0 (size 0)
   -276> 2015-03-11 01:00:14.125540 7f41a1ba4880 10 open_all_classes
   -275> 2015-03-11 01:00:14.125586 7f41a1ba4880 10 open_all_classes found hello
   -274> 2015-03-11 01:00:14.125623 7f41a1ba4880 10 _get_class adding new class name hello 0x4c9e028
   -273> 2015-03-11 01:00:14.125630 7f41a1ba4880 10 _load_class hello from /usr/lib64/rados-classes/libcls_hello.so
   -272> 2015-03-11 01:00:14.126058 7f41a1ba4880  0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
   -271> 2015-03-11 01:00:14.126080 7f41a1ba4880 10 register_class hello status 3
   -270> 2015-03-11 01:00:14.126084 7f41a1ba4880 10 register_cxx_method hello.say_hello flags 1 0x7f4192a10350
   -269> 2015-03-11 01:00:14.126092 7f41a1ba4880 10 register_cxx_method hello.record_hello flags 2 0x7f4192a113c0
   -268> 2015-03-11 01:00:14.126098 7f41a1ba4880 10 register_cxx_method hello.writes_dont_return_data flags 2 0x7f4192a10650
   -267> 2015-03-11 01:00:14.126103 7f41a1ba4880 10 register_cxx_method hello.replay flags 1 0x7f4192a10320
   -266> 2015-03-11 01:00:14.126109 7f41a1ba4880 10 register_cxx_method hello.turn_it_to_11 flags 3 0x7f4192a10920
   -265> 2015-03-11 01:00:14.126114 7f41a1ba4880 10 register_cxx_method hello.bad_reader flags 2 0x7f4192a10310
   -264> 2015-03-11 01:00:14.126119 7f41a1ba4880 10 register_cxx_method hello.bad_writer flags 1 0x7f4192a10300
   -263> 2015-03-11 01:00:14.126127 7f41a1ba4880 10 _load_class hello success
   -262> 2015-03-11 01:00:14.126132 7f41a1ba4880 10 open_all_classes found kvs
   -261> 2015-03-11 01:00:14.126137 7f41a1ba4880 10 _get_class adding new class name kvs 0x4c9e108
   -260> 2015-03-11 01:00:14.126142 7f41a1ba4880 10 _load_class kvs from /usr/lib64/rados-classes/libcls_kvs.so
   -259> 2015-03-11 01:00:14.126655 7f41a1ba4880 10 register_class kvs status 3
   -258> 2015-03-11 01:00:14.126665 7f41a1ba4880 10 register_cxx_method kvs.get_idata_from_key flags 1 0x7f41927ff060
   -257> 2015-03-11 01:00:14.126669 7f41a1ba4880 10 register_cxx_method kvs.get_next_idata flags 1 0x7f4192801050
   -256> 2015-03-11 01:00:14.126672 7f41a1ba4880 10 register_cxx_method kvs.get_prev_idata flags 1 0x7f41927fd570
   -255> 2015-03-11 01:00:14.126675 7f41a1ba4880 10 register_cxx_method kvs.read_many flags 1 0x7f41927f8f30
   -254> 2015-03-11 01:00:14.126678 7f41a1ba4880 10 register_cxx_method kvs.check_writable flags 3 0x7f41927f8ac0
   -253> 2015-03-11 01:00:14.126681 7f41a1ba4880 10 register_cxx_method kvs.assert_size_in_bound flags 2 0x7f41927f8d40
   -252> 2015-03-11 01:00:14.126684 7f41a1ba4880 10 register_cxx_method kvs.omap_insert flags 2 0x7f41927fcb50
   -251> 2015-03-11 01:00:14.126688 7f41a1ba4880 10 register_cxx_method kvs.create_with_omap flags 2 0x7f41927fc0e0
   -250> 2015-03-11 01:00:14.126691 7f41a1ba4880 10 register_cxx_method kvs.omap_remove flags 2 0x7f41927f9e70
   -249> 2015-03-11 01:00:14.126696 7f41a1ba4880 10 register_cxx_method kvs.maybe_read_for_balance flags 1 0x7f41927fab70
   -248> 2015-03-11 01:00:14.126735 7f41a1ba4880 10 _load_class kvs success
   -247> 2015-03-11 01:00:14.126739 7f41a1ba4880 10 open_all_classes found lock
   -246> 2015-03-11 01:00:14.126742 7f41a1ba4880 10 _get_class adding new class name lock 0x4c9e1e8
   -245> 2015-03-11 01:00:14.126745 7f41a1ba4880 10 _load_class lock from /usr/lib64/rados-classes/libcls_lock.so
   -244> 2015-03-11 01:00:14.126972 7f41a1ba4880 10 register_class lock status 3
   -243> 2015-03-11 01:00:14.126977 7f41a1ba4880 10 register_cxx_method lock.lock flags 3 0x7f41925ee800
   -242> 2015-03-11 01:00:14.126981 7f41a1ba4880 10 register_cxx_method lock.unlock flags 3 0x7f41925ed210
   -241> 2015-03-11 01:00:14.126984 7f41a1ba4880 10 register_cxx_method lock.break_lock flags 3 0x7f41925ece60
   -240> 2015-03-11 01:00:14.126986 7f41a1ba4880 10 register_cxx_method lock.get_info flags 1 0x7f41925ed610
   -239> 2015-03-11 01:00:14.126990 7f41a1ba4880 10 register_cxx_method lock.list_locks flags 1 0x7f41925eace0
   -238> 2015-03-11 01:00:14.126993 7f41a1ba4880 10 register_cxx_method lock.assert_locked flags 1 0x7f41925ec330
   -237> 2015-03-11 01:00:14.126996 7f41a1ba4880 10 _load_class lock success
   -236> 2015-03-11 01:00:14.126999 7f41a1ba4880 10 open_all_classes found log
   -235> 2015-03-11 01:00:14.127002 7f41a1ba4880 10 _get_class adding new class name log 0x4c9e2c8
   -234> 2015-03-11 01:00:14.127005 7f41a1ba4880 10 _load_class log from /usr/lib64/rados-classes/libcls_log.so
   -233> 2015-03-11 01:00:14.127213 7f41a1ba4880  1 <cls> cls/log/cls_log.cc:312: Loaded log class!
   -232> 2015-03-11 01:00:14.127223 7f41a1ba4880 10 register_class log status 3
   -231> 2015-03-11 01:00:14.127225 7f41a1ba4880 10 register_cxx_method log.add flags 3 0x7f41923e1560
   -230> 2015-03-11 01:00:14.127228 7f41a1ba4880 10 register_cxx_method log.list flags 1 0x7f41923e2740
   -229> 2015-03-11 01:00:14.127231 7f41a1ba4880 10 register_cxx_method log.trim flags 3 0x7f41923e1f90
   -228> 2015-03-11 01:00:14.127234 7f41a1ba4880 10 register_cxx_method log.info flags 1 0x7f41923e1090
   -227> 2015-03-11 01:00:14.127238 7f41a1ba4880 10 _load_class log success
   -226> 2015-03-11 01:00:14.127241 7f41a1ba4880 10 open_all_classes found rbd
   -225> 2015-03-11 01:00:14.127244 7f41a1ba4880 10 _get_class adding new class name rbd 0x4c9e3a8
   -224> 2015-03-11 01:00:14.127247 7f41a1ba4880 10 _load_class rbd from /usr/lib64/rados-classes/libcls_rbd.so
   -223> 2015-03-11 01:00:14.127735 7f41a1ba4880 10 register_class rbd status 3
   -222> 2015-03-11 01:00:14.127745 7f41a1ba4880 10 register_cxx_method rbd.create flags 3 0x7f41921cc5c0
   -221> 2015-03-11 01:00:14.127748 7f41a1ba4880 10 register_cxx_method rbd.get_features flags 1 0x7f41921d3190
   -220> 2015-03-11 01:00:14.127751 7f41a1ba4880 10 register_cxx_method rbd.get_size flags 1 0x7f41921d2de0
   -219> 2015-03-11 01:00:14.127754 7f41a1ba4880 10 register_cxx_method rbd.set_size flags 3 0x7f41921ca7e0
   -218> 2015-03-11 01:00:14.127758 7f41a1ba4880 10 register_cxx_method rbd.get_snapcontext flags 1 0x7f41921cdb40
   -217> 2015-03-11 01:00:14.127762 7f41a1ba4880 10 register_cxx_method rbd.get_object_prefix flags 1 0x7f41921c9770
   -216> 2015-03-11 01:00:14.127766 7f41a1ba4880 10 register_cxx_method rbd.get_snapshot_name flags 1 0x7f41921d1dd0
   -215> 2015-03-11 01:00:14.127770 7f41a1ba4880 10 register_cxx_method rbd.snapshot_add flags 3 0x7f41921d0830
   -214> 2015-03-11 01:00:14.127773 7f41a1ba4880 10 register_cxx_method rbd.snapshot_remove flags 3 0x7f41921d1b40
   -213> 2015-03-11 01:00:14.127776 7f41a1ba4880 10 register_cxx_method rbd.get_all_features flags 1 0x7f41921c78c0
   -212> 2015-03-11 01:00:14.127779 7f41a1ba4880 10 register_cxx_method rbd.copyup flags 3 0x7f41921c7940
   -211> 2015-03-11 01:00:14.127782 7f41a1ba4880 10 register_cxx_method rbd.get_parent flags 1 0x7f41921d2080
   -210> 2015-03-11 01:00:14.127786 7f41a1ba4880 10 register_cxx_method rbd.set_parent flags 3 0x7f41921ccce0
   -209> 2015-03-11 01:00:14.127789 7f41a1ba4880 10 register_cxx_method rbd.remove_parent flags 3 0x7f41921c9140
   -208> 2015-03-11 01:00:14.127792 7f41a1ba4880 10 register_cxx_method rbd.set_protection_status flags 3 0x7f41921d2770
   -207> 2015-03-11 01:00:14.127815 7f41a1ba4880 10 register_cxx_method rbd.get_protection_status flags 1 0x7f41921d2b60
   -206> 2015-03-11 01:00:14.127818 7f41a1ba4880 10 register_cxx_method rbd.get_stripe_unit_count flags 1 0x7f41921c9330
   -205> 2015-03-11 01:00:14.127822 7f41a1ba4880 10 register_cxx_method rbd.set_stripe_unit_count flags 3 0x7f41921cc070
   -204> 2015-03-11 01:00:14.127825 7f41a1ba4880 10 register_cxx_method rbd.get_flags flags 1 0x7f41921d2470
   -203> 2015-03-11 01:00:14.127828 7f41a1ba4880 10 register_cxx_method rbd.set_flags flags 3 0x7f41921cb370
   -202> 2015-03-11 01:00:14.127831 7f41a1ba4880 10 register_cxx_method rbd.add_child flags 3 0x7f41921cd790
   -201> 2015-03-11 01:00:14.127834 7f41a1ba4880 10 register_cxx_method rbd.remove_child flags 3 0x7f41921cdf50
   -200> 2015-03-11 01:00:14.127838 7f41a1ba4880 10 register_cxx_method rbd.get_children flags 1 0x7f41921cd4e0
   -199> 2015-03-11 01:00:14.127841 7f41a1ba4880 10 register_cxx_method rbd.get_id flags 1 0x7f41921ca580
   -198> 2015-03-11 01:00:14.127844 7f41a1ba4880 10 register_cxx_method rbd.set_id flags 3 0x7f41921cb6c0
   -197> 2015-03-11 01:00:14.127848 7f41a1ba4880 10 register_cxx_method rbd.dir_get_id flags 1 0x7f41921c9f40
   -196> 2015-03-11 01:00:14.127851 7f41a1ba4880 10 register_cxx_method rbd.dir_get_name flags 1 0x7f41921ca210
   -195> 2015-03-11 01:00:14.127855 7f41a1ba4880 10 register_cxx_method rbd.dir_list flags 1 0x7f41921cf9f0
   -194> 2015-03-11 01:00:14.127858 7f41a1ba4880 10 register_cxx_method rbd.dir_add_image flags 3 0x7f41921cf840
   -193> 2015-03-11 01:00:14.127862 7f41a1ba4880 10 register_cxx_method rbd.dir_remove_image flags 3 0x7f41921c9d60
   -192> 2015-03-11 01:00:14.127865 7f41a1ba4880 10 register_cxx_method rbd.dir_rename_image flags 3 0x7f41921cf660
   -191> 2015-03-11 01:00:14.127868 7f41a1ba4880 10 register_cxx_method rbd.object_map_load flags 1 0x7f41921ce5f0
   -190> 2015-03-11 01:00:14.127872 7f41a1ba4880 10 register_cxx_method rbd.object_map_resize flags 3 0x7f41921ceec0
   -189> 2015-03-11 01:00:14.127875 7f41a1ba4880 10 register_cxx_method rbd.object_map_update flags 3 0x7f41921ce720
   -188> 2015-03-11 01:00:14.127879 7f41a1ba4880 10 register_cxx_method rbd.snap_list flags 1 0x7f41921c8540
   -187> 2015-03-11 01:00:14.127882 7f41a1ba4880 10 register_cxx_method rbd.snap_add flags 3 0x7f41921cacc0
   -186> 2015-03-11 01:00:14.127885 7f41a1ba4880 10 register_cxx_method rbd.snap_remove flags 3 0x7f41921cb910
   -185> 2015-03-11 01:00:14.127891 7f41a1ba4880 10 _load_class rbd success
   -184> 2015-03-11 01:00:14.127894 7f41a1ba4880 10 open_all_classes found refcount
   -183> 2015-03-11 01:00:14.127897 7f41a1ba4880 10 _get_class adding new class name refcount 0x4c9e488
   -182> 2015-03-11 01:00:14.127900 7f41a1ba4880 10 _load_class refcount from /usr/lib64/rados-classes/libcls_refcount.so
   -181> 2015-03-11 01:00:14.132905 7f41a1ba4880  1 <cls> cls/refcount/cls_refcount.cc:231: Loaded refcount class!
   -180> 2015-03-11 01:00:14.132942 7f41a1ba4880 10 register_class refcount status 3
   -179> 2015-03-11 01:00:14.132946 7f41a1ba4880 10 register_cxx_method refcount.get flags 3 0x7f4191f32be0
   -178> 2015-03-11 01:00:14.132957 7f41a1ba4880 10 register_cxx_method refcount.put flags 3 0x7f4191f31b30
   -177> 2015-03-11 01:00:14.132960 7f41a1ba4880 10 register_cxx_method refcount.set flags 3 0x7f4191f32510
   -176> 2015-03-11 01:00:14.132963 7f41a1ba4880 10 register_cxx_method refcount.read flags 1 0x7f4191f31fb0
   -175> 2015-03-11 01:00:14.132970 7f41a1ba4880 10 _load_class refcount success
   -174> 2015-03-11 01:00:14.132977 7f41a1ba4880 10 open_all_classes found replica_log
   -173> 2015-03-11 01:00:14.132984 7f41a1ba4880 10 _get_class adding new class name replica_log 0x4c9e568
   -172> 2015-03-11 01:00:14.132987 7f41a1ba4880 10 _load_class replica_log from /usr/lib64/rados-classes/libcls_replica_log.so
   -171> 2015-03-11 01:00:14.133259 7f41a1ba4880  1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class!
   -170> 2015-03-11 01:00:14.133269 7f41a1ba4880 10 register_class replica_log status 3
   -169> 2015-03-11 01:00:14.133271 7f41a1ba4880 10 register_cxx_method replica_log.set flags 3 0x7f4191c7f920
   -168> 2015-03-11 01:00:14.133276 7f41a1ba4880 10 register_cxx_method replica_log.get flags 1 0x7f4191c809f0
   -167> 2015-03-11 01:00:14.133279 7f41a1ba4880 10 register_cxx_method replica_log.delete flags 3 0x7f4191c80400
   -166> 2015-03-11 01:00:14.133283 7f41a1ba4880 10 _load_class replica_log success
   -165> 2015-03-11 01:00:14.133285 7f41a1ba4880 10 open_all_classes found rgw
   -164> 2015-03-11 01:00:14.133288 7f41a1ba4880 10 _get_class adding new class name rgw 0x4c9e728
   -163> 2015-03-11 01:00:14.133291 7f41a1ba4880 10 _load_class rgw from /usr/lib64/rados-classes/libcls_rgw.so
   -162> 2015-03-11 01:00:14.138677 7f41a1ba4880  1 <cls> cls/rgw/cls_rgw.cc:3046: Loaded rgw class!
   -161> 2015-03-11 01:00:14.138712 7f41a1ba4880 10 register_class rgw status 3
   -160> 2015-03-11 01:00:14.138717 7f41a1ba4880 10 register_cxx_method rgw.bucket_init_index flags 3 0x7f41919c0860
   -159> 2015-03-11 01:00:14.138725 7f41a1ba4880 10 register_cxx_method rgw.bucket_set_tag_timeout flags 3 0x7f41919c4a40
   -158> 2015-03-11 01:00:14.138728 7f41a1ba4880 10 register_cxx_method rgw.bucket_list flags 1 0x7f41919c8160
   -157> 2015-03-11 01:00:14.138732 7f41a1ba4880 10 register_cxx_method rgw.bucket_check_index flags 1 0x7f41919ca300
   -156> 2015-03-11 01:00:14.138735 7f41a1ba4880 10 register_cxx_method rgw.bucket_rebuild_index flags 3 0x7f41919ca1a0
   -155> 2015-03-11 01:00:14.138740 7f41a1ba4880 10 register_cxx_method rgw.bucket_prepare_op flags 3 0x7f41919c5180
   -154> 2015-03-11 01:00:14.138745 7f41a1ba4880 10 register_cxx_method rgw.bucket_complete_op flags 3 0x7f41919c5980
   -153> 2015-03-11 01:00:14.138748 7f41a1ba4880 10 register_cxx_method rgw.bucket_link_olh flags 3 0x7f41919cc980
   -152> 2015-03-11 01:00:14.138751 7f41a1ba4880 10 register_cxx_method rgw.bucket_unlink_instance flags 3 0x7f41919cbb80
   -151> 2015-03-11 01:00:14.138754 7f41a1ba4880 10 register_cxx_method rgw.bucket_read_olh_log flags 1 0x7f41919cde40
   -150> 2015-03-11 01:00:14.138757 7f41a1ba4880 10 register_cxx_method rgw.bucket_trim_olh_log flags 3 0x7f41919cb2d0
   -149> 2015-03-11 01:00:14.138760 7f41a1ba4880 10 register_cxx_method rgw.bucket_clear_olh flags 3 0x7f41919cacd0
   -148> 2015-03-11 01:00:14.138763 7f41a1ba4880 10 register_cxx_method rgw.obj_remove flags 3 0x7f41919c3ff0
   -147> 2015-03-11 01:00:14.138771 7f41a1ba4880 10 register_cxx_method rgw.obj_check_attrs_prefix flags 1 0x7f41919c0b60
   -146> 2015-03-11 01:00:14.138774 7f41a1ba4880 10 register_cxx_method rgw.bi_get flags 1 0x7f41919bf050
   -145> 2015-03-11 01:00:14.138777 7f41a1ba4880 10 register_cxx_method rgw.bi_put flags 3 0x7f41919be700
   -144> 2015-03-11 01:00:14.138780 7f41a1ba4880 10 register_cxx_method rgw.bi_list flags 1 0x7f41919c6790
   -143> 2015-03-11 01:00:14.138784 7f41a1ba4880 10 register_cxx_method rgw.bi_log_list flags 1 0x7f41919c2180
   -142> 2015-03-11 01:00:14.138787 7f41a1ba4880 10 register_cxx_method rgw.bi_log_trim flags 3 0x7f41919c26e0
   -141> 2015-03-11 01:00:14.138791 7f41a1ba4880 10 register_cxx_method rgw.dir_suggest_changes flags 3 0x7f41919c9260
   -140> 2015-03-11 01:00:14.138794 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_add flags 3 0x7f41919ce6a0
   -139> 2015-03-11 01:00:14.138797 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_read flags 1 0x7f41919c31c0
   -138> 2015-03-11 01:00:14.138799 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_trim flags 3 0x7f41919c3950
   -137> 2015-03-11 01:00:14.138802 7f41a1ba4880 10 register_cxx_method rgw.gc_set_entry flags 3 0x7f41919c01b0
   -136> 2015-03-11 01:00:14.138806 7f41a1ba4880 10 register_cxx_method rgw.gc_defer_entry flags 3 0x7f41919bfe90
   -135> 2015-03-11 01:00:14.138809 7f41a1ba4880 10 register_cxx_method rgw.gc_list flags 1 0x7f41919c0f80
   -134> 2015-03-11 01:00:14.138812 7f41a1ba4880 10 register_cxx_method rgw.gc_remove flags 3 0x7f41919c0420
   -133> 2015-03-11 01:00:14.138815 7f41a1ba4880 10 _load_class rgw success
   -132> 2015-03-11 01:00:14.138820 7f41a1ba4880 10 open_all_classes found statelog
   -131> 2015-03-11 01:00:14.138826 7f41a1ba4880 10 _get_class adding new class name statelog 0x4c9e808
   -130> 2015-03-11 01:00:14.138830 7f41a1ba4880 10 _load_class statelog from /usr/lib64/rados-classes/libcls_statelog.so
   -129> 2015-03-11 01:00:14.139131 7f41a1ba4880  1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class!
   -128> 2015-03-11 01:00:14.139141 7f41a1ba4880 10 register_class statelog status 3
   -127> 2015-03-11 01:00:14.139143 7f41a1ba4880 10 register_cxx_method statelog.add flags 3 0x7f4191700bc0
   -126> 2015-03-11 01:00:14.139146 7f41a1ba4880 10 register_cxx_method statelog.list flags 1 0x7f4191700f40
   -125> 2015-03-11 01:00:14.139150 7f41a1ba4880 10 register_cxx_method statelog.remove flags 3 0x7f41916ffa80
   -124> 2015-03-11 01:00:14.139153 7f41a1ba4880 10 register_cxx_method statelog.check_state flags 1 0x7f41917002d0
   -123> 2015-03-11 01:00:14.139175 7f41a1ba4880 10 _load_class statelog success
   -122> 2015-03-11 01:00:14.139178 7f41a1ba4880 10 open_all_classes found user
   -121> 2015-03-11 01:00:14.139182 7f41a1ba4880 10 _get_class adding new class name user 0x4c9e8e8
   -120> 2015-03-11 01:00:14.139185 7f41a1ba4880 10 _load_class user from /usr/lib64/rados-classes/libcls_user.so
   -119> 2015-03-11 01:00:14.139377 7f41a1ba4880  1 <cls> cls/user/cls_user.cc:367: Loaded user class!
   -118> 2015-03-11 01:00:14.139386 7f41a1ba4880 10 register_class user status 3
   -117> 2015-03-11 01:00:14.139388 7f41a1ba4880 10 register_cxx_method user.set_buckets_info flags 3 0x7f41914f5180
   -116> 2015-03-11 01:00:14.139392 7f41a1ba4880 10 register_cxx_method user.complete_stats_sync flags 3 0x7f41914f4aa0
   -115> 2015-03-11 01:00:14.139394 7f41a1ba4880 10 register_cxx_method user.remove_bucket flags 3 0x7f41914f3d30
   -114> 2015-03-11 01:00:14.139397 7f41a1ba4880 10 register_cxx_method user.list_buckets flags 1 0x7f41914f64c0
   -113> 2015-03-11 01:00:14.139400 7f41a1ba4880 10 register_cxx_method user.get_header flags 1 0x7f41914f2d50
   -112> 2015-03-11 01:00:14.139403 7f41a1ba4880 10 _load_class user success
   -111> 2015-03-11 01:00:14.139406 7f41a1ba4880 10 open_all_classes found version
   -110> 2015-03-11 01:00:14.139409 7f41a1ba4880 10 _get_class adding new class name version 0x4c9e9c8
   -109> 2015-03-11 01:00:14.139412 7f41a1ba4880 10 _load_class version from /usr/lib64/rados-classes/libcls_version.so
   -108> 2015-03-11 01:00:14.139604 7f41a1ba4880  1 <cls> cls/version/cls_version.cc:227: Loaded version class!
   -107> 2015-03-11 01:00:14.139608 7f41a1ba4880 10 register_class version status 3
   -106> 2015-03-11 01:00:14.139611 7f41a1ba4880 10 register_cxx_method version.set flags 3 0x7f41912ebe60
   -105> 2015-03-11 01:00:14.139615 7f41a1ba4880 10 register_cxx_method version.inc flags 3 0x7f41912ecc80
   -104> 2015-03-11 01:00:14.139618 7f41a1ba4880 10 register_cxx_method version.inc_conds flags 3 0x7f41912ecc80
   -103> 2015-03-11 01:00:14.139621 7f41a1ba4880 10 register_cxx_method version.read flags 1 0x7f41912ebb60
   -102> 2015-03-11 01:00:14.139624 7f41a1ba4880 10 register_cxx_method version.check_conds flags 1 0x7f41912ec260
   -101> 2015-03-11 01:00:14.139627 7f41a1ba4880 10 _load_class version success
   -100> 2015-03-11 01:00:14.139682 7f41a1ba4880 20 osd.3 0 get_map 2757 - loading and decoding 0x4b4cb40
    -99> 2015-03-11 01:00:14.139693 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) read meta/a143428/osdmap.2757/0//-1 0~0
    -98> 2015-03-11 01:00:14.139804 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) FileStore::read meta/a143428/osdmap.2757/0//-1 0~3076/3076
    -97> 2015-03-11 01:00:14.139818 7f41a1ba4880 10 osd.3 0 add_map_bl 2757 3076 bytes
    -96> 2015-03-11 01:00:14.139908 7f41a1ba4880  0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients
    -95> 2015-03-11 01:00:14.139914 7f41a1ba4880  0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons
    -94> 2015-03-11 01:00:14.139919 7f41a1ba4880  0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds
    -93> 2015-03-11 01:00:14.139922 7f41a1ba4880 10 osd.3 2757 create_recoverystate_perf
    -92> 2015-03-11 01:00:14.139939 7f41a1ba4880  0 osd.3 2757 load_pgs
    -91> 2015-03-11 01:00:14.139942 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) list_collections
    -90> 2015-03-11 01:00:14.139981 7f41a1ba4880 10 osd.3 2757 load_pgs ignoring unrecognized meta
    -89> 2015-03-11 01:00:14.139985 7f41a1ba4880  0 osd.3 2757 load_pgs opened 0 pgs
    -88> 2015-03-11 01:00:14.139989 7f41a1ba4880 10 osd.3 2757 build_past_intervals_parallel nothing to build
    -87> 2015-03-11 01:00:14.139992 7f41a1ba4880  2 osd.3 2757 superblock: i am osd.3
    -86> 2015-03-11 01:00:14.139995 7f41a1ba4880 10 osd.3 2757 create_logger
    -85> 2015-03-11 01:00:14.140126 7f41a1ba4880  1 accepter.accepter.start
    -84> 2015-03-11 01:00:14.140269 7f41a1ba4880  1 accepter.accepter.start
    -83> 2015-03-11 01:00:14.140434 7f41a1ba4880  1 accepter.accepter.start
    -82> 2015-03-11 01:00:14.140575 7f41a1ba4880  1 accepter.accepter.start
    -81> 2015-03-11 01:00:14.140676 7f41a1ba4880 10 monclient(hunting): init
    -80> 2015-03-11 01:00:14.140691 7f41a1ba4880  5 adding auth protocol: none
    -79> 2015-03-11 01:00:14.140697 7f41a1ba4880 10 monclient(hunting): auth_supported 1 method none
    -78> 2015-03-11 01:00:14.140803 7f41a1ba4880 10 log_channel(default) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info)
    -77> 2015-03-11 01:00:14.140813 7f41a1ba4880 -1 osd.3 2757 log_to_monitors {default=true}
    -76> 2015-03-11 01:00:14.141325 7f41a1ba4880 10 osd.3 2757 set_disk_tp_priority class  priority -1
    -75> 2015-03-11 01:00:14.141461 7f41a1ba4880 10 monclient(hunting): renew_subs
    -74> 2015-03-11 01:00:14.141466 7f41a1ba4880 10 monclient(hunting): _reopen_session rank -1 name
    -73> 2015-03-11 01:00:14.141564 7f41a1ba4880 10 monclient(hunting): picked mon.a con 0x4cbe3c0 addr 192.168.0.10:6789/0
    -72> 2015-03-11 01:00:14.141594 7f41a1ba4880 10 monclient(hunting): _send_mon_message to mon.a at 192.168.0.10:6789/0
    -71> 2015-03-11 01:00:14.141599 7f41a1ba4880  1 -- 0.0.0.0:6800/2840 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x4b4cd80 con 0x4cbe3c0
    -70> 2015-03-11 01:00:14.141625 7f41a1ba4880 10 monclient(hunting): renew_subs
    -69> 2015-03-11 01:00:14.142020 7f4180ac7700 20 osd.3 2757 update_osd_stat osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist [])
    -68> 2015-03-11 01:00:14.142048 7f4180ac7700  5 osd.3 2757 heartbeat: osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist [])
    -67> 2015-03-11 01:00:14.142221 7f417e2c2700 10 osd.3 2757 agent_entry start
    -66> 2015-03-11 01:00:14.142241 7f417e2c2700 20 osd.3 2757 agent_entry empty queue
    -65> 2015-03-11 01:00:14.158932 7f41a1ba0700  1 -- 192.168.0.250:6800/2840 learned my addr 192.168.0.250:6800/2840
    -64> 2015-03-11 01:00:14.159058 7f41a1ba0700 10 osd.3 2757  new session (outgoing) 0x4b92280 con=0x4b602c0 addr=192.168.0.250:6800/2840
    -63> 2015-03-11 01:00:14.159166 7f41a1ba0700 10 osd.3 2757 OSD::ms_get_authorizer type=mon
    -62> 2015-03-11 01:00:14.173274 7f41912e8700 10 osd.3 2757 ms_handle_connect on mon
    -61> 2015-03-11 01:00:14.173332 7f41912e8700 20 osd.3 2757 send_pg_stats
    -60> 2015-03-11 01:00:14.173348 7f41912e8700 10 monclient(hunting): renew_subs
    -59> 2015-03-11 01:00:14.175783 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x4b4f840 con 0x4cbe3c0
    -58> 2015-03-11 01:00:14.175871 7f41912e8700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
    -57> 2015-03-11 01:00:14.175889 7f41912e8700 10 monclient(hunting):  got monmap 1, mon.a is now rank 0
    -56> 2015-03-11 01:00:14.175893 7f41912e8700 10 monclient(hunting): dump:
epoch 1
fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57
last_changed 2013-09-11 22:41:30.819687
created 2013-09-11 22:41:30.819687
0: 192.168.0.10:6789/0 mon.a

    -55> 2015-03-11 01:00:14.175940 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (335930126 0 0) 0x4b4fa80 con 0x4cbe3c0
    -54> 2015-03-11 01:00:14.175985 7f41912e8700 10 monclient(hunting): my global_id is 14511547
    -53> 2015-03-11 01:00:14.175988 7f41912e8700  1 monclient(hunting): found mon.a
    -52> 2015-03-11 01:00:14.175991 7f41912e8700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
    -51> 2015-03-11 01:00:14.175997 7f41912e8700  1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x4b31a00 con 0x4cbe3c0
    -50> 2015-03-11 01:00:14.176035 7f41912e8700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
    -49> 2015-03-11 01:00:14.176059 7f41912e8700  1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0,osdmap=2757}) v2 -- ?+0 0x4b32800 con 0x4cbe3c0
    -48> 2015-03-11 01:00:14.176077 7f41912e8700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2015-03-11 00:59:44.176076)
    -47> 2015-03-11 01:00:14.176109 7f41a1ba4880  5 monclient: authenticate success, global_id 14511547
    -46> 2015-03-11 01:00:14.176147 7f41a1ba4880 10 osd.3 2757 ensuring pgs have consumed prior maps
    -45> 2015-03-11 01:00:14.176170 7f41a1ba4880  7 osd.3 2757 consume_map version 2757
    -44> 2015-03-11 01:00:14.176187 7f41a1ba4880  0 osd.3 2757 done with init, starting boot process
    -43> 2015-03-11 01:00:14.176191 7f41a1ba4880 10 osd.3 2757 start_boot - have maps 2757..2757
    -42> 2015-03-11 01:00:14.176197 7f41a1ba4880 10 monclient: get_version osdmap req 0x4bad1c0
    -41> 2015-03-11 01:00:14.176207 7f41a1ba4880 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
    -40> 2015-03-11 01:00:14.176221 7f41a1ba4880  1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x4bc41e0 con 0x4cbe3c0
    -39> 2015-03-11 01:00:14.176628 7f41a1ba4880  5 asok(0x4b50000) register_command status hook 0x4b04290
    -38> 2015-03-11 01:00:14.176655 7f41a1ba4880  5 asok(0x4b50000) register_command flush_journal hook 0x4b04290
    -37> 2015-03-11 01:00:14.176666 7f41a1ba4880  5 asok(0x4b50000) register_command dump_ops_in_flight hook 0x4b04290
    -36> 2015-03-11 01:00:14.176686 7f41a1ba4880  5 asok(0x4b50000) register_command ops hook 0x4b04290
    -35> 2015-03-11 01:00:14.176705 7f41a1ba4880  5 asok(0x4b50000) register_command dump_historic_ops hook 0x4b04290
    -34> 2015-03-11 01:00:14.176723 7f41a1ba4880  5 asok(0x4b50000) register_command dump_op_pq_state hook 0x4b04290
    -33> 2015-03-11 01:00:14.176737 7f41a1ba4880  5 asok(0x4b50000) register_command dump_blacklist hook 0x4b04290
    -32> 2015-03-11 01:00:14.176746 7f41a1ba4880  5 asok(0x4b50000) register_command dump_watchers hook 0x4b04290
    -31> 2015-03-11 01:00:14.176760 7f41a1ba4880  5 asok(0x4b50000) register_command dump_reservations hook 0x4b04290
    -30> 2015-03-11 01:00:14.176770 7f41a1ba4880  5 asok(0x4b50000) register_command get_latest_osdmap hook 0x4b04290
    -29> 2015-03-11 01:00:14.176786 7f41a1ba4880  5 asok(0x4b50000) register_command setomapval hook 0x4bad260
    -28> 2015-03-11 01:00:14.176800 7f41a1ba4880  5 asok(0x4b50000) register_command rmomapkey hook 0x4bad260
    -27> 2015-03-11 01:00:14.176817 7f41a1ba4880  5 asok(0x4b50000) register_command setomapheader hook 0x4bad260
    -26> 2015-03-11 01:00:14.176834 7f41a1ba4880  5 asok(0x4b50000) register_command getomap hook 0x4bad260
    -25> 2015-03-11 01:00:14.176850 7f41a1ba4880  5 asok(0x4b50000) register_command truncobj hook 0x4bad260
    -24> 2015-03-11 01:00:14.176866 7f41a1ba4880  5 asok(0x4b50000) register_command injectdataerr hook 0x4bad260
    -23> 2015-03-11 01:00:14.176878 7f41a1ba4880  5 asok(0x4b50000) register_command injectmdataerr hook 0x4bad260
    -22> 2015-03-11 01:00:14.178539 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 3 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x4b4fcc0 con 0x4cbe3c0
    -21> 2015-03-11 01:00:14.178650 7f41912e8700 10 osd.3 2757 do_waiters -- start
    -20> 2015-03-11 01:00:14.178658 7f41912e8700 10 osd.3 2757 do_waiters -- finish
    -19> 2015-03-11 01:00:14.178661 7f41912e8700 20 osd.3 2757 _dispatch 0x4b4fcc0 osd_map(3277..3277 src has 2757..3277) v3
    -18> 2015-03-11 01:00:14.178888 7f41912e8700  3 osd.3 2757 handle_osd_map epochs [3277,3277], i have 2757, src has [2757,3277]
    -17> 2015-03-11 01:00:14.178906 7f41912e8700 10 osd.3 2757 handle_osd_map message skips epochs 2758..3276
    -16> 2015-03-11 01:00:14.178914 7f41912e8700 10 monclient: renew_subs
    -15> 2015-03-11 01:00:14.178922 7f41912e8700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
    -14> 2015-03-11 01:00:14.178954 7f41912e8700  1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=2758}) v2 -- ?+0 0x4b31a00 con 0x4cbe3c0
    -13> 2015-03-11 01:00:14.178991 7f41912e8700 10 osd.3 2757 do_waiters -- start
    -12> 2015-03-11 01:00:14.178996 7f41912e8700 10 osd.3 2757 do_waiters -- finish
    -11> 2015-03-11 01:00:14.179012 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 4 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x4b4f840 con 0x4cbe3c0
    -10> 2015-03-11 01:00:14.179030 7f41912e8700 10 monclient: handle_monmap mon_map magic: 0 v1
     -9> 2015-03-11 01:00:14.179045 7f41912e8700 10 monclient:  got monmap 1, mon.a is now rank 0
     -8> 2015-03-11 01:00:14.179051 7f41912e8700 10 monclient: dump:
epoch 1
fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57
last_changed 2013-09-11 22:41:30.819687
created 2013-09-11 22:41:30.819687
0: 192.168.0.10:6789/0 mon.a

     -7> 2015-03-11 01:00:14.184757 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(2757..2857 src has 2757..3277) v3 ==== 37648+0+0 (2648067016 0 0) 0x4b4fa80 con 0x4cbe3c0
     -6> 2015-03-11 01:00:14.184831 7f41912e8700 10 osd.3 2757 do_waiters -- start
     -5> 2015-03-11 01:00:14.184838 7f41912e8700 10 osd.3 2757 do_waiters -- finish
     -4> 2015-03-11 01:00:14.184841 7f41912e8700 20 osd.3 2757 _dispatch 0x4b4fa80 osd_map(2757..2857 src has 2757..3277) v3
     -3> 2015-03-11 01:00:14.184872 7f41912e8700  3 osd.3 2757 handle_osd_map epochs [2757,2857], i have 2757, src has [2757,3277]
     -2> 2015-03-11 01:00:14.184884 7f41912e8700 10 osd.3 2757 handle_osd_map  got inc map for epoch 2758
     -1> 2015-03-11 01:00:14.185229 7f41912e8700 10 osd.3 2757 handle_osd_map  got inc map for epoch 2759
      0> 2015-03-11 01:00:14.195165 7f41912e8700 -1 *** Caught signal (Aborted) **
  in thread 7f41912e8700

  ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
  1: /usr/bin/ceph-osd() [0xac7cea]
  2: (()+0x10050) [0x7f41a0aaf050]
  3: (gsignal()+0x37) [0x7f419ef820f7]
  4: (abort()+0x13a) [0x7f419ef834ca]
  5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f419f867fe5]
  6: (()+0x63186) [0x7f419f866186]
  7: (()+0x631b3) [0x7f419f8661b3]
  8: (()+0x633d2) [0x7f419f8663d2]
  9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
  10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
  11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
  12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
  13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
  14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
  15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
  16: (DispatchQueue::entry()+0x649) [0xc6fe09]
  17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
  18: (()+0x83a4) [0x7f41a0aa73a4]
  19: (clone()+0x6d) [0x7f419f031a4d]
  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
    0/ 5 none
    0/ 1 lockdep
    0/ 1 context
    1/ 1 crush
    1/ 5 mds
    1/ 5 mds_balancer
    1/ 5 mds_locker
    1/ 5 mds_log
    1/ 5 mds_log_expire
    1/ 5 mds_migrator
    0/ 1 buffer
    0/ 1 timer
    0/ 1 filer
    0/ 1 striper
    0/ 1 objecter
    0/ 5 rados
    0/ 5 rbd
    0/ 5 rbd_replay
    0/ 5 journaler
    0/ 5 objectcacher
    0/ 5 client
   20/20 osd
    0/ 5 optracker
    0/ 5 objclass
   20/20 filestore
    1/ 3 keyvaluestore
    1/ 3 journal
    1/ 1 ms
    1/ 5 mon
    0/10 monc
    1/ 5 paxos
    0/ 5 tp
    1/ 5 auth
    1/ 5 crypto
    1/ 1 finisher
    1/ 5 heartbeatmap
    1/ 5 perfcounter
    1/ 5 rgw
    1/10 civetweb
    1/ 5 javaclient
    1/ 5 asok
    1/ 1 throttle
    0/ 0 refs
    1/ 5 xio
   -2/-2 (syslog threshold)
   -1/-1 (stderr threshold)
   max_recent     10000
   max_new         1000
   log_file /var/log/ceph/ceph-osd.3.log
--- end dump of recent events ---

-----Original Message-----
From: Samuel Just [mailto:sjust@xxxxxxxxxx]
Sent: Wednesday, 11 March 2015 12:59 AM
To: Malcolm Haak
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re:  Issues with fresh 0.93 OSD adding to existing cluster

Can you reproduce this with

debug osd = 20
debug filestore = 20
debug ms = 1

on the crashing osd?  Also, what sha1 are the other osds and mons running?
-Sam

----- Original Message -----
From: "Malcolm Haak" <malcolm@xxxxxxx>
To: ceph-users@xxxxxxxxxxxxxx
Sent: Tuesday, March 10, 2015 3:28:26 AM
Subject:  Issues with fresh 0.93 OSD adding to existing cluster

Hi all,

I've just attempted to add a new node and OSD to an existing ceph cluster (it's a small one I use as a NAS at home, not like the big production ones I normally work on) and it seems to be throwing some odd errors...

Just looking for where to poke it next...

Log is below,

It's a two node cluster with 3 osd's in node A and one osd in the new node (It's going to have more eventually and node one will be retired after node three gets added) And I've hit a weird snag.

I was running 0.80 but I ran into the 'Invalid Command' bug on the new node so I opted to jump to the latest code with the required patches already.

Please let me know what else you need..

This is the log content when attempting to start the new OSD:

2015-03-10 19:28:48.795318 7f0774108880  0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 10810
2015-03-10 19:28:48.817803 7f0774108880  0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342)
2015-03-10 19:28:48.866862 7f0774108880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work
2015-03-10 19:28:48.866920 7f0774108880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2015-03-10 19:28:48.905069 7f0774108880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2015-03-10 19:28:48.905467 7f0774108880  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5
2015-03-10 19:28:49.077872 7f0774108880  0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2015-03-10 19:28:49.078321 7f0774108880 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2015-03-10 19:28:49.078328 7f0774108880  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
2015-03-10 19:28:49.079721 7f0774108880  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
2015-03-10 19:28:49.080948 7f0774108880  0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
2015-03-10 19:28:49.094194 7f0774108880  0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients
2015-03-10 19:28:49.094211 7f0774108880  0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons
2015-03-10 19:28:49.094217 7f0774108880  0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds
2015-03-10 19:28:49.094235 7f0774108880  0 osd.3 2757 load_pgs
2015-03-10 19:28:49.094279 7f0774108880  0 osd.3 2757 load_pgs opened 0 pgs
2015-03-10 19:28:49.095121 7f0774108880 -1 osd.3 2757 log_to_monitors {default=true}
2015-03-10 19:28:49.134104 7f0774108880  0 osd.3 2757 done with init, starting boot process
2015-03-10 19:28:49.149994 7f076384c700 -1 *** Caught signal (Aborted) **  in thread 7f076384c700

  ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
  1: /usr/bin/ceph-osd() [0xac7cea]
  2: (()+0x10050) [0x7f0773013050]
  3: (gsignal()+0x37) [0x7f07714e60f7]
  4: (abort()+0x13a) [0x7f07714e74ca]
  5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f0771dcbfe5]
  6: (()+0x63186) [0x7f0771dca186]
  7: (()+0x631b3) [0x7f0771dca1b3]
  8: (()+0x633d2) [0x7f0771dca3d2]
  9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
  10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
  11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
  12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
  13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
  14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
  15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
  16: (DispatchQueue::entry()+0x649) [0xc6fe09]
  17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
  18: (()+0x83a4) [0x7f077300b3a4]
  19: (clone()+0x6d) [0x7f0771595a4d]
  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -135> 2015-03-10 19:28:48.790490 7f0774108880  5 asok(0x4200000) register_command perfcounters_dump hook 0x41b4030
   -134> 2015-03-10 19:28:48.790565 7f0774108880  5 asok(0x4200000) register_command 1 hook 0x41b4030
   -133> 2015-03-10 19:28:48.790571 7f0774108880  5 asok(0x4200000) register_command perf dump hook 0x41b4030
   -132> 2015-03-10 19:28:48.790583 7f0774108880  5 asok(0x4200000) register_command perfcounters_schema hook 0x41b4030
   -131> 2015-03-10 19:28:48.790599 7f0774108880  5 asok(0x4200000) register_command 2 hook 0x41b4030
   -130> 2015-03-10 19:28:48.790605 7f0774108880  5 asok(0x4200000) register_command perf schema hook 0x41b4030
   -129> 2015-03-10 19:28:48.790635 7f0774108880  5 asok(0x4200000) register_command perf reset hook 0x41b4030
   -128> 2015-03-10 19:28:48.790646 7f0774108880  5 asok(0x4200000) register_command config show hook 0x41b4030
   -127> 2015-03-10 19:28:48.790652 7f0774108880  5 asok(0x4200000) register_command config set hook 0x41b4030
   -126> 2015-03-10 19:28:48.790660 7f0774108880  5 asok(0x4200000) register_command config get hook 0x41b4030
   -125> 2015-03-10 19:28:48.790672 7f0774108880  5 asok(0x4200000) register_command config diff hook 0x41b4030
   -124> 2015-03-10 19:28:48.790682 7f0774108880  5 asok(0x4200000) register_command log flush hook 0x41b4030
   -123> 2015-03-10 19:28:48.790689 7f0774108880  5 asok(0x4200000) register_command log dump hook 0x41b4030
   -122> 2015-03-10 19:28:48.790698 7f0774108880  5 asok(0x4200000) register_command log reopen hook 0x41b4030
   -121> 2015-03-10 19:28:48.795318 7f0774108880  0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 10810
   -120> 2015-03-10 19:28:48.796071 7f0774108880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/10810 need_addr=1
   -119> 2015-03-10 19:28:48.796123 7f0774108880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/10810 need_addr=1
   -118> 2015-03-10 19:28:48.796144 7f0774108880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/10810 need_addr=1
   -117> 2015-03-10 19:28:48.796160 7f0774108880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/10810 need_addr=1
   -116> 2015-03-10 19:28:48.799414 7f0774108880  5 asok(0x4200000) init /var/run/ceph/ceph-osd.3.asok
   -115> 2015-03-10 19:28:48.799446 7f0774108880  5 asok(0x4200000) bind_and_listen /var/run/ceph/ceph-osd.3.asok
   -114> 2015-03-10 19:28:48.799644 7f0774108880  5 asok(0x4200000) register_command 0 hook 0x41b00c0
   -113> 2015-03-10 19:28:48.799659 7f0774108880  5 asok(0x4200000) register_command version hook 0x41b00c0
   -112> 2015-03-10 19:28:48.799665 7f0774108880  5 asok(0x4200000) register_command git_version hook 0x41b00c0
   -111> 2015-03-10 19:28:48.799671 7f0774108880  5 asok(0x4200000) register_command help hook 0x41b4110
   -110> 2015-03-10 19:28:48.799678 7f0774108880  5 asok(0x4200000) register_command get_command_descriptions hook 0x41b4120
   -109> 2015-03-10 19:28:48.799760 7f0774108880 10 monclient(hunting): build_initial_monmap
   -108> 2015-03-10 19:28:48.799757 7f076f4e1700  5 asok(0x4200000) entry start
   -107> 2015-03-10 19:28:48.816916 7f0774108880  5 adding auth protocol: none
   -106> 2015-03-10 19:28:48.816948 7f0774108880  5 adding auth protocol: none
   -105> 2015-03-10 19:28:48.817130 7f0774108880  5 asok(0x4200000) register_command objecter_requests hook 0x41b41b0
   -104> 2015-03-10 19:28:48.817273 7f0774108880  1 -- 0.0.0.0:6800/10810 messenger.start
   -103> 2015-03-10 19:28:48.817334 7f0774108880  1 -- :/0 messenger.start
   -102> 2015-03-10 19:28:48.817378 7f0774108880  1 -- 0.0.0.0:6803/10810 messenger.start
   -101> 2015-03-10 19:28:48.817442 7f0774108880  1 -- 0.0.0.0:6802/10810 messenger.start
   -100> 2015-03-10 19:28:48.817494 7f0774108880  1 -- 0.0.0.0:6801/10810 messenger.start
    -99> 2015-03-10 19:28:48.817546 7f0774108880  1 -- :/0 messenger.start
    -98> 2015-03-10 19:28:48.817666 7f0774108880  2 osd.3 0 mounting /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
    -97> 2015-03-10 19:28:48.817803 7f0774108880  0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342)
    -96> 2015-03-10 19:28:48.866862 7f0774108880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work
    -95> 2015-03-10 19:28:48.866920 7f0774108880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
    -94> 2015-03-10 19:28:48.905069 7f0774108880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
    -93> 2015-03-10 19:28:48.905467 7f0774108880  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5
    -92> 2015-03-10 19:28:49.077872 7f0774108880  0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
    -91> 2015-03-10 19:28:49.078270 7f0774108880  2 journal open /var/lib/ceph/osd/ceph-3/journal fsid 05fb481f-68b3-4442-b7de-af4c867e18d5 fs_op_seq 2
    -90> 2015-03-10 19:28:49.078321 7f0774108880 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
    -89> 2015-03-10 19:28:49.078328 7f0774108880  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
    -88> 2015-03-10 19:28:49.078937 7f0774108880  2 journal read_entry 8192 : seq 3 193 bytes
    -87> 2015-03-10 19:28:49.079000 7f0774108880  2 journal read_entry 8192 : seq 3 193 bytes
    -86> 2015-03-10 19:28:49.079022 7f0774108880  3 journal journal_replay: applying op seq 3
    -85> 2015-03-10 19:28:49.079307 7f0774108880  3 journal journal_replay: r = 0, op_seq now 3
    -84> 2015-03-10 19:28:49.079367 7f0774108880  2 journal read_entry 12288 : seq 4 3829 bytes
    -83> 2015-03-10 19:28:49.079390 7f0774108880  3 journal journal_replay: applying op seq 4
    -82> 2015-03-10 19:28:49.079667 7f0774108880  3 journal journal_replay: r = 0, op_seq now 4
    -81> 2015-03-10 19:28:49.079699 7f0774108880  2 journal No further valid entries found, journal is most likely valid
    -80> 2015-03-10 19:28:49.079703 7f0774108880  3 journal journal_replay: end of journal, done.
    -79> 2015-03-10 19:28:49.079721 7f0774108880  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
    -78> 2015-03-10 19:28:49.080265 7f0774108880  2 osd.3 0 boot
    -77> 2015-03-10 19:28:49.080948 7f0774108880  0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
    -76> 2015-03-10 19:28:49.081757 7f0774108880  1 <cls> cls/log/cls_log.cc:312: Loaded log class!
    -75> 2015-03-10 19:28:49.087397 7f0774108880  1 <cls> cls/refcount/cls_refcount.cc:231: Loaded refcount class!
    -74> 2015-03-10 19:28:49.087735 7f0774108880  1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class!
    -73> 2015-03-10 19:28:49.093126 7f0774108880  1 <cls> cls/rgw/cls_rgw.cc:3046: Loaded rgw class!
    -72> 2015-03-10 19:28:49.093484 7f0774108880  1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class!
    -71> 2015-03-10 19:28:49.093723 7f0774108880  1 <cls> cls/user/cls_user.cc:367: Loaded user class!
    -70> 2015-03-10 19:28:49.093924 7f0774108880  1 <cls> cls/version/cls_version.cc:227: Loaded version class!
    -69> 2015-03-10 19:28:49.094194 7f0774108880  0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients
    -68> 2015-03-10 19:28:49.094211 7f0774108880  0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons
    -67> 2015-03-10 19:28:49.094217 7f0774108880  0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds
    -66> 2015-03-10 19:28:49.094235 7f0774108880  0 osd.3 2757 load_pgs
    -65> 2015-03-10 19:28:49.094279 7f0774108880  0 osd.3 2757 load_pgs opened 0 pgs
    -64> 2015-03-10 19:28:49.094285 7f0774108880  2 osd.3 2757 superblock: i am osd.3
    -63> 2015-03-10 19:28:49.094424 7f0774108880  1 accepter.accepter.start
    -62> 2015-03-10 19:28:49.094573 7f0774108880  1 accepter.accepter.start
    -61> 2015-03-10 19:28:49.094748 7f0774108880  1 accepter.accepter.start
    -60> 2015-03-10 19:28:49.094864 7f0774108880  1 accepter.accepter.start
    -59> 2015-03-10 19:28:49.094963 7f0774108880 10 monclient(hunting): init
    -58> 2015-03-10 19:28:49.094980 7f0774108880  5 adding auth protocol: none
    -57> 2015-03-10 19:28:49.095005 7f0774108880 10 monclient(hunting): auth_supported 1 method none
    -56> 2015-03-10 19:28:49.095110 7f0774108880 10 log_channel(default) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info)
    -55> 2015-03-10 19:28:49.095121 7f0774108880 -1 osd.3 2757 log_to_monitors {default=true}
    -54> 2015-03-10 19:28:49.095899 7f0774108880 10 monclient(hunting): renew_subs
    -53> 2015-03-10 19:28:49.095914 7f0774108880 10 monclient(hunting): _reopen_session rank -1 name
    -52> 2015-03-10 19:28:49.096003 7f0774108880 10 monclient(hunting): picked mon.a con 0x43623c0 addr 192.168.0.10:6789/0
    -51> 2015-03-10 19:28:49.096086 7f0774108880 10 monclient(hunting): _send_mon_message to mon.a at 192.168.0.10:6789/0
    -50> 2015-03-10 19:28:49.096098 7f0774108880  1 -- 0.0.0.0:6800/10810 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x41fcd80 con 0x43623c0
    -49> 2015-03-10 19:28:49.096132 7f0774108880 10 monclient(hunting): renew_subs
    -48> 2015-03-10 19:28:49.096501 7f075302b700  5 osd.3 2757 heartbeat: osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist [])
    -47> 2015-03-10 19:28:49.113808 7f0774104700  1 -- 192.168.0.250:6800/10810 learned my addr 192.168.0.250:6800/10810
    -46> 2015-03-10 19:28:49.131334 7f076384c700 10 monclient(hunting): renew_subs
    -45> 2015-03-10 19:28:49.133513 7f076384c700  1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x41ff840 con 0x43623c0
    -44> 2015-03-10 19:28:49.133572 7f076384c700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
    -43> 2015-03-10 19:28:49.133587 7f076384c700 10 monclient(hunting):  got monmap 1, mon.a is now rank 0
    -42> 2015-03-10 19:28:49.133597 7f076384c700 10 monclient(hunting): dump:
epoch 1
fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57
last_changed 2013-09-11 22:41:30.819687
created 2013-09-11 22:41:30.819687
0: 192.168.0.10:6789/0 mon.a

    -41> 2015-03-10 19:28:49.133678 7f076384c700  1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (1541693634 0 0) 0x41ffa80 con 0x43623c0
    -40> 2015-03-10 19:28:49.133716 7f076384c700 10 monclient(hunting): my global_id is 14511543
    -39> 2015-03-10 19:28:49.133721 7f076384c700  1 monclient(hunting): found mon.a
    -38> 2015-03-10 19:28:49.133803 7f076384c700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
    -37> 2015-03-10 19:28:49.133820 7f076384c700  1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x41e1a00 con 0x43623c0
    -36> 2015-03-10 19:28:49.133906 7f076384c700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
    -35> 2015-03-10 19:28:49.133925 7f076384c700  1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0,osdmap=2757}) v2 -- ?+0 0x41e2800 con 0x43623c0
    -34> 2015-03-10 19:28:49.134016 7f076384c700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2015-03-10 19:28:19.134014)
    -33> 2015-03-10 19:28:49.134061 7f0774108880  5 monclient: authenticate success, global_id 14511543
    -32> 2015-03-10 19:28:49.134104 7f0774108880  0 osd.3 2757 done with init, starting boot process
    -31> 2015-03-10 19:28:49.134118 7f0774108880 10 monclient: get_version osdmap req 0x4259160
    -30> 2015-03-10 19:28:49.134126 7f0774108880 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
    -29> 2015-03-10 19:28:49.134136 7f0774108880  1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x42741e0 con 0x43623c0
    -28> 2015-03-10 19:28:49.134362 7f0774108880  5 asok(0x4200000) register_command status hook 0x41b4290
    -27> 2015-03-10 19:28:49.134388 7f0774108880  5 asok(0x4200000) register_command flush_journal hook 0x41b4290
    -26> 2015-03-10 19:28:49.134405 7f0774108880  5 asok(0x4200000) register_command dump_ops_in_flight hook 0x41b4290
    -25> 2015-03-10 19:28:49.134421 7f0774108880  5 asok(0x4200000) register_command ops hook 0x41b4290
    -24> 2015-03-10 19:28:49.134437 7f0774108880  5 asok(0x4200000) register_command dump_historic_ops hook 0x41b4290
    -23> 2015-03-10 19:28:49.134454 7f0774108880  5 asok(0x4200000) register_command dump_op_pq_state hook 0x41b4290
    -22> 2015-03-10 19:28:49.134469 7f0774108880  5 asok(0x4200000) register_command dump_blacklist hook 0x41b4290
    -21> 2015-03-10 19:28:49.134484 7f0774108880  5 asok(0x4200000) register_command dump_watchers hook 0x41b4290
    -20> 2015-03-10 19:28:49.134500 7f0774108880  5 asok(0x4200000) register_command dump_reservations hook 0x41b4290
    -19> 2015-03-10 19:28:49.134516 7f0774108880  5 asok(0x4200000) register_command get_latest_osdmap hook 0x41b4290
    -18> 2015-03-10 19:28:49.134532 7f0774108880  5 asok(0x4200000) register_command setomapval hook 0x4259200
    -17> 2015-03-10 19:28:49.134546 7f0774108880  5 asok(0x4200000) register_command rmomapkey hook 0x4259200
    -16> 2015-03-10 19:28:49.134563 7f0774108880  5 asok(0x4200000) register_command setomapheader hook 0x4259200
    -15> 2015-03-10 19:28:49.134580 7f0774108880  5 asok(0x4200000) register_command getomap hook 0x4259200
    -14> 2015-03-10 19:28:49.134595 7f0774108880  5 asok(0x4200000) register_command truncobj hook 0x4259200
    -13> 2015-03-10 19:28:49.134609 7f0774108880  5 asok(0x4200000) register_command injectdataerr hook 0x4259200
    -12> 2015-03-10 19:28:49.134624 7f0774108880  5 asok(0x4200000) register_command injectmdataerr hook 0x4259200
    -11> 2015-03-10 19:28:49.136325 7f076384c700  1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 3 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x41ffcc0 con 0x43623c0
    -10> 2015-03-10 19:28:49.136622 7f076384c700  3 osd.3 2757 handle_osd_map epochs [3277,3277], i have 2757, src has [2757,3277]
     -9> 2015-03-10 19:28:49.136643 7f076384c700 10 monclient: renew_subs
     -8> 2015-03-10 19:28:49.136652 7f076384c700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
     -7> 2015-03-10 19:28:49.136679 7f076384c700  1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=2758}) v2 -- ?+0 0x41e1a00 con 0x43623c0
     -6> 2015-03-10 19:28:49.136943 7f076384c700  1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 4 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x41ff840 con 0x43623c0
     -5> 2015-03-10 19:28:49.136994 7f076384c700 10 monclient: handle_monmap mon_map magic: 0 v1
     -4> 2015-03-10 19:28:49.137010 7f076384c700 10 monclient:  got monmap 1, mon.a is now rank 0
     -3> 2015-03-10 19:28:49.137022 7f076384c700 10 monclient: dump:
epoch 1
fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57
last_changed 2013-09-11 22:41:30.819687
created 2013-09-11 22:41:30.819687
0: 192.168.0.10:6789/0 mon.a

     -2> 2015-03-10 19:28:49.142203 7f076384c700  1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(2757..2857 src has 2757..3277) v3 ==== 37648+0+0 (2648067016 0 0) 0x41ffa80 con 0x43623c0
     -1> 2015-03-10 19:28:49.142316 7f076384c700  3 osd.3 2757 handle_osd_map epochs [2757,2857], i have 2757, src has [2757,3277]
      0> 2015-03-10 19:28:49.149994 7f076384c700 -1 *** Caught signal (Aborted) **  in thread 7f076384c700

  ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
  1: /usr/bin/ceph-osd() [0xac7cea]
  2: (()+0x10050) [0x7f0773013050]
  3: (gsignal()+0x37) [0x7f07714e60f7]
  4: (abort()+0x13a) [0x7f07714e74ca]
  5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f0771dcbfe5]
  6: (()+0x63186) [0x7f0771dca186]
  7: (()+0x631b3) [0x7f0771dca1b3]
  8: (()+0x633d2) [0x7f0771dca3d2]
  9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
  10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
  11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
  12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
  13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
  14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
  15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
  16: (DispatchQueue::entry()+0x649) [0xc6fe09]
  17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
  18: (()+0x83a4) [0x7f077300b3a4]
  19: (clone()+0x6d) [0x7f0771595a4d]
  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
    0/ 5 none
    0/ 1 lockdep
    0/ 1 context
    1/ 1 crush
    1/ 5 mds
    1/ 5 mds_balancer
    1/ 5 mds_locker
    1/ 5 mds_log
    1/ 5 mds_log_expire
    1/ 5 mds_migrator
    0/ 1 buffer
    0/ 1 timer
    0/ 1 filer
    0/ 1 striper
    0/ 1 objecter
    0/ 5 rados
    0/ 5 rbd
    0/ 5 rbd_replay
    0/ 5 journaler
    0/ 5 objectcacher
    0/ 5 client
    0/ 5 osd
    0/ 5 optracker
    0/ 5 objclass
    1/ 3 filestore
    1/ 3 keyvaluestore
    1/ 3 journal
    0/ 5 ms
    1/ 5 mon
    0/10 monc
    1/ 5 paxos
    0/ 5 tp
    1/ 5 auth
    1/ 5 crypto
    1/ 1 finisher
    1/ 5 heartbeatmap
    1/ 5 perfcounter
    1/ 5 rgw
    1/10 civetweb
    1/ 5 javaclient
    1/ 5 asok
    1/ 1 throttle
    0/ 0 refs
    1/ 5 xio
   -2/-2 (syslog threshold)
   -1/-1 (stderr threshold)
   max_recent     10000
   max_new         1000
   log_file /var/log/ceph/ceph-osd.3.log
--- end dump of recent events ---
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com