Re: Issues with fresh 0.93 OSD adding to existing cluster

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Joao, it looks like map 2759 is causing trouble, how would he get the
full and incremental maps for that out of the mons?
-Sam

On Tue, 2015-03-10 at 14:12 +0000, Malcolm Haak wrote:
> Hi Samuel,
> 
> The sha1? I'm going to admit ignorance as to what you are looking for. They are all running the same release if that is what you are asking. 
> Same tarball built into rpms using rpmbuild on both nodes... 
> Only difference being that the other node has been upgraded and the problem node is fresh.
> 
> added the requested config here is the command line output
> 
> microserver-1:/etc # /etc/init.d/ceph start osd.3
> === osd.3 === 
> Mounting xfs on microserver-1:/var/lib/ceph/osd/ceph-3
> 2015-03-11 01:00:13.492279 7f05b2f72700  1 -- :/0 messenger.start
> 2015-03-11 01:00:13.492823 7f05b2f72700  1 -- :/1002795 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x7f05ac0290b0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.510814 7f05b07ef700  1 -- 192.168.0.250:0/1002795 learned my addr 192.168.0.250:0/1002795
> 2015-03-11 01:00:13.527653 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x7f05a0000ab0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.527899 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (3859410672 0 0) 0x7f05a0000e70 con 0x7f05ac027c40
> 2015-03-11 01:00:13.527973 7f05abfff700  1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x7f05ac029730 con 0x7f05ac027c40
> 2015-03-11 01:00:13.528124 7f05b2f72700  1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x7f05ac029a50 con 0x7f05ac027c40
> 2015-03-11 01:00:13.528265 7f05b2f72700  1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x7f05ac029f20 con 0x7f05ac027c40
> 2015-03-11 01:00:13.530359 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 3 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x7f05a0000ab0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.530548 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 4 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3648139960 0 0) 0x7f05a0000fb0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.531114 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x7f05a0002800 con 0x7f05ac027c40
> 2015-03-11 01:00:13.531772 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3648139960 0 0) 0x7f05a0000fb0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.532186 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 7 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x7f05a0001250 con 0x7f05ac027c40
> 2015-03-11 01:00:13.532260 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3648139960 0 0) 0x7f05a0000fb0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.556748 7f05b2f72700  1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- ?+0 0x7f05ac016ac0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.564968 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0  v0) v1 ==== 72+0+34995 (1092875540 0 1727986498) 0x7f05a0000a70 con 0x7f05ac027c40
> 2015-03-11 01:00:13.770122 7f05b2f72700  1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_command({"prefix": "osd crush create-or-move", "args": ["host=microserver-1", "root=default"], "id": 3, "weight": 1.81} v 0) v1 -- ?+0 0x7f05ac016ac0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.772299 7f05abfff700  1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "args": ["host=microserver-1", "root=default"], "id": 3, "weight": 1.81}]=0 create-or-move updated item name 'osd.3' weight 1.81 at location {host=microserver-1,root=default} to crush map v3277) v1 ==== 256+0+0 (1191546821 0 0) 0x7f05a0001000 con 0x7f05ac027c40
> create-or-move updated item name 'osd.3' weight 1.81 at location {host=microserver-1,root=default} to crush map
> 2015-03-11 01:00:13.776891 7f05b2f72700  1 -- 192.168.0.250:0/1002795 mark_down 0x7f05ac027c40 -- 0x7f05ac0239a0
> 2015-03-11 01:00:13.777212 7f05b2f72700  1 -- 192.168.0.250:0/1002795 mark_down_all
> 2015-03-11 01:00:13.778120 7f05b2f72700  1 -- 192.168.0.250:0/1002795 shutdown complete.
> Starting Ceph osd.3 on microserver-1...
> microserver-1:/etc #
> 
> 
> Log file
> 
> 
> 2015-03-11 01:00:13.876152 7f41a1ba4880  0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 2840
> 2015-03-11 01:00:13.877059 7f41a1ba4880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/2840 need_addr=1
> 2015-03-11 01:00:13.877111 7f41a1ba4880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/2840 need_addr=1
> 2015-03-11 01:00:13.877140 7f41a1ba4880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/2840 need_addr=1
> 2015-03-11 01:00:13.877169 7f41a1ba4880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/2840 need_addr=1
> 2015-03-11 01:00:13.880323 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) dump_stop
> 2015-03-11 01:00:13.890355 7f41a1ba4880 10 ErasureCodePluginSelectJerasure: load: jerasure_generic 
> 2015-03-11 01:00:13.897843 7f41a1ba4880 10 load: jerasure load: lrc load: isa 
> 2015-03-11 01:00:13.898172 7f41a1ba4880  5 filestore(/var/lib/ceph/osd/ceph-3) test_mount basedir /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal
> 2015-03-11 01:00:13.898290 7f41a1ba4880  1 -- 0.0.0.0:6800/2840 messenger.start
> 2015-03-11 01:00:13.898362 7f41a1ba4880  1 -- :/0 messenger.start
> 2015-03-11 01:00:13.898450 7f41a1ba4880  1 -- 0.0.0.0:6803/2840 messenger.start
> 2015-03-11 01:00:13.898527 7f41a1ba4880  1 -- 0.0.0.0:6802/2840 messenger.start
> 2015-03-11 01:00:13.898609 7f41a1ba4880  1 -- 0.0.0.0:6801/2840 messenger.start
> 2015-03-11 01:00:13.898672 7f41a1ba4880  1 -- :/0 messenger.start
> 2015-03-11 01:00:13.898792 7f41a1ba4880  2 osd.3 0 mounting /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
> 2015-03-11 01:00:13.898826 7f41a1ba4880  5 filestore(/var/lib/ceph/osd/ceph-3) basedir /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal
> 2015-03-11 01:00:13.898868 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) mount fsid is 05fb481f-68b3-4442-b7de-af4c867e18d5
> 2015-03-11 01:00:13.898937 7f41a1ba4880  0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342)
> 2015-03-11 01:00:13.941605 7f41a1ba4880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work
> 2015-03-11 01:00:13.941665 7f41a1ba4880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
> 2015-03-11 01:00:13.970056 7f41a1ba4880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
> 2015-03-11 01:00:13.970442 7f41a1ba4880  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5
> 2015-03-11 01:00:14.001199 7f41a1ba4880  5 filestore(/var/lib/ceph/osd/ceph-3) mount op_seq is 2
> 2015-03-11 01:00:14.122464 7f41a1ba4880 20 filestore (init)dbobjectmap: seq is 1
> 2015-03-11 01:00:14.122580 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) open_journal at /var/lib/ceph/osd/ceph-3/journal
> 2015-03-11 01:00:14.122674 7f41a1ba4880  0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
> 2015-03-11 01:00:14.122682 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) list_collections
> 2015-03-11 01:00:14.123024 7f41a1ba4880 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
> 2015-03-11 01:00:14.123040 7f41a1ba4880  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> 2015-03-11 01:00:14.123003 7f4196c1b700 20 filestore(/var/lib/ceph/osd/ceph-3) sync_entry waiting for max_interval 5.000000
> 2015-03-11 01:00:14.123724 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x4b9b6c0
> 2015-03-11 01:00:14.123837 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
> 2015-03-11 01:00:14.123971 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
> 2015-03-11 01:00:14.124020 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) touch meta/a468ec03/snapmapper/0//-1
> 2015-03-11 01:00:14.124103 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) touch meta/a468ec03/snapmapper/0//-1 = 0
> 2015-03-11 01:00:14.124247 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x4b9b6c0
> 2015-03-11 01:00:14.124282 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
> 2015-03-11 01:00:14.124332 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
> 2015-03-11 01:00:14.124348 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) write meta/a143428/osdmap.2757/0//-1 0~3076
> 2015-03-11 01:00:14.124420 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) write meta/a143428/osdmap.2757/0//-1 0~3076 = 3076
> 2015-03-11 01:00:14.124461 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
> 2015-03-11 01:00:14.124547 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
> 2015-03-11 01:00:14.124557 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) write meta/23c2fcde/osd_superblock/0//-1 0~413
> 2015-03-11 01:00:14.124615 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) write meta/23c2fcde/osd_superblock/0//-1 0~413 = 413
> 2015-03-11 01:00:14.124716 7f41a1ba4880  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> 2015-03-11 01:00:14.125247 7f41a1ba4880  2 osd.3 0 boot
> 2015-03-11 01:00:14.125270 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) read meta/23c2fcde/osd_superblock/0//-1 0~0
> 2015-03-11 01:00:14.125343 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) FileStore::read meta/23c2fcde/osd_superblock/0//-1 0~413/413
> 2015-03-11 01:00:14.125393 7f41a1ba4880 10 osd.3 0 read_superblock sb(3bf19e54-97e2-4a5f-b811-7c3780cd5e57 osd.3 05fb481f-68b3-4442-b7de-af4c867e18d5 e2757 [2757,2757] lci=[0,0])
> 2015-03-11 01:00:14.125521 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) stat meta/a468ec03/snapmapper/0//-1 = 0 (size 0)
> 2015-03-11 01:00:14.125540 7f41a1ba4880 10 open_all_classes
> 2015-03-11 01:00:14.125586 7f41a1ba4880 10 open_all_classes found hello
> 2015-03-11 01:00:14.125623 7f41a1ba4880 10 _get_class adding new class name hello 0x4c9e028
> 2015-03-11 01:00:14.125630 7f41a1ba4880 10 _load_class hello from /usr/lib64/rados-classes/libcls_hello.so
> 2015-03-11 01:00:14.126058 7f41a1ba4880  0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
> 2015-03-11 01:00:14.126080 7f41a1ba4880 10 register_class hello status 3
> 2015-03-11 01:00:14.126084 7f41a1ba4880 10 register_cxx_method hello.say_hello flags 1 0x7f4192a10350
> 2015-03-11 01:00:14.126092 7f41a1ba4880 10 register_cxx_method hello.record_hello flags 2 0x7f4192a113c0
> 2015-03-11 01:00:14.126098 7f41a1ba4880 10 register_cxx_method hello.writes_dont_return_data flags 2 0x7f4192a10650
> 2015-03-11 01:00:14.126103 7f41a1ba4880 10 register_cxx_method hello.replay flags 1 0x7f4192a10320
> 2015-03-11 01:00:14.126109 7f41a1ba4880 10 register_cxx_method hello.turn_it_to_11 flags 3 0x7f4192a10920
> 2015-03-11 01:00:14.126114 7f41a1ba4880 10 register_cxx_method hello.bad_reader flags 2 0x7f4192a10310
> 2015-03-11 01:00:14.126119 7f41a1ba4880 10 register_cxx_method hello.bad_writer flags 1 0x7f4192a10300
> 2015-03-11 01:00:14.126127 7f41a1ba4880 10 _load_class hello success
> 2015-03-11 01:00:14.126132 7f41a1ba4880 10 open_all_classes found kvs
> 2015-03-11 01:00:14.126137 7f41a1ba4880 10 _get_class adding new class name kvs 0x4c9e108
> 2015-03-11 01:00:14.126142 7f41a1ba4880 10 _load_class kvs from /usr/lib64/rados-classes/libcls_kvs.so
> 2015-03-11 01:00:14.126655 7f41a1ba4880 10 register_class kvs status 3
> 2015-03-11 01:00:14.126665 7f41a1ba4880 10 register_cxx_method kvs.get_idata_from_key flags 1 0x7f41927ff060
> 2015-03-11 01:00:14.126669 7f41a1ba4880 10 register_cxx_method kvs.get_next_idata flags 1 0x7f4192801050
> 2015-03-11 01:00:14.126672 7f41a1ba4880 10 register_cxx_method kvs.get_prev_idata flags 1 0x7f41927fd570
> 2015-03-11 01:00:14.126675 7f41a1ba4880 10 register_cxx_method kvs.read_many flags 1 0x7f41927f8f30
> 2015-03-11 01:00:14.126678 7f41a1ba4880 10 register_cxx_method kvs.check_writable flags 3 0x7f41927f8ac0
> 2015-03-11 01:00:14.126681 7f41a1ba4880 10 register_cxx_method kvs.assert_size_in_bound flags 2 0x7f41927f8d40
> 2015-03-11 01:00:14.126684 7f41a1ba4880 10 register_cxx_method kvs.omap_insert flags 2 0x7f41927fcb50
> 2015-03-11 01:00:14.126688 7f41a1ba4880 10 register_cxx_method kvs.create_with_omap flags 2 0x7f41927fc0e0
> 2015-03-11 01:00:14.126691 7f41a1ba4880 10 register_cxx_method kvs.omap_remove flags 2 0x7f41927f9e70
> 2015-03-11 01:00:14.126696 7f41a1ba4880 10 register_cxx_method kvs.maybe_read_for_balance flags 1 0x7f41927fab70
> 2015-03-11 01:00:14.126735 7f41a1ba4880 10 _load_class kvs success
> 2015-03-11 01:00:14.126739 7f41a1ba4880 10 open_all_classes found lock
> 2015-03-11 01:00:14.126742 7f41a1ba4880 10 _get_class adding new class name lock 0x4c9e1e8
> 2015-03-11 01:00:14.126745 7f41a1ba4880 10 _load_class lock from /usr/lib64/rados-classes/libcls_lock.so
> 2015-03-11 01:00:14.126972 7f41a1ba4880 10 register_class lock status 3
> 2015-03-11 01:00:14.126977 7f41a1ba4880 10 register_cxx_method lock.lock flags 3 0x7f41925ee800
> 2015-03-11 01:00:14.126981 7f41a1ba4880 10 register_cxx_method lock.unlock flags 3 0x7f41925ed210
> 2015-03-11 01:00:14.126984 7f41a1ba4880 10 register_cxx_method lock.break_lock flags 3 0x7f41925ece60
> 2015-03-11 01:00:14.126986 7f41a1ba4880 10 register_cxx_method lock.get_info flags 1 0x7f41925ed610
> 2015-03-11 01:00:14.126990 7f41a1ba4880 10 register_cxx_method lock.list_locks flags 1 0x7f41925eace0
> 2015-03-11 01:00:14.126993 7f41a1ba4880 10 register_cxx_method lock.assert_locked flags 1 0x7f41925ec330
> 2015-03-11 01:00:14.126996 7f41a1ba4880 10 _load_class lock success
> 2015-03-11 01:00:14.126999 7f41a1ba4880 10 open_all_classes found log
> 2015-03-11 01:00:14.127002 7f41a1ba4880 10 _get_class adding new class name log 0x4c9e2c8
> 2015-03-11 01:00:14.127005 7f41a1ba4880 10 _load_class log from /usr/lib64/rados-classes/libcls_log.so
> 2015-03-11 01:00:14.127223 7f41a1ba4880 10 register_class log status 3
> 2015-03-11 01:00:14.127225 7f41a1ba4880 10 register_cxx_method log.add flags 3 0x7f41923e1560
> 2015-03-11 01:00:14.127228 7f41a1ba4880 10 register_cxx_method log.list flags 1 0x7f41923e2740
> 2015-03-11 01:00:14.127231 7f41a1ba4880 10 register_cxx_method log.trim flags 3 0x7f41923e1f90
> 2015-03-11 01:00:14.127234 7f41a1ba4880 10 register_cxx_method log.info flags 1 0x7f41923e1090
> 2015-03-11 01:00:14.127238 7f41a1ba4880 10 _load_class log success
> 2015-03-11 01:00:14.127241 7f41a1ba4880 10 open_all_classes found rbd
> 2015-03-11 01:00:14.127244 7f41a1ba4880 10 _get_class adding new class name rbd 0x4c9e3a8
> 2015-03-11 01:00:14.127247 7f41a1ba4880 10 _load_class rbd from /usr/lib64/rados-classes/libcls_rbd.so
> 2015-03-11 01:00:14.127735 7f41a1ba4880 10 register_class rbd status 3
> 2015-03-11 01:00:14.127745 7f41a1ba4880 10 register_cxx_method rbd.create flags 3 0x7f41921cc5c0
> 2015-03-11 01:00:14.127748 7f41a1ba4880 10 register_cxx_method rbd.get_features flags 1 0x7f41921d3190
> 2015-03-11 01:00:14.127751 7f41a1ba4880 10 register_cxx_method rbd.get_size flags 1 0x7f41921d2de0
> 2015-03-11 01:00:14.127754 7f41a1ba4880 10 register_cxx_method rbd.set_size flags 3 0x7f41921ca7e0
> 2015-03-11 01:00:14.127758 7f41a1ba4880 10 register_cxx_method rbd.get_snapcontext flags 1 0x7f41921cdb40
> 2015-03-11 01:00:14.127762 7f41a1ba4880 10 register_cxx_method rbd.get_object_prefix flags 1 0x7f41921c9770
> 2015-03-11 01:00:14.127766 7f41a1ba4880 10 register_cxx_method rbd.get_snapshot_name flags 1 0x7f41921d1dd0
> 2015-03-11 01:00:14.127770 7f41a1ba4880 10 register_cxx_method rbd.snapshot_add flags 3 0x7f41921d0830
> 2015-03-11 01:00:14.127773 7f41a1ba4880 10 register_cxx_method rbd.snapshot_remove flags 3 0x7f41921d1b40
> 2015-03-11 01:00:14.127776 7f41a1ba4880 10 register_cxx_method rbd.get_all_features flags 1 0x7f41921c78c0
> 2015-03-11 01:00:14.127779 7f41a1ba4880 10 register_cxx_method rbd.copyup flags 3 0x7f41921c7940
> 2015-03-11 01:00:14.127782 7f41a1ba4880 10 register_cxx_method rbd.get_parent flags 1 0x7f41921d2080
> 2015-03-11 01:00:14.127786 7f41a1ba4880 10 register_cxx_method rbd.set_parent flags 3 0x7f41921ccce0
> 2015-03-11 01:00:14.127789 7f41a1ba4880 10 register_cxx_method rbd.remove_parent flags 3 0x7f41921c9140
> 2015-03-11 01:00:14.127792 7f41a1ba4880 10 register_cxx_method rbd.set_protection_status flags 3 0x7f41921d2770
> 2015-03-11 01:00:14.127815 7f41a1ba4880 10 register_cxx_method rbd.get_protection_status flags 1 0x7f41921d2b60
> 2015-03-11 01:00:14.127818 7f41a1ba4880 10 register_cxx_method rbd.get_stripe_unit_count flags 1 0x7f41921c9330
> 2015-03-11 01:00:14.127822 7f41a1ba4880 10 register_cxx_method rbd.set_stripe_unit_count flags 3 0x7f41921cc070
> 2015-03-11 01:00:14.127825 7f41a1ba4880 10 register_cxx_method rbd.get_flags flags 1 0x7f41921d2470
> 2015-03-11 01:00:14.127828 7f41a1ba4880 10 register_cxx_method rbd.set_flags flags 3 0x7f41921cb370
> 2015-03-11 01:00:14.127831 7f41a1ba4880 10 register_cxx_method rbd.add_child flags 3 0x7f41921cd790
> 2015-03-11 01:00:14.127834 7f41a1ba4880 10 register_cxx_method rbd.remove_child flags 3 0x7f41921cdf50
> 2015-03-11 01:00:14.127838 7f41a1ba4880 10 register_cxx_method rbd.get_children flags 1 0x7f41921cd4e0
> 2015-03-11 01:00:14.127841 7f41a1ba4880 10 register_cxx_method rbd.get_id flags 1 0x7f41921ca580
> 2015-03-11 01:00:14.127844 7f41a1ba4880 10 register_cxx_method rbd.set_id flags 3 0x7f41921cb6c0
> 2015-03-11 01:00:14.127848 7f41a1ba4880 10 register_cxx_method rbd.dir_get_id flags 1 0x7f41921c9f40
> 2015-03-11 01:00:14.127851 7f41a1ba4880 10 register_cxx_method rbd.dir_get_name flags 1 0x7f41921ca210
> 2015-03-11 01:00:14.127855 7f41a1ba4880 10 register_cxx_method rbd.dir_list flags 1 0x7f41921cf9f0
> 2015-03-11 01:00:14.127858 7f41a1ba4880 10 register_cxx_method rbd.dir_add_image flags 3 0x7f41921cf840
> 2015-03-11 01:00:14.127862 7f41a1ba4880 10 register_cxx_method rbd.dir_remove_image flags 3 0x7f41921c9d60
> 2015-03-11 01:00:14.127865 7f41a1ba4880 10 register_cxx_method rbd.dir_rename_image flags 3 0x7f41921cf660
> 2015-03-11 01:00:14.127868 7f41a1ba4880 10 register_cxx_method rbd.object_map_load flags 1 0x7f41921ce5f0
> 2015-03-11 01:00:14.127872 7f41a1ba4880 10 register_cxx_method rbd.object_map_resize flags 3 0x7f41921ceec0
> 2015-03-11 01:00:14.127875 7f41a1ba4880 10 register_cxx_method rbd.object_map_update flags 3 0x7f41921ce720
> 2015-03-11 01:00:14.127879 7f41a1ba4880 10 register_cxx_method rbd.snap_list flags 1 0x7f41921c8540
> 2015-03-11 01:00:14.127882 7f41a1ba4880 10 register_cxx_method rbd.snap_add flags 3 0x7f41921cacc0
> 2015-03-11 01:00:14.127885 7f41a1ba4880 10 register_cxx_method rbd.snap_remove flags 3 0x7f41921cb910
> 2015-03-11 01:00:14.127891 7f41a1ba4880 10 _load_class rbd success
> 2015-03-11 01:00:14.127894 7f41a1ba4880 10 open_all_classes found refcount
> 2015-03-11 01:00:14.127897 7f41a1ba4880 10 _get_class adding new class name refcount 0x4c9e488
> 2015-03-11 01:00:14.127900 7f41a1ba4880 10 _load_class refcount from /usr/lib64/rados-classes/libcls_refcount.so
> 2015-03-11 01:00:14.132942 7f41a1ba4880 10 register_class refcount status 3
> 2015-03-11 01:00:14.132946 7f41a1ba4880 10 register_cxx_method refcount.get flags 3 0x7f4191f32be0
> 2015-03-11 01:00:14.132957 7f41a1ba4880 10 register_cxx_method refcount.put flags 3 0x7f4191f31b30
> 2015-03-11 01:00:14.132960 7f41a1ba4880 10 register_cxx_method refcount.set flags 3 0x7f4191f32510
> 2015-03-11 01:00:14.132963 7f41a1ba4880 10 register_cxx_method refcount.read flags 1 0x7f4191f31fb0
> 2015-03-11 01:00:14.132970 7f41a1ba4880 10 _load_class refcount success
> 2015-03-11 01:00:14.132977 7f41a1ba4880 10 open_all_classes found replica_log
> 2015-03-11 01:00:14.132984 7f41a1ba4880 10 _get_class adding new class name replica_log 0x4c9e568
> 2015-03-11 01:00:14.132987 7f41a1ba4880 10 _load_class replica_log from /usr/lib64/rados-classes/libcls_replica_log.so
> 2015-03-11 01:00:14.133269 7f41a1ba4880 10 register_class replica_log status 3
> 2015-03-11 01:00:14.133271 7f41a1ba4880 10 register_cxx_method replica_log.set flags 3 0x7f4191c7f920
> 2015-03-11 01:00:14.133276 7f41a1ba4880 10 register_cxx_method replica_log.get flags 1 0x7f4191c809f0
> 2015-03-11 01:00:14.133279 7f41a1ba4880 10 register_cxx_method replica_log.delete flags 3 0x7f4191c80400
> 2015-03-11 01:00:14.133283 7f41a1ba4880 10 _load_class replica_log success
> 2015-03-11 01:00:14.133285 7f41a1ba4880 10 open_all_classes found rgw
> 2015-03-11 01:00:14.133288 7f41a1ba4880 10 _get_class adding new class name rgw 0x4c9e728
> 2015-03-11 01:00:14.133291 7f41a1ba4880 10 _load_class rgw from /usr/lib64/rados-classes/libcls_rgw.so
> 2015-03-11 01:00:14.138712 7f41a1ba4880 10 register_class rgw status 3
> 2015-03-11 01:00:14.138717 7f41a1ba4880 10 register_cxx_method rgw.bucket_init_index flags 3 0x7f41919c0860
> 2015-03-11 01:00:14.138725 7f41a1ba4880 10 register_cxx_method rgw.bucket_set_tag_timeout flags 3 0x7f41919c4a40
> 2015-03-11 01:00:14.138728 7f41a1ba4880 10 register_cxx_method rgw.bucket_list flags 1 0x7f41919c8160
> 2015-03-11 01:00:14.138732 7f41a1ba4880 10 register_cxx_method rgw.bucket_check_index flags 1 0x7f41919ca300
> 2015-03-11 01:00:14.138735 7f41a1ba4880 10 register_cxx_method rgw.bucket_rebuild_index flags 3 0x7f41919ca1a0
> 2015-03-11 01:00:14.138740 7f41a1ba4880 10 register_cxx_method rgw.bucket_prepare_op flags 3 0x7f41919c5180
> 2015-03-11 01:00:14.138745 7f41a1ba4880 10 register_cxx_method rgw.bucket_complete_op flags 3 0x7f41919c5980
> 2015-03-11 01:00:14.138748 7f41a1ba4880 10 register_cxx_method rgw.bucket_link_olh flags 3 0x7f41919cc980
> 2015-03-11 01:00:14.138751 7f41a1ba4880 10 register_cxx_method rgw.bucket_unlink_instance flags 3 0x7f41919cbb80
> 2015-03-11 01:00:14.138754 7f41a1ba4880 10 register_cxx_method rgw.bucket_read_olh_log flags 1 0x7f41919cde40
> 2015-03-11 01:00:14.138757 7f41a1ba4880 10 register_cxx_method rgw.bucket_trim_olh_log flags 3 0x7f41919cb2d0
> 2015-03-11 01:00:14.138760 7f41a1ba4880 10 register_cxx_method rgw.bucket_clear_olh flags 3 0x7f41919cacd0
> 2015-03-11 01:00:14.138763 7f41a1ba4880 10 register_cxx_method rgw.obj_remove flags 3 0x7f41919c3ff0
> 2015-03-11 01:00:14.138771 7f41a1ba4880 10 register_cxx_method rgw.obj_check_attrs_prefix flags 1 0x7f41919c0b60
> 2015-03-11 01:00:14.138774 7f41a1ba4880 10 register_cxx_method rgw.bi_get flags 1 0x7f41919bf050
> 2015-03-11 01:00:14.138777 7f41a1ba4880 10 register_cxx_method rgw.bi_put flags 3 0x7f41919be700
> 2015-03-11 01:00:14.138780 7f41a1ba4880 10 register_cxx_method rgw.bi_list flags 1 0x7f41919c6790
> 2015-03-11 01:00:14.138784 7f41a1ba4880 10 register_cxx_method rgw.bi_log_list flags 1 0x7f41919c2180
> 2015-03-11 01:00:14.138787 7f41a1ba4880 10 register_cxx_method rgw.bi_log_trim flags 3 0x7f41919c26e0
> 2015-03-11 01:00:14.138791 7f41a1ba4880 10 register_cxx_method rgw.dir_suggest_changes flags 3 0x7f41919c9260
> 2015-03-11 01:00:14.138794 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_add flags 3 0x7f41919ce6a0
> 2015-03-11 01:00:14.138797 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_read flags 1 0x7f41919c31c0
> 2015-03-11 01:00:14.138799 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_trim flags 3 0x7f41919c3950
> 2015-03-11 01:00:14.138802 7f41a1ba4880 10 register_cxx_method rgw.gc_set_entry flags 3 0x7f41919c01b0
> 2015-03-11 01:00:14.138806 7f41a1ba4880 10 register_cxx_method rgw.gc_defer_entry flags 3 0x7f41919bfe90
> 2015-03-11 01:00:14.138809 7f41a1ba4880 10 register_cxx_method rgw.gc_list flags 1 0x7f41919c0f80
> 2015-03-11 01:00:14.138812 7f41a1ba4880 10 register_cxx_method rgw.gc_remove flags 3 0x7f41919c0420
> 2015-03-11 01:00:14.138815 7f41a1ba4880 10 _load_class rgw success
> 2015-03-11 01:00:14.138820 7f41a1ba4880 10 open_all_classes found statelog
> 2015-03-11 01:00:14.138826 7f41a1ba4880 10 _get_class adding new class name statelog 0x4c9e808
> 2015-03-11 01:00:14.138830 7f41a1ba4880 10 _load_class statelog from /usr/lib64/rados-classes/libcls_statelog.so
> 2015-03-11 01:00:14.139141 7f41a1ba4880 10 register_class statelog status 3
> 2015-03-11 01:00:14.139143 7f41a1ba4880 10 register_cxx_method statelog.add flags 3 0x7f4191700bc0
> 2015-03-11 01:00:14.139146 7f41a1ba4880 10 register_cxx_method statelog.list flags 1 0x7f4191700f40
> 2015-03-11 01:00:14.139150 7f41a1ba4880 10 register_cxx_method statelog.remove flags 3 0x7f41916ffa80
> 2015-03-11 01:00:14.139153 7f41a1ba4880 10 register_cxx_method statelog.check_state flags 1 0x7f41917002d0
> 2015-03-11 01:00:14.139175 7f41a1ba4880 10 _load_class statelog success
> 2015-03-11 01:00:14.139178 7f41a1ba4880 10 open_all_classes found user
> 2015-03-11 01:00:14.139182 7f41a1ba4880 10 _get_class adding new class name user 0x4c9e8e8
> 2015-03-11 01:00:14.139185 7f41a1ba4880 10 _load_class user from /usr/lib64/rados-classes/libcls_user.so
> 2015-03-11 01:00:14.139386 7f41a1ba4880 10 register_class user status 3
> 2015-03-11 01:00:14.139388 7f41a1ba4880 10 register_cxx_method user.set_buckets_info flags 3 0x7f41914f5180
> 2015-03-11 01:00:14.139392 7f41a1ba4880 10 register_cxx_method user.complete_stats_sync flags 3 0x7f41914f4aa0
> 2015-03-11 01:00:14.139394 7f41a1ba4880 10 register_cxx_method user.remove_bucket flags 3 0x7f41914f3d30
> 2015-03-11 01:00:14.139397 7f41a1ba4880 10 register_cxx_method user.list_buckets flags 1 0x7f41914f64c0
> 2015-03-11 01:00:14.139400 7f41a1ba4880 10 register_cxx_method user.get_header flags 1 0x7f41914f2d50
> 2015-03-11 01:00:14.139403 7f41a1ba4880 10 _load_class user success
> 2015-03-11 01:00:14.139406 7f41a1ba4880 10 open_all_classes found version
> 2015-03-11 01:00:14.139409 7f41a1ba4880 10 _get_class adding new class name version 0x4c9e9c8
> 2015-03-11 01:00:14.139412 7f41a1ba4880 10 _load_class version from /usr/lib64/rados-classes/libcls_version.so
> 2015-03-11 01:00:14.139608 7f41a1ba4880 10 register_class version status 3
> 2015-03-11 01:00:14.139611 7f41a1ba4880 10 register_cxx_method version.set flags 3 0x7f41912ebe60
> 2015-03-11 01:00:14.139615 7f41a1ba4880 10 register_cxx_method version.inc flags 3 0x7f41912ecc80
> 2015-03-11 01:00:14.139618 7f41a1ba4880 10 register_cxx_method version.inc_conds flags 3 0x7f41912ecc80
> 2015-03-11 01:00:14.139621 7f41a1ba4880 10 register_cxx_method version.read flags 1 0x7f41912ebb60
> 2015-03-11 01:00:14.139624 7f41a1ba4880 10 register_cxx_method version.check_conds flags 1 0x7f41912ec260
> 2015-03-11 01:00:14.139627 7f41a1ba4880 10 _load_class version success
> 2015-03-11 01:00:14.139682 7f41a1ba4880 20 osd.3 0 get_map 2757 - loading and decoding 0x4b4cb40
> 2015-03-11 01:00:14.139693 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) read meta/a143428/osdmap.2757/0//-1 0~0
> 2015-03-11 01:00:14.139804 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) FileStore::read meta/a143428/osdmap.2757/0//-1 0~3076/3076
> 2015-03-11 01:00:14.139818 7f41a1ba4880 10 osd.3 0 add_map_bl 2757 3076 bytes
> 2015-03-11 01:00:14.139908 7f41a1ba4880  0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients
> 2015-03-11 01:00:14.139914 7f41a1ba4880  0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons
> 2015-03-11 01:00:14.139919 7f41a1ba4880  0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds
> 2015-03-11 01:00:14.139922 7f41a1ba4880 10 osd.3 2757 create_recoverystate_perf
> 2015-03-11 01:00:14.139939 7f41a1ba4880  0 osd.3 2757 load_pgs
> 2015-03-11 01:00:14.139942 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) list_collections
> 2015-03-11 01:00:14.139981 7f41a1ba4880 10 osd.3 2757 load_pgs ignoring unrecognized meta
> 2015-03-11 01:00:14.139985 7f41a1ba4880  0 osd.3 2757 load_pgs opened 0 pgs
> 2015-03-11 01:00:14.139989 7f41a1ba4880 10 osd.3 2757 build_past_intervals_parallel nothing to build
> 2015-03-11 01:00:14.139992 7f41a1ba4880  2 osd.3 2757 superblock: i am osd.3
> 2015-03-11 01:00:14.139995 7f41a1ba4880 10 osd.3 2757 create_logger
> 2015-03-11 01:00:14.140126 7f41a1ba4880  1 accepter.accepter.start
> 2015-03-11 01:00:14.140269 7f41a1ba4880  1 accepter.accepter.start
> 2015-03-11 01:00:14.140434 7f41a1ba4880  1 accepter.accepter.start
> 2015-03-11 01:00:14.140575 7f41a1ba4880  1 accepter.accepter.start
> 2015-03-11 01:00:14.140813 7f41a1ba4880 -1 osd.3 2757 log_to_monitors {default=true}
> 2015-03-11 01:00:14.141325 7f41a1ba4880 10 osd.3 2757 set_disk_tp_priority class  priority -1
> 2015-03-11 01:00:14.141599 7f41a1ba4880  1 -- 0.0.0.0:6800/2840 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x4b4cd80 con 0x4cbe3c0
> 2015-03-11 01:00:14.142020 7f4180ac7700 20 osd.3 2757 update_osd_stat osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist [])
> 2015-03-11 01:00:14.142048 7f4180ac7700  5 osd.3 2757 heartbeat: osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist [])
> 2015-03-11 01:00:14.142221 7f417e2c2700 10 osd.3 2757 agent_entry start
> 2015-03-11 01:00:14.142241 7f417e2c2700 20 osd.3 2757 agent_entry empty queue
> 2015-03-11 01:00:14.158932 7f41a1ba0700  1 -- 192.168.0.250:6800/2840 learned my addr 192.168.0.250:6800/2840
> 2015-03-11 01:00:14.159058 7f41a1ba0700 10 osd.3 2757  new session (outgoing) 0x4b92280 con=0x4b602c0 addr=192.168.0.250:6800/2840
> 2015-03-11 01:00:14.159166 7f41a1ba0700 10 osd.3 2757 OSD::ms_get_authorizer type=mon
> 2015-03-11 01:00:14.173274 7f41912e8700 10 osd.3 2757 ms_handle_connect on mon
> 2015-03-11 01:00:14.173332 7f41912e8700 20 osd.3 2757 send_pg_stats
> 2015-03-11 01:00:14.175783 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x4b4f840 con 0x4cbe3c0
> 2015-03-11 01:00:14.175940 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (335930126 0 0) 0x4b4fa80 con 0x4cbe3c0
> 2015-03-11 01:00:14.175997 7f41912e8700  1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x4b31a00 con 0x4cbe3c0
> 2015-03-11 01:00:14.176059 7f41912e8700  1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0,osdmap=2757}) v2 -- ?+0 0x4b32800 con 0x4cbe3c0
> 2015-03-11 01:00:14.176147 7f41a1ba4880 10 osd.3 2757 ensuring pgs have consumed prior maps
> 2015-03-11 01:00:14.176170 7f41a1ba4880  7 osd.3 2757 consume_map version 2757
> 2015-03-11 01:00:14.176187 7f41a1ba4880  0 osd.3 2757 done with init, starting boot process
> 2015-03-11 01:00:14.176191 7f41a1ba4880 10 osd.3 2757 start_boot - have maps 2757..2757
> 2015-03-11 01:00:14.176221 7f41a1ba4880  1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x4bc41e0 con 0x4cbe3c0
> 2015-03-11 01:00:14.178539 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 3 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x4b4fcc0 con 0x4cbe3c0
> 2015-03-11 01:00:14.178650 7f41912e8700 10 osd.3 2757 do_waiters -- start
> 2015-03-11 01:00:14.178658 7f41912e8700 10 osd.3 2757 do_waiters -- finish
> 2015-03-11 01:00:14.178661 7f41912e8700 20 osd.3 2757 _dispatch 0x4b4fcc0 osd_map(3277..3277 src has 2757..3277) v3
> 2015-03-11 01:00:14.178888 7f41912e8700  3 osd.3 2757 handle_osd_map epochs [3277,3277], i have 2757, src has [2757,3277]
> 2015-03-11 01:00:14.178906 7f41912e8700 10 osd.3 2757 handle_osd_map message skips epochs 2758..3276
> 2015-03-11 01:00:14.178954 7f41912e8700  1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=2758}) v2 -- ?+0 0x4b31a00 con 0x4cbe3c0
> 2015-03-11 01:00:14.178991 7f41912e8700 10 osd.3 2757 do_waiters -- start
> 2015-03-11 01:00:14.178996 7f41912e8700 10 osd.3 2757 do_waiters -- finish
> 2015-03-11 01:00:14.179012 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 4 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x4b4f840 con 0x4cbe3c0
> 2015-03-11 01:00:14.184757 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(2757..2857 src has 2757..3277) v3 ==== 37648+0+0 (2648067016 0 0) 0x4b4fa80 con 0x4cbe3c0
> 2015-03-11 01:00:14.184831 7f41912e8700 10 osd.3 2757 do_waiters -- start
> 2015-03-11 01:00:14.184838 7f41912e8700 10 osd.3 2757 do_waiters -- finish
> 2015-03-11 01:00:14.184841 7f41912e8700 20 osd.3 2757 _dispatch 0x4b4fa80 osd_map(2757..2857 src has 2757..3277) v3
> 2015-03-11 01:00:14.184872 7f41912e8700  3 osd.3 2757 handle_osd_map epochs [2757,2857], i have 2757, src has [2757,3277]
> 2015-03-11 01:00:14.184884 7f41912e8700 10 osd.3 2757 handle_osd_map  got inc map for epoch 2758
> 2015-03-11 01:00:14.185229 7f41912e8700 10 osd.3 2757 handle_osd_map  got inc map for epoch 2759
> 2015-03-11 01:00:14.195165 7f41912e8700 -1 *** Caught signal (Aborted) **
>  in thread 7f41912e8700
> 
>  ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
>  1: /usr/bin/ceph-osd() [0xac7cea]
>  2: (()+0x10050) [0x7f41a0aaf050]
>  3: (gsignal()+0x37) [0x7f419ef820f7]
>  4: (abort()+0x13a) [0x7f419ef834ca]
>  5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f419f867fe5]
>  6: (()+0x63186) [0x7f419f866186]
>  7: (()+0x631b3) [0x7f419f8661b3]
>  8: (()+0x633d2) [0x7f419f8663d2]
>  9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
>  10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
>  11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
>  12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
>  13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
>  14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
>  15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
>  16: (DispatchQueue::entry()+0x649) [0xc6fe09]
>  17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
>  18: (()+0x83a4) [0x7f41a0aa73a4]
>  19: (clone()+0x6d) [0x7f419f031a4d]
>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
> 
> --- begin dump of recent events ---
>   -363> 2015-03-11 01:00:13.871379 7f41a1ba4880  5 asok(0x4b50000) register_command perfcounters_dump hook 0x4b04030
>   -362> 2015-03-11 01:00:13.871462 7f41a1ba4880  5 asok(0x4b50000) register_command 1 hook 0x4b04030
>   -361> 2015-03-11 01:00:13.871471 7f41a1ba4880  5 asok(0x4b50000) register_command perf dump hook 0x4b04030
>   -360> 2015-03-11 01:00:13.871484 7f41a1ba4880  5 asok(0x4b50000) register_command perfcounters_schema hook 0x4b04030
>   -359> 2015-03-11 01:00:13.871507 7f41a1ba4880  5 asok(0x4b50000) register_command 2 hook 0x4b04030
>   -358> 2015-03-11 01:00:13.871512 7f41a1ba4880  5 asok(0x4b50000) register_command perf schema hook 0x4b04030
>   -357> 2015-03-11 01:00:13.871545 7f41a1ba4880  5 asok(0x4b50000) register_command perf reset hook 0x4b04030
>   -356> 2015-03-11 01:00:13.871555 7f41a1ba4880  5 asok(0x4b50000) register_command config show hook 0x4b04030
>   -355> 2015-03-11 01:00:13.871562 7f41a1ba4880  5 asok(0x4b50000) register_command config set hook 0x4b04030
>   -354> 2015-03-11 01:00:13.871571 7f41a1ba4880  5 asok(0x4b50000) register_command config get hook 0x4b04030
>   -353> 2015-03-11 01:00:13.871576 7f41a1ba4880  5 asok(0x4b50000) register_command config diff hook 0x4b04030
>   -352> 2015-03-11 01:00:13.871585 7f41a1ba4880  5 asok(0x4b50000) register_command log flush hook 0x4b04030
>   -351> 2015-03-11 01:00:13.871590 7f41a1ba4880  5 asok(0x4b50000) register_command log dump hook 0x4b04030
>   -350> 2015-03-11 01:00:13.871600 7f41a1ba4880  5 asok(0x4b50000) register_command log reopen hook 0x4b04030
>   -349> 2015-03-11 01:00:13.876152 7f41a1ba4880  0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 2840
>   -348> 2015-03-11 01:00:13.877059 7f41a1ba4880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/2840 need_addr=1
>   -347> 2015-03-11 01:00:13.877111 7f41a1ba4880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/2840 need_addr=1
>   -346> 2015-03-11 01:00:13.877140 7f41a1ba4880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/2840 need_addr=1
>   -345> 2015-03-11 01:00:13.877169 7f41a1ba4880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/2840 need_addr=1
>   -344> 2015-03-11 01:00:13.880323 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) dump_stop
>   -343> 2015-03-11 01:00:13.880362 7f41a1ba4880  5 asok(0x4b50000) init /var/run/ceph/ceph-osd.3.asok
>   -342> 2015-03-11 01:00:13.880382 7f41a1ba4880  5 asok(0x4b50000) bind_and_listen /var/run/ceph/ceph-osd.3.asok
>   -341> 2015-03-11 01:00:13.880604 7f41a1ba4880  5 asok(0x4b50000) register_command 0 hook 0x4b000c0
>   -340> 2015-03-11 01:00:13.880620 7f41a1ba4880  5 asok(0x4b50000) register_command version hook 0x4b000c0
>   -339> 2015-03-11 01:00:13.880626 7f41a1ba4880  5 asok(0x4b50000) register_command git_version hook 0x4b000c0
>   -338> 2015-03-11 01:00:13.880633 7f41a1ba4880  5 asok(0x4b50000) register_command help hook 0x4b04110
>   -337> 2015-03-11 01:00:13.880639 7f41a1ba4880  5 asok(0x4b50000) register_command get_command_descriptions hook 0x4b04120
>   -336> 2015-03-11 01:00:13.880703 7f41a1ba4880 10 monclient(hunting): build_initial_monmap
>   -335> 2015-03-11 01:00:13.880697 7f419cf7d700  5 asok(0x4b50000) entry start
>   -334> 2015-03-11 01:00:13.890355 7f41a1ba4880 10 ErasureCodePluginSelectJerasure: load: jerasure_generic 
>   -333> 2015-03-11 01:00:13.897843 7f41a1ba4880 10 load: jerasure load: lrc load: isa 
>   -332> 2015-03-11 01:00:13.897900 7f41a1ba4880  5 adding auth protocol: none
>   -331> 2015-03-11 01:00:13.897905 7f41a1ba4880  5 adding auth protocol: none
>   -330> 2015-03-11 01:00:13.898086 7f41a1ba4880  5 asok(0x4b50000) register_command objecter_requests hook 0x4b041b0
>   -329> 2015-03-11 01:00:13.898172 7f41a1ba4880  5 filestore(/var/lib/ceph/osd/ceph-3) test_mount basedir /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal
>   -328> 2015-03-11 01:00:13.898290 7f41a1ba4880  1 -- 0.0.0.0:6800/2840 messenger.start
>   -327> 2015-03-11 01:00:13.898362 7f41a1ba4880  1 -- :/0 messenger.start
>   -326> 2015-03-11 01:00:13.898450 7f41a1ba4880  1 -- 0.0.0.0:6803/2840 messenger.start
>   -325> 2015-03-11 01:00:13.898527 7f41a1ba4880  1 -- 0.0.0.0:6802/2840 messenger.start
>   -324> 2015-03-11 01:00:13.898609 7f41a1ba4880  1 -- 0.0.0.0:6801/2840 messenger.start
>   -323> 2015-03-11 01:00:13.898672 7f41a1ba4880  1 -- :/0 messenger.start
>   -322> 2015-03-11 01:00:13.898792 7f41a1ba4880  2 osd.3 0 mounting /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
>   -321> 2015-03-11 01:00:13.898826 7f41a1ba4880  5 filestore(/var/lib/ceph/osd/ceph-3) basedir /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal
>   -320> 2015-03-11 01:00:13.898868 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) mount fsid is 05fb481f-68b3-4442-b7de-af4c867e18d5
>   -319> 2015-03-11 01:00:13.898937 7f41a1ba4880  0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342)
>   -318> 2015-03-11 01:00:13.941605 7f41a1ba4880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work
>   -317> 2015-03-11 01:00:13.941665 7f41a1ba4880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
>   -316> 2015-03-11 01:00:13.970056 7f41a1ba4880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
>   -315> 2015-03-11 01:00:13.970442 7f41a1ba4880  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5
>   -314> 2015-03-11 01:00:14.001199 7f41a1ba4880  5 filestore(/var/lib/ceph/osd/ceph-3) mount op_seq is 2
>   -313> 2015-03-11 01:00:14.122464 7f41a1ba4880 20 filestore (init)dbobjectmap: seq is 1
>   -312> 2015-03-11 01:00:14.122580 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) open_journal at /var/lib/ceph/osd/ceph-3/journal
>   -311> 2015-03-11 01:00:14.122674 7f41a1ba4880  0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
>   -310> 2015-03-11 01:00:14.122682 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) list_collections
>   -309> 2015-03-11 01:00:14.122987 7f41a1ba4880  2 journal open /var/lib/ceph/osd/ceph-3/journal fsid 05fb481f-68b3-4442-b7de-af4c867e18d5 fs_op_seq 2
>   -308> 2015-03-11 01:00:14.123024 7f41a1ba4880 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
>   -307> 2015-03-11 01:00:14.123040 7f41a1ba4880  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
>   -306> 2015-03-11 01:00:14.123003 7f4196c1b700 20 filestore(/var/lib/ceph/osd/ceph-3) sync_entry waiting for max_interval 5.000000
>   -305> 2015-03-11 01:00:14.123590 7f41a1ba4880  2 journal read_entry 8192 : seq 3 193 bytes
>   -304> 2015-03-11 01:00:14.123639 7f41a1ba4880  2 journal read_entry 8192 : seq 3 193 bytes
>   -303> 2015-03-11 01:00:14.123661 7f41a1ba4880  3 journal journal_replay: applying op seq 3
>   -302> 2015-03-11 01:00:14.123724 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x4b9b6c0
>   -301> 2015-03-11 01:00:14.123837 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
>   -300> 2015-03-11 01:00:14.123971 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
>   -299> 2015-03-11 01:00:14.124020 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) touch meta/a468ec03/snapmapper/0//-1
>   -298> 2015-03-11 01:00:14.124103 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) touch meta/a468ec03/snapmapper/0//-1 = 0
>   -297> 2015-03-11 01:00:14.124163 7f41a1ba4880  3 journal journal_replay: r = 0, op_seq now 3
>   -296> 2015-03-11 01:00:14.124220 7f41a1ba4880  2 journal read_entry 12288 : seq 4 3829 bytes
>   -295> 2015-03-11 01:00:14.124231 7f41a1ba4880  3 journal journal_replay: applying op seq 4
>   -294> 2015-03-11 01:00:14.124247 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x4b9b6c0
>   -293> 2015-03-11 01:00:14.124282 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
>   -292> 2015-03-11 01:00:14.124332 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
>   -291> 2015-03-11 01:00:14.124348 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) write meta/a143428/osdmap.2757/0//-1 0~3076
>   -290> 2015-03-11 01:00:14.124420 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) write meta/a143428/osdmap.2757/0//-1 0~3076 = 3076
>   -289> 2015-03-11 01:00:14.124461 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
>   -288> 2015-03-11 01:00:14.124547 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
>   -287> 2015-03-11 01:00:14.124557 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) write meta/23c2fcde/osd_superblock/0//-1 0~413
>   -286> 2015-03-11 01:00:14.124615 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) write meta/23c2fcde/osd_superblock/0//-1 0~413 = 413
>   -285> 2015-03-11 01:00:14.124676 7f41a1ba4880  3 journal journal_replay: r = 0, op_seq now 4
>   -284> 2015-03-11 01:00:14.124697 7f41a1ba4880  2 journal No further valid entries found, journal is most likely valid
>   -283> 2015-03-11 01:00:14.124700 7f41a1ba4880  3 journal journal_replay: end of journal, done.
>   -282> 2015-03-11 01:00:14.124716 7f41a1ba4880  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
>   -281> 2015-03-11 01:00:14.125247 7f41a1ba4880  2 osd.3 0 boot
>   -280> 2015-03-11 01:00:14.125270 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) read meta/23c2fcde/osd_superblock/0//-1 0~0
>   -279> 2015-03-11 01:00:14.125343 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) FileStore::read meta/23c2fcde/osd_superblock/0//-1 0~413/413
>   -278> 2015-03-11 01:00:14.125393 7f41a1ba4880 10 osd.3 0 read_superblock sb(3bf19e54-97e2-4a5f-b811-7c3780cd5e57 osd.3 05fb481f-68b3-4442-b7de-af4c867e18d5 e2757 [2757,2757] lci=[0,0])
>   -277> 2015-03-11 01:00:14.125521 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) stat meta/a468ec03/snapmapper/0//-1 = 0 (size 0)
>   -276> 2015-03-11 01:00:14.125540 7f41a1ba4880 10 open_all_classes
>   -275> 2015-03-11 01:00:14.125586 7f41a1ba4880 10 open_all_classes found hello
>   -274> 2015-03-11 01:00:14.125623 7f41a1ba4880 10 _get_class adding new class name hello 0x4c9e028
>   -273> 2015-03-11 01:00:14.125630 7f41a1ba4880 10 _load_class hello from /usr/lib64/rados-classes/libcls_hello.so
>   -272> 2015-03-11 01:00:14.126058 7f41a1ba4880  0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
>   -271> 2015-03-11 01:00:14.126080 7f41a1ba4880 10 register_class hello status 3
>   -270> 2015-03-11 01:00:14.126084 7f41a1ba4880 10 register_cxx_method hello.say_hello flags 1 0x7f4192a10350
>   -269> 2015-03-11 01:00:14.126092 7f41a1ba4880 10 register_cxx_method hello.record_hello flags 2 0x7f4192a113c0
>   -268> 2015-03-11 01:00:14.126098 7f41a1ba4880 10 register_cxx_method hello.writes_dont_return_data flags 2 0x7f4192a10650
>   -267> 2015-03-11 01:00:14.126103 7f41a1ba4880 10 register_cxx_method hello.replay flags 1 0x7f4192a10320
>   -266> 2015-03-11 01:00:14.126109 7f41a1ba4880 10 register_cxx_method hello.turn_it_to_11 flags 3 0x7f4192a10920
>   -265> 2015-03-11 01:00:14.126114 7f41a1ba4880 10 register_cxx_method hello.bad_reader flags 2 0x7f4192a10310
>   -264> 2015-03-11 01:00:14.126119 7f41a1ba4880 10 register_cxx_method hello.bad_writer flags 1 0x7f4192a10300
>   -263> 2015-03-11 01:00:14.126127 7f41a1ba4880 10 _load_class hello success
>   -262> 2015-03-11 01:00:14.126132 7f41a1ba4880 10 open_all_classes found kvs
>   -261> 2015-03-11 01:00:14.126137 7f41a1ba4880 10 _get_class adding new class name kvs 0x4c9e108
>   -260> 2015-03-11 01:00:14.126142 7f41a1ba4880 10 _load_class kvs from /usr/lib64/rados-classes/libcls_kvs.so
>   -259> 2015-03-11 01:00:14.126655 7f41a1ba4880 10 register_class kvs status 3
>   -258> 2015-03-11 01:00:14.126665 7f41a1ba4880 10 register_cxx_method kvs.get_idata_from_key flags 1 0x7f41927ff060
>   -257> 2015-03-11 01:00:14.126669 7f41a1ba4880 10 register_cxx_method kvs.get_next_idata flags 1 0x7f4192801050
>   -256> 2015-03-11 01:00:14.126672 7f41a1ba4880 10 register_cxx_method kvs.get_prev_idata flags 1 0x7f41927fd570
>   -255> 2015-03-11 01:00:14.126675 7f41a1ba4880 10 register_cxx_method kvs.read_many flags 1 0x7f41927f8f30
>   -254> 2015-03-11 01:00:14.126678 7f41a1ba4880 10 register_cxx_method kvs.check_writable flags 3 0x7f41927f8ac0
>   -253> 2015-03-11 01:00:14.126681 7f41a1ba4880 10 register_cxx_method kvs.assert_size_in_bound flags 2 0x7f41927f8d40
>   -252> 2015-03-11 01:00:14.126684 7f41a1ba4880 10 register_cxx_method kvs.omap_insert flags 2 0x7f41927fcb50
>   -251> 2015-03-11 01:00:14.126688 7f41a1ba4880 10 register_cxx_method kvs.create_with_omap flags 2 0x7f41927fc0e0
>   -250> 2015-03-11 01:00:14.126691 7f41a1ba4880 10 register_cxx_method kvs.omap_remove flags 2 0x7f41927f9e70
>   -249> 2015-03-11 01:00:14.126696 7f41a1ba4880 10 register_cxx_method kvs.maybe_read_for_balance flags 1 0x7f41927fab70
>   -248> 2015-03-11 01:00:14.126735 7f41a1ba4880 10 _load_class kvs success
>   -247> 2015-03-11 01:00:14.126739 7f41a1ba4880 10 open_all_classes found lock
>   -246> 2015-03-11 01:00:14.126742 7f41a1ba4880 10 _get_class adding new class name lock 0x4c9e1e8
>   -245> 2015-03-11 01:00:14.126745 7f41a1ba4880 10 _load_class lock from /usr/lib64/rados-classes/libcls_lock.so
>   -244> 2015-03-11 01:00:14.126972 7f41a1ba4880 10 register_class lock status 3
>   -243> 2015-03-11 01:00:14.126977 7f41a1ba4880 10 register_cxx_method lock.lock flags 3 0x7f41925ee800
>   -242> 2015-03-11 01:00:14.126981 7f41a1ba4880 10 register_cxx_method lock.unlock flags 3 0x7f41925ed210
>   -241> 2015-03-11 01:00:14.126984 7f41a1ba4880 10 register_cxx_method lock.break_lock flags 3 0x7f41925ece60
>   -240> 2015-03-11 01:00:14.126986 7f41a1ba4880 10 register_cxx_method lock.get_info flags 1 0x7f41925ed610
>   -239> 2015-03-11 01:00:14.126990 7f41a1ba4880 10 register_cxx_method lock.list_locks flags 1 0x7f41925eace0
>   -238> 2015-03-11 01:00:14.126993 7f41a1ba4880 10 register_cxx_method lock.assert_locked flags 1 0x7f41925ec330
>   -237> 2015-03-11 01:00:14.126996 7f41a1ba4880 10 _load_class lock success
>   -236> 2015-03-11 01:00:14.126999 7f41a1ba4880 10 open_all_classes found log
>   -235> 2015-03-11 01:00:14.127002 7f41a1ba4880 10 _get_class adding new class name log 0x4c9e2c8
>   -234> 2015-03-11 01:00:14.127005 7f41a1ba4880 10 _load_class log from /usr/lib64/rados-classes/libcls_log.so
>   -233> 2015-03-11 01:00:14.127213 7f41a1ba4880  1 <cls> cls/log/cls_log.cc:312: Loaded log class!
>   -232> 2015-03-11 01:00:14.127223 7f41a1ba4880 10 register_class log status 3
>   -231> 2015-03-11 01:00:14.127225 7f41a1ba4880 10 register_cxx_method log.add flags 3 0x7f41923e1560
>   -230> 2015-03-11 01:00:14.127228 7f41a1ba4880 10 register_cxx_method log.list flags 1 0x7f41923e2740
>   -229> 2015-03-11 01:00:14.127231 7f41a1ba4880 10 register_cxx_method log.trim flags 3 0x7f41923e1f90
>   -228> 2015-03-11 01:00:14.127234 7f41a1ba4880 10 register_cxx_method log.info flags 1 0x7f41923e1090
>   -227> 2015-03-11 01:00:14.127238 7f41a1ba4880 10 _load_class log success
>   -226> 2015-03-11 01:00:14.127241 7f41a1ba4880 10 open_all_classes found rbd
>   -225> 2015-03-11 01:00:14.127244 7f41a1ba4880 10 _get_class adding new class name rbd 0x4c9e3a8
>   -224> 2015-03-11 01:00:14.127247 7f41a1ba4880 10 _load_class rbd from /usr/lib64/rados-classes/libcls_rbd.so
>   -223> 2015-03-11 01:00:14.127735 7f41a1ba4880 10 register_class rbd status 3
>   -222> 2015-03-11 01:00:14.127745 7f41a1ba4880 10 register_cxx_method rbd.create flags 3 0x7f41921cc5c0
>   -221> 2015-03-11 01:00:14.127748 7f41a1ba4880 10 register_cxx_method rbd.get_features flags 1 0x7f41921d3190
>   -220> 2015-03-11 01:00:14.127751 7f41a1ba4880 10 register_cxx_method rbd.get_size flags 1 0x7f41921d2de0
>   -219> 2015-03-11 01:00:14.127754 7f41a1ba4880 10 register_cxx_method rbd.set_size flags 3 0x7f41921ca7e0
>   -218> 2015-03-11 01:00:14.127758 7f41a1ba4880 10 register_cxx_method rbd.get_snapcontext flags 1 0x7f41921cdb40
>   -217> 2015-03-11 01:00:14.127762 7f41a1ba4880 10 register_cxx_method rbd.get_object_prefix flags 1 0x7f41921c9770
>   -216> 2015-03-11 01:00:14.127766 7f41a1ba4880 10 register_cxx_method rbd.get_snapshot_name flags 1 0x7f41921d1dd0
>   -215> 2015-03-11 01:00:14.127770 7f41a1ba4880 10 register_cxx_method rbd.snapshot_add flags 3 0x7f41921d0830
>   -214> 2015-03-11 01:00:14.127773 7f41a1ba4880 10 register_cxx_method rbd.snapshot_remove flags 3 0x7f41921d1b40
>   -213> 2015-03-11 01:00:14.127776 7f41a1ba4880 10 register_cxx_method rbd.get_all_features flags 1 0x7f41921c78c0
>   -212> 2015-03-11 01:00:14.127779 7f41a1ba4880 10 register_cxx_method rbd.copyup flags 3 0x7f41921c7940
>   -211> 2015-03-11 01:00:14.127782 7f41a1ba4880 10 register_cxx_method rbd.get_parent flags 1 0x7f41921d2080
>   -210> 2015-03-11 01:00:14.127786 7f41a1ba4880 10 register_cxx_method rbd.set_parent flags 3 0x7f41921ccce0
>   -209> 2015-03-11 01:00:14.127789 7f41a1ba4880 10 register_cxx_method rbd.remove_parent flags 3 0x7f41921c9140
>   -208> 2015-03-11 01:00:14.127792 7f41a1ba4880 10 register_cxx_method rbd.set_protection_status flags 3 0x7f41921d2770
>   -207> 2015-03-11 01:00:14.127815 7f41a1ba4880 10 register_cxx_method rbd.get_protection_status flags 1 0x7f41921d2b60
>   -206> 2015-03-11 01:00:14.127818 7f41a1ba4880 10 register_cxx_method rbd.get_stripe_unit_count flags 1 0x7f41921c9330
>   -205> 2015-03-11 01:00:14.127822 7f41a1ba4880 10 register_cxx_method rbd.set_stripe_unit_count flags 3 0x7f41921cc070
>   -204> 2015-03-11 01:00:14.127825 7f41a1ba4880 10 register_cxx_method rbd.get_flags flags 1 0x7f41921d2470
>   -203> 2015-03-11 01:00:14.127828 7f41a1ba4880 10 register_cxx_method rbd.set_flags flags 3 0x7f41921cb370
>   -202> 2015-03-11 01:00:14.127831 7f41a1ba4880 10 register_cxx_method rbd.add_child flags 3 0x7f41921cd790
>   -201> 2015-03-11 01:00:14.127834 7f41a1ba4880 10 register_cxx_method rbd.remove_child flags 3 0x7f41921cdf50
>   -200> 2015-03-11 01:00:14.127838 7f41a1ba4880 10 register_cxx_method rbd.get_children flags 1 0x7f41921cd4e0
>   -199> 2015-03-11 01:00:14.127841 7f41a1ba4880 10 register_cxx_method rbd.get_id flags 1 0x7f41921ca580
>   -198> 2015-03-11 01:00:14.127844 7f41a1ba4880 10 register_cxx_method rbd.set_id flags 3 0x7f41921cb6c0
>   -197> 2015-03-11 01:00:14.127848 7f41a1ba4880 10 register_cxx_method rbd.dir_get_id flags 1 0x7f41921c9f40
>   -196> 2015-03-11 01:00:14.127851 7f41a1ba4880 10 register_cxx_method rbd.dir_get_name flags 1 0x7f41921ca210
>   -195> 2015-03-11 01:00:14.127855 7f41a1ba4880 10 register_cxx_method rbd.dir_list flags 1 0x7f41921cf9f0
>   -194> 2015-03-11 01:00:14.127858 7f41a1ba4880 10 register_cxx_method rbd.dir_add_image flags 3 0x7f41921cf840
>   -193> 2015-03-11 01:00:14.127862 7f41a1ba4880 10 register_cxx_method rbd.dir_remove_image flags 3 0x7f41921c9d60
>   -192> 2015-03-11 01:00:14.127865 7f41a1ba4880 10 register_cxx_method rbd.dir_rename_image flags 3 0x7f41921cf660
>   -191> 2015-03-11 01:00:14.127868 7f41a1ba4880 10 register_cxx_method rbd.object_map_load flags 1 0x7f41921ce5f0
>   -190> 2015-03-11 01:00:14.127872 7f41a1ba4880 10 register_cxx_method rbd.object_map_resize flags 3 0x7f41921ceec0
>   -189> 2015-03-11 01:00:14.127875 7f41a1ba4880 10 register_cxx_method rbd.object_map_update flags 3 0x7f41921ce720
>   -188> 2015-03-11 01:00:14.127879 7f41a1ba4880 10 register_cxx_method rbd.snap_list flags 1 0x7f41921c8540
>   -187> 2015-03-11 01:00:14.127882 7f41a1ba4880 10 register_cxx_method rbd.snap_add flags 3 0x7f41921cacc0
>   -186> 2015-03-11 01:00:14.127885 7f41a1ba4880 10 register_cxx_method rbd.snap_remove flags 3 0x7f41921cb910
>   -185> 2015-03-11 01:00:14.127891 7f41a1ba4880 10 _load_class rbd success
>   -184> 2015-03-11 01:00:14.127894 7f41a1ba4880 10 open_all_classes found refcount
>   -183> 2015-03-11 01:00:14.127897 7f41a1ba4880 10 _get_class adding new class name refcount 0x4c9e488
>   -182> 2015-03-11 01:00:14.127900 7f41a1ba4880 10 _load_class refcount from /usr/lib64/rados-classes/libcls_refcount.so
>   -181> 2015-03-11 01:00:14.132905 7f41a1ba4880  1 <cls> cls/refcount/cls_refcount.cc:231: Loaded refcount class!
>   -180> 2015-03-11 01:00:14.132942 7f41a1ba4880 10 register_class refcount status 3
>   -179> 2015-03-11 01:00:14.132946 7f41a1ba4880 10 register_cxx_method refcount.get flags 3 0x7f4191f32be0
>   -178> 2015-03-11 01:00:14.132957 7f41a1ba4880 10 register_cxx_method refcount.put flags 3 0x7f4191f31b30
>   -177> 2015-03-11 01:00:14.132960 7f41a1ba4880 10 register_cxx_method refcount.set flags 3 0x7f4191f32510
>   -176> 2015-03-11 01:00:14.132963 7f41a1ba4880 10 register_cxx_method refcount.read flags 1 0x7f4191f31fb0
>   -175> 2015-03-11 01:00:14.132970 7f41a1ba4880 10 _load_class refcount success
>   -174> 2015-03-11 01:00:14.132977 7f41a1ba4880 10 open_all_classes found replica_log
>   -173> 2015-03-11 01:00:14.132984 7f41a1ba4880 10 _get_class adding new class name replica_log 0x4c9e568
>   -172> 2015-03-11 01:00:14.132987 7f41a1ba4880 10 _load_class replica_log from /usr/lib64/rados-classes/libcls_replica_log.so
>   -171> 2015-03-11 01:00:14.133259 7f41a1ba4880  1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class!
>   -170> 2015-03-11 01:00:14.133269 7f41a1ba4880 10 register_class replica_log status 3
>   -169> 2015-03-11 01:00:14.133271 7f41a1ba4880 10 register_cxx_method replica_log.set flags 3 0x7f4191c7f920
>   -168> 2015-03-11 01:00:14.133276 7f41a1ba4880 10 register_cxx_method replica_log.get flags 1 0x7f4191c809f0
>   -167> 2015-03-11 01:00:14.133279 7f41a1ba4880 10 register_cxx_method replica_log.delete flags 3 0x7f4191c80400
>   -166> 2015-03-11 01:00:14.133283 7f41a1ba4880 10 _load_class replica_log success
>   -165> 2015-03-11 01:00:14.133285 7f41a1ba4880 10 open_all_classes found rgw
>   -164> 2015-03-11 01:00:14.133288 7f41a1ba4880 10 _get_class adding new class name rgw 0x4c9e728
>   -163> 2015-03-11 01:00:14.133291 7f41a1ba4880 10 _load_class rgw from /usr/lib64/rados-classes/libcls_rgw.so
>   -162> 2015-03-11 01:00:14.138677 7f41a1ba4880  1 <cls> cls/rgw/cls_rgw.cc:3046: Loaded rgw class!
>   -161> 2015-03-11 01:00:14.138712 7f41a1ba4880 10 register_class rgw status 3
>   -160> 2015-03-11 01:00:14.138717 7f41a1ba4880 10 register_cxx_method rgw.bucket_init_index flags 3 0x7f41919c0860
>   -159> 2015-03-11 01:00:14.138725 7f41a1ba4880 10 register_cxx_method rgw.bucket_set_tag_timeout flags 3 0x7f41919c4a40
>   -158> 2015-03-11 01:00:14.138728 7f41a1ba4880 10 register_cxx_method rgw.bucket_list flags 1 0x7f41919c8160
>   -157> 2015-03-11 01:00:14.138732 7f41a1ba4880 10 register_cxx_method rgw.bucket_check_index flags 1 0x7f41919ca300
>   -156> 2015-03-11 01:00:14.138735 7f41a1ba4880 10 register_cxx_method rgw.bucket_rebuild_index flags 3 0x7f41919ca1a0
>   -155> 2015-03-11 01:00:14.138740 7f41a1ba4880 10 register_cxx_method rgw.bucket_prepare_op flags 3 0x7f41919c5180
>   -154> 2015-03-11 01:00:14.138745 7f41a1ba4880 10 register_cxx_method rgw.bucket_complete_op flags 3 0x7f41919c5980
>   -153> 2015-03-11 01:00:14.138748 7f41a1ba4880 10 register_cxx_method rgw.bucket_link_olh flags 3 0x7f41919cc980
>   -152> 2015-03-11 01:00:14.138751 7f41a1ba4880 10 register_cxx_method rgw.bucket_unlink_instance flags 3 0x7f41919cbb80
>   -151> 2015-03-11 01:00:14.138754 7f41a1ba4880 10 register_cxx_method rgw.bucket_read_olh_log flags 1 0x7f41919cde40
>   -150> 2015-03-11 01:00:14.138757 7f41a1ba4880 10 register_cxx_method rgw.bucket_trim_olh_log flags 3 0x7f41919cb2d0
>   -149> 2015-03-11 01:00:14.138760 7f41a1ba4880 10 register_cxx_method rgw.bucket_clear_olh flags 3 0x7f41919cacd0
>   -148> 2015-03-11 01:00:14.138763 7f41a1ba4880 10 register_cxx_method rgw.obj_remove flags 3 0x7f41919c3ff0
>   -147> 2015-03-11 01:00:14.138771 7f41a1ba4880 10 register_cxx_method rgw.obj_check_attrs_prefix flags 1 0x7f41919c0b60
>   -146> 2015-03-11 01:00:14.138774 7f41a1ba4880 10 register_cxx_method rgw.bi_get flags 1 0x7f41919bf050
>   -145> 2015-03-11 01:00:14.138777 7f41a1ba4880 10 register_cxx_method rgw.bi_put flags 3 0x7f41919be700
>   -144> 2015-03-11 01:00:14.138780 7f41a1ba4880 10 register_cxx_method rgw.bi_list flags 1 0x7f41919c6790
>   -143> 2015-03-11 01:00:14.138784 7f41a1ba4880 10 register_cxx_method rgw.bi_log_list flags 1 0x7f41919c2180
>   -142> 2015-03-11 01:00:14.138787 7f41a1ba4880 10 register_cxx_method rgw.bi_log_trim flags 3 0x7f41919c26e0
>   -141> 2015-03-11 01:00:14.138791 7f41a1ba4880 10 register_cxx_method rgw.dir_suggest_changes flags 3 0x7f41919c9260
>   -140> 2015-03-11 01:00:14.138794 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_add flags 3 0x7f41919ce6a0
>   -139> 2015-03-11 01:00:14.138797 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_read flags 1 0x7f41919c31c0
>   -138> 2015-03-11 01:00:14.138799 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_trim flags 3 0x7f41919c3950
>   -137> 2015-03-11 01:00:14.138802 7f41a1ba4880 10 register_cxx_method rgw.gc_set_entry flags 3 0x7f41919c01b0
>   -136> 2015-03-11 01:00:14.138806 7f41a1ba4880 10 register_cxx_method rgw.gc_defer_entry flags 3 0x7f41919bfe90
>   -135> 2015-03-11 01:00:14.138809 7f41a1ba4880 10 register_cxx_method rgw.gc_list flags 1 0x7f41919c0f80
>   -134> 2015-03-11 01:00:14.138812 7f41a1ba4880 10 register_cxx_method rgw.gc_remove flags 3 0x7f41919c0420
>   -133> 2015-03-11 01:00:14.138815 7f41a1ba4880 10 _load_class rgw success
>   -132> 2015-03-11 01:00:14.138820 7f41a1ba4880 10 open_all_classes found statelog
>   -131> 2015-03-11 01:00:14.138826 7f41a1ba4880 10 _get_class adding new class name statelog 0x4c9e808
>   -130> 2015-03-11 01:00:14.138830 7f41a1ba4880 10 _load_class statelog from /usr/lib64/rados-classes/libcls_statelog.so
>   -129> 2015-03-11 01:00:14.139131 7f41a1ba4880  1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class!
>   -128> 2015-03-11 01:00:14.139141 7f41a1ba4880 10 register_class statelog status 3
>   -127> 2015-03-11 01:00:14.139143 7f41a1ba4880 10 register_cxx_method statelog.add flags 3 0x7f4191700bc0
>   -126> 2015-03-11 01:00:14.139146 7f41a1ba4880 10 register_cxx_method statelog.list flags 1 0x7f4191700f40
>   -125> 2015-03-11 01:00:14.139150 7f41a1ba4880 10 register_cxx_method statelog.remove flags 3 0x7f41916ffa80
>   -124> 2015-03-11 01:00:14.139153 7f41a1ba4880 10 register_cxx_method statelog.check_state flags 1 0x7f41917002d0
>   -123> 2015-03-11 01:00:14.139175 7f41a1ba4880 10 _load_class statelog success
>   -122> 2015-03-11 01:00:14.139178 7f41a1ba4880 10 open_all_classes found user
>   -121> 2015-03-11 01:00:14.139182 7f41a1ba4880 10 _get_class adding new class name user 0x4c9e8e8
>   -120> 2015-03-11 01:00:14.139185 7f41a1ba4880 10 _load_class user from /usr/lib64/rados-classes/libcls_user.so
>   -119> 2015-03-11 01:00:14.139377 7f41a1ba4880  1 <cls> cls/user/cls_user.cc:367: Loaded user class!
>   -118> 2015-03-11 01:00:14.139386 7f41a1ba4880 10 register_class user status 3
>   -117> 2015-03-11 01:00:14.139388 7f41a1ba4880 10 register_cxx_method user.set_buckets_info flags 3 0x7f41914f5180
>   -116> 2015-03-11 01:00:14.139392 7f41a1ba4880 10 register_cxx_method user.complete_stats_sync flags 3 0x7f41914f4aa0
>   -115> 2015-03-11 01:00:14.139394 7f41a1ba4880 10 register_cxx_method user.remove_bucket flags 3 0x7f41914f3d30
>   -114> 2015-03-11 01:00:14.139397 7f41a1ba4880 10 register_cxx_method user.list_buckets flags 1 0x7f41914f64c0
>   -113> 2015-03-11 01:00:14.139400 7f41a1ba4880 10 register_cxx_method user.get_header flags 1 0x7f41914f2d50
>   -112> 2015-03-11 01:00:14.139403 7f41a1ba4880 10 _load_class user success
>   -111> 2015-03-11 01:00:14.139406 7f41a1ba4880 10 open_all_classes found version
>   -110> 2015-03-11 01:00:14.139409 7f41a1ba4880 10 _get_class adding new class name version 0x4c9e9c8
>   -109> 2015-03-11 01:00:14.139412 7f41a1ba4880 10 _load_class version from /usr/lib64/rados-classes/libcls_version.so
>   -108> 2015-03-11 01:00:14.139604 7f41a1ba4880  1 <cls> cls/version/cls_version.cc:227: Loaded version class!
>   -107> 2015-03-11 01:00:14.139608 7f41a1ba4880 10 register_class version status 3
>   -106> 2015-03-11 01:00:14.139611 7f41a1ba4880 10 register_cxx_method version.set flags 3 0x7f41912ebe60
>   -105> 2015-03-11 01:00:14.139615 7f41a1ba4880 10 register_cxx_method version.inc flags 3 0x7f41912ecc80
>   -104> 2015-03-11 01:00:14.139618 7f41a1ba4880 10 register_cxx_method version.inc_conds flags 3 0x7f41912ecc80
>   -103> 2015-03-11 01:00:14.139621 7f41a1ba4880 10 register_cxx_method version.read flags 1 0x7f41912ebb60
>   -102> 2015-03-11 01:00:14.139624 7f41a1ba4880 10 register_cxx_method version.check_conds flags 1 0x7f41912ec260
>   -101> 2015-03-11 01:00:14.139627 7f41a1ba4880 10 _load_class version success
>   -100> 2015-03-11 01:00:14.139682 7f41a1ba4880 20 osd.3 0 get_map 2757 - loading and decoding 0x4b4cb40
>    -99> 2015-03-11 01:00:14.139693 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) read meta/a143428/osdmap.2757/0//-1 0~0
>    -98> 2015-03-11 01:00:14.139804 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) FileStore::read meta/a143428/osdmap.2757/0//-1 0~3076/3076
>    -97> 2015-03-11 01:00:14.139818 7f41a1ba4880 10 osd.3 0 add_map_bl 2757 3076 bytes
>    -96> 2015-03-11 01:00:14.139908 7f41a1ba4880  0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients
>    -95> 2015-03-11 01:00:14.139914 7f41a1ba4880  0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons
>    -94> 2015-03-11 01:00:14.139919 7f41a1ba4880  0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds
>    -93> 2015-03-11 01:00:14.139922 7f41a1ba4880 10 osd.3 2757 create_recoverystate_perf
>    -92> 2015-03-11 01:00:14.139939 7f41a1ba4880  0 osd.3 2757 load_pgs
>    -91> 2015-03-11 01:00:14.139942 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) list_collections
>    -90> 2015-03-11 01:00:14.139981 7f41a1ba4880 10 osd.3 2757 load_pgs ignoring unrecognized meta
>    -89> 2015-03-11 01:00:14.139985 7f41a1ba4880  0 osd.3 2757 load_pgs opened 0 pgs
>    -88> 2015-03-11 01:00:14.139989 7f41a1ba4880 10 osd.3 2757 build_past_intervals_parallel nothing to build
>    -87> 2015-03-11 01:00:14.139992 7f41a1ba4880  2 osd.3 2757 superblock: i am osd.3
>    -86> 2015-03-11 01:00:14.139995 7f41a1ba4880 10 osd.3 2757 create_logger
>    -85> 2015-03-11 01:00:14.140126 7f41a1ba4880  1 accepter.accepter.start
>    -84> 2015-03-11 01:00:14.140269 7f41a1ba4880  1 accepter.accepter.start
>    -83> 2015-03-11 01:00:14.140434 7f41a1ba4880  1 accepter.accepter.start
>    -82> 2015-03-11 01:00:14.140575 7f41a1ba4880  1 accepter.accepter.start
>    -81> 2015-03-11 01:00:14.140676 7f41a1ba4880 10 monclient(hunting): init
>    -80> 2015-03-11 01:00:14.140691 7f41a1ba4880  5 adding auth protocol: none
>    -79> 2015-03-11 01:00:14.140697 7f41a1ba4880 10 monclient(hunting): auth_supported 1 method none
>    -78> 2015-03-11 01:00:14.140803 7f41a1ba4880 10 log_channel(default) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info)
>    -77> 2015-03-11 01:00:14.140813 7f41a1ba4880 -1 osd.3 2757 log_to_monitors {default=true}
>    -76> 2015-03-11 01:00:14.141325 7f41a1ba4880 10 osd.3 2757 set_disk_tp_priority class  priority -1
>    -75> 2015-03-11 01:00:14.141461 7f41a1ba4880 10 monclient(hunting): renew_subs
>    -74> 2015-03-11 01:00:14.141466 7f41a1ba4880 10 monclient(hunting): _reopen_session rank -1 name 
>    -73> 2015-03-11 01:00:14.141564 7f41a1ba4880 10 monclient(hunting): picked mon.a con 0x4cbe3c0 addr 192.168.0.10:6789/0
>    -72> 2015-03-11 01:00:14.141594 7f41a1ba4880 10 monclient(hunting): _send_mon_message to mon.a at 192.168.0.10:6789/0
>    -71> 2015-03-11 01:00:14.141599 7f41a1ba4880  1 -- 0.0.0.0:6800/2840 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x4b4cd80 con 0x4cbe3c0
>    -70> 2015-03-11 01:00:14.141625 7f41a1ba4880 10 monclient(hunting): renew_subs
>    -69> 2015-03-11 01:00:14.142020 7f4180ac7700 20 osd.3 2757 update_osd_stat osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist [])
>    -68> 2015-03-11 01:00:14.142048 7f4180ac7700  5 osd.3 2757 heartbeat: osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist [])
>    -67> 2015-03-11 01:00:14.142221 7f417e2c2700 10 osd.3 2757 agent_entry start
>    -66> 2015-03-11 01:00:14.142241 7f417e2c2700 20 osd.3 2757 agent_entry empty queue
>    -65> 2015-03-11 01:00:14.158932 7f41a1ba0700  1 -- 192.168.0.250:6800/2840 learned my addr 192.168.0.250:6800/2840
>    -64> 2015-03-11 01:00:14.159058 7f41a1ba0700 10 osd.3 2757  new session (outgoing) 0x4b92280 con=0x4b602c0 addr=192.168.0.250:6800/2840
>    -63> 2015-03-11 01:00:14.159166 7f41a1ba0700 10 osd.3 2757 OSD::ms_get_authorizer type=mon
>    -62> 2015-03-11 01:00:14.173274 7f41912e8700 10 osd.3 2757 ms_handle_connect on mon
>    -61> 2015-03-11 01:00:14.173332 7f41912e8700 20 osd.3 2757 send_pg_stats
>    -60> 2015-03-11 01:00:14.173348 7f41912e8700 10 monclient(hunting): renew_subs
>    -59> 2015-03-11 01:00:14.175783 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x4b4f840 con 0x4cbe3c0
>    -58> 2015-03-11 01:00:14.175871 7f41912e8700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
>    -57> 2015-03-11 01:00:14.175889 7f41912e8700 10 monclient(hunting):  got monmap 1, mon.a is now rank 0
>    -56> 2015-03-11 01:00:14.175893 7f41912e8700 10 monclient(hunting): dump:
> epoch 1
> fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57
> last_changed 2013-09-11 22:41:30.819687
> created 2013-09-11 22:41:30.819687
> 0: 192.168.0.10:6789/0 mon.a
> 
>    -55> 2015-03-11 01:00:14.175940 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (335930126 0 0) 0x4b4fa80 con 0x4cbe3c0
>    -54> 2015-03-11 01:00:14.175985 7f41912e8700 10 monclient(hunting): my global_id is 14511547
>    -53> 2015-03-11 01:00:14.175988 7f41912e8700  1 monclient(hunting): found mon.a
>    -52> 2015-03-11 01:00:14.175991 7f41912e8700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
>    -51> 2015-03-11 01:00:14.175997 7f41912e8700  1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x4b31a00 con 0x4cbe3c0
>    -50> 2015-03-11 01:00:14.176035 7f41912e8700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
>    -49> 2015-03-11 01:00:14.176059 7f41912e8700  1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0,osdmap=2757}) v2 -- ?+0 0x4b32800 con 0x4cbe3c0
>    -48> 2015-03-11 01:00:14.176077 7f41912e8700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2015-03-11 00:59:44.176076)
>    -47> 2015-03-11 01:00:14.176109 7f41a1ba4880  5 monclient: authenticate success, global_id 14511547
>    -46> 2015-03-11 01:00:14.176147 7f41a1ba4880 10 osd.3 2757 ensuring pgs have consumed prior maps
>    -45> 2015-03-11 01:00:14.176170 7f41a1ba4880  7 osd.3 2757 consume_map version 2757
>    -44> 2015-03-11 01:00:14.176187 7f41a1ba4880  0 osd.3 2757 done with init, starting boot process
>    -43> 2015-03-11 01:00:14.176191 7f41a1ba4880 10 osd.3 2757 start_boot - have maps 2757..2757
>    -42> 2015-03-11 01:00:14.176197 7f41a1ba4880 10 monclient: get_version osdmap req 0x4bad1c0
>    -41> 2015-03-11 01:00:14.176207 7f41a1ba4880 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
>    -40> 2015-03-11 01:00:14.176221 7f41a1ba4880  1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x4bc41e0 con 0x4cbe3c0
>    -39> 2015-03-11 01:00:14.176628 7f41a1ba4880  5 asok(0x4b50000) register_command status hook 0x4b04290
>    -38> 2015-03-11 01:00:14.176655 7f41a1ba4880  5 asok(0x4b50000) register_command flush_journal hook 0x4b04290
>    -37> 2015-03-11 01:00:14.176666 7f41a1ba4880  5 asok(0x4b50000) register_command dump_ops_in_flight hook 0x4b04290
>    -36> 2015-03-11 01:00:14.176686 7f41a1ba4880  5 asok(0x4b50000) register_command ops hook 0x4b04290
>    -35> 2015-03-11 01:00:14.176705 7f41a1ba4880  5 asok(0x4b50000) register_command dump_historic_ops hook 0x4b04290
>    -34> 2015-03-11 01:00:14.176723 7f41a1ba4880  5 asok(0x4b50000) register_command dump_op_pq_state hook 0x4b04290
>    -33> 2015-03-11 01:00:14.176737 7f41a1ba4880  5 asok(0x4b50000) register_command dump_blacklist hook 0x4b04290
>    -32> 2015-03-11 01:00:14.176746 7f41a1ba4880  5 asok(0x4b50000) register_command dump_watchers hook 0x4b04290
>    -31> 2015-03-11 01:00:14.176760 7f41a1ba4880  5 asok(0x4b50000) register_command dump_reservations hook 0x4b04290
>    -30> 2015-03-11 01:00:14.176770 7f41a1ba4880  5 asok(0x4b50000) register_command get_latest_osdmap hook 0x4b04290
>    -29> 2015-03-11 01:00:14.176786 7f41a1ba4880  5 asok(0x4b50000) register_command setomapval hook 0x4bad260
>    -28> 2015-03-11 01:00:14.176800 7f41a1ba4880  5 asok(0x4b50000) register_command rmomapkey hook 0x4bad260
>    -27> 2015-03-11 01:00:14.176817 7f41a1ba4880  5 asok(0x4b50000) register_command setomapheader hook 0x4bad260
>    -26> 2015-03-11 01:00:14.176834 7f41a1ba4880  5 asok(0x4b50000) register_command getomap hook 0x4bad260
>    -25> 2015-03-11 01:00:14.176850 7f41a1ba4880  5 asok(0x4b50000) register_command truncobj hook 0x4bad260
>    -24> 2015-03-11 01:00:14.176866 7f41a1ba4880  5 asok(0x4b50000) register_command injectdataerr hook 0x4bad260
>    -23> 2015-03-11 01:00:14.176878 7f41a1ba4880  5 asok(0x4b50000) register_command injectmdataerr hook 0x4bad260
>    -22> 2015-03-11 01:00:14.178539 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 3 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x4b4fcc0 con 0x4cbe3c0
>    -21> 2015-03-11 01:00:14.178650 7f41912e8700 10 osd.3 2757 do_waiters -- start
>    -20> 2015-03-11 01:00:14.178658 7f41912e8700 10 osd.3 2757 do_waiters -- finish
>    -19> 2015-03-11 01:00:14.178661 7f41912e8700 20 osd.3 2757 _dispatch 0x4b4fcc0 osd_map(3277..3277 src has 2757..3277) v3
>    -18> 2015-03-11 01:00:14.178888 7f41912e8700  3 osd.3 2757 handle_osd_map epochs [3277,3277], i have 2757, src has [2757,3277]
>    -17> 2015-03-11 01:00:14.178906 7f41912e8700 10 osd.3 2757 handle_osd_map message skips epochs 2758..3276
>    -16> 2015-03-11 01:00:14.178914 7f41912e8700 10 monclient: renew_subs
>    -15> 2015-03-11 01:00:14.178922 7f41912e8700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
>    -14> 2015-03-11 01:00:14.178954 7f41912e8700  1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=2758}) v2 -- ?+0 0x4b31a00 con 0x4cbe3c0
>    -13> 2015-03-11 01:00:14.178991 7f41912e8700 10 osd.3 2757 do_waiters -- start
>    -12> 2015-03-11 01:00:14.178996 7f41912e8700 10 osd.3 2757 do_waiters -- finish
>    -11> 2015-03-11 01:00:14.179012 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 4 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x4b4f840 con 0x4cbe3c0
>    -10> 2015-03-11 01:00:14.179030 7f41912e8700 10 monclient: handle_monmap mon_map magic: 0 v1
>     -9> 2015-03-11 01:00:14.179045 7f41912e8700 10 monclient:  got monmap 1, mon.a is now rank 0
>     -8> 2015-03-11 01:00:14.179051 7f41912e8700 10 monclient: dump:
> epoch 1
> fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57
> last_changed 2013-09-11 22:41:30.819687
> created 2013-09-11 22:41:30.819687
> 0: 192.168.0.10:6789/0 mon.a
> 
>     -7> 2015-03-11 01:00:14.184757 7f41912e8700  1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(2757..2857 src has 2757..3277) v3 ==== 37648+0+0 (2648067016 0 0) 0x4b4fa80 con 0x4cbe3c0
>     -6> 2015-03-11 01:00:14.184831 7f41912e8700 10 osd.3 2757 do_waiters -- start
>     -5> 2015-03-11 01:00:14.184838 7f41912e8700 10 osd.3 2757 do_waiters -- finish
>     -4> 2015-03-11 01:00:14.184841 7f41912e8700 20 osd.3 2757 _dispatch 0x4b4fa80 osd_map(2757..2857 src has 2757..3277) v3
>     -3> 2015-03-11 01:00:14.184872 7f41912e8700  3 osd.3 2757 handle_osd_map epochs [2757,2857], i have 2757, src has [2757,3277]
>     -2> 2015-03-11 01:00:14.184884 7f41912e8700 10 osd.3 2757 handle_osd_map  got inc map for epoch 2758
>     -1> 2015-03-11 01:00:14.185229 7f41912e8700 10 osd.3 2757 handle_osd_map  got inc map for epoch 2759
>      0> 2015-03-11 01:00:14.195165 7f41912e8700 -1 *** Caught signal (Aborted) **
>  in thread 7f41912e8700
> 
>  ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
>  1: /usr/bin/ceph-osd() [0xac7cea]
>  2: (()+0x10050) [0x7f41a0aaf050]
>  3: (gsignal()+0x37) [0x7f419ef820f7]
>  4: (abort()+0x13a) [0x7f419ef834ca]
>  5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f419f867fe5]
>  6: (()+0x63186) [0x7f419f866186]
>  7: (()+0x631b3) [0x7f419f8661b3]
>  8: (()+0x633d2) [0x7f419f8663d2]
>  9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
>  10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
>  11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
>  12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
>  13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
>  14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
>  15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
>  16: (DispatchQueue::entry()+0x649) [0xc6fe09]
>  17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
>  18: (()+0x83a4) [0x7f41a0aa73a4]
>  19: (clone()+0x6d) [0x7f419f031a4d]
>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
> 
> --- logging levels ---
>    0/ 5 none
>    0/ 1 lockdep
>    0/ 1 context
>    1/ 1 crush
>    1/ 5 mds
>    1/ 5 mds_balancer
>    1/ 5 mds_locker
>    1/ 5 mds_log
>    1/ 5 mds_log_expire
>    1/ 5 mds_migrator
>    0/ 1 buffer
>    0/ 1 timer
>    0/ 1 filer
>    0/ 1 striper
>    0/ 1 objecter
>    0/ 5 rados
>    0/ 5 rbd
>    0/ 5 rbd_replay
>    0/ 5 journaler
>    0/ 5 objectcacher
>    0/ 5 client
>   20/20 osd
>    0/ 5 optracker
>    0/ 5 objclass
>   20/20 filestore
>    1/ 3 keyvaluestore
>    1/ 3 journal
>    1/ 1 ms
>    1/ 5 mon
>    0/10 monc
>    1/ 5 paxos
>    0/ 5 tp
>    1/ 5 auth
>    1/ 5 crypto
>    1/ 1 finisher
>    1/ 5 heartbeatmap
>    1/ 5 perfcounter
>    1/ 5 rgw
>    1/10 civetweb
>    1/ 5 javaclient
>    1/ 5 asok
>    1/ 1 throttle
>    0/ 0 refs
>    1/ 5 xio
>   -2/-2 (syslog threshold)
>   -1/-1 (stderr threshold)
>   max_recent     10000
>   max_new         1000
>   log_file /var/log/ceph/ceph-osd.3.log
> --- end dump of recent events ---
> 
> 
> -----Original Message-----
> From: Samuel Just [mailto:sjust@xxxxxxxxxx] 
> Sent: Wednesday, 11 March 2015 12:59 AM
> To: Malcolm Haak
> Cc: ceph-users@xxxxxxxxxxxxxx
> Subject: Re:  Issues with fresh 0.93 OSD adding to existing cluster
> 
> Can you reproduce this with
> 
> debug osd = 20
> debug filestore = 20
> debug ms = 1
> 
> on the crashing osd?  Also, what sha1 are the other osds and mons running?
> -Sam
> 
> ----- Original Message -----
> From: "Malcolm Haak" <malcolm@xxxxxxx>
> To: ceph-users@xxxxxxxxxxxxxx
> Sent: Tuesday, March 10, 2015 3:28:26 AM
> Subject:  Issues with fresh 0.93 OSD adding to existing cluster
> 
> Hi all,
> 
> I've just attempted to add a new node and OSD to an existing ceph cluster (it's a small one I use as a NAS at home, not like the big production ones I normally work on) and it seems to be throwing some odd errors...
> 
> Just looking for where to poke it next... 
> 
> Log is below,
> 
> It's a two node cluster with 3 osd's in node A and one osd in the new node (It's going to have more eventually and node one will be retired after node three gets added) And I've hit a weird snag.
> 
> I was running 0.80 but I ran into the 'Invalid Command' bug on the new node so I opted to jump to the latest code with the required patches already. 
> 
> Please let me know what else you need..
> 
> This is the log content when attempting to start the new OSD:
> 
> 2015-03-10 19:28:48.795318 7f0774108880  0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 10810
> 2015-03-10 19:28:48.817803 7f0774108880  0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342)
> 2015-03-10 19:28:48.866862 7f0774108880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work
> 2015-03-10 19:28:48.866920 7f0774108880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
> 2015-03-10 19:28:48.905069 7f0774108880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
> 2015-03-10 19:28:48.905467 7f0774108880  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5
> 2015-03-10 19:28:49.077872 7f0774108880  0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
> 2015-03-10 19:28:49.078321 7f0774108880 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
> 2015-03-10 19:28:49.078328 7f0774108880  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> 2015-03-10 19:28:49.079721 7f0774108880  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> 2015-03-10 19:28:49.080948 7f0774108880  0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
> 2015-03-10 19:28:49.094194 7f0774108880  0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients
> 2015-03-10 19:28:49.094211 7f0774108880  0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons
> 2015-03-10 19:28:49.094217 7f0774108880  0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds
> 2015-03-10 19:28:49.094235 7f0774108880  0 osd.3 2757 load_pgs
> 2015-03-10 19:28:49.094279 7f0774108880  0 osd.3 2757 load_pgs opened 0 pgs
> 2015-03-10 19:28:49.095121 7f0774108880 -1 osd.3 2757 log_to_monitors {default=true}
> 2015-03-10 19:28:49.134104 7f0774108880  0 osd.3 2757 done with init, starting boot process
> 2015-03-10 19:28:49.149994 7f076384c700 -1 *** Caught signal (Aborted) **  in thread 7f076384c700
> 
>  ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
>  1: /usr/bin/ceph-osd() [0xac7cea]
>  2: (()+0x10050) [0x7f0773013050]
>  3: (gsignal()+0x37) [0x7f07714e60f7]
>  4: (abort()+0x13a) [0x7f07714e74ca]
>  5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f0771dcbfe5]
>  6: (()+0x63186) [0x7f0771dca186]
>  7: (()+0x631b3) [0x7f0771dca1b3]
>  8: (()+0x633d2) [0x7f0771dca3d2]
>  9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
>  10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
>  11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
>  12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
>  13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
>  14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
>  15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
>  16: (DispatchQueue::entry()+0x649) [0xc6fe09]
>  17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
>  18: (()+0x83a4) [0x7f077300b3a4]
>  19: (clone()+0x6d) [0x7f0771595a4d]
>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
> 
> --- begin dump of recent events ---
>   -135> 2015-03-10 19:28:48.790490 7f0774108880  5 asok(0x4200000) register_command perfcounters_dump hook 0x41b4030
>   -134> 2015-03-10 19:28:48.790565 7f0774108880  5 asok(0x4200000) register_command 1 hook 0x41b4030
>   -133> 2015-03-10 19:28:48.790571 7f0774108880  5 asok(0x4200000) register_command perf dump hook 0x41b4030
>   -132> 2015-03-10 19:28:48.790583 7f0774108880  5 asok(0x4200000) register_command perfcounters_schema hook 0x41b4030
>   -131> 2015-03-10 19:28:48.790599 7f0774108880  5 asok(0x4200000) register_command 2 hook 0x41b4030
>   -130> 2015-03-10 19:28:48.790605 7f0774108880  5 asok(0x4200000) register_command perf schema hook 0x41b4030
>   -129> 2015-03-10 19:28:48.790635 7f0774108880  5 asok(0x4200000) register_command perf reset hook 0x41b4030
>   -128> 2015-03-10 19:28:48.790646 7f0774108880  5 asok(0x4200000) register_command config show hook 0x41b4030
>   -127> 2015-03-10 19:28:48.790652 7f0774108880  5 asok(0x4200000) register_command config set hook 0x41b4030
>   -126> 2015-03-10 19:28:48.790660 7f0774108880  5 asok(0x4200000) register_command config get hook 0x41b4030
>   -125> 2015-03-10 19:28:48.790672 7f0774108880  5 asok(0x4200000) register_command config diff hook 0x41b4030
>   -124> 2015-03-10 19:28:48.790682 7f0774108880  5 asok(0x4200000) register_command log flush hook 0x41b4030
>   -123> 2015-03-10 19:28:48.790689 7f0774108880  5 asok(0x4200000) register_command log dump hook 0x41b4030
>   -122> 2015-03-10 19:28:48.790698 7f0774108880  5 asok(0x4200000) register_command log reopen hook 0x41b4030
>   -121> 2015-03-10 19:28:48.795318 7f0774108880  0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 10810
>   -120> 2015-03-10 19:28:48.796071 7f0774108880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/10810 need_addr=1
>   -119> 2015-03-10 19:28:48.796123 7f0774108880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/10810 need_addr=1
>   -118> 2015-03-10 19:28:48.796144 7f0774108880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/10810 need_addr=1
>   -117> 2015-03-10 19:28:48.796160 7f0774108880  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/10810 need_addr=1
>   -116> 2015-03-10 19:28:48.799414 7f0774108880  5 asok(0x4200000) init /var/run/ceph/ceph-osd.3.asok
>   -115> 2015-03-10 19:28:48.799446 7f0774108880  5 asok(0x4200000) bind_and_listen /var/run/ceph/ceph-osd.3.asok
>   -114> 2015-03-10 19:28:48.799644 7f0774108880  5 asok(0x4200000) register_command 0 hook 0x41b00c0
>   -113> 2015-03-10 19:28:48.799659 7f0774108880  5 asok(0x4200000) register_command version hook 0x41b00c0
>   -112> 2015-03-10 19:28:48.799665 7f0774108880  5 asok(0x4200000) register_command git_version hook 0x41b00c0
>   -111> 2015-03-10 19:28:48.799671 7f0774108880  5 asok(0x4200000) register_command help hook 0x41b4110
>   -110> 2015-03-10 19:28:48.799678 7f0774108880  5 asok(0x4200000) register_command get_command_descriptions hook 0x41b4120
>   -109> 2015-03-10 19:28:48.799760 7f0774108880 10 monclient(hunting): build_initial_monmap
>   -108> 2015-03-10 19:28:48.799757 7f076f4e1700  5 asok(0x4200000) entry start
>   -107> 2015-03-10 19:28:48.816916 7f0774108880  5 adding auth protocol: none
>   -106> 2015-03-10 19:28:48.816948 7f0774108880  5 adding auth protocol: none
>   -105> 2015-03-10 19:28:48.817130 7f0774108880  5 asok(0x4200000) register_command objecter_requests hook 0x41b41b0
>   -104> 2015-03-10 19:28:48.817273 7f0774108880  1 -- 0.0.0.0:6800/10810 messenger.start
>   -103> 2015-03-10 19:28:48.817334 7f0774108880  1 -- :/0 messenger.start
>   -102> 2015-03-10 19:28:48.817378 7f0774108880  1 -- 0.0.0.0:6803/10810 messenger.start
>   -101> 2015-03-10 19:28:48.817442 7f0774108880  1 -- 0.0.0.0:6802/10810 messenger.start
>   -100> 2015-03-10 19:28:48.817494 7f0774108880  1 -- 0.0.0.0:6801/10810 messenger.start
>    -99> 2015-03-10 19:28:48.817546 7f0774108880  1 -- :/0 messenger.start
>    -98> 2015-03-10 19:28:48.817666 7f0774108880  2 osd.3 0 mounting /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
>    -97> 2015-03-10 19:28:48.817803 7f0774108880  0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342)
>    -96> 2015-03-10 19:28:48.866862 7f0774108880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work
>    -95> 2015-03-10 19:28:48.866920 7f0774108880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
>    -94> 2015-03-10 19:28:48.905069 7f0774108880  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
>    -93> 2015-03-10 19:28:48.905467 7f0774108880  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5
>    -92> 2015-03-10 19:28:49.077872 7f0774108880  0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
>    -91> 2015-03-10 19:28:49.078270 7f0774108880  2 journal open /var/lib/ceph/osd/ceph-3/journal fsid 05fb481f-68b3-4442-b7de-af4c867e18d5 fs_op_seq 2
>    -90> 2015-03-10 19:28:49.078321 7f0774108880 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
>    -89> 2015-03-10 19:28:49.078328 7f0774108880  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
>    -88> 2015-03-10 19:28:49.078937 7f0774108880  2 journal read_entry 8192 : seq 3 193 bytes
>    -87> 2015-03-10 19:28:49.079000 7f0774108880  2 journal read_entry 8192 : seq 3 193 bytes
>    -86> 2015-03-10 19:28:49.079022 7f0774108880  3 journal journal_replay: applying op seq 3
>    -85> 2015-03-10 19:28:49.079307 7f0774108880  3 journal journal_replay: r = 0, op_seq now 3
>    -84> 2015-03-10 19:28:49.079367 7f0774108880  2 journal read_entry 12288 : seq 4 3829 bytes
>    -83> 2015-03-10 19:28:49.079390 7f0774108880  3 journal journal_replay: applying op seq 4
>    -82> 2015-03-10 19:28:49.079667 7f0774108880  3 journal journal_replay: r = 0, op_seq now 4
>    -81> 2015-03-10 19:28:49.079699 7f0774108880  2 journal No further valid entries found, journal is most likely valid
>    -80> 2015-03-10 19:28:49.079703 7f0774108880  3 journal journal_replay: end of journal, done.
>    -79> 2015-03-10 19:28:49.079721 7f0774108880  1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
>    -78> 2015-03-10 19:28:49.080265 7f0774108880  2 osd.3 0 boot
>    -77> 2015-03-10 19:28:49.080948 7f0774108880  0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
>    -76> 2015-03-10 19:28:49.081757 7f0774108880  1 <cls> cls/log/cls_log.cc:312: Loaded log class!
>    -75> 2015-03-10 19:28:49.087397 7f0774108880  1 <cls> cls/refcount/cls_refcount.cc:231: Loaded refcount class!
>    -74> 2015-03-10 19:28:49.087735 7f0774108880  1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class!
>    -73> 2015-03-10 19:28:49.093126 7f0774108880  1 <cls> cls/rgw/cls_rgw.cc:3046: Loaded rgw class!
>    -72> 2015-03-10 19:28:49.093484 7f0774108880  1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class!
>    -71> 2015-03-10 19:28:49.093723 7f0774108880  1 <cls> cls/user/cls_user.cc:367: Loaded user class!
>    -70> 2015-03-10 19:28:49.093924 7f0774108880  1 <cls> cls/version/cls_version.cc:227: Loaded version class!
>    -69> 2015-03-10 19:28:49.094194 7f0774108880  0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients
>    -68> 2015-03-10 19:28:49.094211 7f0774108880  0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons
>    -67> 2015-03-10 19:28:49.094217 7f0774108880  0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds
>    -66> 2015-03-10 19:28:49.094235 7f0774108880  0 osd.3 2757 load_pgs
>    -65> 2015-03-10 19:28:49.094279 7f0774108880  0 osd.3 2757 load_pgs opened 0 pgs
>    -64> 2015-03-10 19:28:49.094285 7f0774108880  2 osd.3 2757 superblock: i am osd.3
>    -63> 2015-03-10 19:28:49.094424 7f0774108880  1 accepter.accepter.start
>    -62> 2015-03-10 19:28:49.094573 7f0774108880  1 accepter.accepter.start
>    -61> 2015-03-10 19:28:49.094748 7f0774108880  1 accepter.accepter.start
>    -60> 2015-03-10 19:28:49.094864 7f0774108880  1 accepter.accepter.start
>    -59> 2015-03-10 19:28:49.094963 7f0774108880 10 monclient(hunting): init
>    -58> 2015-03-10 19:28:49.094980 7f0774108880  5 adding auth protocol: none
>    -57> 2015-03-10 19:28:49.095005 7f0774108880 10 monclient(hunting): auth_supported 1 method none
>    -56> 2015-03-10 19:28:49.095110 7f0774108880 10 log_channel(default) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info)
>    -55> 2015-03-10 19:28:49.095121 7f0774108880 -1 osd.3 2757 log_to_monitors {default=true}
>    -54> 2015-03-10 19:28:49.095899 7f0774108880 10 monclient(hunting): renew_subs
>    -53> 2015-03-10 19:28:49.095914 7f0774108880 10 monclient(hunting): _reopen_session rank -1 name 
>    -52> 2015-03-10 19:28:49.096003 7f0774108880 10 monclient(hunting): picked mon.a con 0x43623c0 addr 192.168.0.10:6789/0
>    -51> 2015-03-10 19:28:49.096086 7f0774108880 10 monclient(hunting): _send_mon_message to mon.a at 192.168.0.10:6789/0
>    -50> 2015-03-10 19:28:49.096098 7f0774108880  1 -- 0.0.0.0:6800/10810 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x41fcd80 con 0x43623c0
>    -49> 2015-03-10 19:28:49.096132 7f0774108880 10 monclient(hunting): renew_subs
>    -48> 2015-03-10 19:28:49.096501 7f075302b700  5 osd.3 2757 heartbeat: osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist [])
>    -47> 2015-03-10 19:28:49.113808 7f0774104700  1 -- 192.168.0.250:6800/10810 learned my addr 192.168.0.250:6800/10810
>    -46> 2015-03-10 19:28:49.131334 7f076384c700 10 monclient(hunting): renew_subs
>    -45> 2015-03-10 19:28:49.133513 7f076384c700  1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x41ff840 con 0x43623c0
>    -44> 2015-03-10 19:28:49.133572 7f076384c700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
>    -43> 2015-03-10 19:28:49.133587 7f076384c700 10 monclient(hunting):  got monmap 1, mon.a is now rank 0
>    -42> 2015-03-10 19:28:49.133597 7f076384c700 10 monclient(hunting): dump:
> epoch 1
> fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57
> last_changed 2013-09-11 22:41:30.819687
> created 2013-09-11 22:41:30.819687
> 0: 192.168.0.10:6789/0 mon.a
> 
>    -41> 2015-03-10 19:28:49.133678 7f076384c700  1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (1541693634 0 0) 0x41ffa80 con 0x43623c0
>    -40> 2015-03-10 19:28:49.133716 7f076384c700 10 monclient(hunting): my global_id is 14511543
>    -39> 2015-03-10 19:28:49.133721 7f076384c700  1 monclient(hunting): found mon.a
>    -38> 2015-03-10 19:28:49.133803 7f076384c700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
>    -37> 2015-03-10 19:28:49.133820 7f076384c700  1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x41e1a00 con 0x43623c0
>    -36> 2015-03-10 19:28:49.133906 7f076384c700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
>    -35> 2015-03-10 19:28:49.133925 7f076384c700  1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0,osdmap=2757}) v2 -- ?+0 0x41e2800 con 0x43623c0
>    -34> 2015-03-10 19:28:49.134016 7f076384c700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2015-03-10 19:28:19.134014)
>    -33> 2015-03-10 19:28:49.134061 7f0774108880  5 monclient: authenticate success, global_id 14511543
>    -32> 2015-03-10 19:28:49.134104 7f0774108880  0 osd.3 2757 done with init, starting boot process
>    -31> 2015-03-10 19:28:49.134118 7f0774108880 10 monclient: get_version osdmap req 0x4259160
>    -30> 2015-03-10 19:28:49.134126 7f0774108880 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
>    -29> 2015-03-10 19:28:49.134136 7f0774108880  1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x42741e0 con 0x43623c0
>    -28> 2015-03-10 19:28:49.134362 7f0774108880  5 asok(0x4200000) register_command status hook 0x41b4290
>    -27> 2015-03-10 19:28:49.134388 7f0774108880  5 asok(0x4200000) register_command flush_journal hook 0x41b4290
>    -26> 2015-03-10 19:28:49.134405 7f0774108880  5 asok(0x4200000) register_command dump_ops_in_flight hook 0x41b4290
>    -25> 2015-03-10 19:28:49.134421 7f0774108880  5 asok(0x4200000) register_command ops hook 0x41b4290
>    -24> 2015-03-10 19:28:49.134437 7f0774108880  5 asok(0x4200000) register_command dump_historic_ops hook 0x41b4290
>    -23> 2015-03-10 19:28:49.134454 7f0774108880  5 asok(0x4200000) register_command dump_op_pq_state hook 0x41b4290
>    -22> 2015-03-10 19:28:49.134469 7f0774108880  5 asok(0x4200000) register_command dump_blacklist hook 0x41b4290
>    -21> 2015-03-10 19:28:49.134484 7f0774108880  5 asok(0x4200000) register_command dump_watchers hook 0x41b4290
>    -20> 2015-03-10 19:28:49.134500 7f0774108880  5 asok(0x4200000) register_command dump_reservations hook 0x41b4290
>    -19> 2015-03-10 19:28:49.134516 7f0774108880  5 asok(0x4200000) register_command get_latest_osdmap hook 0x41b4290
>    -18> 2015-03-10 19:28:49.134532 7f0774108880  5 asok(0x4200000) register_command setomapval hook 0x4259200
>    -17> 2015-03-10 19:28:49.134546 7f0774108880  5 asok(0x4200000) register_command rmomapkey hook 0x4259200
>    -16> 2015-03-10 19:28:49.134563 7f0774108880  5 asok(0x4200000) register_command setomapheader hook 0x4259200
>    -15> 2015-03-10 19:28:49.134580 7f0774108880  5 asok(0x4200000) register_command getomap hook 0x4259200
>    -14> 2015-03-10 19:28:49.134595 7f0774108880  5 asok(0x4200000) register_command truncobj hook 0x4259200
>    -13> 2015-03-10 19:28:49.134609 7f0774108880  5 asok(0x4200000) register_command injectdataerr hook 0x4259200
>    -12> 2015-03-10 19:28:49.134624 7f0774108880  5 asok(0x4200000) register_command injectmdataerr hook 0x4259200
>    -11> 2015-03-10 19:28:49.136325 7f076384c700  1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 3 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x41ffcc0 con 0x43623c0
>    -10> 2015-03-10 19:28:49.136622 7f076384c700  3 osd.3 2757 handle_osd_map epochs [3277,3277], i have 2757, src has [2757,3277]
>     -9> 2015-03-10 19:28:49.136643 7f076384c700 10 monclient: renew_subs
>     -8> 2015-03-10 19:28:49.136652 7f076384c700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
>     -7> 2015-03-10 19:28:49.136679 7f076384c700  1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=2758}) v2 -- ?+0 0x41e1a00 con 0x43623c0
>     -6> 2015-03-10 19:28:49.136943 7f076384c700  1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 4 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x41ff840 con 0x43623c0
>     -5> 2015-03-10 19:28:49.136994 7f076384c700 10 monclient: handle_monmap mon_map magic: 0 v1
>     -4> 2015-03-10 19:28:49.137010 7f076384c700 10 monclient:  got monmap 1, mon.a is now rank 0
>     -3> 2015-03-10 19:28:49.137022 7f076384c700 10 monclient: dump:
> epoch 1
> fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57
> last_changed 2013-09-11 22:41:30.819687
> created 2013-09-11 22:41:30.819687
> 0: 192.168.0.10:6789/0 mon.a
> 
>     -2> 2015-03-10 19:28:49.142203 7f076384c700  1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(2757..2857 src has 2757..3277) v3 ==== 37648+0+0 (2648067016 0 0) 0x41ffa80 con 0x43623c0
>     -1> 2015-03-10 19:28:49.142316 7f076384c700  3 osd.3 2757 handle_osd_map epochs [2757,2857], i have 2757, src has [2757,3277]
>      0> 2015-03-10 19:28:49.149994 7f076384c700 -1 *** Caught signal (Aborted) **  in thread 7f076384c700
> 
>  ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
>  1: /usr/bin/ceph-osd() [0xac7cea]
>  2: (()+0x10050) [0x7f0773013050]
>  3: (gsignal()+0x37) [0x7f07714e60f7]
>  4: (abort()+0x13a) [0x7f07714e74ca]
>  5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f0771dcbfe5]
>  6: (()+0x63186) [0x7f0771dca186]
>  7: (()+0x631b3) [0x7f0771dca1b3]
>  8: (()+0x633d2) [0x7f0771dca3d2]
>  9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
>  10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
>  11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
>  12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
>  13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
>  14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
>  15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
>  16: (DispatchQueue::entry()+0x649) [0xc6fe09]
>  17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
>  18: (()+0x83a4) [0x7f077300b3a4]
>  19: (clone()+0x6d) [0x7f0771595a4d]
>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
> 
> --- logging levels ---
>    0/ 5 none
>    0/ 1 lockdep
>    0/ 1 context
>    1/ 1 crush
>    1/ 5 mds
>    1/ 5 mds_balancer
>    1/ 5 mds_locker
>    1/ 5 mds_log
>    1/ 5 mds_log_expire
>    1/ 5 mds_migrator
>    0/ 1 buffer
>    0/ 1 timer
>    0/ 1 filer
>    0/ 1 striper
>    0/ 1 objecter
>    0/ 5 rados
>    0/ 5 rbd
>    0/ 5 rbd_replay
>    0/ 5 journaler
>    0/ 5 objectcacher
>    0/ 5 client
>    0/ 5 osd
>    0/ 5 optracker
>    0/ 5 objclass
>    1/ 3 filestore
>    1/ 3 keyvaluestore
>    1/ 3 journal
>    0/ 5 ms
>    1/ 5 mon
>    0/10 monc
>    1/ 5 paxos
>    0/ 5 tp
>    1/ 5 auth
>    1/ 5 crypto
>    1/ 1 finisher
>    1/ 5 heartbeatmap
>    1/ 5 perfcounter
>    1/ 5 rgw
>    1/10 civetweb
>    1/ 5 javaclient
>    1/ 5 asok
>    1/ 1 throttle
>    0/ 0 refs
>    1/ 5 xio
>   -2/-2 (syslog threshold)
>   -1/-1 (stderr threshold)
>   max_recent     10000
>   max_new         1000
>   log_file /var/log/ceph/ceph-osd.3.log
> --- end dump of recent events ---
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux