Joao, it looks like map 2759 is causing trouble, how would he get the
full and incremental maps for that out of the mons?
-Sam
On Tue, 2015-03-10 at 14:12 +0000, Malcolm Haak wrote:
> Hi Samuel,
>
> The sha1? I'm going to admit ignorance as to what you are looking for. They are all running the same release if that is what you are asking.
> Same tarball built into rpms using rpmbuild on both nodes...
> Only difference being that the other node has been upgraded and the problem node is fresh.
>
> added the requested config here is the command line output
>
> microserver-1:/etc # /etc/init.d/ceph start osd.3
> === osd.3 ===
> Mounting xfs on microserver-1:/var/lib/ceph/osd/ceph-3
> 2015-03-11 01:00:13.492279 7f05b2f72700 1 -- :/0 messenger.start
> 2015-03-11 01:00:13.492823 7f05b2f72700 1 -- :/1002795 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x7f05ac0290b0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.510814 7f05b07ef700 1 -- 192.168.0.250:0/1002795 learned my addr 192.168.0.250:0/1002795
> 2015-03-11 01:00:13.527653 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x7f05a0000ab0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.527899 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (3859410672 0 0) 0x7f05a0000e70 con 0x7f05ac027c40
> 2015-03-11 01:00:13.527973 7f05abfff700 1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x7f05ac029730 con 0x7f05ac027c40
> 2015-03-11 01:00:13.528124 7f05b2f72700 1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x7f05ac029a50 con 0x7f05ac027c40
> 2015-03-11 01:00:13.528265 7f05b2f72700 1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x7f05ac029f20 con 0x7f05ac027c40
> 2015-03-11 01:00:13.530359 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 3 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x7f05a0000ab0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.530548 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 4 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3648139960 0 0) 0x7f05a0000fb0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.531114 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x7f05a0002800 con 0x7f05ac027c40
> 2015-03-11 01:00:13.531772 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3648139960 0 0) 0x7f05a0000fb0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.532186 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 7 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x7f05a0001250 con 0x7f05ac027c40
> 2015-03-11 01:00:13.532260 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3648139960 0 0) 0x7f05a0000fb0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.556748 7f05b2f72700 1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- ?+0 0x7f05ac016ac0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.564968 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+34995 (1092875540 0 1727986498) 0x7f05a0000a70 con 0x7f05ac027c40
> 2015-03-11 01:00:13.770122 7f05b2f72700 1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_command({"prefix": "osd crush create-or-move", "args": ["host=microserver-1", "root=default"], "id": 3, "weight": 1.81} v 0) v1 -- ?+0 0x7f05ac016ac0 con 0x7f05ac027c40
> 2015-03-11 01:00:13.772299 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "args": ["host=microserver-1", "root=default"], "id": 3, "weight": 1.81}]=0 create-or-move updated item name 'osd.3' weight 1.81 at location {host=microserver-1,root=default} to crush map v3277) v1 ==== 256+0+0 (1191546821 0 0) 0x7f05a0001000 con 0x7f05ac027c40
> create-or-move updated item name 'osd.3' weight 1.81 at location {host=microserver-1,root=default} to crush map
> 2015-03-11 01:00:13.776891 7f05b2f72700 1 -- 192.168.0.250:0/1002795 mark_down 0x7f05ac027c40 -- 0x7f05ac0239a0
> 2015-03-11 01:00:13.777212 7f05b2f72700 1 -- 192.168.0.250:0/1002795 mark_down_all
> 2015-03-11 01:00:13.778120 7f05b2f72700 1 -- 192.168.0.250:0/1002795 shutdown complete.
> Starting Ceph osd.3 on microserver-1...
> microserver-1:/etc #
>
>
> Log file
>
>
> 2015-03-11 01:00:13.876152 7f41a1ba4880 0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 2840
> 2015-03-11 01:00:13.877059 7f41a1ba4880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/2840 need_addr=1
> 2015-03-11 01:00:13.877111 7f41a1ba4880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/2840 need_addr=1
> 2015-03-11 01:00:13.877140 7f41a1ba4880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/2840 need_addr=1
> 2015-03-11 01:00:13.877169 7f41a1ba4880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/2840 need_addr=1
> 2015-03-11 01:00:13.880323 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) dump_stop
> 2015-03-11 01:00:13.890355 7f41a1ba4880 10 ErasureCodePluginSelectJerasure: load: jerasure_generic
> 2015-03-11 01:00:13.897843 7f41a1ba4880 10 load: jerasure load: lrc load: isa
> 2015-03-11 01:00:13.898172 7f41a1ba4880 5 filestore(/var/lib/ceph/osd/ceph-3) test_mount basedir /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal
> 2015-03-11 01:00:13.898290 7f41a1ba4880 1 -- 0.0.0.0:6800/2840 messenger.start
> 2015-03-11 01:00:13.898362 7f41a1ba4880 1 -- :/0 messenger.start
> 2015-03-11 01:00:13.898450 7f41a1ba4880 1 -- 0.0.0.0:6803/2840 messenger.start
> 2015-03-11 01:00:13.898527 7f41a1ba4880 1 -- 0.0.0.0:6802/2840 messenger.start
> 2015-03-11 01:00:13.898609 7f41a1ba4880 1 -- 0.0.0.0:6801/2840 messenger.start
> 2015-03-11 01:00:13.898672 7f41a1ba4880 1 -- :/0 messenger.start
> 2015-03-11 01:00:13.898792 7f41a1ba4880 2 osd.3 0 mounting /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
> 2015-03-11 01:00:13.898826 7f41a1ba4880 5 filestore(/var/lib/ceph/osd/ceph-3) basedir /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal
> 2015-03-11 01:00:13.898868 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) mount fsid is 05fb481f-68b3-4442-b7de-af4c867e18d5
> 2015-03-11 01:00:13.898937 7f41a1ba4880 0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342)
> 2015-03-11 01:00:13.941605 7f41a1ba4880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work
> 2015-03-11 01:00:13.941665 7f41a1ba4880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
> 2015-03-11 01:00:13.970056 7f41a1ba4880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
> 2015-03-11 01:00:13.970442 7f41a1ba4880 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5
> 2015-03-11 01:00:14.001199 7f41a1ba4880 5 filestore(/var/lib/ceph/osd/ceph-3) mount op_seq is 2
> 2015-03-11 01:00:14.122464 7f41a1ba4880 20 filestore (init)dbobjectmap: seq is 1
> 2015-03-11 01:00:14.122580 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) open_journal at /var/lib/ceph/osd/ceph-3/journal
> 2015-03-11 01:00:14.122674 7f41a1ba4880 0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
> 2015-03-11 01:00:14.122682 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) list_collections
> 2015-03-11 01:00:14.123024 7f41a1ba4880 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
> 2015-03-11 01:00:14.123040 7f41a1ba4880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> 2015-03-11 01:00:14.123003 7f4196c1b700 20 filestore(/var/lib/ceph/osd/ceph-3) sync_entry waiting for max_interval 5.000000
> 2015-03-11 01:00:14.123724 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x4b9b6c0
> 2015-03-11 01:00:14.123837 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
> 2015-03-11 01:00:14.123971 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
> 2015-03-11 01:00:14.124020 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) touch meta/a468ec03/snapmapper/0//-1
> 2015-03-11 01:00:14.124103 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) touch meta/a468ec03/snapmapper/0//-1 = 0
> 2015-03-11 01:00:14.124247 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x4b9b6c0
> 2015-03-11 01:00:14.124282 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
> 2015-03-11 01:00:14.124332 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
> 2015-03-11 01:00:14.124348 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) write meta/a143428/osdmap.2757/0//-1 0~3076
> 2015-03-11 01:00:14.124420 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) write meta/a143428/osdmap.2757/0//-1 0~3076 = 3076
> 2015-03-11 01:00:14.124461 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
> 2015-03-11 01:00:14.124547 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
> 2015-03-11 01:00:14.124557 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) write meta/23c2fcde/osd_superblock/0//-1 0~413
> 2015-03-11 01:00:14.124615 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) write meta/23c2fcde/osd_superblock/0//-1 0~413 = 413
> 2015-03-11 01:00:14.124716 7f41a1ba4880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> 2015-03-11 01:00:14.125247 7f41a1ba4880 2 osd.3 0 boot
> 2015-03-11 01:00:14.125270 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) read meta/23c2fcde/osd_superblock/0//-1 0~0
> 2015-03-11 01:00:14.125343 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) FileStore::read meta/23c2fcde/osd_superblock/0//-1 0~413/413
> 2015-03-11 01:00:14.125393 7f41a1ba4880 10 osd.3 0 read_superblock sb(3bf19e54-97e2-4a5f-b811-7c3780cd5e57 osd.3 05fb481f-68b3-4442-b7de-af4c867e18d5 e2757 [2757,2757] lci=[0,0])
> 2015-03-11 01:00:14.125521 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) stat meta/a468ec03/snapmapper/0//-1 = 0 (size 0)
> 2015-03-11 01:00:14.125540 7f41a1ba4880 10 open_all_classes
> 2015-03-11 01:00:14.125586 7f41a1ba4880 10 open_all_classes found hello
> 2015-03-11 01:00:14.125623 7f41a1ba4880 10 _get_class adding new class name hello 0x4c9e028
> 2015-03-11 01:00:14.125630 7f41a1ba4880 10 _load_class hello from /usr/lib64/rados-classes/libcls_hello.so
> 2015-03-11 01:00:14.126058 7f41a1ba4880 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
> 2015-03-11 01:00:14.126080 7f41a1ba4880 10 register_class hello status 3
> 2015-03-11 01:00:14.126084 7f41a1ba4880 10 register_cxx_method hello.say_hello flags 1 0x7f4192a10350
> 2015-03-11 01:00:14.126092 7f41a1ba4880 10 register_cxx_method hello.record_hello flags 2 0x7f4192a113c0
> 2015-03-11 01:00:14.126098 7f41a1ba4880 10 register_cxx_method hello.writes_dont_return_data flags 2 0x7f4192a10650
> 2015-03-11 01:00:14.126103 7f41a1ba4880 10 register_cxx_method hello.replay flags 1 0x7f4192a10320
> 2015-03-11 01:00:14.126109 7f41a1ba4880 10 register_cxx_method hello.turn_it_to_11 flags 3 0x7f4192a10920
> 2015-03-11 01:00:14.126114 7f41a1ba4880 10 register_cxx_method hello.bad_reader flags 2 0x7f4192a10310
> 2015-03-11 01:00:14.126119 7f41a1ba4880 10 register_cxx_method hello.bad_writer flags 1 0x7f4192a10300
> 2015-03-11 01:00:14.126127 7f41a1ba4880 10 _load_class hello success
> 2015-03-11 01:00:14.126132 7f41a1ba4880 10 open_all_classes found kvs
> 2015-03-11 01:00:14.126137 7f41a1ba4880 10 _get_class adding new class name kvs 0x4c9e108
> 2015-03-11 01:00:14.126142 7f41a1ba4880 10 _load_class kvs from /usr/lib64/rados-classes/libcls_kvs.so
> 2015-03-11 01:00:14.126655 7f41a1ba4880 10 register_class kvs status 3
> 2015-03-11 01:00:14.126665 7f41a1ba4880 10 register_cxx_method kvs.get_idata_from_key flags 1 0x7f41927ff060
> 2015-03-11 01:00:14.126669 7f41a1ba4880 10 register_cxx_method kvs.get_next_idata flags 1 0x7f4192801050
> 2015-03-11 01:00:14.126672 7f41a1ba4880 10 register_cxx_method kvs.get_prev_idata flags 1 0x7f41927fd570
> 2015-03-11 01:00:14.126675 7f41a1ba4880 10 register_cxx_method kvs.read_many flags 1 0x7f41927f8f30
> 2015-03-11 01:00:14.126678 7f41a1ba4880 10 register_cxx_method kvs.check_writable flags 3 0x7f41927f8ac0
> 2015-03-11 01:00:14.126681 7f41a1ba4880 10 register_cxx_method kvs.assert_size_in_bound flags 2 0x7f41927f8d40
> 2015-03-11 01:00:14.126684 7f41a1ba4880 10 register_cxx_method kvs.omap_insert flags 2 0x7f41927fcb50
> 2015-03-11 01:00:14.126688 7f41a1ba4880 10 register_cxx_method kvs.create_with_omap flags 2 0x7f41927fc0e0
> 2015-03-11 01:00:14.126691 7f41a1ba4880 10 register_cxx_method kvs.omap_remove flags 2 0x7f41927f9e70
> 2015-03-11 01:00:14.126696 7f41a1ba4880 10 register_cxx_method kvs.maybe_read_for_balance flags 1 0x7f41927fab70
> 2015-03-11 01:00:14.126735 7f41a1ba4880 10 _load_class kvs success
> 2015-03-11 01:00:14.126739 7f41a1ba4880 10 open_all_classes found lock
> 2015-03-11 01:00:14.126742 7f41a1ba4880 10 _get_class adding new class name lock 0x4c9e1e8
> 2015-03-11 01:00:14.126745 7f41a1ba4880 10 _load_class lock from /usr/lib64/rados-classes/libcls_lock.so
> 2015-03-11 01:00:14.126972 7f41a1ba4880 10 register_class lock status 3
> 2015-03-11 01:00:14.126977 7f41a1ba4880 10 register_cxx_method lock.lock flags 3 0x7f41925ee800
> 2015-03-11 01:00:14.126981 7f41a1ba4880 10 register_cxx_method lock.unlock flags 3 0x7f41925ed210
> 2015-03-11 01:00:14.126984 7f41a1ba4880 10 register_cxx_method lock.break_lock flags 3 0x7f41925ece60
> 2015-03-11 01:00:14.126986 7f41a1ba4880 10 register_cxx_method lock.get_info flags 1 0x7f41925ed610
> 2015-03-11 01:00:14.126990 7f41a1ba4880 10 register_cxx_method lock.list_locks flags 1 0x7f41925eace0
> 2015-03-11 01:00:14.126993 7f41a1ba4880 10 register_cxx_method lock.assert_locked flags 1 0x7f41925ec330
> 2015-03-11 01:00:14.126996 7f41a1ba4880 10 _load_class lock success
> 2015-03-11 01:00:14.126999 7f41a1ba4880 10 open_all_classes found log
> 2015-03-11 01:00:14.127002 7f41a1ba4880 10 _get_class adding new class name log 0x4c9e2c8
> 2015-03-11 01:00:14.127005 7f41a1ba4880 10 _load_class log from /usr/lib64/rados-classes/libcls_log.so
> 2015-03-11 01:00:14.127223 7f41a1ba4880 10 register_class log status 3
> 2015-03-11 01:00:14.127225 7f41a1ba4880 10 register_cxx_method log.add flags 3 0x7f41923e1560
> 2015-03-11 01:00:14.127228 7f41a1ba4880 10 register_cxx_method log.list flags 1 0x7f41923e2740
> 2015-03-11 01:00:14.127231 7f41a1ba4880 10 register_cxx_method log.trim flags 3 0x7f41923e1f90
> 2015-03-11 01:00:14.127234 7f41a1ba4880 10 register_cxx_method log.info flags 1 0x7f41923e1090
> 2015-03-11 01:00:14.127238 7f41a1ba4880 10 _load_class log success
> 2015-03-11 01:00:14.127241 7f41a1ba4880 10 open_all_classes found rbd
> 2015-03-11 01:00:14.127244 7f41a1ba4880 10 _get_class adding new class name rbd 0x4c9e3a8
> 2015-03-11 01:00:14.127247 7f41a1ba4880 10 _load_class rbd from /usr/lib64/rados-classes/libcls_rbd.so
> 2015-03-11 01:00:14.127735 7f41a1ba4880 10 register_class rbd status 3
> 2015-03-11 01:00:14.127745 7f41a1ba4880 10 register_cxx_method rbd.create flags 3 0x7f41921cc5c0
> 2015-03-11 01:00:14.127748 7f41a1ba4880 10 register_cxx_method rbd.get_features flags 1 0x7f41921d3190
> 2015-03-11 01:00:14.127751 7f41a1ba4880 10 register_cxx_method rbd.get_size flags 1 0x7f41921d2de0
> 2015-03-11 01:00:14.127754 7f41a1ba4880 10 register_cxx_method rbd.set_size flags 3 0x7f41921ca7e0
> 2015-03-11 01:00:14.127758 7f41a1ba4880 10 register_cxx_method rbd.get_snapcontext flags 1 0x7f41921cdb40
> 2015-03-11 01:00:14.127762 7f41a1ba4880 10 register_cxx_method rbd.get_object_prefix flags 1 0x7f41921c9770
> 2015-03-11 01:00:14.127766 7f41a1ba4880 10 register_cxx_method rbd.get_snapshot_name flags 1 0x7f41921d1dd0
> 2015-03-11 01:00:14.127770 7f41a1ba4880 10 register_cxx_method rbd.snapshot_add flags 3 0x7f41921d0830
> 2015-03-11 01:00:14.127773 7f41a1ba4880 10 register_cxx_method rbd.snapshot_remove flags 3 0x7f41921d1b40
> 2015-03-11 01:00:14.127776 7f41a1ba4880 10 register_cxx_method rbd.get_all_features flags 1 0x7f41921c78c0
> 2015-03-11 01:00:14.127779 7f41a1ba4880 10 register_cxx_method rbd.copyup flags 3 0x7f41921c7940
> 2015-03-11 01:00:14.127782 7f41a1ba4880 10 register_cxx_method rbd.get_parent flags 1 0x7f41921d2080
> 2015-03-11 01:00:14.127786 7f41a1ba4880 10 register_cxx_method rbd.set_parent flags 3 0x7f41921ccce0
> 2015-03-11 01:00:14.127789 7f41a1ba4880 10 register_cxx_method rbd.remove_parent flags 3 0x7f41921c9140
> 2015-03-11 01:00:14.127792 7f41a1ba4880 10 register_cxx_method rbd.set_protection_status flags 3 0x7f41921d2770
> 2015-03-11 01:00:14.127815 7f41a1ba4880 10 register_cxx_method rbd.get_protection_status flags 1 0x7f41921d2b60
> 2015-03-11 01:00:14.127818 7f41a1ba4880 10 register_cxx_method rbd.get_stripe_unit_count flags 1 0x7f41921c9330
> 2015-03-11 01:00:14.127822 7f41a1ba4880 10 register_cxx_method rbd.set_stripe_unit_count flags 3 0x7f41921cc070
> 2015-03-11 01:00:14.127825 7f41a1ba4880 10 register_cxx_method rbd.get_flags flags 1 0x7f41921d2470
> 2015-03-11 01:00:14.127828 7f41a1ba4880 10 register_cxx_method rbd.set_flags flags 3 0x7f41921cb370
> 2015-03-11 01:00:14.127831 7f41a1ba4880 10 register_cxx_method rbd.add_child flags 3 0x7f41921cd790
> 2015-03-11 01:00:14.127834 7f41a1ba4880 10 register_cxx_method rbd.remove_child flags 3 0x7f41921cdf50
> 2015-03-11 01:00:14.127838 7f41a1ba4880 10 register_cxx_method rbd.get_children flags 1 0x7f41921cd4e0
> 2015-03-11 01:00:14.127841 7f41a1ba4880 10 register_cxx_method rbd.get_id flags 1 0x7f41921ca580
> 2015-03-11 01:00:14.127844 7f41a1ba4880 10 register_cxx_method rbd.set_id flags 3 0x7f41921cb6c0
> 2015-03-11 01:00:14.127848 7f41a1ba4880 10 register_cxx_method rbd.dir_get_id flags 1 0x7f41921c9f40
> 2015-03-11 01:00:14.127851 7f41a1ba4880 10 register_cxx_method rbd.dir_get_name flags 1 0x7f41921ca210
> 2015-03-11 01:00:14.127855 7f41a1ba4880 10 register_cxx_method rbd.dir_list flags 1 0x7f41921cf9f0
> 2015-03-11 01:00:14.127858 7f41a1ba4880 10 register_cxx_method rbd.dir_add_image flags 3 0x7f41921cf840
> 2015-03-11 01:00:14.127862 7f41a1ba4880 10 register_cxx_method rbd.dir_remove_image flags 3 0x7f41921c9d60
> 2015-03-11 01:00:14.127865 7f41a1ba4880 10 register_cxx_method rbd.dir_rename_image flags 3 0x7f41921cf660
> 2015-03-11 01:00:14.127868 7f41a1ba4880 10 register_cxx_method rbd.object_map_load flags 1 0x7f41921ce5f0
> 2015-03-11 01:00:14.127872 7f41a1ba4880 10 register_cxx_method rbd.object_map_resize flags 3 0x7f41921ceec0
> 2015-03-11 01:00:14.127875 7f41a1ba4880 10 register_cxx_method rbd.object_map_update flags 3 0x7f41921ce720
> 2015-03-11 01:00:14.127879 7f41a1ba4880 10 register_cxx_method rbd.snap_list flags 1 0x7f41921c8540
> 2015-03-11 01:00:14.127882 7f41a1ba4880 10 register_cxx_method rbd.snap_add flags 3 0x7f41921cacc0
> 2015-03-11 01:00:14.127885 7f41a1ba4880 10 register_cxx_method rbd.snap_remove flags 3 0x7f41921cb910
> 2015-03-11 01:00:14.127891 7f41a1ba4880 10 _load_class rbd success
> 2015-03-11 01:00:14.127894 7f41a1ba4880 10 open_all_classes found refcount
> 2015-03-11 01:00:14.127897 7f41a1ba4880 10 _get_class adding new class name refcount 0x4c9e488
> 2015-03-11 01:00:14.127900 7f41a1ba4880 10 _load_class refcount from /usr/lib64/rados-classes/libcls_refcount.so
> 2015-03-11 01:00:14.132942 7f41a1ba4880 10 register_class refcount status 3
> 2015-03-11 01:00:14.132946 7f41a1ba4880 10 register_cxx_method refcount.get flags 3 0x7f4191f32be0
> 2015-03-11 01:00:14.132957 7f41a1ba4880 10 register_cxx_method refcount.put flags 3 0x7f4191f31b30
> 2015-03-11 01:00:14.132960 7f41a1ba4880 10 register_cxx_method refcount.set flags 3 0x7f4191f32510
> 2015-03-11 01:00:14.132963 7f41a1ba4880 10 register_cxx_method refcount.read flags 1 0x7f4191f31fb0
> 2015-03-11 01:00:14.132970 7f41a1ba4880 10 _load_class refcount success
> 2015-03-11 01:00:14.132977 7f41a1ba4880 10 open_all_classes found replica_log
> 2015-03-11 01:00:14.132984 7f41a1ba4880 10 _get_class adding new class name replica_log 0x4c9e568
> 2015-03-11 01:00:14.132987 7f41a1ba4880 10 _load_class replica_log from /usr/lib64/rados-classes/libcls_replica_log.so
> 2015-03-11 01:00:14.133269 7f41a1ba4880 10 register_class replica_log status 3
> 2015-03-11 01:00:14.133271 7f41a1ba4880 10 register_cxx_method replica_log.set flags 3 0x7f4191c7f920
> 2015-03-11 01:00:14.133276 7f41a1ba4880 10 register_cxx_method replica_log.get flags 1 0x7f4191c809f0
> 2015-03-11 01:00:14.133279 7f41a1ba4880 10 register_cxx_method replica_log.delete flags 3 0x7f4191c80400
> 2015-03-11 01:00:14.133283 7f41a1ba4880 10 _load_class replica_log success
> 2015-03-11 01:00:14.133285 7f41a1ba4880 10 open_all_classes found rgw
> 2015-03-11 01:00:14.133288 7f41a1ba4880 10 _get_class adding new class name rgw 0x4c9e728
> 2015-03-11 01:00:14.133291 7f41a1ba4880 10 _load_class rgw from /usr/lib64/rados-classes/libcls_rgw.so
> 2015-03-11 01:00:14.138712 7f41a1ba4880 10 register_class rgw status 3
> 2015-03-11 01:00:14.138717 7f41a1ba4880 10 register_cxx_method rgw.bucket_init_index flags 3 0x7f41919c0860
> 2015-03-11 01:00:14.138725 7f41a1ba4880 10 register_cxx_method rgw.bucket_set_tag_timeout flags 3 0x7f41919c4a40
> 2015-03-11 01:00:14.138728 7f41a1ba4880 10 register_cxx_method rgw.bucket_list flags 1 0x7f41919c8160
> 2015-03-11 01:00:14.138732 7f41a1ba4880 10 register_cxx_method rgw.bucket_check_index flags 1 0x7f41919ca300
> 2015-03-11 01:00:14.138735 7f41a1ba4880 10 register_cxx_method rgw.bucket_rebuild_index flags 3 0x7f41919ca1a0
> 2015-03-11 01:00:14.138740 7f41a1ba4880 10 register_cxx_method rgw.bucket_prepare_op flags 3 0x7f41919c5180
> 2015-03-11 01:00:14.138745 7f41a1ba4880 10 register_cxx_method rgw.bucket_complete_op flags 3 0x7f41919c5980
> 2015-03-11 01:00:14.138748 7f41a1ba4880 10 register_cxx_method rgw.bucket_link_olh flags 3 0x7f41919cc980
> 2015-03-11 01:00:14.138751 7f41a1ba4880 10 register_cxx_method rgw.bucket_unlink_instance flags 3 0x7f41919cbb80
> 2015-03-11 01:00:14.138754 7f41a1ba4880 10 register_cxx_method rgw.bucket_read_olh_log flags 1 0x7f41919cde40
> 2015-03-11 01:00:14.138757 7f41a1ba4880 10 register_cxx_method rgw.bucket_trim_olh_log flags 3 0x7f41919cb2d0
> 2015-03-11 01:00:14.138760 7f41a1ba4880 10 register_cxx_method rgw.bucket_clear_olh flags 3 0x7f41919cacd0
> 2015-03-11 01:00:14.138763 7f41a1ba4880 10 register_cxx_method rgw.obj_remove flags 3 0x7f41919c3ff0
> 2015-03-11 01:00:14.138771 7f41a1ba4880 10 register_cxx_method rgw.obj_check_attrs_prefix flags 1 0x7f41919c0b60
> 2015-03-11 01:00:14.138774 7f41a1ba4880 10 register_cxx_method rgw.bi_get flags 1 0x7f41919bf050
> 2015-03-11 01:00:14.138777 7f41a1ba4880 10 register_cxx_method rgw.bi_put flags 3 0x7f41919be700
> 2015-03-11 01:00:14.138780 7f41a1ba4880 10 register_cxx_method rgw.bi_list flags 1 0x7f41919c6790
> 2015-03-11 01:00:14.138784 7f41a1ba4880 10 register_cxx_method rgw.bi_log_list flags 1 0x7f41919c2180
> 2015-03-11 01:00:14.138787 7f41a1ba4880 10 register_cxx_method rgw.bi_log_trim flags 3 0x7f41919c26e0
> 2015-03-11 01:00:14.138791 7f41a1ba4880 10 register_cxx_method rgw.dir_suggest_changes flags 3 0x7f41919c9260
> 2015-03-11 01:00:14.138794 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_add flags 3 0x7f41919ce6a0
> 2015-03-11 01:00:14.138797 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_read flags 1 0x7f41919c31c0
> 2015-03-11 01:00:14.138799 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_trim flags 3 0x7f41919c3950
> 2015-03-11 01:00:14.138802 7f41a1ba4880 10 register_cxx_method rgw.gc_set_entry flags 3 0x7f41919c01b0
> 2015-03-11 01:00:14.138806 7f41a1ba4880 10 register_cxx_method rgw.gc_defer_entry flags 3 0x7f41919bfe90
> 2015-03-11 01:00:14.138809 7f41a1ba4880 10 register_cxx_method rgw.gc_list flags 1 0x7f41919c0f80
> 2015-03-11 01:00:14.138812 7f41a1ba4880 10 register_cxx_method rgw.gc_remove flags 3 0x7f41919c0420
> 2015-03-11 01:00:14.138815 7f41a1ba4880 10 _load_class rgw success
> 2015-03-11 01:00:14.138820 7f41a1ba4880 10 open_all_classes found statelog
> 2015-03-11 01:00:14.138826 7f41a1ba4880 10 _get_class adding new class name statelog 0x4c9e808
> 2015-03-11 01:00:14.138830 7f41a1ba4880 10 _load_class statelog from /usr/lib64/rados-classes/libcls_statelog.so
> 2015-03-11 01:00:14.139141 7f41a1ba4880 10 register_class statelog status 3
> 2015-03-11 01:00:14.139143 7f41a1ba4880 10 register_cxx_method statelog.add flags 3 0x7f4191700bc0
> 2015-03-11 01:00:14.139146 7f41a1ba4880 10 register_cxx_method statelog.list flags 1 0x7f4191700f40
> 2015-03-11 01:00:14.139150 7f41a1ba4880 10 register_cxx_method statelog.remove flags 3 0x7f41916ffa80
> 2015-03-11 01:00:14.139153 7f41a1ba4880 10 register_cxx_method statelog.check_state flags 1 0x7f41917002d0
> 2015-03-11 01:00:14.139175 7f41a1ba4880 10 _load_class statelog success
> 2015-03-11 01:00:14.139178 7f41a1ba4880 10 open_all_classes found user
> 2015-03-11 01:00:14.139182 7f41a1ba4880 10 _get_class adding new class name user 0x4c9e8e8
> 2015-03-11 01:00:14.139185 7f41a1ba4880 10 _load_class user from /usr/lib64/rados-classes/libcls_user.so
> 2015-03-11 01:00:14.139386 7f41a1ba4880 10 register_class user status 3
> 2015-03-11 01:00:14.139388 7f41a1ba4880 10 register_cxx_method user.set_buckets_info flags 3 0x7f41914f5180
> 2015-03-11 01:00:14.139392 7f41a1ba4880 10 register_cxx_method user.complete_stats_sync flags 3 0x7f41914f4aa0
> 2015-03-11 01:00:14.139394 7f41a1ba4880 10 register_cxx_method user.remove_bucket flags 3 0x7f41914f3d30
> 2015-03-11 01:00:14.139397 7f41a1ba4880 10 register_cxx_method user.list_buckets flags 1 0x7f41914f64c0
> 2015-03-11 01:00:14.139400 7f41a1ba4880 10 register_cxx_method user.get_header flags 1 0x7f41914f2d50
> 2015-03-11 01:00:14.139403 7f41a1ba4880 10 _load_class user success
> 2015-03-11 01:00:14.139406 7f41a1ba4880 10 open_all_classes found version
> 2015-03-11 01:00:14.139409 7f41a1ba4880 10 _get_class adding new class name version 0x4c9e9c8
> 2015-03-11 01:00:14.139412 7f41a1ba4880 10 _load_class version from /usr/lib64/rados-classes/libcls_version.so
> 2015-03-11 01:00:14.139608 7f41a1ba4880 10 register_class version status 3
> 2015-03-11 01:00:14.139611 7f41a1ba4880 10 register_cxx_method version.set flags 3 0x7f41912ebe60
> 2015-03-11 01:00:14.139615 7f41a1ba4880 10 register_cxx_method version.inc flags 3 0x7f41912ecc80
> 2015-03-11 01:00:14.139618 7f41a1ba4880 10 register_cxx_method version.inc_conds flags 3 0x7f41912ecc80
> 2015-03-11 01:00:14.139621 7f41a1ba4880 10 register_cxx_method version.read flags 1 0x7f41912ebb60
> 2015-03-11 01:00:14.139624 7f41a1ba4880 10 register_cxx_method version.check_conds flags 1 0x7f41912ec260
> 2015-03-11 01:00:14.139627 7f41a1ba4880 10 _load_class version success
> 2015-03-11 01:00:14.139682 7f41a1ba4880 20 osd.3 0 get_map 2757 - loading and decoding 0x4b4cb40
> 2015-03-11 01:00:14.139693 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) read meta/a143428/osdmap.2757/0//-1 0~0
> 2015-03-11 01:00:14.139804 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) FileStore::read meta/a143428/osdmap.2757/0//-1 0~3076/3076
> 2015-03-11 01:00:14.139818 7f41a1ba4880 10 osd.3 0 add_map_bl 2757 3076 bytes
> 2015-03-11 01:00:14.139908 7f41a1ba4880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients
> 2015-03-11 01:00:14.139914 7f41a1ba4880 0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons
> 2015-03-11 01:00:14.139919 7f41a1ba4880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds
> 2015-03-11 01:00:14.139922 7f41a1ba4880 10 osd.3 2757 create_recoverystate_perf
> 2015-03-11 01:00:14.139939 7f41a1ba4880 0 osd.3 2757 load_pgs
> 2015-03-11 01:00:14.139942 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) list_collections
> 2015-03-11 01:00:14.139981 7f41a1ba4880 10 osd.3 2757 load_pgs ignoring unrecognized meta
> 2015-03-11 01:00:14.139985 7f41a1ba4880 0 osd.3 2757 load_pgs opened 0 pgs
> 2015-03-11 01:00:14.139989 7f41a1ba4880 10 osd.3 2757 build_past_intervals_parallel nothing to build
> 2015-03-11 01:00:14.139992 7f41a1ba4880 2 osd.3 2757 superblock: i am osd.3
> 2015-03-11 01:00:14.139995 7f41a1ba4880 10 osd.3 2757 create_logger
> 2015-03-11 01:00:14.140126 7f41a1ba4880 1 accepter.accepter.start
> 2015-03-11 01:00:14.140269 7f41a1ba4880 1 accepter.accepter.start
> 2015-03-11 01:00:14.140434 7f41a1ba4880 1 accepter.accepter.start
> 2015-03-11 01:00:14.140575 7f41a1ba4880 1 accepter.accepter.start
> 2015-03-11 01:00:14.140813 7f41a1ba4880 -1 osd.3 2757 log_to_monitors {default=true}
> 2015-03-11 01:00:14.141325 7f41a1ba4880 10 osd.3 2757 set_disk_tp_priority class priority -1
> 2015-03-11 01:00:14.141599 7f41a1ba4880 1 -- 0.0.0.0:6800/2840 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x4b4cd80 con 0x4cbe3c0
> 2015-03-11 01:00:14.142020 7f4180ac7700 20 osd.3 2757 update_osd_stat osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist [])
> 2015-03-11 01:00:14.142048 7f4180ac7700 5 osd.3 2757 heartbeat: osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist [])
> 2015-03-11 01:00:14.142221 7f417e2c2700 10 osd.3 2757 agent_entry start
> 2015-03-11 01:00:14.142241 7f417e2c2700 20 osd.3 2757 agent_entry empty queue
> 2015-03-11 01:00:14.158932 7f41a1ba0700 1 -- 192.168.0.250:6800/2840 learned my addr 192.168.0.250:6800/2840
> 2015-03-11 01:00:14.159058 7f41a1ba0700 10 osd.3 2757 new session (outgoing) 0x4b92280 con=0x4b602c0 addr=192.168.0.250:6800/2840
> 2015-03-11 01:00:14.159166 7f41a1ba0700 10 osd.3 2757 OSD::ms_get_authorizer type=mon
> 2015-03-11 01:00:14.173274 7f41912e8700 10 osd.3 2757 ms_handle_connect on mon
> 2015-03-11 01:00:14.173332 7f41912e8700 20 osd.3 2757 send_pg_stats
> 2015-03-11 01:00:14.175783 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x4b4f840 con 0x4cbe3c0
> 2015-03-11 01:00:14.175940 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (335930126 0 0) 0x4b4fa80 con 0x4cbe3c0
> 2015-03-11 01:00:14.175997 7f41912e8700 1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x4b31a00 con 0x4cbe3c0
> 2015-03-11 01:00:14.176059 7f41912e8700 1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0,osdmap=2757}) v2 -- ?+0 0x4b32800 con 0x4cbe3c0
> 2015-03-11 01:00:14.176147 7f41a1ba4880 10 osd.3 2757 ensuring pgs have consumed prior maps
> 2015-03-11 01:00:14.176170 7f41a1ba4880 7 osd.3 2757 consume_map version 2757
> 2015-03-11 01:00:14.176187 7f41a1ba4880 0 osd.3 2757 done with init, starting boot process
> 2015-03-11 01:00:14.176191 7f41a1ba4880 10 osd.3 2757 start_boot - have maps 2757..2757
> 2015-03-11 01:00:14.176221 7f41a1ba4880 1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x4bc41e0 con 0x4cbe3c0
> 2015-03-11 01:00:14.178539 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 3 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x4b4fcc0 con 0x4cbe3c0
> 2015-03-11 01:00:14.178650 7f41912e8700 10 osd.3 2757 do_waiters -- start
> 2015-03-11 01:00:14.178658 7f41912e8700 10 osd.3 2757 do_waiters -- finish
> 2015-03-11 01:00:14.178661 7f41912e8700 20 osd.3 2757 _dispatch 0x4b4fcc0 osd_map(3277..3277 src has 2757..3277) v3
> 2015-03-11 01:00:14.178888 7f41912e8700 3 osd.3 2757 handle_osd_map epochs [3277,3277], i have 2757, src has [2757,3277]
> 2015-03-11 01:00:14.178906 7f41912e8700 10 osd.3 2757 handle_osd_map message skips epochs 2758..3276
> 2015-03-11 01:00:14.178954 7f41912e8700 1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=2758}) v2 -- ?+0 0x4b31a00 con 0x4cbe3c0
> 2015-03-11 01:00:14.178991 7f41912e8700 10 osd.3 2757 do_waiters -- start
> 2015-03-11 01:00:14.178996 7f41912e8700 10 osd.3 2757 do_waiters -- finish
> 2015-03-11 01:00:14.179012 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 4 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x4b4f840 con 0x4cbe3c0
> 2015-03-11 01:00:14.184757 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(2757..2857 src has 2757..3277) v3 ==== 37648+0+0 (2648067016 0 0) 0x4b4fa80 con 0x4cbe3c0
> 2015-03-11 01:00:14.184831 7f41912e8700 10 osd.3 2757 do_waiters -- start
> 2015-03-11 01:00:14.184838 7f41912e8700 10 osd.3 2757 do_waiters -- finish
> 2015-03-11 01:00:14.184841 7f41912e8700 20 osd.3 2757 _dispatch 0x4b4fa80 osd_map(2757..2857 src has 2757..3277) v3
> 2015-03-11 01:00:14.184872 7f41912e8700 3 osd.3 2757 handle_osd_map epochs [2757,2857], i have 2757, src has [2757,3277]
> 2015-03-11 01:00:14.184884 7f41912e8700 10 osd.3 2757 handle_osd_map got inc map for epoch 2758
> 2015-03-11 01:00:14.185229 7f41912e8700 10 osd.3 2757 handle_osd_map got inc map for epoch 2759
> 2015-03-11 01:00:14.195165 7f41912e8700 -1 *** Caught signal (Aborted) **
> in thread 7f41912e8700
>
> ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
> 1: /usr/bin/ceph-osd() [0xac7cea]
> 2: (()+0x10050) [0x7f41a0aaf050]
> 3: (gsignal()+0x37) [0x7f419ef820f7]
> 4: (abort()+0x13a) [0x7f419ef834ca]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f419f867fe5]
> 6: (()+0x63186) [0x7f419f866186]
> 7: (()+0x631b3) [0x7f419f8661b3]
> 8: (()+0x633d2) [0x7f419f8663d2]
> 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
> 10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
> 11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
> 12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
> 13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
> 14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
> 15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
> 16: (DispatchQueue::entry()+0x649) [0xc6fe09]
> 17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
> 18: (()+0x83a4) [0x7f41a0aa73a4]
> 19: (clone()+0x6d) [0x7f419f031a4d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>
> --- begin dump of recent events ---
> -363> 2015-03-11 01:00:13.871379 7f41a1ba4880 5 asok(0x4b50000) register_command perfcounters_dump hook 0x4b04030
> -362> 2015-03-11 01:00:13.871462 7f41a1ba4880 5 asok(0x4b50000) register_command 1 hook 0x4b04030
> -361> 2015-03-11 01:00:13.871471 7f41a1ba4880 5 asok(0x4b50000) register_command perf dump hook 0x4b04030
> -360> 2015-03-11 01:00:13.871484 7f41a1ba4880 5 asok(0x4b50000) register_command perfcounters_schema hook 0x4b04030
> -359> 2015-03-11 01:00:13.871507 7f41a1ba4880 5 asok(0x4b50000) register_command 2 hook 0x4b04030
> -358> 2015-03-11 01:00:13.871512 7f41a1ba4880 5 asok(0x4b50000) register_command perf schema hook 0x4b04030
> -357> 2015-03-11 01:00:13.871545 7f41a1ba4880 5 asok(0x4b50000) register_command perf reset hook 0x4b04030
> -356> 2015-03-11 01:00:13.871555 7f41a1ba4880 5 asok(0x4b50000) register_command config show hook 0x4b04030
> -355> 2015-03-11 01:00:13.871562 7f41a1ba4880 5 asok(0x4b50000) register_command config set hook 0x4b04030
> -354> 2015-03-11 01:00:13.871571 7f41a1ba4880 5 asok(0x4b50000) register_command config get hook 0x4b04030
> -353> 2015-03-11 01:00:13.871576 7f41a1ba4880 5 asok(0x4b50000) register_command config diff hook 0x4b04030
> -352> 2015-03-11 01:00:13.871585 7f41a1ba4880 5 asok(0x4b50000) register_command log flush hook 0x4b04030
> -351> 2015-03-11 01:00:13.871590 7f41a1ba4880 5 asok(0x4b50000) register_command log dump hook 0x4b04030
> -350> 2015-03-11 01:00:13.871600 7f41a1ba4880 5 asok(0x4b50000) register_command log reopen hook 0x4b04030
> -349> 2015-03-11 01:00:13.876152 7f41a1ba4880 0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 2840
> -348> 2015-03-11 01:00:13.877059 7f41a1ba4880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/2840 need_addr=1
> -347> 2015-03-11 01:00:13.877111 7f41a1ba4880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/2840 need_addr=1
> -346> 2015-03-11 01:00:13.877140 7f41a1ba4880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/2840 need_addr=1
> -345> 2015-03-11 01:00:13.877169 7f41a1ba4880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/2840 need_addr=1
> -344> 2015-03-11 01:00:13.880323 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) dump_stop
> -343> 2015-03-11 01:00:13.880362 7f41a1ba4880 5 asok(0x4b50000) init /var/run/ceph/ceph-osd.3.asok
> -342> 2015-03-11 01:00:13.880382 7f41a1ba4880 5 asok(0x4b50000) bind_and_listen /var/run/ceph/ceph-osd.3.asok
> -341> 2015-03-11 01:00:13.880604 7f41a1ba4880 5 asok(0x4b50000) register_command 0 hook 0x4b000c0
> -340> 2015-03-11 01:00:13.880620 7f41a1ba4880 5 asok(0x4b50000) register_command version hook 0x4b000c0
> -339> 2015-03-11 01:00:13.880626 7f41a1ba4880 5 asok(0x4b50000) register_command git_version hook 0x4b000c0
> -338> 2015-03-11 01:00:13.880633 7f41a1ba4880 5 asok(0x4b50000) register_command help hook 0x4b04110
> -337> 2015-03-11 01:00:13.880639 7f41a1ba4880 5 asok(0x4b50000) register_command get_command_descriptions hook 0x4b04120
> -336> 2015-03-11 01:00:13.880703 7f41a1ba4880 10 monclient(hunting): build_initial_monmap
> -335> 2015-03-11 01:00:13.880697 7f419cf7d700 5 asok(0x4b50000) entry start
> -334> 2015-03-11 01:00:13.890355 7f41a1ba4880 10 ErasureCodePluginSelectJerasure: load: jerasure_generic
> -333> 2015-03-11 01:00:13.897843 7f41a1ba4880 10 load: jerasure load: lrc load: isa
> -332> 2015-03-11 01:00:13.897900 7f41a1ba4880 5 adding auth protocol: none
> -331> 2015-03-11 01:00:13.897905 7f41a1ba4880 5 adding auth protocol: none
> -330> 2015-03-11 01:00:13.898086 7f41a1ba4880 5 asok(0x4b50000) register_command objecter_requests hook 0x4b041b0
> -329> 2015-03-11 01:00:13.898172 7f41a1ba4880 5 filestore(/var/lib/ceph/osd/ceph-3) test_mount basedir /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal
> -328> 2015-03-11 01:00:13.898290 7f41a1ba4880 1 -- 0.0.0.0:6800/2840 messenger.start
> -327> 2015-03-11 01:00:13.898362 7f41a1ba4880 1 -- :/0 messenger.start
> -326> 2015-03-11 01:00:13.898450 7f41a1ba4880 1 -- 0.0.0.0:6803/2840 messenger.start
> -325> 2015-03-11 01:00:13.898527 7f41a1ba4880 1 -- 0.0.0.0:6802/2840 messenger.start
> -324> 2015-03-11 01:00:13.898609 7f41a1ba4880 1 -- 0.0.0.0:6801/2840 messenger.start
> -323> 2015-03-11 01:00:13.898672 7f41a1ba4880 1 -- :/0 messenger.start
> -322> 2015-03-11 01:00:13.898792 7f41a1ba4880 2 osd.3 0 mounting /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
> -321> 2015-03-11 01:00:13.898826 7f41a1ba4880 5 filestore(/var/lib/ceph/osd/ceph-3) basedir /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal
> -320> 2015-03-11 01:00:13.898868 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) mount fsid is 05fb481f-68b3-4442-b7de-af4c867e18d5
> -319> 2015-03-11 01:00:13.898937 7f41a1ba4880 0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342)
> -318> 2015-03-11 01:00:13.941605 7f41a1ba4880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work
> -317> 2015-03-11 01:00:13.941665 7f41a1ba4880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
> -316> 2015-03-11 01:00:13.970056 7f41a1ba4880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
> -315> 2015-03-11 01:00:13.970442 7f41a1ba4880 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5
> -314> 2015-03-11 01:00:14.001199 7f41a1ba4880 5 filestore(/var/lib/ceph/osd/ceph-3) mount op_seq is 2
> -313> 2015-03-11 01:00:14.122464 7f41a1ba4880 20 filestore (init)dbobjectmap: seq is 1
> -312> 2015-03-11 01:00:14.122580 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) open_journal at /var/lib/ceph/osd/ceph-3/journal
> -311> 2015-03-11 01:00:14.122674 7f41a1ba4880 0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
> -310> 2015-03-11 01:00:14.122682 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) list_collections
> -309> 2015-03-11 01:00:14.122987 7f41a1ba4880 2 journal open /var/lib/ceph/osd/ceph-3/journal fsid 05fb481f-68b3-4442-b7de-af4c867e18d5 fs_op_seq 2
> -308> 2015-03-11 01:00:14.123024 7f41a1ba4880 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
> -307> 2015-03-11 01:00:14.123040 7f41a1ba4880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> -306> 2015-03-11 01:00:14.123003 7f4196c1b700 20 filestore(/var/lib/ceph/osd/ceph-3) sync_entry waiting for max_interval 5.000000
> -305> 2015-03-11 01:00:14.123590 7f41a1ba4880 2 journal read_entry 8192 : seq 3 193 bytes
> -304> 2015-03-11 01:00:14.123639 7f41a1ba4880 2 journal read_entry 8192 : seq 3 193 bytes
> -303> 2015-03-11 01:00:14.123661 7f41a1ba4880 3 journal journal_replay: applying op seq 3
> -302> 2015-03-11 01:00:14.123724 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x4b9b6c0
> -301> 2015-03-11 01:00:14.123837 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
> -300> 2015-03-11 01:00:14.123971 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
> -299> 2015-03-11 01:00:14.124020 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) touch meta/a468ec03/snapmapper/0//-1
> -298> 2015-03-11 01:00:14.124103 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) touch meta/a468ec03/snapmapper/0//-1 = 0
> -297> 2015-03-11 01:00:14.124163 7f41a1ba4880 3 journal journal_replay: r = 0, op_seq now 3
> -296> 2015-03-11 01:00:14.124220 7f41a1ba4880 2 journal read_entry 12288 : seq 4 3829 bytes
> -295> 2015-03-11 01:00:14.124231 7f41a1ba4880 3 journal journal_replay: applying op seq 4
> -294> 2015-03-11 01:00:14.124247 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x4b9b6c0
> -293> 2015-03-11 01:00:14.124282 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
> -292> 2015-03-11 01:00:14.124332 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
> -291> 2015-03-11 01:00:14.124348 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) write meta/a143428/osdmap.2757/0//-1 0~3076
> -290> 2015-03-11 01:00:14.124420 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) write meta/a143428/osdmap.2757/0//-1 0~3076 = 3076
> -289> 2015-03-11 01:00:14.124461 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr
> -288> 2015-03-11 01:00:14.124547 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr
> -287> 2015-03-11 01:00:14.124557 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) write meta/23c2fcde/osd_superblock/0//-1 0~413
> -286> 2015-03-11 01:00:14.124615 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) write meta/23c2fcde/osd_superblock/0//-1 0~413 = 413
> -285> 2015-03-11 01:00:14.124676 7f41a1ba4880 3 journal journal_replay: r = 0, op_seq now 4
> -284> 2015-03-11 01:00:14.124697 7f41a1ba4880 2 journal No further valid entries found, journal is most likely valid
> -283> 2015-03-11 01:00:14.124700 7f41a1ba4880 3 journal journal_replay: end of journal, done.
> -282> 2015-03-11 01:00:14.124716 7f41a1ba4880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> -281> 2015-03-11 01:00:14.125247 7f41a1ba4880 2 osd.3 0 boot
> -280> 2015-03-11 01:00:14.125270 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) read meta/23c2fcde/osd_superblock/0//-1 0~0
> -279> 2015-03-11 01:00:14.125343 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) FileStore::read meta/23c2fcde/osd_superblock/0//-1 0~413/413
> -278> 2015-03-11 01:00:14.125393 7f41a1ba4880 10 osd.3 0 read_superblock sb(3bf19e54-97e2-4a5f-b811-7c3780cd5e57 osd.3 05fb481f-68b3-4442-b7de-af4c867e18d5 e2757 [2757,2757] lci=[0,0])
> -277> 2015-03-11 01:00:14.125521 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) stat meta/a468ec03/snapmapper/0//-1 = 0 (size 0)
> -276> 2015-03-11 01:00:14.125540 7f41a1ba4880 10 open_all_classes
> -275> 2015-03-11 01:00:14.125586 7f41a1ba4880 10 open_all_classes found hello
> -274> 2015-03-11 01:00:14.125623 7f41a1ba4880 10 _get_class adding new class name hello 0x4c9e028
> -273> 2015-03-11 01:00:14.125630 7f41a1ba4880 10 _load_class hello from /usr/lib64/rados-classes/libcls_hello.so
> -272> 2015-03-11 01:00:14.126058 7f41a1ba4880 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
> -271> 2015-03-11 01:00:14.126080 7f41a1ba4880 10 register_class hello status 3
> -270> 2015-03-11 01:00:14.126084 7f41a1ba4880 10 register_cxx_method hello.say_hello flags 1 0x7f4192a10350
> -269> 2015-03-11 01:00:14.126092 7f41a1ba4880 10 register_cxx_method hello.record_hello flags 2 0x7f4192a113c0
> -268> 2015-03-11 01:00:14.126098 7f41a1ba4880 10 register_cxx_method hello.writes_dont_return_data flags 2 0x7f4192a10650
> -267> 2015-03-11 01:00:14.126103 7f41a1ba4880 10 register_cxx_method hello.replay flags 1 0x7f4192a10320
> -266> 2015-03-11 01:00:14.126109 7f41a1ba4880 10 register_cxx_method hello.turn_it_to_11 flags 3 0x7f4192a10920
> -265> 2015-03-11 01:00:14.126114 7f41a1ba4880 10 register_cxx_method hello.bad_reader flags 2 0x7f4192a10310
> -264> 2015-03-11 01:00:14.126119 7f41a1ba4880 10 register_cxx_method hello.bad_writer flags 1 0x7f4192a10300
> -263> 2015-03-11 01:00:14.126127 7f41a1ba4880 10 _load_class hello success
> -262> 2015-03-11 01:00:14.126132 7f41a1ba4880 10 open_all_classes found kvs
> -261> 2015-03-11 01:00:14.126137 7f41a1ba4880 10 _get_class adding new class name kvs 0x4c9e108
> -260> 2015-03-11 01:00:14.126142 7f41a1ba4880 10 _load_class kvs from /usr/lib64/rados-classes/libcls_kvs.so
> -259> 2015-03-11 01:00:14.126655 7f41a1ba4880 10 register_class kvs status 3
> -258> 2015-03-11 01:00:14.126665 7f41a1ba4880 10 register_cxx_method kvs.get_idata_from_key flags 1 0x7f41927ff060
> -257> 2015-03-11 01:00:14.126669 7f41a1ba4880 10 register_cxx_method kvs.get_next_idata flags 1 0x7f4192801050
> -256> 2015-03-11 01:00:14.126672 7f41a1ba4880 10 register_cxx_method kvs.get_prev_idata flags 1 0x7f41927fd570
> -255> 2015-03-11 01:00:14.126675 7f41a1ba4880 10 register_cxx_method kvs.read_many flags 1 0x7f41927f8f30
> -254> 2015-03-11 01:00:14.126678 7f41a1ba4880 10 register_cxx_method kvs.check_writable flags 3 0x7f41927f8ac0
> -253> 2015-03-11 01:00:14.126681 7f41a1ba4880 10 register_cxx_method kvs.assert_size_in_bound flags 2 0x7f41927f8d40
> -252> 2015-03-11 01:00:14.126684 7f41a1ba4880 10 register_cxx_method kvs.omap_insert flags 2 0x7f41927fcb50
> -251> 2015-03-11 01:00:14.126688 7f41a1ba4880 10 register_cxx_method kvs.create_with_omap flags 2 0x7f41927fc0e0
> -250> 2015-03-11 01:00:14.126691 7f41a1ba4880 10 register_cxx_method kvs.omap_remove flags 2 0x7f41927f9e70
> -249> 2015-03-11 01:00:14.126696 7f41a1ba4880 10 register_cxx_method kvs.maybe_read_for_balance flags 1 0x7f41927fab70
> -248> 2015-03-11 01:00:14.126735 7f41a1ba4880 10 _load_class kvs success
> -247> 2015-03-11 01:00:14.126739 7f41a1ba4880 10 open_all_classes found lock
> -246> 2015-03-11 01:00:14.126742 7f41a1ba4880 10 _get_class adding new class name lock 0x4c9e1e8
> -245> 2015-03-11 01:00:14.126745 7f41a1ba4880 10 _load_class lock from /usr/lib64/rados-classes/libcls_lock.so
> -244> 2015-03-11 01:00:14.126972 7f41a1ba4880 10 register_class lock status 3
> -243> 2015-03-11 01:00:14.126977 7f41a1ba4880 10 register_cxx_method lock.lock flags 3 0x7f41925ee800
> -242> 2015-03-11 01:00:14.126981 7f41a1ba4880 10 register_cxx_method lock.unlock flags 3 0x7f41925ed210
> -241> 2015-03-11 01:00:14.126984 7f41a1ba4880 10 register_cxx_method lock.break_lock flags 3 0x7f41925ece60
> -240> 2015-03-11 01:00:14.126986 7f41a1ba4880 10 register_cxx_method lock.get_info flags 1 0x7f41925ed610
> -239> 2015-03-11 01:00:14.126990 7f41a1ba4880 10 register_cxx_method lock.list_locks flags 1 0x7f41925eace0
> -238> 2015-03-11 01:00:14.126993 7f41a1ba4880 10 register_cxx_method lock.assert_locked flags 1 0x7f41925ec330
> -237> 2015-03-11 01:00:14.126996 7f41a1ba4880 10 _load_class lock success
> -236> 2015-03-11 01:00:14.126999 7f41a1ba4880 10 open_all_classes found log
> -235> 2015-03-11 01:00:14.127002 7f41a1ba4880 10 _get_class adding new class name log 0x4c9e2c8
> -234> 2015-03-11 01:00:14.127005 7f41a1ba4880 10 _load_class log from /usr/lib64/rados-classes/libcls_log.so
> -233> 2015-03-11 01:00:14.127213 7f41a1ba4880 1 <cls> cls/log/cls_log.cc:312: Loaded log class!
> -232> 2015-03-11 01:00:14.127223 7f41a1ba4880 10 register_class log status 3
> -231> 2015-03-11 01:00:14.127225 7f41a1ba4880 10 register_cxx_method log.add flags 3 0x7f41923e1560
> -230> 2015-03-11 01:00:14.127228 7f41a1ba4880 10 register_cxx_method log.list flags 1 0x7f41923e2740
> -229> 2015-03-11 01:00:14.127231 7f41a1ba4880 10 register_cxx_method log.trim flags 3 0x7f41923e1f90
> -228> 2015-03-11 01:00:14.127234 7f41a1ba4880 10 register_cxx_method log.info flags 1 0x7f41923e1090
> -227> 2015-03-11 01:00:14.127238 7f41a1ba4880 10 _load_class log success
> -226> 2015-03-11 01:00:14.127241 7f41a1ba4880 10 open_all_classes found rbd
> -225> 2015-03-11 01:00:14.127244 7f41a1ba4880 10 _get_class adding new class name rbd 0x4c9e3a8
> -224> 2015-03-11 01:00:14.127247 7f41a1ba4880 10 _load_class rbd from /usr/lib64/rados-classes/libcls_rbd.so
> -223> 2015-03-11 01:00:14.127735 7f41a1ba4880 10 register_class rbd status 3
> -222> 2015-03-11 01:00:14.127745 7f41a1ba4880 10 register_cxx_method rbd.create flags 3 0x7f41921cc5c0
> -221> 2015-03-11 01:00:14.127748 7f41a1ba4880 10 register_cxx_method rbd.get_features flags 1 0x7f41921d3190
> -220> 2015-03-11 01:00:14.127751 7f41a1ba4880 10 register_cxx_method rbd.get_size flags 1 0x7f41921d2de0
> -219> 2015-03-11 01:00:14.127754 7f41a1ba4880 10 register_cxx_method rbd.set_size flags 3 0x7f41921ca7e0
> -218> 2015-03-11 01:00:14.127758 7f41a1ba4880 10 register_cxx_method rbd.get_snapcontext flags 1 0x7f41921cdb40
> -217> 2015-03-11 01:00:14.127762 7f41a1ba4880 10 register_cxx_method rbd.get_object_prefix flags 1 0x7f41921c9770
> -216> 2015-03-11 01:00:14.127766 7f41a1ba4880 10 register_cxx_method rbd.get_snapshot_name flags 1 0x7f41921d1dd0
> -215> 2015-03-11 01:00:14.127770 7f41a1ba4880 10 register_cxx_method rbd.snapshot_add flags 3 0x7f41921d0830
> -214> 2015-03-11 01:00:14.127773 7f41a1ba4880 10 register_cxx_method rbd.snapshot_remove flags 3 0x7f41921d1b40
> -213> 2015-03-11 01:00:14.127776 7f41a1ba4880 10 register_cxx_method rbd.get_all_features flags 1 0x7f41921c78c0
> -212> 2015-03-11 01:00:14.127779 7f41a1ba4880 10 register_cxx_method rbd.copyup flags 3 0x7f41921c7940
> -211> 2015-03-11 01:00:14.127782 7f41a1ba4880 10 register_cxx_method rbd.get_parent flags 1 0x7f41921d2080
> -210> 2015-03-11 01:00:14.127786 7f41a1ba4880 10 register_cxx_method rbd.set_parent flags 3 0x7f41921ccce0
> -209> 2015-03-11 01:00:14.127789 7f41a1ba4880 10 register_cxx_method rbd.remove_parent flags 3 0x7f41921c9140
> -208> 2015-03-11 01:00:14.127792 7f41a1ba4880 10 register_cxx_method rbd.set_protection_status flags 3 0x7f41921d2770
> -207> 2015-03-11 01:00:14.127815 7f41a1ba4880 10 register_cxx_method rbd.get_protection_status flags 1 0x7f41921d2b60
> -206> 2015-03-11 01:00:14.127818 7f41a1ba4880 10 register_cxx_method rbd.get_stripe_unit_count flags 1 0x7f41921c9330
> -205> 2015-03-11 01:00:14.127822 7f41a1ba4880 10 register_cxx_method rbd.set_stripe_unit_count flags 3 0x7f41921cc070
> -204> 2015-03-11 01:00:14.127825 7f41a1ba4880 10 register_cxx_method rbd.get_flags flags 1 0x7f41921d2470
> -203> 2015-03-11 01:00:14.127828 7f41a1ba4880 10 register_cxx_method rbd.set_flags flags 3 0x7f41921cb370
> -202> 2015-03-11 01:00:14.127831 7f41a1ba4880 10 register_cxx_method rbd.add_child flags 3 0x7f41921cd790
> -201> 2015-03-11 01:00:14.127834 7f41a1ba4880 10 register_cxx_method rbd.remove_child flags 3 0x7f41921cdf50
> -200> 2015-03-11 01:00:14.127838 7f41a1ba4880 10 register_cxx_method rbd.get_children flags 1 0x7f41921cd4e0
> -199> 2015-03-11 01:00:14.127841 7f41a1ba4880 10 register_cxx_method rbd.get_id flags 1 0x7f41921ca580
> -198> 2015-03-11 01:00:14.127844 7f41a1ba4880 10 register_cxx_method rbd.set_id flags 3 0x7f41921cb6c0
> -197> 2015-03-11 01:00:14.127848 7f41a1ba4880 10 register_cxx_method rbd.dir_get_id flags 1 0x7f41921c9f40
> -196> 2015-03-11 01:00:14.127851 7f41a1ba4880 10 register_cxx_method rbd.dir_get_name flags 1 0x7f41921ca210
> -195> 2015-03-11 01:00:14.127855 7f41a1ba4880 10 register_cxx_method rbd.dir_list flags 1 0x7f41921cf9f0
> -194> 2015-03-11 01:00:14.127858 7f41a1ba4880 10 register_cxx_method rbd.dir_add_image flags 3 0x7f41921cf840
> -193> 2015-03-11 01:00:14.127862 7f41a1ba4880 10 register_cxx_method rbd.dir_remove_image flags 3 0x7f41921c9d60
> -192> 2015-03-11 01:00:14.127865 7f41a1ba4880 10 register_cxx_method rbd.dir_rename_image flags 3 0x7f41921cf660
> -191> 2015-03-11 01:00:14.127868 7f41a1ba4880 10 register_cxx_method rbd.object_map_load flags 1 0x7f41921ce5f0
> -190> 2015-03-11 01:00:14.127872 7f41a1ba4880 10 register_cxx_method rbd.object_map_resize flags 3 0x7f41921ceec0
> -189> 2015-03-11 01:00:14.127875 7f41a1ba4880 10 register_cxx_method rbd.object_map_update flags 3 0x7f41921ce720
> -188> 2015-03-11 01:00:14.127879 7f41a1ba4880 10 register_cxx_method rbd.snap_list flags 1 0x7f41921c8540
> -187> 2015-03-11 01:00:14.127882 7f41a1ba4880 10 register_cxx_method rbd.snap_add flags 3 0x7f41921cacc0
> -186> 2015-03-11 01:00:14.127885 7f41a1ba4880 10 register_cxx_method rbd.snap_remove flags 3 0x7f41921cb910
> -185> 2015-03-11 01:00:14.127891 7f41a1ba4880 10 _load_class rbd success
> -184> 2015-03-11 01:00:14.127894 7f41a1ba4880 10 open_all_classes found refcount
> -183> 2015-03-11 01:00:14.127897 7f41a1ba4880 10 _get_class adding new class name refcount 0x4c9e488
> -182> 2015-03-11 01:00:14.127900 7f41a1ba4880 10 _load_class refcount from /usr/lib64/rados-classes/libcls_refcount.so
> -181> 2015-03-11 01:00:14.132905 7f41a1ba4880 1 <cls> cls/refcount/cls_refcount.cc:231: Loaded refcount class!
> -180> 2015-03-11 01:00:14.132942 7f41a1ba4880 10 register_class refcount status 3
> -179> 2015-03-11 01:00:14.132946 7f41a1ba4880 10 register_cxx_method refcount.get flags 3 0x7f4191f32be0
> -178> 2015-03-11 01:00:14.132957 7f41a1ba4880 10 register_cxx_method refcount.put flags 3 0x7f4191f31b30
> -177> 2015-03-11 01:00:14.132960 7f41a1ba4880 10 register_cxx_method refcount.set flags 3 0x7f4191f32510
> -176> 2015-03-11 01:00:14.132963 7f41a1ba4880 10 register_cxx_method refcount.read flags 1 0x7f4191f31fb0
> -175> 2015-03-11 01:00:14.132970 7f41a1ba4880 10 _load_class refcount success
> -174> 2015-03-11 01:00:14.132977 7f41a1ba4880 10 open_all_classes found replica_log
> -173> 2015-03-11 01:00:14.132984 7f41a1ba4880 10 _get_class adding new class name replica_log 0x4c9e568
> -172> 2015-03-11 01:00:14.132987 7f41a1ba4880 10 _load_class replica_log from /usr/lib64/rados-classes/libcls_replica_log.so
> -171> 2015-03-11 01:00:14.133259 7f41a1ba4880 1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class!
> -170> 2015-03-11 01:00:14.133269 7f41a1ba4880 10 register_class replica_log status 3
> -169> 2015-03-11 01:00:14.133271 7f41a1ba4880 10 register_cxx_method replica_log.set flags 3 0x7f4191c7f920
> -168> 2015-03-11 01:00:14.133276 7f41a1ba4880 10 register_cxx_method replica_log.get flags 1 0x7f4191c809f0
> -167> 2015-03-11 01:00:14.133279 7f41a1ba4880 10 register_cxx_method replica_log.delete flags 3 0x7f4191c80400
> -166> 2015-03-11 01:00:14.133283 7f41a1ba4880 10 _load_class replica_log success
> -165> 2015-03-11 01:00:14.133285 7f41a1ba4880 10 open_all_classes found rgw
> -164> 2015-03-11 01:00:14.133288 7f41a1ba4880 10 _get_class adding new class name rgw 0x4c9e728
> -163> 2015-03-11 01:00:14.133291 7f41a1ba4880 10 _load_class rgw from /usr/lib64/rados-classes/libcls_rgw.so
> -162> 2015-03-11 01:00:14.138677 7f41a1ba4880 1 <cls> cls/rgw/cls_rgw.cc:3046: Loaded rgw class!
> -161> 2015-03-11 01:00:14.138712 7f41a1ba4880 10 register_class rgw status 3
> -160> 2015-03-11 01:00:14.138717 7f41a1ba4880 10 register_cxx_method rgw.bucket_init_index flags 3 0x7f41919c0860
> -159> 2015-03-11 01:00:14.138725 7f41a1ba4880 10 register_cxx_method rgw.bucket_set_tag_timeout flags 3 0x7f41919c4a40
> -158> 2015-03-11 01:00:14.138728 7f41a1ba4880 10 register_cxx_method rgw.bucket_list flags 1 0x7f41919c8160
> -157> 2015-03-11 01:00:14.138732 7f41a1ba4880 10 register_cxx_method rgw.bucket_check_index flags 1 0x7f41919ca300
> -156> 2015-03-11 01:00:14.138735 7f41a1ba4880 10 register_cxx_method rgw.bucket_rebuild_index flags 3 0x7f41919ca1a0
> -155> 2015-03-11 01:00:14.138740 7f41a1ba4880 10 register_cxx_method rgw.bucket_prepare_op flags 3 0x7f41919c5180
> -154> 2015-03-11 01:00:14.138745 7f41a1ba4880 10 register_cxx_method rgw.bucket_complete_op flags 3 0x7f41919c5980
> -153> 2015-03-11 01:00:14.138748 7f41a1ba4880 10 register_cxx_method rgw.bucket_link_olh flags 3 0x7f41919cc980
> -152> 2015-03-11 01:00:14.138751 7f41a1ba4880 10 register_cxx_method rgw.bucket_unlink_instance flags 3 0x7f41919cbb80
> -151> 2015-03-11 01:00:14.138754 7f41a1ba4880 10 register_cxx_method rgw.bucket_read_olh_log flags 1 0x7f41919cde40
> -150> 2015-03-11 01:00:14.138757 7f41a1ba4880 10 register_cxx_method rgw.bucket_trim_olh_log flags 3 0x7f41919cb2d0
> -149> 2015-03-11 01:00:14.138760 7f41a1ba4880 10 register_cxx_method rgw.bucket_clear_olh flags 3 0x7f41919cacd0
> -148> 2015-03-11 01:00:14.138763 7f41a1ba4880 10 register_cxx_method rgw.obj_remove flags 3 0x7f41919c3ff0
> -147> 2015-03-11 01:00:14.138771 7f41a1ba4880 10 register_cxx_method rgw.obj_check_attrs_prefix flags 1 0x7f41919c0b60
> -146> 2015-03-11 01:00:14.138774 7f41a1ba4880 10 register_cxx_method rgw.bi_get flags 1 0x7f41919bf050
> -145> 2015-03-11 01:00:14.138777 7f41a1ba4880 10 register_cxx_method rgw.bi_put flags 3 0x7f41919be700
> -144> 2015-03-11 01:00:14.138780 7f41a1ba4880 10 register_cxx_method rgw.bi_list flags 1 0x7f41919c6790
> -143> 2015-03-11 01:00:14.138784 7f41a1ba4880 10 register_cxx_method rgw.bi_log_list flags 1 0x7f41919c2180
> -142> 2015-03-11 01:00:14.138787 7f41a1ba4880 10 register_cxx_method rgw.bi_log_trim flags 3 0x7f41919c26e0
> -141> 2015-03-11 01:00:14.138791 7f41a1ba4880 10 register_cxx_method rgw.dir_suggest_changes flags 3 0x7f41919c9260
> -140> 2015-03-11 01:00:14.138794 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_add flags 3 0x7f41919ce6a0
> -139> 2015-03-11 01:00:14.138797 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_read flags 1 0x7f41919c31c0
> -138> 2015-03-11 01:00:14.138799 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_trim flags 3 0x7f41919c3950
> -137> 2015-03-11 01:00:14.138802 7f41a1ba4880 10 register_cxx_method rgw.gc_set_entry flags 3 0x7f41919c01b0
> -136> 2015-03-11 01:00:14.138806 7f41a1ba4880 10 register_cxx_method rgw.gc_defer_entry flags 3 0x7f41919bfe90
> -135> 2015-03-11 01:00:14.138809 7f41a1ba4880 10 register_cxx_method rgw.gc_list flags 1 0x7f41919c0f80
> -134> 2015-03-11 01:00:14.138812 7f41a1ba4880 10 register_cxx_method rgw.gc_remove flags 3 0x7f41919c0420
> -133> 2015-03-11 01:00:14.138815 7f41a1ba4880 10 _load_class rgw success
> -132> 2015-03-11 01:00:14.138820 7f41a1ba4880 10 open_all_classes found statelog
> -131> 2015-03-11 01:00:14.138826 7f41a1ba4880 10 _get_class adding new class name statelog 0x4c9e808
> -130> 2015-03-11 01:00:14.138830 7f41a1ba4880 10 _load_class statelog from /usr/lib64/rados-classes/libcls_statelog.so
> -129> 2015-03-11 01:00:14.139131 7f41a1ba4880 1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class!
> -128> 2015-03-11 01:00:14.139141 7f41a1ba4880 10 register_class statelog status 3
> -127> 2015-03-11 01:00:14.139143 7f41a1ba4880 10 register_cxx_method statelog.add flags 3 0x7f4191700bc0
> -126> 2015-03-11 01:00:14.139146 7f41a1ba4880 10 register_cxx_method statelog.list flags 1 0x7f4191700f40
> -125> 2015-03-11 01:00:14.139150 7f41a1ba4880 10 register_cxx_method statelog.remove flags 3 0x7f41916ffa80
> -124> 2015-03-11 01:00:14.139153 7f41a1ba4880 10 register_cxx_method statelog.check_state flags 1 0x7f41917002d0
> -123> 2015-03-11 01:00:14.139175 7f41a1ba4880 10 _load_class statelog success
> -122> 2015-03-11 01:00:14.139178 7f41a1ba4880 10 open_all_classes found user
> -121> 2015-03-11 01:00:14.139182 7f41a1ba4880 10 _get_class adding new class name user 0x4c9e8e8
> -120> 2015-03-11 01:00:14.139185 7f41a1ba4880 10 _load_class user from /usr/lib64/rados-classes/libcls_user.so
> -119> 2015-03-11 01:00:14.139377 7f41a1ba4880 1 <cls> cls/user/cls_user.cc:367: Loaded user class!
> -118> 2015-03-11 01:00:14.139386 7f41a1ba4880 10 register_class user status 3
> -117> 2015-03-11 01:00:14.139388 7f41a1ba4880 10 register_cxx_method user.set_buckets_info flags 3 0x7f41914f5180
> -116> 2015-03-11 01:00:14.139392 7f41a1ba4880 10 register_cxx_method user.complete_stats_sync flags 3 0x7f41914f4aa0
> -115> 2015-03-11 01:00:14.139394 7f41a1ba4880 10 register_cxx_method user.remove_bucket flags 3 0x7f41914f3d30
> -114> 2015-03-11 01:00:14.139397 7f41a1ba4880 10 register_cxx_method user.list_buckets flags 1 0x7f41914f64c0
> -113> 2015-03-11 01:00:14.139400 7f41a1ba4880 10 register_cxx_method user.get_header flags 1 0x7f41914f2d50
> -112> 2015-03-11 01:00:14.139403 7f41a1ba4880 10 _load_class user success
> -111> 2015-03-11 01:00:14.139406 7f41a1ba4880 10 open_all_classes found version
> -110> 2015-03-11 01:00:14.139409 7f41a1ba4880 10 _get_class adding new class name version 0x4c9e9c8
> -109> 2015-03-11 01:00:14.139412 7f41a1ba4880 10 _load_class version from /usr/lib64/rados-classes/libcls_version.so
> -108> 2015-03-11 01:00:14.139604 7f41a1ba4880 1 <cls> cls/version/cls_version.cc:227: Loaded version class!
> -107> 2015-03-11 01:00:14.139608 7f41a1ba4880 10 register_class version status 3
> -106> 2015-03-11 01:00:14.139611 7f41a1ba4880 10 register_cxx_method version.set flags 3 0x7f41912ebe60
> -105> 2015-03-11 01:00:14.139615 7f41a1ba4880 10 register_cxx_method version.inc flags 3 0x7f41912ecc80
> -104> 2015-03-11 01:00:14.139618 7f41a1ba4880 10 register_cxx_method version.inc_conds flags 3 0x7f41912ecc80
> -103> 2015-03-11 01:00:14.139621 7f41a1ba4880 10 register_cxx_method version.read flags 1 0x7f41912ebb60
> -102> 2015-03-11 01:00:14.139624 7f41a1ba4880 10 register_cxx_method version.check_conds flags 1 0x7f41912ec260
> -101> 2015-03-11 01:00:14.139627 7f41a1ba4880 10 _load_class version success
> -100> 2015-03-11 01:00:14.139682 7f41a1ba4880 20 osd.3 0 get_map 2757 - loading and decoding 0x4b4cb40
> -99> 2015-03-11 01:00:14.139693 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) read meta/a143428/osdmap.2757/0//-1 0~0
> -98> 2015-03-11 01:00:14.139804 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) FileStore::read meta/a143428/osdmap.2757/0//-1 0~3076/3076
> -97> 2015-03-11 01:00:14.139818 7f41a1ba4880 10 osd.3 0 add_map_bl 2757 3076 bytes
> -96> 2015-03-11 01:00:14.139908 7f41a1ba4880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients
> -95> 2015-03-11 01:00:14.139914 7f41a1ba4880 0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons
> -94> 2015-03-11 01:00:14.139919 7f41a1ba4880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds
> -93> 2015-03-11 01:00:14.139922 7f41a1ba4880 10 osd.3 2757 create_recoverystate_perf
> -92> 2015-03-11 01:00:14.139939 7f41a1ba4880 0 osd.3 2757 load_pgs
> -91> 2015-03-11 01:00:14.139942 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) list_collections
> -90> 2015-03-11 01:00:14.139981 7f41a1ba4880 10 osd.3 2757 load_pgs ignoring unrecognized meta
> -89> 2015-03-11 01:00:14.139985 7f41a1ba4880 0 osd.3 2757 load_pgs opened 0 pgs
> -88> 2015-03-11 01:00:14.139989 7f41a1ba4880 10 osd.3 2757 build_past_intervals_parallel nothing to build
> -87> 2015-03-11 01:00:14.139992 7f41a1ba4880 2 osd.3 2757 superblock: i am osd.3
> -86> 2015-03-11 01:00:14.139995 7f41a1ba4880 10 osd.3 2757 create_logger
> -85> 2015-03-11 01:00:14.140126 7f41a1ba4880 1 accepter.accepter.start
> -84> 2015-03-11 01:00:14.140269 7f41a1ba4880 1 accepter.accepter.start
> -83> 2015-03-11 01:00:14.140434 7f41a1ba4880 1 accepter.accepter.start
> -82> 2015-03-11 01:00:14.140575 7f41a1ba4880 1 accepter.accepter.start
> -81> 2015-03-11 01:00:14.140676 7f41a1ba4880 10 monclient(hunting): init
> -80> 2015-03-11 01:00:14.140691 7f41a1ba4880 5 adding auth protocol: none
> -79> 2015-03-11 01:00:14.140697 7f41a1ba4880 10 monclient(hunting): auth_supported 1 method none
> -78> 2015-03-11 01:00:14.140803 7f41a1ba4880 10 log_channel(default) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info)
> -77> 2015-03-11 01:00:14.140813 7f41a1ba4880 -1 osd.3 2757 log_to_monitors {default=true}
> -76> 2015-03-11 01:00:14.141325 7f41a1ba4880 10 osd.3 2757 set_disk_tp_priority class priority -1
> -75> 2015-03-11 01:00:14.141461 7f41a1ba4880 10 monclient(hunting): renew_subs
> -74> 2015-03-11 01:00:14.141466 7f41a1ba4880 10 monclient(hunting): _reopen_session rank -1 name
> -73> 2015-03-11 01:00:14.141564 7f41a1ba4880 10 monclient(hunting): picked mon.a con 0x4cbe3c0 addr 192.168.0.10:6789/0
> -72> 2015-03-11 01:00:14.141594 7f41a1ba4880 10 monclient(hunting): _send_mon_message to mon.a at 192.168.0.10:6789/0
> -71> 2015-03-11 01:00:14.141599 7f41a1ba4880 1 -- 0.0.0.0:6800/2840 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x4b4cd80 con 0x4cbe3c0
> -70> 2015-03-11 01:00:14.141625 7f41a1ba4880 10 monclient(hunting): renew_subs
> -69> 2015-03-11 01:00:14.142020 7f4180ac7700 20 osd.3 2757 update_osd_stat osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist [])
> -68> 2015-03-11 01:00:14.142048 7f4180ac7700 5 osd.3 2757 heartbeat: osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist [])
> -67> 2015-03-11 01:00:14.142221 7f417e2c2700 10 osd.3 2757 agent_entry start
> -66> 2015-03-11 01:00:14.142241 7f417e2c2700 20 osd.3 2757 agent_entry empty queue
> -65> 2015-03-11 01:00:14.158932 7f41a1ba0700 1 -- 192.168.0.250:6800/2840 learned my addr 192.168.0.250:6800/2840
> -64> 2015-03-11 01:00:14.159058 7f41a1ba0700 10 osd.3 2757 new session (outgoing) 0x4b92280 con=0x4b602c0 addr=192.168.0.250:6800/2840
> -63> 2015-03-11 01:00:14.159166 7f41a1ba0700 10 osd.3 2757 OSD::ms_get_authorizer type=mon
> -62> 2015-03-11 01:00:14.173274 7f41912e8700 10 osd.3 2757 ms_handle_connect on mon
> -61> 2015-03-11 01:00:14.173332 7f41912e8700 20 osd.3 2757 send_pg_stats
> -60> 2015-03-11 01:00:14.173348 7f41912e8700 10 monclient(hunting): renew_subs
> -59> 2015-03-11 01:00:14.175783 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x4b4f840 con 0x4cbe3c0
> -58> 2015-03-11 01:00:14.175871 7f41912e8700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
> -57> 2015-03-11 01:00:14.175889 7f41912e8700 10 monclient(hunting): got monmap 1, mon.a is now rank 0
> -56> 2015-03-11 01:00:14.175893 7f41912e8700 10 monclient(hunting): dump:
> epoch 1
> fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57
> last_changed 2013-09-11 22:41:30.819687
> created 2013-09-11 22:41:30.819687
> 0: 192.168.0.10:6789/0 mon.a
>
> -55> 2015-03-11 01:00:14.175940 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (335930126 0 0) 0x4b4fa80 con 0x4cbe3c0
> -54> 2015-03-11 01:00:14.175985 7f41912e8700 10 monclient(hunting): my global_id is 14511547
> -53> 2015-03-11 01:00:14.175988 7f41912e8700 1 monclient(hunting): found mon.a
> -52> 2015-03-11 01:00:14.175991 7f41912e8700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
> -51> 2015-03-11 01:00:14.175997 7f41912e8700 1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x4b31a00 con 0x4cbe3c0
> -50> 2015-03-11 01:00:14.176035 7f41912e8700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
> -49> 2015-03-11 01:00:14.176059 7f41912e8700 1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0,osdmap=2757}) v2 -- ?+0 0x4b32800 con 0x4cbe3c0
> -48> 2015-03-11 01:00:14.176077 7f41912e8700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2015-03-11 00:59:44.176076)
> -47> 2015-03-11 01:00:14.176109 7f41a1ba4880 5 monclient: authenticate success, global_id 14511547
> -46> 2015-03-11 01:00:14.176147 7f41a1ba4880 10 osd.3 2757 ensuring pgs have consumed prior maps
> -45> 2015-03-11 01:00:14.176170 7f41a1ba4880 7 osd.3 2757 consume_map version 2757
> -44> 2015-03-11 01:00:14.176187 7f41a1ba4880 0 osd.3 2757 done with init, starting boot process
> -43> 2015-03-11 01:00:14.176191 7f41a1ba4880 10 osd.3 2757 start_boot - have maps 2757..2757
> -42> 2015-03-11 01:00:14.176197 7f41a1ba4880 10 monclient: get_version osdmap req 0x4bad1c0
> -41> 2015-03-11 01:00:14.176207 7f41a1ba4880 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
> -40> 2015-03-11 01:00:14.176221 7f41a1ba4880 1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x4bc41e0 con 0x4cbe3c0
> -39> 2015-03-11 01:00:14.176628 7f41a1ba4880 5 asok(0x4b50000) register_command status hook 0x4b04290
> -38> 2015-03-11 01:00:14.176655 7f41a1ba4880 5 asok(0x4b50000) register_command flush_journal hook 0x4b04290
> -37> 2015-03-11 01:00:14.176666 7f41a1ba4880 5 asok(0x4b50000) register_command dump_ops_in_flight hook 0x4b04290
> -36> 2015-03-11 01:00:14.176686 7f41a1ba4880 5 asok(0x4b50000) register_command ops hook 0x4b04290
> -35> 2015-03-11 01:00:14.176705 7f41a1ba4880 5 asok(0x4b50000) register_command dump_historic_ops hook 0x4b04290
> -34> 2015-03-11 01:00:14.176723 7f41a1ba4880 5 asok(0x4b50000) register_command dump_op_pq_state hook 0x4b04290
> -33> 2015-03-11 01:00:14.176737 7f41a1ba4880 5 asok(0x4b50000) register_command dump_blacklist hook 0x4b04290
> -32> 2015-03-11 01:00:14.176746 7f41a1ba4880 5 asok(0x4b50000) register_command dump_watchers hook 0x4b04290
> -31> 2015-03-11 01:00:14.176760 7f41a1ba4880 5 asok(0x4b50000) register_command dump_reservations hook 0x4b04290
> -30> 2015-03-11 01:00:14.176770 7f41a1ba4880 5 asok(0x4b50000) register_command get_latest_osdmap hook 0x4b04290
> -29> 2015-03-11 01:00:14.176786 7f41a1ba4880 5 asok(0x4b50000) register_command setomapval hook 0x4bad260
> -28> 2015-03-11 01:00:14.176800 7f41a1ba4880 5 asok(0x4b50000) register_command rmomapkey hook 0x4bad260
> -27> 2015-03-11 01:00:14.176817 7f41a1ba4880 5 asok(0x4b50000) register_command setomapheader hook 0x4bad260
> -26> 2015-03-11 01:00:14.176834 7f41a1ba4880 5 asok(0x4b50000) register_command getomap hook 0x4bad260
> -25> 2015-03-11 01:00:14.176850 7f41a1ba4880 5 asok(0x4b50000) register_command truncobj hook 0x4bad260
> -24> 2015-03-11 01:00:14.176866 7f41a1ba4880 5 asok(0x4b50000) register_command injectdataerr hook 0x4bad260
> -23> 2015-03-11 01:00:14.176878 7f41a1ba4880 5 asok(0x4b50000) register_command injectmdataerr hook 0x4bad260
> -22> 2015-03-11 01:00:14.178539 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 3 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x4b4fcc0 con 0x4cbe3c0
> -21> 2015-03-11 01:00:14.178650 7f41912e8700 10 osd.3 2757 do_waiters -- start
> -20> 2015-03-11 01:00:14.178658 7f41912e8700 10 osd.3 2757 do_waiters -- finish
> -19> 2015-03-11 01:00:14.178661 7f41912e8700 20 osd.3 2757 _dispatch 0x4b4fcc0 osd_map(3277..3277 src has 2757..3277) v3
> -18> 2015-03-11 01:00:14.178888 7f41912e8700 3 osd.3 2757 handle_osd_map epochs [3277,3277], i have 2757, src has [2757,3277]
> -17> 2015-03-11 01:00:14.178906 7f41912e8700 10 osd.3 2757 handle_osd_map message skips epochs 2758..3276
> -16> 2015-03-11 01:00:14.178914 7f41912e8700 10 monclient: renew_subs
> -15> 2015-03-11 01:00:14.178922 7f41912e8700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
> -14> 2015-03-11 01:00:14.178954 7f41912e8700 1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=2758}) v2 -- ?+0 0x4b31a00 con 0x4cbe3c0
> -13> 2015-03-11 01:00:14.178991 7f41912e8700 10 osd.3 2757 do_waiters -- start
> -12> 2015-03-11 01:00:14.178996 7f41912e8700 10 osd.3 2757 do_waiters -- finish
> -11> 2015-03-11 01:00:14.179012 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 4 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x4b4f840 con 0x4cbe3c0
> -10> 2015-03-11 01:00:14.179030 7f41912e8700 10 monclient: handle_monmap mon_map magic: 0 v1
> -9> 2015-03-11 01:00:14.179045 7f41912e8700 10 monclient: got monmap 1, mon.a is now rank 0
> -8> 2015-03-11 01:00:14.179051 7f41912e8700 10 monclient: dump:
> epoch 1
> fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57
> last_changed 2013-09-11 22:41:30.819687
> created 2013-09-11 22:41:30.819687
> 0: 192.168.0.10:6789/0 mon.a
>
> -7> 2015-03-11 01:00:14.184757 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(2757..2857 src has 2757..3277) v3 ==== 37648+0+0 (2648067016 0 0) 0x4b4fa80 con 0x4cbe3c0
> -6> 2015-03-11 01:00:14.184831 7f41912e8700 10 osd.3 2757 do_waiters -- start
> -5> 2015-03-11 01:00:14.184838 7f41912e8700 10 osd.3 2757 do_waiters -- finish
> -4> 2015-03-11 01:00:14.184841 7f41912e8700 20 osd.3 2757 _dispatch 0x4b4fa80 osd_map(2757..2857 src has 2757..3277) v3
> -3> 2015-03-11 01:00:14.184872 7f41912e8700 3 osd.3 2757 handle_osd_map epochs [2757,2857], i have 2757, src has [2757,3277]
> -2> 2015-03-11 01:00:14.184884 7f41912e8700 10 osd.3 2757 handle_osd_map got inc map for epoch 2758
> -1> 2015-03-11 01:00:14.185229 7f41912e8700 10 osd.3 2757 handle_osd_map got inc map for epoch 2759
> 0> 2015-03-11 01:00:14.195165 7f41912e8700 -1 *** Caught signal (Aborted) **
> in thread 7f41912e8700
>
> ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
> 1: /usr/bin/ceph-osd() [0xac7cea]
> 2: (()+0x10050) [0x7f41a0aaf050]
> 3: (gsignal()+0x37) [0x7f419ef820f7]
> 4: (abort()+0x13a) [0x7f419ef834ca]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f419f867fe5]
> 6: (()+0x63186) [0x7f419f866186]
> 7: (()+0x631b3) [0x7f419f8661b3]
> 8: (()+0x633d2) [0x7f419f8663d2]
> 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
> 10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
> 11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
> 12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
> 13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
> 14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
> 15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
> 16: (DispatchQueue::entry()+0x649) [0xc6fe09]
> 17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
> 18: (()+0x83a4) [0x7f41a0aa73a4]
> 19: (clone()+0x6d) [0x7f419f031a4d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>
> --- logging levels ---
> 0/ 5 none
> 0/ 1 lockdep
> 0/ 1 context
> 1/ 1 crush
> 1/ 5 mds
> 1/ 5 mds_balancer
> 1/ 5 mds_locker
> 1/ 5 mds_log
> 1/ 5 mds_log_expire
> 1/ 5 mds_migrator
> 0/ 1 buffer
> 0/ 1 timer
> 0/ 1 filer
> 0/ 1 striper
> 0/ 1 objecter
> 0/ 5 rados
> 0/ 5 rbd
> 0/ 5 rbd_replay
> 0/ 5 journaler
> 0/ 5 objectcacher
> 0/ 5 client
> 20/20 osd
> 0/ 5 optracker
> 0/ 5 objclass
> 20/20 filestore
> 1/ 3 keyvaluestore
> 1/ 3 journal
> 1/ 1 ms
> 1/ 5 mon
> 0/10 monc
> 1/ 5 paxos
> 0/ 5 tp
> 1/ 5 auth
> 1/ 5 crypto
> 1/ 1 finisher
> 1/ 5 heartbeatmap
> 1/ 5 perfcounter
> 1/ 5 rgw
> 1/10 civetweb
> 1/ 5 javaclient
> 1/ 5 asok
> 1/ 1 throttle
> 0/ 0 refs
> 1/ 5 xio
> -2/-2 (syslog threshold)
> -1/-1 (stderr threshold)
> max_recent 10000
> max_new 1000
> log_file /var/log/ceph/ceph-osd.3.log
> --- end dump of recent events ---
>
>
> -----Original Message-----
> From: Samuel Just [mailto:sjust@xxxxxxxxxx]
> Sent: Wednesday, 11 March 2015 12:59 AM
> To: Malcolm Haak
> Cc: ceph-users@xxxxxxxxxxxxxx
> Subject: Re: Issues with fresh 0.93 OSD adding to existing cluster
>
> Can you reproduce this with
>
> debug osd = 20
> debug filestore = 20
> debug ms = 1
>
> on the crashing osd? Also, what sha1 are the other osds and mons running?
> -Sam
>
> ----- Original Message -----
> From: "Malcolm Haak" <malcolm@xxxxxxx>
> To: ceph-users@xxxxxxxxxxxxxx
> Sent: Tuesday, March 10, 2015 3:28:26 AM
> Subject: Issues with fresh 0.93 OSD adding to existing cluster
>
> Hi all,
>
> I've just attempted to add a new node and OSD to an existing ceph cluster (it's a small one I use as a NAS at home, not like the big production ones I normally work on) and it seems to be throwing some odd errors...
>
> Just looking for where to poke it next...
>
> Log is below,
>
> It's a two node cluster with 3 osd's in node A and one osd in the new node (It's going to have more eventually and node one will be retired after node three gets added) And I've hit a weird snag.
>
> I was running 0.80 but I ran into the 'Invalid Command' bug on the new node so I opted to jump to the latest code with the required patches already.
>
> Please let me know what else you need..
>
> This is the log content when attempting to start the new OSD:
>
> 2015-03-10 19:28:48.795318 7f0774108880 0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 10810
> 2015-03-10 19:28:48.817803 7f0774108880 0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342)
> 2015-03-10 19:28:48.866862 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work
> 2015-03-10 19:28:48.866920 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
> 2015-03-10 19:28:48.905069 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
> 2015-03-10 19:28:48.905467 7f0774108880 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5
> 2015-03-10 19:28:49.077872 7f0774108880 0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
> 2015-03-10 19:28:49.078321 7f0774108880 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
> 2015-03-10 19:28:49.078328 7f0774108880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> 2015-03-10 19:28:49.079721 7f0774108880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> 2015-03-10 19:28:49.080948 7f0774108880 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
> 2015-03-10 19:28:49.094194 7f0774108880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients
> 2015-03-10 19:28:49.094211 7f0774108880 0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons
> 2015-03-10 19:28:49.094217 7f0774108880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds
> 2015-03-10 19:28:49.094235 7f0774108880 0 osd.3 2757 load_pgs
> 2015-03-10 19:28:49.094279 7f0774108880 0 osd.3 2757 load_pgs opened 0 pgs
> 2015-03-10 19:28:49.095121 7f0774108880 -1 osd.3 2757 log_to_monitors {default=true}
> 2015-03-10 19:28:49.134104 7f0774108880 0 osd.3 2757 done with init, starting boot process
> 2015-03-10 19:28:49.149994 7f076384c700 -1 *** Caught signal (Aborted) ** in thread 7f076384c700
>
> ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
> 1: /usr/bin/ceph-osd() [0xac7cea]
> 2: (()+0x10050) [0x7f0773013050]
> 3: (gsignal()+0x37) [0x7f07714e60f7]
> 4: (abort()+0x13a) [0x7f07714e74ca]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f0771dcbfe5]
> 6: (()+0x63186) [0x7f0771dca186]
> 7: (()+0x631b3) [0x7f0771dca1b3]
> 8: (()+0x633d2) [0x7f0771dca3d2]
> 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
> 10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
> 11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
> 12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
> 13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
> 14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
> 15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
> 16: (DispatchQueue::entry()+0x649) [0xc6fe09]
> 17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
> 18: (()+0x83a4) [0x7f077300b3a4]
> 19: (clone()+0x6d) [0x7f0771595a4d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>
> --- begin dump of recent events ---
> -135> 2015-03-10 19:28:48.790490 7f0774108880 5 asok(0x4200000) register_command perfcounters_dump hook 0x41b4030
> -134> 2015-03-10 19:28:48.790565 7f0774108880 5 asok(0x4200000) register_command 1 hook 0x41b4030
> -133> 2015-03-10 19:28:48.790571 7f0774108880 5 asok(0x4200000) register_command perf dump hook 0x41b4030
> -132> 2015-03-10 19:28:48.790583 7f0774108880 5 asok(0x4200000) register_command perfcounters_schema hook 0x41b4030
> -131> 2015-03-10 19:28:48.790599 7f0774108880 5 asok(0x4200000) register_command 2 hook 0x41b4030
> -130> 2015-03-10 19:28:48.790605 7f0774108880 5 asok(0x4200000) register_command perf schema hook 0x41b4030
> -129> 2015-03-10 19:28:48.790635 7f0774108880 5 asok(0x4200000) register_command perf reset hook 0x41b4030
> -128> 2015-03-10 19:28:48.790646 7f0774108880 5 asok(0x4200000) register_command config show hook 0x41b4030
> -127> 2015-03-10 19:28:48.790652 7f0774108880 5 asok(0x4200000) register_command config set hook 0x41b4030
> -126> 2015-03-10 19:28:48.790660 7f0774108880 5 asok(0x4200000) register_command config get hook 0x41b4030
> -125> 2015-03-10 19:28:48.790672 7f0774108880 5 asok(0x4200000) register_command config diff hook 0x41b4030
> -124> 2015-03-10 19:28:48.790682 7f0774108880 5 asok(0x4200000) register_command log flush hook 0x41b4030
> -123> 2015-03-10 19:28:48.790689 7f0774108880 5 asok(0x4200000) register_command log dump hook 0x41b4030
> -122> 2015-03-10 19:28:48.790698 7f0774108880 5 asok(0x4200000) register_command log reopen hook 0x41b4030
> -121> 2015-03-10 19:28:48.795318 7f0774108880 0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 10810
> -120> 2015-03-10 19:28:48.796071 7f0774108880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/10810 need_addr=1
> -119> 2015-03-10 19:28:48.796123 7f0774108880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/10810 need_addr=1
> -118> 2015-03-10 19:28:48.796144 7f0774108880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/10810 need_addr=1
> -117> 2015-03-10 19:28:48.796160 7f0774108880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/10810 need_addr=1
> -116> 2015-03-10 19:28:48.799414 7f0774108880 5 asok(0x4200000) init /var/run/ceph/ceph-osd.3.asok
> -115> 2015-03-10 19:28:48.799446 7f0774108880 5 asok(0x4200000) bind_and_listen /var/run/ceph/ceph-osd.3.asok
> -114> 2015-03-10 19:28:48.799644 7f0774108880 5 asok(0x4200000) register_command 0 hook 0x41b00c0
> -113> 2015-03-10 19:28:48.799659 7f0774108880 5 asok(0x4200000) register_command version hook 0x41b00c0
> -112> 2015-03-10 19:28:48.799665 7f0774108880 5 asok(0x4200000) register_command git_version hook 0x41b00c0
> -111> 2015-03-10 19:28:48.799671 7f0774108880 5 asok(0x4200000) register_command help hook 0x41b4110
> -110> 2015-03-10 19:28:48.799678 7f0774108880 5 asok(0x4200000) register_command get_command_descriptions hook 0x41b4120
> -109> 2015-03-10 19:28:48.799760 7f0774108880 10 monclient(hunting): build_initial_monmap
> -108> 2015-03-10 19:28:48.799757 7f076f4e1700 5 asok(0x4200000) entry start
> -107> 2015-03-10 19:28:48.816916 7f0774108880 5 adding auth protocol: none
> -106> 2015-03-10 19:28:48.816948 7f0774108880 5 adding auth protocol: none
> -105> 2015-03-10 19:28:48.817130 7f0774108880 5 asok(0x4200000) register_command objecter_requests hook 0x41b41b0
> -104> 2015-03-10 19:28:48.817273 7f0774108880 1 -- 0.0.0.0:6800/10810 messenger.start
> -103> 2015-03-10 19:28:48.817334 7f0774108880 1 -- :/0 messenger.start
> -102> 2015-03-10 19:28:48.817378 7f0774108880 1 -- 0.0.0.0:6803/10810 messenger.start
> -101> 2015-03-10 19:28:48.817442 7f0774108880 1 -- 0.0.0.0:6802/10810 messenger.start
> -100> 2015-03-10 19:28:48.817494 7f0774108880 1 -- 0.0.0.0:6801/10810 messenger.start
> -99> 2015-03-10 19:28:48.817546 7f0774108880 1 -- :/0 messenger.start
> -98> 2015-03-10 19:28:48.817666 7f0774108880 2 osd.3 0 mounting /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
> -97> 2015-03-10 19:28:48.817803 7f0774108880 0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342)
> -96> 2015-03-10 19:28:48.866862 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work
> -95> 2015-03-10 19:28:48.866920 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
> -94> 2015-03-10 19:28:48.905069 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
> -93> 2015-03-10 19:28:48.905467 7f0774108880 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5
> -92> 2015-03-10 19:28:49.077872 7f0774108880 0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
> -91> 2015-03-10 19:28:49.078270 7f0774108880 2 journal open /var/lib/ceph/osd/ceph-3/journal fsid 05fb481f-68b3-4442-b7de-af4c867e18d5 fs_op_seq 2
> -90> 2015-03-10 19:28:49.078321 7f0774108880 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
> -89> 2015-03-10 19:28:49.078328 7f0774108880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> -88> 2015-03-10 19:28:49.078937 7f0774108880 2 journal read_entry 8192 : seq 3 193 bytes
> -87> 2015-03-10 19:28:49.079000 7f0774108880 2 journal read_entry 8192 : seq 3 193 bytes
> -86> 2015-03-10 19:28:49.079022 7f0774108880 3 journal journal_replay: applying op seq 3
> -85> 2015-03-10 19:28:49.079307 7f0774108880 3 journal journal_replay: r = 0, op_seq now 3
> -84> 2015-03-10 19:28:49.079367 7f0774108880 2 journal read_entry 12288 : seq 4 3829 bytes
> -83> 2015-03-10 19:28:49.079390 7f0774108880 3 journal journal_replay: applying op seq 4
> -82> 2015-03-10 19:28:49.079667 7f0774108880 3 journal journal_replay: r = 0, op_seq now 4
> -81> 2015-03-10 19:28:49.079699 7f0774108880 2 journal No further valid entries found, journal is most likely valid
> -80> 2015-03-10 19:28:49.079703 7f0774108880 3 journal journal_replay: end of journal, done.
> -79> 2015-03-10 19:28:49.079721 7f0774108880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> -78> 2015-03-10 19:28:49.080265 7f0774108880 2 osd.3 0 boot
> -77> 2015-03-10 19:28:49.080948 7f0774108880 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
> -76> 2015-03-10 19:28:49.081757 7f0774108880 1 <cls> cls/log/cls_log.cc:312: Loaded log class!
> -75> 2015-03-10 19:28:49.087397 7f0774108880 1 <cls> cls/refcount/cls_refcount.cc:231: Loaded refcount class!
> -74> 2015-03-10 19:28:49.087735 7f0774108880 1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class!
> -73> 2015-03-10 19:28:49.093126 7f0774108880 1 <cls> cls/rgw/cls_rgw.cc:3046: Loaded rgw class!
> -72> 2015-03-10 19:28:49.093484 7f0774108880 1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class!
> -71> 2015-03-10 19:28:49.093723 7f0774108880 1 <cls> cls/user/cls_user.cc:367: Loaded user class!
> -70> 2015-03-10 19:28:49.093924 7f0774108880 1 <cls> cls/version/cls_version.cc:227: Loaded version class!
> -69> 2015-03-10 19:28:49.094194 7f0774108880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients
> -68> 2015-03-10 19:28:49.094211 7f0774108880 0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons
> -67> 2015-03-10 19:28:49.094217 7f0774108880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds
> -66> 2015-03-10 19:28:49.094235 7f0774108880 0 osd.3 2757 load_pgs
> -65> 2015-03-10 19:28:49.094279 7f0774108880 0 osd.3 2757 load_pgs opened 0 pgs
> -64> 2015-03-10 19:28:49.094285 7f0774108880 2 osd.3 2757 superblock: i am osd.3
> -63> 2015-03-10 19:28:49.094424 7f0774108880 1 accepter.accepter.start
> -62> 2015-03-10 19:28:49.094573 7f0774108880 1 accepter.accepter.start
> -61> 2015-03-10 19:28:49.094748 7f0774108880 1 accepter.accepter.start
> -60> 2015-03-10 19:28:49.094864 7f0774108880 1 accepter.accepter.start
> -59> 2015-03-10 19:28:49.094963 7f0774108880 10 monclient(hunting): init
> -58> 2015-03-10 19:28:49.094980 7f0774108880 5 adding auth protocol: none
> -57> 2015-03-10 19:28:49.095005 7f0774108880 10 monclient(hunting): auth_supported 1 method none
> -56> 2015-03-10 19:28:49.095110 7f0774108880 10 log_channel(default) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info)
> -55> 2015-03-10 19:28:49.095121 7f0774108880 -1 osd.3 2757 log_to_monitors {default=true}
> -54> 2015-03-10 19:28:49.095899 7f0774108880 10 monclient(hunting): renew_subs
> -53> 2015-03-10 19:28:49.095914 7f0774108880 10 monclient(hunting): _reopen_session rank -1 name
> -52> 2015-03-10 19:28:49.096003 7f0774108880 10 monclient(hunting): picked mon.a con 0x43623c0 addr 192.168.0.10:6789/0
> -51> 2015-03-10 19:28:49.096086 7f0774108880 10 monclient(hunting): _send_mon_message to mon.a at 192.168.0.10:6789/0
> -50> 2015-03-10 19:28:49.096098 7f0774108880 1 -- 0.0.0.0:6800/10810 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x41fcd80 con 0x43623c0
> -49> 2015-03-10 19:28:49.096132 7f0774108880 10 monclient(hunting): renew_subs
> -48> 2015-03-10 19:28:49.096501 7f075302b700 5 osd.3 2757 heartbeat: osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist [])
> -47> 2015-03-10 19:28:49.113808 7f0774104700 1 -- 192.168.0.250:6800/10810 learned my addr 192.168.0.250:6800/10810
> -46> 2015-03-10 19:28:49.131334 7f076384c700 10 monclient(hunting): renew_subs
> -45> 2015-03-10 19:28:49.133513 7f076384c700 1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x41ff840 con 0x43623c0
> -44> 2015-03-10 19:28:49.133572 7f076384c700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
> -43> 2015-03-10 19:28:49.133587 7f076384c700 10 monclient(hunting): got monmap 1, mon.a is now rank 0
> -42> 2015-03-10 19:28:49.133597 7f076384c700 10 monclient(hunting): dump:
> epoch 1
> fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57
> last_changed 2013-09-11 22:41:30.819687
> created 2013-09-11 22:41:30.819687
> 0: 192.168.0.10:6789/0 mon.a
>
> -41> 2015-03-10 19:28:49.133678 7f076384c700 1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (1541693634 0 0) 0x41ffa80 con 0x43623c0
> -40> 2015-03-10 19:28:49.133716 7f076384c700 10 monclient(hunting): my global_id is 14511543
> -39> 2015-03-10 19:28:49.133721 7f076384c700 1 monclient(hunting): found mon.a
> -38> 2015-03-10 19:28:49.133803 7f076384c700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
> -37> 2015-03-10 19:28:49.133820 7f076384c700 1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x41e1a00 con 0x43623c0
> -36> 2015-03-10 19:28:49.133906 7f076384c700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
> -35> 2015-03-10 19:28:49.133925 7f076384c700 1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0,osdmap=2757}) v2 -- ?+0 0x41e2800 con 0x43623c0
> -34> 2015-03-10 19:28:49.134016 7f076384c700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2015-03-10 19:28:19.134014)
> -33> 2015-03-10 19:28:49.134061 7f0774108880 5 monclient: authenticate success, global_id 14511543
> -32> 2015-03-10 19:28:49.134104 7f0774108880 0 osd.3 2757 done with init, starting boot process
> -31> 2015-03-10 19:28:49.134118 7f0774108880 10 monclient: get_version osdmap req 0x4259160
> -30> 2015-03-10 19:28:49.134126 7f0774108880 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
> -29> 2015-03-10 19:28:49.134136 7f0774108880 1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x42741e0 con 0x43623c0
> -28> 2015-03-10 19:28:49.134362 7f0774108880 5 asok(0x4200000) register_command status hook 0x41b4290
> -27> 2015-03-10 19:28:49.134388 7f0774108880 5 asok(0x4200000) register_command flush_journal hook 0x41b4290
> -26> 2015-03-10 19:28:49.134405 7f0774108880 5 asok(0x4200000) register_command dump_ops_in_flight hook 0x41b4290
> -25> 2015-03-10 19:28:49.134421 7f0774108880 5 asok(0x4200000) register_command ops hook 0x41b4290
> -24> 2015-03-10 19:28:49.134437 7f0774108880 5 asok(0x4200000) register_command dump_historic_ops hook 0x41b4290
> -23> 2015-03-10 19:28:49.134454 7f0774108880 5 asok(0x4200000) register_command dump_op_pq_state hook 0x41b4290
> -22> 2015-03-10 19:28:49.134469 7f0774108880 5 asok(0x4200000) register_command dump_blacklist hook 0x41b4290
> -21> 2015-03-10 19:28:49.134484 7f0774108880 5 asok(0x4200000) register_command dump_watchers hook 0x41b4290
> -20> 2015-03-10 19:28:49.134500 7f0774108880 5 asok(0x4200000) register_command dump_reservations hook 0x41b4290
> -19> 2015-03-10 19:28:49.134516 7f0774108880 5 asok(0x4200000) register_command get_latest_osdmap hook 0x41b4290
> -18> 2015-03-10 19:28:49.134532 7f0774108880 5 asok(0x4200000) register_command setomapval hook 0x4259200
> -17> 2015-03-10 19:28:49.134546 7f0774108880 5 asok(0x4200000) register_command rmomapkey hook 0x4259200
> -16> 2015-03-10 19:28:49.134563 7f0774108880 5 asok(0x4200000) register_command setomapheader hook 0x4259200
> -15> 2015-03-10 19:28:49.134580 7f0774108880 5 asok(0x4200000) register_command getomap hook 0x4259200
> -14> 2015-03-10 19:28:49.134595 7f0774108880 5 asok(0x4200000) register_command truncobj hook 0x4259200
> -13> 2015-03-10 19:28:49.134609 7f0774108880 5 asok(0x4200000) register_command injectdataerr hook 0x4259200
> -12> 2015-03-10 19:28:49.134624 7f0774108880 5 asok(0x4200000) register_command injectmdataerr hook 0x4259200
> -11> 2015-03-10 19:28:49.136325 7f076384c700 1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 3 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x41ffcc0 con 0x43623c0
> -10> 2015-03-10 19:28:49.136622 7f076384c700 3 osd.3 2757 handle_osd_map epochs [3277,3277], i have 2757, src has [2757,3277]
> -9> 2015-03-10 19:28:49.136643 7f076384c700 10 monclient: renew_subs
> -8> 2015-03-10 19:28:49.136652 7f076384c700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0
> -7> 2015-03-10 19:28:49.136679 7f076384c700 1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=2758}) v2 -- ?+0 0x41e1a00 con 0x43623c0
> -6> 2015-03-10 19:28:49.136943 7f076384c700 1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 4 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x41ff840 con 0x43623c0
> -5> 2015-03-10 19:28:49.136994 7f076384c700 10 monclient: handle_monmap mon_map magic: 0 v1
> -4> 2015-03-10 19:28:49.137010 7f076384c700 10 monclient: got monmap 1, mon.a is now rank 0
> -3> 2015-03-10 19:28:49.137022 7f076384c700 10 monclient: dump:
> epoch 1
> fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57
> last_changed 2013-09-11 22:41:30.819687
> created 2013-09-11 22:41:30.819687
> 0: 192.168.0.10:6789/0 mon.a
>
> -2> 2015-03-10 19:28:49.142203 7f076384c700 1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(2757..2857 src has 2757..3277) v3 ==== 37648+0+0 (2648067016 0 0) 0x41ffa80 con 0x43623c0
> -1> 2015-03-10 19:28:49.142316 7f076384c700 3 osd.3 2757 handle_osd_map epochs [2757,2857], i have 2757, src has [2757,3277]
> 0> 2015-03-10 19:28:49.149994 7f076384c700 -1 *** Caught signal (Aborted) ** in thread 7f076384c700
>
> ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4)
> 1: /usr/bin/ceph-osd() [0xac7cea]
> 2: (()+0x10050) [0x7f0773013050]
> 3: (gsignal()+0x37) [0x7f07714e60f7]
> 4: (abort()+0x13a) [0x7f07714e74ca]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f0771dcbfe5]
> 6: (()+0x63186) [0x7f0771dca186]
> 7: (()+0x631b3) [0x7f0771dca1b3]
> 8: (()+0x633d2) [0x7f0771dca3d2]
> 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7]
> 10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5]
> 11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc]
> 12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf]
> 13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7]
> 14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb]
> 15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007]
> 16: (DispatchQueue::entry()+0x649) [0xc6fe09]
> 17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d]
> 18: (()+0x83a4) [0x7f077300b3a4]
> 19: (clone()+0x6d) [0x7f0771595a4d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>
> --- logging levels ---
> 0/ 5 none
> 0/ 1 lockdep
> 0/ 1 context
> 1/ 1 crush
> 1/ 5 mds
> 1/ 5 mds_balancer
> 1/ 5 mds_locker
> 1/ 5 mds_log
> 1/ 5 mds_log_expire
> 1/ 5 mds_migrator
> 0/ 1 buffer
> 0/ 1 timer
> 0/ 1 filer
> 0/ 1 striper
> 0/ 1 objecter
> 0/ 5 rados
> 0/ 5 rbd
> 0/ 5 rbd_replay
> 0/ 5 journaler
> 0/ 5 objectcacher
> 0/ 5 client
> 0/ 5 osd
> 0/ 5 optracker
> 0/ 5 objclass
> 1/ 3 filestore
> 1/ 3 keyvaluestore
> 1/ 3 journal
> 0/ 5 ms
> 1/ 5 mon
> 0/10 monc
> 1/ 5 paxos
> 0/ 5 tp
> 1/ 5 auth
> 1/ 5 crypto
> 1/ 1 finisher
> 1/ 5 heartbeatmap
> 1/ 5 perfcounter
> 1/ 5 rgw
> 1/10 civetweb
> 1/ 5 javaclient
> 1/ 5 asok
> 1/ 1 throttle
> 0/ 0 refs
> 1/ 5 xio
> -2/-2 (syslog threshold)
> -1/-1 (stderr threshold)
> max_recent 10000
> max_new 1000
> log_file /var/log/ceph/ceph-osd.3.log
> --- end dump of recent events ---
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
