Hi all, So the init script issue is sorted.. my grep binary is not working correctly. I've replaced it and everything seems to be fine. Which now has me wondering if the binaries I generated are any good... the bad grep might have caused issues with the build... I'm going to recompile after some more sanity testing.. -----Original Message----- From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Malcolm Haak Sent: Wednesday, 11 March 2015 8:56 PM To: Samuel Just; jluis@xxxxxxxxxx Cc: ceph-users@xxxxxxxxxxxxxx Subject: Re: Issues with fresh 0.93 OSD adding to existing cluster I ran ceph-osd via the command line... It's not really given me much more to go off... Well except that it's hitting an early end of buffer for some reason. Also I've hit another issue... The /etc/init.d/ceph script is not seeing my new mon (I decided to add more mon's to see if it would help since the mon map looks like it is the issue) The script starts the mon fine. And the new mon (on the same host as this problem osd) appears to be good. The issue is when you do /etc/init.d/ceph status It tells you the mon.b is dead.. It seems to be one of the greps that is failing Specifically grep -qwe -i.$daemon_id /proc/\$pid/cmdline returns 1 What's odd is the same grep works on the other node for mon.a it just doesn't work on this node for mon.b I'm wondering if there is something odd happening. Anyway here is the output of the manual start of ceph-osd # /usr/bin/ceph-osd -i 3 --pid-file /var/run/ceph/osd.3.pid -c /etc/ceph/ceph.conf --cluster ceph -f starting osd.3 at :/0 osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal 2015-03-11 20:38:56.401205 7f04221e6880 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio t o force use of aio anyway 2015-03-11 20:38:56.418747 7f04221e6880 -1 osd.3 2757 log_to_monitors {default=true} terminate called after throwing an instance of 'ceph::buffer::end_of_buffer' what(): buffer::end_of_buffer *** Caught signal (Aborted) ** in thread 7f041192a700 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4) 1: /usr/bin/ceph-osd() [0xac7cea] 2: (()+0x10050) [0x7f04210f1050] 3: (gsignal()+0x37) [0x7f041f5c40f7] 4: (abort()+0x13a) [0x7f041f5c54ca] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f041fea9fe5] 6: (()+0x63186) [0x7f041fea8186] 7: (()+0x631b3) [0x7f041fea81b3] 8: (()+0x633d2) [0x7f041fea83d2] 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7] 10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5] 11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc] 12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf] 13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7] 14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb] 15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007] 16: (DispatchQueue::entry()+0x649) [0xc6fe09] 17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d] 18: (()+0x83a4) [0x7f04210e93a4] 19: (clone()+0x6d) [0x7f041f673a4d] 2015-03-11 20:38:56.471624 7f041192a700 -1 *** Caught signal (Aborted) ** in thread 7f041192a700 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4) 1: /usr/bin/ceph-osd() [0xac7cea] 2: (()+0x10050) [0x7f04210f1050] 3: (gsignal()+0x37) [0x7f041f5c40f7] 4: (abort()+0x13a) [0x7f041f5c54ca] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f041fea9fe5] 6: (()+0x63186) [0x7f041fea8186] 7: (()+0x631b3) [0x7f041fea81b3] 8: (()+0x633d2) [0x7f041fea83d2] 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7] 10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5] 11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc] 12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf] 13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7] 14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb] 15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007] 16: (DispatchQueue::entry()+0x649) [0xc6fe09] 17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d] 18: (()+0x83a4) [0x7f04210e93a4] 19: (clone()+0x6d) [0x7f041f673a4d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -308> 2015-03-11 20:38:56.401205 7f04221e6880 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_for ce_aio to force use of aio anyway -77> 2015-03-11 20:38:56.418747 7f04221e6880 -1 osd.3 2757 log_to_monitors {default=true} 0> 2015-03-11 20:38:56.471624 7f041192a700 -1 *** Caught signal (Aborted) ** in thread 7f041192a700 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4) 1: /usr/bin/ceph-osd() [0xac7cea] 2: (()+0x10050) [0x7f04210f1050] 3: (gsignal()+0x37) [0x7f041f5c40f7] 4: (abort()+0x13a) [0x7f041f5c54ca] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f041fea9fe5] 6: (()+0x63186) [0x7f041fea8186] 7: (()+0x631b3) [0x7f041fea81b3] 8: (()+0x633d2) [0x7f041fea83d2] 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7] 10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5] 11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc] 12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf] 13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7] 14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb] 15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007] 16: (DispatchQueue::entry()+0x649) [0xc6fe09] 17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d] 18: (()+0x83a4) [0x7f04210e93a4] 19: (clone()+0x6d) [0x7f041f673a4d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -----Original Message----- From: Samuel Just [mailto:sjust@xxxxxxxxxx] Sent: Wednesday, 11 March 2015 1:41 AM To: Malcolm Haak; jluis@xxxxxxxxxx Cc: ceph-users@xxxxxxxxxxxxxx Subject: Re: Issues with fresh 0.93 OSD adding to existing cluster Joao, it looks like map 2759 is causing trouble, how would he get the full and incremental maps for that out of the mons? -Sam On Tue, 2015-03-10 at 14:12 +0000, Malcolm Haak wrote: > Hi Samuel, > > The sha1? I'm going to admit ignorance as to what you are looking for. They are all running the same release if that is what you are asking. > Same tarball built into rpms using rpmbuild on both nodes... > Only difference being that the other node has been upgraded and the problem node is fresh. > > added the requested config here is the command line output > > microserver-1:/etc # /etc/init.d/ceph start osd.3 > === osd.3 === > Mounting xfs on microserver-1:/var/lib/ceph/osd/ceph-3 > 2015-03-11 01:00:13.492279 7f05b2f72700 1 -- :/0 messenger.start > 2015-03-11 01:00:13.492823 7f05b2f72700 1 -- :/1002795 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x7f05ac0290b0 con 0x7f05ac027c40 > 2015-03-11 01:00:13.510814 7f05b07ef700 1 -- 192.168.0.250:0/1002795 learned my addr 192.168.0.250:0/1002795 > 2015-03-11 01:00:13.527653 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x7f05a0000ab0 con 0x7f05ac027c40 > 2015-03-11 01:00:13.527899 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (3859410672 0 0) 0x7f05a0000e70 con 0x7f05ac027c40 > 2015-03-11 01:00:13.527973 7f05abfff700 1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x7f05ac029730 con 0x7f05ac027c40 > 2015-03-11 01:00:13.528124 7f05b2f72700 1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x7f05ac029a50 con 0x7f05ac027c40 > 2015-03-11 01:00:13.528265 7f05b2f72700 1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x7f05ac029f20 con 0x7f05ac027c40 > 2015-03-11 01:00:13.530359 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 3 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x7f05a0000ab0 con 0x7f05ac027c40 > 2015-03-11 01:00:13.530548 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 4 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3648139960 0 0) 0x7f05a0000fb0 con 0x7f05ac027c40 > 2015-03-11 01:00:13.531114 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x7f05a0002800 con 0x7f05ac027c40 > 2015-03-11 01:00:13.531772 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3648139960 0 0) 0x7f05a0000fb0 con 0x7f05ac027c40 > 2015-03-11 01:00:13.532186 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 7 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x7f05a0001250 con 0x7f05ac027c40 > 2015-03-11 01:00:13.532260 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3648139960 0 0) 0x7f05a0000fb0 con 0x7f05ac027c40 > 2015-03-11 01:00:13.556748 7f05b2f72700 1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- ?+0 0x7f05ac016ac0 con 0x7f05ac027c40 > 2015-03-11 01:00:13.564968 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+34995 (1092875540 0 1727986498) 0x7f05a0000a70 con 0x7f05ac027c40 > 2015-03-11 01:00:13.770122 7f05b2f72700 1 -- 192.168.0.250:0/1002795 --> 192.168.0.10:6789/0 -- mon_command({"prefix": "osd crush create-or-move", "args": ["host=microserver-1", "root=default"], "id": 3, "weight": 1.81} v 0) v1 -- ?+0 0x7f05ac016ac0 con 0x7f05ac027c40 > 2015-03-11 01:00:13.772299 7f05abfff700 1 -- 192.168.0.250:0/1002795 <== mon.0 192.168.0.10:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "args": ["host=microserver-1", "root=default"], "id": 3, "weight": 1.81}]=0 create-or-move updated item name 'osd.3' weight 1.81 at location {host=microserver-1,root=default} to crush map v3277) v1 ==== 256+0+0 (1191546821 0 0) 0x7f05a0001000 con 0x7f05ac027c40 > create-or-move updated item name 'osd.3' weight 1.81 at location {host=microserver-1,root=default} to crush map > 2015-03-11 01:00:13.776891 7f05b2f72700 1 -- 192.168.0.250:0/1002795 mark_down 0x7f05ac027c40 -- 0x7f05ac0239a0 > 2015-03-11 01:00:13.777212 7f05b2f72700 1 -- 192.168.0.250:0/1002795 mark_down_all > 2015-03-11 01:00:13.778120 7f05b2f72700 1 -- 192.168.0.250:0/1002795 shutdown complete. > Starting Ceph osd.3 on microserver-1... > microserver-1:/etc # > > > Log file > > > 2015-03-11 01:00:13.876152 7f41a1ba4880 0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 2840 > 2015-03-11 01:00:13.877059 7f41a1ba4880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/2840 need_addr=1 > 2015-03-11 01:00:13.877111 7f41a1ba4880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/2840 need_addr=1 > 2015-03-11 01:00:13.877140 7f41a1ba4880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/2840 need_addr=1 > 2015-03-11 01:00:13.877169 7f41a1ba4880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/2840 need_addr=1 > 2015-03-11 01:00:13.880323 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) dump_stop > 2015-03-11 01:00:13.890355 7f41a1ba4880 10 ErasureCodePluginSelectJerasure: load: jerasure_generic > 2015-03-11 01:00:13.897843 7f41a1ba4880 10 load: jerasure load: lrc load: isa > 2015-03-11 01:00:13.898172 7f41a1ba4880 5 filestore(/var/lib/ceph/osd/ceph-3) test_mount basedir /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal > 2015-03-11 01:00:13.898290 7f41a1ba4880 1 -- 0.0.0.0:6800/2840 messenger.start > 2015-03-11 01:00:13.898362 7f41a1ba4880 1 -- :/0 messenger.start > 2015-03-11 01:00:13.898450 7f41a1ba4880 1 -- 0.0.0.0:6803/2840 messenger.start > 2015-03-11 01:00:13.898527 7f41a1ba4880 1 -- 0.0.0.0:6802/2840 messenger.start > 2015-03-11 01:00:13.898609 7f41a1ba4880 1 -- 0.0.0.0:6801/2840 messenger.start > 2015-03-11 01:00:13.898672 7f41a1ba4880 1 -- :/0 messenger.start > 2015-03-11 01:00:13.898792 7f41a1ba4880 2 osd.3 0 mounting /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal > 2015-03-11 01:00:13.898826 7f41a1ba4880 5 filestore(/var/lib/ceph/osd/ceph-3) basedir /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal > 2015-03-11 01:00:13.898868 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) mount fsid is 05fb481f-68b3-4442-b7de-af4c867e18d5 > 2015-03-11 01:00:13.898937 7f41a1ba4880 0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) > 2015-03-11 01:00:13.941605 7f41a1ba4880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work > 2015-03-11 01:00:13.941665 7f41a1ba4880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option > 2015-03-11 01:00:13.970056 7f41a1ba4880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) > 2015-03-11 01:00:13.970442 7f41a1ba4880 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5 > 2015-03-11 01:00:14.001199 7f41a1ba4880 5 filestore(/var/lib/ceph/osd/ceph-3) mount op_seq is 2 > 2015-03-11 01:00:14.122464 7f41a1ba4880 20 filestore (init)dbobjectmap: seq is 1 > 2015-03-11 01:00:14.122580 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) open_journal at /var/lib/ceph/osd/ceph-3/journal > 2015-03-11 01:00:14.122674 7f41a1ba4880 0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled > 2015-03-11 01:00:14.122682 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) list_collections > 2015-03-11 01:00:14.123024 7f41a1ba4880 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway > 2015-03-11 01:00:14.123040 7f41a1ba4880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0 > 2015-03-11 01:00:14.123003 7f4196c1b700 20 filestore(/var/lib/ceph/osd/ceph-3) sync_entry waiting for max_interval 5.000000 > 2015-03-11 01:00:14.123724 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x4b9b6c0 > 2015-03-11 01:00:14.123837 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr > 2015-03-11 01:00:14.123971 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr > 2015-03-11 01:00:14.124020 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) touch meta/a468ec03/snapmapper/0//-1 > 2015-03-11 01:00:14.124103 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) touch meta/a468ec03/snapmapper/0//-1 = 0 > 2015-03-11 01:00:14.124247 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x4b9b6c0 > 2015-03-11 01:00:14.124282 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr > 2015-03-11 01:00:14.124332 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr > 2015-03-11 01:00:14.124348 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) write meta/a143428/osdmap.2757/0//-1 0~3076 > 2015-03-11 01:00:14.124420 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) write meta/a143428/osdmap.2757/0//-1 0~3076 = 3076 > 2015-03-11 01:00:14.124461 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr > 2015-03-11 01:00:14.124547 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr > 2015-03-11 01:00:14.124557 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) write meta/23c2fcde/osd_superblock/0//-1 0~413 > 2015-03-11 01:00:14.124615 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) write meta/23c2fcde/osd_superblock/0//-1 0~413 = 413 > 2015-03-11 01:00:14.124716 7f41a1ba4880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0 > 2015-03-11 01:00:14.125247 7f41a1ba4880 2 osd.3 0 boot > 2015-03-11 01:00:14.125270 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) read meta/23c2fcde/osd_superblock/0//-1 0~0 > 2015-03-11 01:00:14.125343 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) FileStore::read meta/23c2fcde/osd_superblock/0//-1 0~413/413 > 2015-03-11 01:00:14.125393 7f41a1ba4880 10 osd.3 0 read_superblock sb(3bf19e54-97e2-4a5f-b811-7c3780cd5e57 osd.3 05fb481f-68b3-4442-b7de-af4c867e18d5 e2757 [2757,2757] lci=[0,0]) > 2015-03-11 01:00:14.125521 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) stat meta/a468ec03/snapmapper/0//-1 = 0 (size 0) > 2015-03-11 01:00:14.125540 7f41a1ba4880 10 open_all_classes > 2015-03-11 01:00:14.125586 7f41a1ba4880 10 open_all_classes found hello > 2015-03-11 01:00:14.125623 7f41a1ba4880 10 _get_class adding new class name hello 0x4c9e028 > 2015-03-11 01:00:14.125630 7f41a1ba4880 10 _load_class hello from /usr/lib64/rados-classes/libcls_hello.so > 2015-03-11 01:00:14.126058 7f41a1ba4880 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello > 2015-03-11 01:00:14.126080 7f41a1ba4880 10 register_class hello status 3 > 2015-03-11 01:00:14.126084 7f41a1ba4880 10 register_cxx_method hello.say_hello flags 1 0x7f4192a10350 > 2015-03-11 01:00:14.126092 7f41a1ba4880 10 register_cxx_method hello.record_hello flags 2 0x7f4192a113c0 > 2015-03-11 01:00:14.126098 7f41a1ba4880 10 register_cxx_method hello.writes_dont_return_data flags 2 0x7f4192a10650 > 2015-03-11 01:00:14.126103 7f41a1ba4880 10 register_cxx_method hello.replay flags 1 0x7f4192a10320 > 2015-03-11 01:00:14.126109 7f41a1ba4880 10 register_cxx_method hello.turn_it_to_11 flags 3 0x7f4192a10920 > 2015-03-11 01:00:14.126114 7f41a1ba4880 10 register_cxx_method hello.bad_reader flags 2 0x7f4192a10310 > 2015-03-11 01:00:14.126119 7f41a1ba4880 10 register_cxx_method hello.bad_writer flags 1 0x7f4192a10300 > 2015-03-11 01:00:14.126127 7f41a1ba4880 10 _load_class hello success > 2015-03-11 01:00:14.126132 7f41a1ba4880 10 open_all_classes found kvs > 2015-03-11 01:00:14.126137 7f41a1ba4880 10 _get_class adding new class name kvs 0x4c9e108 > 2015-03-11 01:00:14.126142 7f41a1ba4880 10 _load_class kvs from /usr/lib64/rados-classes/libcls_kvs.so > 2015-03-11 01:00:14.126655 7f41a1ba4880 10 register_class kvs status 3 > 2015-03-11 01:00:14.126665 7f41a1ba4880 10 register_cxx_method kvs.get_idata_from_key flags 1 0x7f41927ff060 > 2015-03-11 01:00:14.126669 7f41a1ba4880 10 register_cxx_method kvs.get_next_idata flags 1 0x7f4192801050 > 2015-03-11 01:00:14.126672 7f41a1ba4880 10 register_cxx_method kvs.get_prev_idata flags 1 0x7f41927fd570 > 2015-03-11 01:00:14.126675 7f41a1ba4880 10 register_cxx_method kvs.read_many flags 1 0x7f41927f8f30 > 2015-03-11 01:00:14.126678 7f41a1ba4880 10 register_cxx_method kvs.check_writable flags 3 0x7f41927f8ac0 > 2015-03-11 01:00:14.126681 7f41a1ba4880 10 register_cxx_method kvs.assert_size_in_bound flags 2 0x7f41927f8d40 > 2015-03-11 01:00:14.126684 7f41a1ba4880 10 register_cxx_method kvs.omap_insert flags 2 0x7f41927fcb50 > 2015-03-11 01:00:14.126688 7f41a1ba4880 10 register_cxx_method kvs.create_with_omap flags 2 0x7f41927fc0e0 > 2015-03-11 01:00:14.126691 7f41a1ba4880 10 register_cxx_method kvs.omap_remove flags 2 0x7f41927f9e70 > 2015-03-11 01:00:14.126696 7f41a1ba4880 10 register_cxx_method kvs.maybe_read_for_balance flags 1 0x7f41927fab70 > 2015-03-11 01:00:14.126735 7f41a1ba4880 10 _load_class kvs success > 2015-03-11 01:00:14.126739 7f41a1ba4880 10 open_all_classes found lock > 2015-03-11 01:00:14.126742 7f41a1ba4880 10 _get_class adding new class name lock 0x4c9e1e8 > 2015-03-11 01:00:14.126745 7f41a1ba4880 10 _load_class lock from /usr/lib64/rados-classes/libcls_lock.so > 2015-03-11 01:00:14.126972 7f41a1ba4880 10 register_class lock status 3 > 2015-03-11 01:00:14.126977 7f41a1ba4880 10 register_cxx_method lock.lock flags 3 0x7f41925ee800 > 2015-03-11 01:00:14.126981 7f41a1ba4880 10 register_cxx_method lock.unlock flags 3 0x7f41925ed210 > 2015-03-11 01:00:14.126984 7f41a1ba4880 10 register_cxx_method lock.break_lock flags 3 0x7f41925ece60 > 2015-03-11 01:00:14.126986 7f41a1ba4880 10 register_cxx_method lock.get_info flags 1 0x7f41925ed610 > 2015-03-11 01:00:14.126990 7f41a1ba4880 10 register_cxx_method lock.list_locks flags 1 0x7f41925eace0 > 2015-03-11 01:00:14.126993 7f41a1ba4880 10 register_cxx_method lock.assert_locked flags 1 0x7f41925ec330 > 2015-03-11 01:00:14.126996 7f41a1ba4880 10 _load_class lock success > 2015-03-11 01:00:14.126999 7f41a1ba4880 10 open_all_classes found log > 2015-03-11 01:00:14.127002 7f41a1ba4880 10 _get_class adding new class name log 0x4c9e2c8 > 2015-03-11 01:00:14.127005 7f41a1ba4880 10 _load_class log from /usr/lib64/rados-classes/libcls_log.so > 2015-03-11 01:00:14.127223 7f41a1ba4880 10 register_class log status 3 > 2015-03-11 01:00:14.127225 7f41a1ba4880 10 register_cxx_method log.add flags 3 0x7f41923e1560 > 2015-03-11 01:00:14.127228 7f41a1ba4880 10 register_cxx_method log.list flags 1 0x7f41923e2740 > 2015-03-11 01:00:14.127231 7f41a1ba4880 10 register_cxx_method log.trim flags 3 0x7f41923e1f90 > 2015-03-11 01:00:14.127234 7f41a1ba4880 10 register_cxx_method log.info flags 1 0x7f41923e1090 > 2015-03-11 01:00:14.127238 7f41a1ba4880 10 _load_class log success > 2015-03-11 01:00:14.127241 7f41a1ba4880 10 open_all_classes found rbd > 2015-03-11 01:00:14.127244 7f41a1ba4880 10 _get_class adding new class name rbd 0x4c9e3a8 > 2015-03-11 01:00:14.127247 7f41a1ba4880 10 _load_class rbd from /usr/lib64/rados-classes/libcls_rbd.so > 2015-03-11 01:00:14.127735 7f41a1ba4880 10 register_class rbd status 3 > 2015-03-11 01:00:14.127745 7f41a1ba4880 10 register_cxx_method rbd.create flags 3 0x7f41921cc5c0 > 2015-03-11 01:00:14.127748 7f41a1ba4880 10 register_cxx_method rbd.get_features flags 1 0x7f41921d3190 > 2015-03-11 01:00:14.127751 7f41a1ba4880 10 register_cxx_method rbd.get_size flags 1 0x7f41921d2de0 > 2015-03-11 01:00:14.127754 7f41a1ba4880 10 register_cxx_method rbd.set_size flags 3 0x7f41921ca7e0 > 2015-03-11 01:00:14.127758 7f41a1ba4880 10 register_cxx_method rbd.get_snapcontext flags 1 0x7f41921cdb40 > 2015-03-11 01:00:14.127762 7f41a1ba4880 10 register_cxx_method rbd.get_object_prefix flags 1 0x7f41921c9770 > 2015-03-11 01:00:14.127766 7f41a1ba4880 10 register_cxx_method rbd.get_snapshot_name flags 1 0x7f41921d1dd0 > 2015-03-11 01:00:14.127770 7f41a1ba4880 10 register_cxx_method rbd.snapshot_add flags 3 0x7f41921d0830 > 2015-03-11 01:00:14.127773 7f41a1ba4880 10 register_cxx_method rbd.snapshot_remove flags 3 0x7f41921d1b40 > 2015-03-11 01:00:14.127776 7f41a1ba4880 10 register_cxx_method rbd.get_all_features flags 1 0x7f41921c78c0 > 2015-03-11 01:00:14.127779 7f41a1ba4880 10 register_cxx_method rbd.copyup flags 3 0x7f41921c7940 > 2015-03-11 01:00:14.127782 7f41a1ba4880 10 register_cxx_method rbd.get_parent flags 1 0x7f41921d2080 > 2015-03-11 01:00:14.127786 7f41a1ba4880 10 register_cxx_method rbd.set_parent flags 3 0x7f41921ccce0 > 2015-03-11 01:00:14.127789 7f41a1ba4880 10 register_cxx_method rbd.remove_parent flags 3 0x7f41921c9140 > 2015-03-11 01:00:14.127792 7f41a1ba4880 10 register_cxx_method rbd.set_protection_status flags 3 0x7f41921d2770 > 2015-03-11 01:00:14.127815 7f41a1ba4880 10 register_cxx_method rbd.get_protection_status flags 1 0x7f41921d2b60 > 2015-03-11 01:00:14.127818 7f41a1ba4880 10 register_cxx_method rbd.get_stripe_unit_count flags 1 0x7f41921c9330 > 2015-03-11 01:00:14.127822 7f41a1ba4880 10 register_cxx_method rbd.set_stripe_unit_count flags 3 0x7f41921cc070 > 2015-03-11 01:00:14.127825 7f41a1ba4880 10 register_cxx_method rbd.get_flags flags 1 0x7f41921d2470 > 2015-03-11 01:00:14.127828 7f41a1ba4880 10 register_cxx_method rbd.set_flags flags 3 0x7f41921cb370 > 2015-03-11 01:00:14.127831 7f41a1ba4880 10 register_cxx_method rbd.add_child flags 3 0x7f41921cd790 > 2015-03-11 01:00:14.127834 7f41a1ba4880 10 register_cxx_method rbd.remove_child flags 3 0x7f41921cdf50 > 2015-03-11 01:00:14.127838 7f41a1ba4880 10 register_cxx_method rbd.get_children flags 1 0x7f41921cd4e0 > 2015-03-11 01:00:14.127841 7f41a1ba4880 10 register_cxx_method rbd.get_id flags 1 0x7f41921ca580 > 2015-03-11 01:00:14.127844 7f41a1ba4880 10 register_cxx_method rbd.set_id flags 3 0x7f41921cb6c0 > 2015-03-11 01:00:14.127848 7f41a1ba4880 10 register_cxx_method rbd.dir_get_id flags 1 0x7f41921c9f40 > 2015-03-11 01:00:14.127851 7f41a1ba4880 10 register_cxx_method rbd.dir_get_name flags 1 0x7f41921ca210 > 2015-03-11 01:00:14.127855 7f41a1ba4880 10 register_cxx_method rbd.dir_list flags 1 0x7f41921cf9f0 > 2015-03-11 01:00:14.127858 7f41a1ba4880 10 register_cxx_method rbd.dir_add_image flags 3 0x7f41921cf840 > 2015-03-11 01:00:14.127862 7f41a1ba4880 10 register_cxx_method rbd.dir_remove_image flags 3 0x7f41921c9d60 > 2015-03-11 01:00:14.127865 7f41a1ba4880 10 register_cxx_method rbd.dir_rename_image flags 3 0x7f41921cf660 > 2015-03-11 01:00:14.127868 7f41a1ba4880 10 register_cxx_method rbd.object_map_load flags 1 0x7f41921ce5f0 > 2015-03-11 01:00:14.127872 7f41a1ba4880 10 register_cxx_method rbd.object_map_resize flags 3 0x7f41921ceec0 > 2015-03-11 01:00:14.127875 7f41a1ba4880 10 register_cxx_method rbd.object_map_update flags 3 0x7f41921ce720 > 2015-03-11 01:00:14.127879 7f41a1ba4880 10 register_cxx_method rbd.snap_list flags 1 0x7f41921c8540 > 2015-03-11 01:00:14.127882 7f41a1ba4880 10 register_cxx_method rbd.snap_add flags 3 0x7f41921cacc0 > 2015-03-11 01:00:14.127885 7f41a1ba4880 10 register_cxx_method rbd.snap_remove flags 3 0x7f41921cb910 > 2015-03-11 01:00:14.127891 7f41a1ba4880 10 _load_class rbd success > 2015-03-11 01:00:14.127894 7f41a1ba4880 10 open_all_classes found refcount > 2015-03-11 01:00:14.127897 7f41a1ba4880 10 _get_class adding new class name refcount 0x4c9e488 > 2015-03-11 01:00:14.127900 7f41a1ba4880 10 _load_class refcount from /usr/lib64/rados-classes/libcls_refcount.so > 2015-03-11 01:00:14.132942 7f41a1ba4880 10 register_class refcount status 3 > 2015-03-11 01:00:14.132946 7f41a1ba4880 10 register_cxx_method refcount.get flags 3 0x7f4191f32be0 > 2015-03-11 01:00:14.132957 7f41a1ba4880 10 register_cxx_method refcount.put flags 3 0x7f4191f31b30 > 2015-03-11 01:00:14.132960 7f41a1ba4880 10 register_cxx_method refcount.set flags 3 0x7f4191f32510 > 2015-03-11 01:00:14.132963 7f41a1ba4880 10 register_cxx_method refcount.read flags 1 0x7f4191f31fb0 > 2015-03-11 01:00:14.132970 7f41a1ba4880 10 _load_class refcount success > 2015-03-11 01:00:14.132977 7f41a1ba4880 10 open_all_classes found replica_log > 2015-03-11 01:00:14.132984 7f41a1ba4880 10 _get_class adding new class name replica_log 0x4c9e568 > 2015-03-11 01:00:14.132987 7f41a1ba4880 10 _load_class replica_log from /usr/lib64/rados-classes/libcls_replica_log.so > 2015-03-11 01:00:14.133269 7f41a1ba4880 10 register_class replica_log status 3 > 2015-03-11 01:00:14.133271 7f41a1ba4880 10 register_cxx_method replica_log.set flags 3 0x7f4191c7f920 > 2015-03-11 01:00:14.133276 7f41a1ba4880 10 register_cxx_method replica_log.get flags 1 0x7f4191c809f0 > 2015-03-11 01:00:14.133279 7f41a1ba4880 10 register_cxx_method replica_log.delete flags 3 0x7f4191c80400 > 2015-03-11 01:00:14.133283 7f41a1ba4880 10 _load_class replica_log success > 2015-03-11 01:00:14.133285 7f41a1ba4880 10 open_all_classes found rgw > 2015-03-11 01:00:14.133288 7f41a1ba4880 10 _get_class adding new class name rgw 0x4c9e728 > 2015-03-11 01:00:14.133291 7f41a1ba4880 10 _load_class rgw from /usr/lib64/rados-classes/libcls_rgw.so > 2015-03-11 01:00:14.138712 7f41a1ba4880 10 register_class rgw status 3 > 2015-03-11 01:00:14.138717 7f41a1ba4880 10 register_cxx_method rgw.bucket_init_index flags 3 0x7f41919c0860 > 2015-03-11 01:00:14.138725 7f41a1ba4880 10 register_cxx_method rgw.bucket_set_tag_timeout flags 3 0x7f41919c4a40 > 2015-03-11 01:00:14.138728 7f41a1ba4880 10 register_cxx_method rgw.bucket_list flags 1 0x7f41919c8160 > 2015-03-11 01:00:14.138732 7f41a1ba4880 10 register_cxx_method rgw.bucket_check_index flags 1 0x7f41919ca300 > 2015-03-11 01:00:14.138735 7f41a1ba4880 10 register_cxx_method rgw.bucket_rebuild_index flags 3 0x7f41919ca1a0 > 2015-03-11 01:00:14.138740 7f41a1ba4880 10 register_cxx_method rgw.bucket_prepare_op flags 3 0x7f41919c5180 > 2015-03-11 01:00:14.138745 7f41a1ba4880 10 register_cxx_method rgw.bucket_complete_op flags 3 0x7f41919c5980 > 2015-03-11 01:00:14.138748 7f41a1ba4880 10 register_cxx_method rgw.bucket_link_olh flags 3 0x7f41919cc980 > 2015-03-11 01:00:14.138751 7f41a1ba4880 10 register_cxx_method rgw.bucket_unlink_instance flags 3 0x7f41919cbb80 > 2015-03-11 01:00:14.138754 7f41a1ba4880 10 register_cxx_method rgw.bucket_read_olh_log flags 1 0x7f41919cde40 > 2015-03-11 01:00:14.138757 7f41a1ba4880 10 register_cxx_method rgw.bucket_trim_olh_log flags 3 0x7f41919cb2d0 > 2015-03-11 01:00:14.138760 7f41a1ba4880 10 register_cxx_method rgw.bucket_clear_olh flags 3 0x7f41919cacd0 > 2015-03-11 01:00:14.138763 7f41a1ba4880 10 register_cxx_method rgw.obj_remove flags 3 0x7f41919c3ff0 > 2015-03-11 01:00:14.138771 7f41a1ba4880 10 register_cxx_method rgw.obj_check_attrs_prefix flags 1 0x7f41919c0b60 > 2015-03-11 01:00:14.138774 7f41a1ba4880 10 register_cxx_method rgw.bi_get flags 1 0x7f41919bf050 > 2015-03-11 01:00:14.138777 7f41a1ba4880 10 register_cxx_method rgw.bi_put flags 3 0x7f41919be700 > 2015-03-11 01:00:14.138780 7f41a1ba4880 10 register_cxx_method rgw.bi_list flags 1 0x7f41919c6790 > 2015-03-11 01:00:14.138784 7f41a1ba4880 10 register_cxx_method rgw.bi_log_list flags 1 0x7f41919c2180 > 2015-03-11 01:00:14.138787 7f41a1ba4880 10 register_cxx_method rgw.bi_log_trim flags 3 0x7f41919c26e0 > 2015-03-11 01:00:14.138791 7f41a1ba4880 10 register_cxx_method rgw.dir_suggest_changes flags 3 0x7f41919c9260 > 2015-03-11 01:00:14.138794 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_add flags 3 0x7f41919ce6a0 > 2015-03-11 01:00:14.138797 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_read flags 1 0x7f41919c31c0 > 2015-03-11 01:00:14.138799 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_trim flags 3 0x7f41919c3950 > 2015-03-11 01:00:14.138802 7f41a1ba4880 10 register_cxx_method rgw.gc_set_entry flags 3 0x7f41919c01b0 > 2015-03-11 01:00:14.138806 7f41a1ba4880 10 register_cxx_method rgw.gc_defer_entry flags 3 0x7f41919bfe90 > 2015-03-11 01:00:14.138809 7f41a1ba4880 10 register_cxx_method rgw.gc_list flags 1 0x7f41919c0f80 > 2015-03-11 01:00:14.138812 7f41a1ba4880 10 register_cxx_method rgw.gc_remove flags 3 0x7f41919c0420 > 2015-03-11 01:00:14.138815 7f41a1ba4880 10 _load_class rgw success > 2015-03-11 01:00:14.138820 7f41a1ba4880 10 open_all_classes found statelog > 2015-03-11 01:00:14.138826 7f41a1ba4880 10 _get_class adding new class name statelog 0x4c9e808 > 2015-03-11 01:00:14.138830 7f41a1ba4880 10 _load_class statelog from /usr/lib64/rados-classes/libcls_statelog.so > 2015-03-11 01:00:14.139141 7f41a1ba4880 10 register_class statelog status 3 > 2015-03-11 01:00:14.139143 7f41a1ba4880 10 register_cxx_method statelog.add flags 3 0x7f4191700bc0 > 2015-03-11 01:00:14.139146 7f41a1ba4880 10 register_cxx_method statelog.list flags 1 0x7f4191700f40 > 2015-03-11 01:00:14.139150 7f41a1ba4880 10 register_cxx_method statelog.remove flags 3 0x7f41916ffa80 > 2015-03-11 01:00:14.139153 7f41a1ba4880 10 register_cxx_method statelog.check_state flags 1 0x7f41917002d0 > 2015-03-11 01:00:14.139175 7f41a1ba4880 10 _load_class statelog success > 2015-03-11 01:00:14.139178 7f41a1ba4880 10 open_all_classes found user > 2015-03-11 01:00:14.139182 7f41a1ba4880 10 _get_class adding new class name user 0x4c9e8e8 > 2015-03-11 01:00:14.139185 7f41a1ba4880 10 _load_class user from /usr/lib64/rados-classes/libcls_user.so > 2015-03-11 01:00:14.139386 7f41a1ba4880 10 register_class user status 3 > 2015-03-11 01:00:14.139388 7f41a1ba4880 10 register_cxx_method user.set_buckets_info flags 3 0x7f41914f5180 > 2015-03-11 01:00:14.139392 7f41a1ba4880 10 register_cxx_method user.complete_stats_sync flags 3 0x7f41914f4aa0 > 2015-03-11 01:00:14.139394 7f41a1ba4880 10 register_cxx_method user.remove_bucket flags 3 0x7f41914f3d30 > 2015-03-11 01:00:14.139397 7f41a1ba4880 10 register_cxx_method user.list_buckets flags 1 0x7f41914f64c0 > 2015-03-11 01:00:14.139400 7f41a1ba4880 10 register_cxx_method user.get_header flags 1 0x7f41914f2d50 > 2015-03-11 01:00:14.139403 7f41a1ba4880 10 _load_class user success > 2015-03-11 01:00:14.139406 7f41a1ba4880 10 open_all_classes found version > 2015-03-11 01:00:14.139409 7f41a1ba4880 10 _get_class adding new class name version 0x4c9e9c8 > 2015-03-11 01:00:14.139412 7f41a1ba4880 10 _load_class version from /usr/lib64/rados-classes/libcls_version.so > 2015-03-11 01:00:14.139608 7f41a1ba4880 10 register_class version status 3 > 2015-03-11 01:00:14.139611 7f41a1ba4880 10 register_cxx_method version.set flags 3 0x7f41912ebe60 > 2015-03-11 01:00:14.139615 7f41a1ba4880 10 register_cxx_method version.inc flags 3 0x7f41912ecc80 > 2015-03-11 01:00:14.139618 7f41a1ba4880 10 register_cxx_method version.inc_conds flags 3 0x7f41912ecc80 > 2015-03-11 01:00:14.139621 7f41a1ba4880 10 register_cxx_method version.read flags 1 0x7f41912ebb60 > 2015-03-11 01:00:14.139624 7f41a1ba4880 10 register_cxx_method version.check_conds flags 1 0x7f41912ec260 > 2015-03-11 01:00:14.139627 7f41a1ba4880 10 _load_class version success > 2015-03-11 01:00:14.139682 7f41a1ba4880 20 osd.3 0 get_map 2757 - loading and decoding 0x4b4cb40 > 2015-03-11 01:00:14.139693 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) read meta/a143428/osdmap.2757/0//-1 0~0 > 2015-03-11 01:00:14.139804 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) FileStore::read meta/a143428/osdmap.2757/0//-1 0~3076/3076 > 2015-03-11 01:00:14.139818 7f41a1ba4880 10 osd.3 0 add_map_bl 2757 3076 bytes > 2015-03-11 01:00:14.139908 7f41a1ba4880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients > 2015-03-11 01:00:14.139914 7f41a1ba4880 0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons > 2015-03-11 01:00:14.139919 7f41a1ba4880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds > 2015-03-11 01:00:14.139922 7f41a1ba4880 10 osd.3 2757 create_recoverystate_perf > 2015-03-11 01:00:14.139939 7f41a1ba4880 0 osd.3 2757 load_pgs > 2015-03-11 01:00:14.139942 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) list_collections > 2015-03-11 01:00:14.139981 7f41a1ba4880 10 osd.3 2757 load_pgs ignoring unrecognized meta > 2015-03-11 01:00:14.139985 7f41a1ba4880 0 osd.3 2757 load_pgs opened 0 pgs > 2015-03-11 01:00:14.139989 7f41a1ba4880 10 osd.3 2757 build_past_intervals_parallel nothing to build > 2015-03-11 01:00:14.139992 7f41a1ba4880 2 osd.3 2757 superblock: i am osd.3 > 2015-03-11 01:00:14.139995 7f41a1ba4880 10 osd.3 2757 create_logger > 2015-03-11 01:00:14.140126 7f41a1ba4880 1 accepter.accepter.start > 2015-03-11 01:00:14.140269 7f41a1ba4880 1 accepter.accepter.start > 2015-03-11 01:00:14.140434 7f41a1ba4880 1 accepter.accepter.start > 2015-03-11 01:00:14.140575 7f41a1ba4880 1 accepter.accepter.start > 2015-03-11 01:00:14.140813 7f41a1ba4880 -1 osd.3 2757 log_to_monitors {default=true} > 2015-03-11 01:00:14.141325 7f41a1ba4880 10 osd.3 2757 set_disk_tp_priority class priority -1 > 2015-03-11 01:00:14.141599 7f41a1ba4880 1 -- 0.0.0.0:6800/2840 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x4b4cd80 con 0x4cbe3c0 > 2015-03-11 01:00:14.142020 7f4180ac7700 20 osd.3 2757 update_osd_stat osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist []) > 2015-03-11 01:00:14.142048 7f4180ac7700 5 osd.3 2757 heartbeat: osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist []) > 2015-03-11 01:00:14.142221 7f417e2c2700 10 osd.3 2757 agent_entry start > 2015-03-11 01:00:14.142241 7f417e2c2700 20 osd.3 2757 agent_entry empty queue > 2015-03-11 01:00:14.158932 7f41a1ba0700 1 -- 192.168.0.250:6800/2840 learned my addr 192.168.0.250:6800/2840 > 2015-03-11 01:00:14.159058 7f41a1ba0700 10 osd.3 2757 new session (outgoing) 0x4b92280 con=0x4b602c0 addr=192.168.0.250:6800/2840 > 2015-03-11 01:00:14.159166 7f41a1ba0700 10 osd.3 2757 OSD::ms_get_authorizer type=mon > 2015-03-11 01:00:14.173274 7f41912e8700 10 osd.3 2757 ms_handle_connect on mon > 2015-03-11 01:00:14.173332 7f41912e8700 20 osd.3 2757 send_pg_stats > 2015-03-11 01:00:14.175783 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x4b4f840 con 0x4cbe3c0 > 2015-03-11 01:00:14.175940 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (335930126 0 0) 0x4b4fa80 con 0x4cbe3c0 > 2015-03-11 01:00:14.175997 7f41912e8700 1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x4b31a00 con 0x4cbe3c0 > 2015-03-11 01:00:14.176059 7f41912e8700 1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0,osdmap=2757}) v2 -- ?+0 0x4b32800 con 0x4cbe3c0 > 2015-03-11 01:00:14.176147 7f41a1ba4880 10 osd.3 2757 ensuring pgs have consumed prior maps > 2015-03-11 01:00:14.176170 7f41a1ba4880 7 osd.3 2757 consume_map version 2757 > 2015-03-11 01:00:14.176187 7f41a1ba4880 0 osd.3 2757 done with init, starting boot process > 2015-03-11 01:00:14.176191 7f41a1ba4880 10 osd.3 2757 start_boot - have maps 2757..2757 > 2015-03-11 01:00:14.176221 7f41a1ba4880 1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x4bc41e0 con 0x4cbe3c0 > 2015-03-11 01:00:14.178539 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 3 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x4b4fcc0 con 0x4cbe3c0 > 2015-03-11 01:00:14.178650 7f41912e8700 10 osd.3 2757 do_waiters -- start > 2015-03-11 01:00:14.178658 7f41912e8700 10 osd.3 2757 do_waiters -- finish > 2015-03-11 01:00:14.178661 7f41912e8700 20 osd.3 2757 _dispatch 0x4b4fcc0 osd_map(3277..3277 src has 2757..3277) v3 > 2015-03-11 01:00:14.178888 7f41912e8700 3 osd.3 2757 handle_osd_map epochs [3277,3277], i have 2757, src has [2757,3277] > 2015-03-11 01:00:14.178906 7f41912e8700 10 osd.3 2757 handle_osd_map message skips epochs 2758..3276 > 2015-03-11 01:00:14.178954 7f41912e8700 1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=2758}) v2 -- ?+0 0x4b31a00 con 0x4cbe3c0 > 2015-03-11 01:00:14.178991 7f41912e8700 10 osd.3 2757 do_waiters -- start > 2015-03-11 01:00:14.178996 7f41912e8700 10 osd.3 2757 do_waiters -- finish > 2015-03-11 01:00:14.179012 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 4 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x4b4f840 con 0x4cbe3c0 > 2015-03-11 01:00:14.184757 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(2757..2857 src has 2757..3277) v3 ==== 37648+0+0 (2648067016 0 0) 0x4b4fa80 con 0x4cbe3c0 > 2015-03-11 01:00:14.184831 7f41912e8700 10 osd.3 2757 do_waiters -- start > 2015-03-11 01:00:14.184838 7f41912e8700 10 osd.3 2757 do_waiters -- finish > 2015-03-11 01:00:14.184841 7f41912e8700 20 osd.3 2757 _dispatch 0x4b4fa80 osd_map(2757..2857 src has 2757..3277) v3 > 2015-03-11 01:00:14.184872 7f41912e8700 3 osd.3 2757 handle_osd_map epochs [2757,2857], i have 2757, src has [2757,3277] > 2015-03-11 01:00:14.184884 7f41912e8700 10 osd.3 2757 handle_osd_map got inc map for epoch 2758 > 2015-03-11 01:00:14.185229 7f41912e8700 10 osd.3 2757 handle_osd_map got inc map for epoch 2759 > 2015-03-11 01:00:14.195165 7f41912e8700 -1 *** Caught signal (Aborted) ** > in thread 7f41912e8700 > > ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4) > 1: /usr/bin/ceph-osd() [0xac7cea] > 2: (()+0x10050) [0x7f41a0aaf050] > 3: (gsignal()+0x37) [0x7f419ef820f7] > 4: (abort()+0x13a) [0x7f419ef834ca] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f419f867fe5] > 6: (()+0x63186) [0x7f419f866186] > 7: (()+0x631b3) [0x7f419f8661b3] > 8: (()+0x633d2) [0x7f419f8663d2] > 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7] > 10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5] > 11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc] > 12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf] > 13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7] > 14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb] > 15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007] > 16: (DispatchQueue::entry()+0x649) [0xc6fe09] > 17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d] > 18: (()+0x83a4) [0x7f41a0aa73a4] > 19: (clone()+0x6d) [0x7f419f031a4d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > --- begin dump of recent events --- > -363> 2015-03-11 01:00:13.871379 7f41a1ba4880 5 asok(0x4b50000) register_command perfcounters_dump hook 0x4b04030 > -362> 2015-03-11 01:00:13.871462 7f41a1ba4880 5 asok(0x4b50000) register_command 1 hook 0x4b04030 > -361> 2015-03-11 01:00:13.871471 7f41a1ba4880 5 asok(0x4b50000) register_command perf dump hook 0x4b04030 > -360> 2015-03-11 01:00:13.871484 7f41a1ba4880 5 asok(0x4b50000) register_command perfcounters_schema hook 0x4b04030 > -359> 2015-03-11 01:00:13.871507 7f41a1ba4880 5 asok(0x4b50000) register_command 2 hook 0x4b04030 > -358> 2015-03-11 01:00:13.871512 7f41a1ba4880 5 asok(0x4b50000) register_command perf schema hook 0x4b04030 > -357> 2015-03-11 01:00:13.871545 7f41a1ba4880 5 asok(0x4b50000) register_command perf reset hook 0x4b04030 > -356> 2015-03-11 01:00:13.871555 7f41a1ba4880 5 asok(0x4b50000) register_command config show hook 0x4b04030 > -355> 2015-03-11 01:00:13.871562 7f41a1ba4880 5 asok(0x4b50000) register_command config set hook 0x4b04030 > -354> 2015-03-11 01:00:13.871571 7f41a1ba4880 5 asok(0x4b50000) register_command config get hook 0x4b04030 > -353> 2015-03-11 01:00:13.871576 7f41a1ba4880 5 asok(0x4b50000) register_command config diff hook 0x4b04030 > -352> 2015-03-11 01:00:13.871585 7f41a1ba4880 5 asok(0x4b50000) register_command log flush hook 0x4b04030 > -351> 2015-03-11 01:00:13.871590 7f41a1ba4880 5 asok(0x4b50000) register_command log dump hook 0x4b04030 > -350> 2015-03-11 01:00:13.871600 7f41a1ba4880 5 asok(0x4b50000) register_command log reopen hook 0x4b04030 > -349> 2015-03-11 01:00:13.876152 7f41a1ba4880 0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 2840 > -348> 2015-03-11 01:00:13.877059 7f41a1ba4880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/2840 need_addr=1 > -347> 2015-03-11 01:00:13.877111 7f41a1ba4880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/2840 need_addr=1 > -346> 2015-03-11 01:00:13.877140 7f41a1ba4880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/2840 need_addr=1 > -345> 2015-03-11 01:00:13.877169 7f41a1ba4880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/2840 need_addr=1 > -344> 2015-03-11 01:00:13.880323 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) dump_stop > -343> 2015-03-11 01:00:13.880362 7f41a1ba4880 5 asok(0x4b50000) init /var/run/ceph/ceph-osd.3.asok > -342> 2015-03-11 01:00:13.880382 7f41a1ba4880 5 asok(0x4b50000) bind_and_listen /var/run/ceph/ceph-osd.3.asok > -341> 2015-03-11 01:00:13.880604 7f41a1ba4880 5 asok(0x4b50000) register_command 0 hook 0x4b000c0 > -340> 2015-03-11 01:00:13.880620 7f41a1ba4880 5 asok(0x4b50000) register_command version hook 0x4b000c0 > -339> 2015-03-11 01:00:13.880626 7f41a1ba4880 5 asok(0x4b50000) register_command git_version hook 0x4b000c0 > -338> 2015-03-11 01:00:13.880633 7f41a1ba4880 5 asok(0x4b50000) register_command help hook 0x4b04110 > -337> 2015-03-11 01:00:13.880639 7f41a1ba4880 5 asok(0x4b50000) register_command get_command_descriptions hook 0x4b04120 > -336> 2015-03-11 01:00:13.880703 7f41a1ba4880 10 monclient(hunting): build_initial_monmap > -335> 2015-03-11 01:00:13.880697 7f419cf7d700 5 asok(0x4b50000) entry start > -334> 2015-03-11 01:00:13.890355 7f41a1ba4880 10 ErasureCodePluginSelectJerasure: load: jerasure_generic > -333> 2015-03-11 01:00:13.897843 7f41a1ba4880 10 load: jerasure load: lrc load: isa > -332> 2015-03-11 01:00:13.897900 7f41a1ba4880 5 adding auth protocol: none > -331> 2015-03-11 01:00:13.897905 7f41a1ba4880 5 adding auth protocol: none > -330> 2015-03-11 01:00:13.898086 7f41a1ba4880 5 asok(0x4b50000) register_command objecter_requests hook 0x4b041b0 > -329> 2015-03-11 01:00:13.898172 7f41a1ba4880 5 filestore(/var/lib/ceph/osd/ceph-3) test_mount basedir /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal > -328> 2015-03-11 01:00:13.898290 7f41a1ba4880 1 -- 0.0.0.0:6800/2840 messenger.start > -327> 2015-03-11 01:00:13.898362 7f41a1ba4880 1 -- :/0 messenger.start > -326> 2015-03-11 01:00:13.898450 7f41a1ba4880 1 -- 0.0.0.0:6803/2840 messenger.start > -325> 2015-03-11 01:00:13.898527 7f41a1ba4880 1 -- 0.0.0.0:6802/2840 messenger.start > -324> 2015-03-11 01:00:13.898609 7f41a1ba4880 1 -- 0.0.0.0:6801/2840 messenger.start > -323> 2015-03-11 01:00:13.898672 7f41a1ba4880 1 -- :/0 messenger.start > -322> 2015-03-11 01:00:13.898792 7f41a1ba4880 2 osd.3 0 mounting /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal > -321> 2015-03-11 01:00:13.898826 7f41a1ba4880 5 filestore(/var/lib/ceph/osd/ceph-3) basedir /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal > -320> 2015-03-11 01:00:13.898868 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) mount fsid is 05fb481f-68b3-4442-b7de-af4c867e18d5 > -319> 2015-03-11 01:00:13.898937 7f41a1ba4880 0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) > -318> 2015-03-11 01:00:13.941605 7f41a1ba4880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work > -317> 2015-03-11 01:00:13.941665 7f41a1ba4880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option > -316> 2015-03-11 01:00:13.970056 7f41a1ba4880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) > -315> 2015-03-11 01:00:13.970442 7f41a1ba4880 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5 > -314> 2015-03-11 01:00:14.001199 7f41a1ba4880 5 filestore(/var/lib/ceph/osd/ceph-3) mount op_seq is 2 > -313> 2015-03-11 01:00:14.122464 7f41a1ba4880 20 filestore (init)dbobjectmap: seq is 1 > -312> 2015-03-11 01:00:14.122580 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) open_journal at /var/lib/ceph/osd/ceph-3/journal > -311> 2015-03-11 01:00:14.122674 7f41a1ba4880 0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled > -310> 2015-03-11 01:00:14.122682 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) list_collections > -309> 2015-03-11 01:00:14.122987 7f41a1ba4880 2 journal open /var/lib/ceph/osd/ceph-3/journal fsid 05fb481f-68b3-4442-b7de-af4c867e18d5 fs_op_seq 2 > -308> 2015-03-11 01:00:14.123024 7f41a1ba4880 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway > -307> 2015-03-11 01:00:14.123040 7f41a1ba4880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0 > -306> 2015-03-11 01:00:14.123003 7f4196c1b700 20 filestore(/var/lib/ceph/osd/ceph-3) sync_entry waiting for max_interval 5.000000 > -305> 2015-03-11 01:00:14.123590 7f41a1ba4880 2 journal read_entry 8192 : seq 3 193 bytes > -304> 2015-03-11 01:00:14.123639 7f41a1ba4880 2 journal read_entry 8192 : seq 3 193 bytes > -303> 2015-03-11 01:00:14.123661 7f41a1ba4880 3 journal journal_replay: applying op seq 3 > -302> 2015-03-11 01:00:14.123724 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x4b9b6c0 > -301> 2015-03-11 01:00:14.123837 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr > -300> 2015-03-11 01:00:14.123971 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr > -299> 2015-03-11 01:00:14.124020 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) touch meta/a468ec03/snapmapper/0//-1 > -298> 2015-03-11 01:00:14.124103 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) touch meta/a468ec03/snapmapper/0//-1 = 0 > -297> 2015-03-11 01:00:14.124163 7f41a1ba4880 3 journal journal_replay: r = 0, op_seq now 3 > -296> 2015-03-11 01:00:14.124220 7f41a1ba4880 2 journal read_entry 12288 : seq 4 3829 bytes > -295> 2015-03-11 01:00:14.124231 7f41a1ba4880 3 journal journal_replay: applying op seq 4 > -294> 2015-03-11 01:00:14.124247 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) _do_transaction on 0x4b9b6c0 > -293> 2015-03-11 01:00:14.124282 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr > -292> 2015-03-11 01:00:14.124332 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr > -291> 2015-03-11 01:00:14.124348 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) write meta/a143428/osdmap.2757/0//-1 0~3076 > -290> 2015-03-11 01:00:14.124420 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) write meta/a143428/osdmap.2757/0//-1 0~3076 = 3076 > -289> 2015-03-11 01:00:14.124461 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_global_replay_guard no xattr > -288> 2015-03-11 01:00:14.124547 7f41a1ba4880 20 filestore(/var/lib/ceph/osd/ceph-3) _check_replay_guard no xattr > -287> 2015-03-11 01:00:14.124557 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) write meta/23c2fcde/osd_superblock/0//-1 0~413 > -286> 2015-03-11 01:00:14.124615 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) write meta/23c2fcde/osd_superblock/0//-1 0~413 = 413 > -285> 2015-03-11 01:00:14.124676 7f41a1ba4880 3 journal journal_replay: r = 0, op_seq now 4 > -284> 2015-03-11 01:00:14.124697 7f41a1ba4880 2 journal No further valid entries found, journal is most likely valid > -283> 2015-03-11 01:00:14.124700 7f41a1ba4880 3 journal journal_replay: end of journal, done. > -282> 2015-03-11 01:00:14.124716 7f41a1ba4880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0 > -281> 2015-03-11 01:00:14.125247 7f41a1ba4880 2 osd.3 0 boot > -280> 2015-03-11 01:00:14.125270 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) read meta/23c2fcde/osd_superblock/0//-1 0~0 > -279> 2015-03-11 01:00:14.125343 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) FileStore::read meta/23c2fcde/osd_superblock/0//-1 0~413/413 > -278> 2015-03-11 01:00:14.125393 7f41a1ba4880 10 osd.3 0 read_superblock sb(3bf19e54-97e2-4a5f-b811-7c3780cd5e57 osd.3 05fb481f-68b3-4442-b7de-af4c867e18d5 e2757 [2757,2757] lci=[0,0]) > -277> 2015-03-11 01:00:14.125521 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) stat meta/a468ec03/snapmapper/0//-1 = 0 (size 0) > -276> 2015-03-11 01:00:14.125540 7f41a1ba4880 10 open_all_classes > -275> 2015-03-11 01:00:14.125586 7f41a1ba4880 10 open_all_classes found hello > -274> 2015-03-11 01:00:14.125623 7f41a1ba4880 10 _get_class adding new class name hello 0x4c9e028 > -273> 2015-03-11 01:00:14.125630 7f41a1ba4880 10 _load_class hello from /usr/lib64/rados-classes/libcls_hello.so > -272> 2015-03-11 01:00:14.126058 7f41a1ba4880 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello > -271> 2015-03-11 01:00:14.126080 7f41a1ba4880 10 register_class hello status 3 > -270> 2015-03-11 01:00:14.126084 7f41a1ba4880 10 register_cxx_method hello.say_hello flags 1 0x7f4192a10350 > -269> 2015-03-11 01:00:14.126092 7f41a1ba4880 10 register_cxx_method hello.record_hello flags 2 0x7f4192a113c0 > -268> 2015-03-11 01:00:14.126098 7f41a1ba4880 10 register_cxx_method hello.writes_dont_return_data flags 2 0x7f4192a10650 > -267> 2015-03-11 01:00:14.126103 7f41a1ba4880 10 register_cxx_method hello.replay flags 1 0x7f4192a10320 > -266> 2015-03-11 01:00:14.126109 7f41a1ba4880 10 register_cxx_method hello.turn_it_to_11 flags 3 0x7f4192a10920 > -265> 2015-03-11 01:00:14.126114 7f41a1ba4880 10 register_cxx_method hello.bad_reader flags 2 0x7f4192a10310 > -264> 2015-03-11 01:00:14.126119 7f41a1ba4880 10 register_cxx_method hello.bad_writer flags 1 0x7f4192a10300 > -263> 2015-03-11 01:00:14.126127 7f41a1ba4880 10 _load_class hello success > -262> 2015-03-11 01:00:14.126132 7f41a1ba4880 10 open_all_classes found kvs > -261> 2015-03-11 01:00:14.126137 7f41a1ba4880 10 _get_class adding new class name kvs 0x4c9e108 > -260> 2015-03-11 01:00:14.126142 7f41a1ba4880 10 _load_class kvs from /usr/lib64/rados-classes/libcls_kvs.so > -259> 2015-03-11 01:00:14.126655 7f41a1ba4880 10 register_class kvs status 3 > -258> 2015-03-11 01:00:14.126665 7f41a1ba4880 10 register_cxx_method kvs.get_idata_from_key flags 1 0x7f41927ff060 > -257> 2015-03-11 01:00:14.126669 7f41a1ba4880 10 register_cxx_method kvs.get_next_idata flags 1 0x7f4192801050 > -256> 2015-03-11 01:00:14.126672 7f41a1ba4880 10 register_cxx_method kvs.get_prev_idata flags 1 0x7f41927fd570 > -255> 2015-03-11 01:00:14.126675 7f41a1ba4880 10 register_cxx_method kvs.read_many flags 1 0x7f41927f8f30 > -254> 2015-03-11 01:00:14.126678 7f41a1ba4880 10 register_cxx_method kvs.check_writable flags 3 0x7f41927f8ac0 > -253> 2015-03-11 01:00:14.126681 7f41a1ba4880 10 register_cxx_method kvs.assert_size_in_bound flags 2 0x7f41927f8d40 > -252> 2015-03-11 01:00:14.126684 7f41a1ba4880 10 register_cxx_method kvs.omap_insert flags 2 0x7f41927fcb50 > -251> 2015-03-11 01:00:14.126688 7f41a1ba4880 10 register_cxx_method kvs.create_with_omap flags 2 0x7f41927fc0e0 > -250> 2015-03-11 01:00:14.126691 7f41a1ba4880 10 register_cxx_method kvs.omap_remove flags 2 0x7f41927f9e70 > -249> 2015-03-11 01:00:14.126696 7f41a1ba4880 10 register_cxx_method kvs.maybe_read_for_balance flags 1 0x7f41927fab70 > -248> 2015-03-11 01:00:14.126735 7f41a1ba4880 10 _load_class kvs success > -247> 2015-03-11 01:00:14.126739 7f41a1ba4880 10 open_all_classes found lock > -246> 2015-03-11 01:00:14.126742 7f41a1ba4880 10 _get_class adding new class name lock 0x4c9e1e8 > -245> 2015-03-11 01:00:14.126745 7f41a1ba4880 10 _load_class lock from /usr/lib64/rados-classes/libcls_lock.so > -244> 2015-03-11 01:00:14.126972 7f41a1ba4880 10 register_class lock status 3 > -243> 2015-03-11 01:00:14.126977 7f41a1ba4880 10 register_cxx_method lock.lock flags 3 0x7f41925ee800 > -242> 2015-03-11 01:00:14.126981 7f41a1ba4880 10 register_cxx_method lock.unlock flags 3 0x7f41925ed210 > -241> 2015-03-11 01:00:14.126984 7f41a1ba4880 10 register_cxx_method lock.break_lock flags 3 0x7f41925ece60 > -240> 2015-03-11 01:00:14.126986 7f41a1ba4880 10 register_cxx_method lock.get_info flags 1 0x7f41925ed610 > -239> 2015-03-11 01:00:14.126990 7f41a1ba4880 10 register_cxx_method lock.list_locks flags 1 0x7f41925eace0 > -238> 2015-03-11 01:00:14.126993 7f41a1ba4880 10 register_cxx_method lock.assert_locked flags 1 0x7f41925ec330 > -237> 2015-03-11 01:00:14.126996 7f41a1ba4880 10 _load_class lock success > -236> 2015-03-11 01:00:14.126999 7f41a1ba4880 10 open_all_classes found log > -235> 2015-03-11 01:00:14.127002 7f41a1ba4880 10 _get_class adding new class name log 0x4c9e2c8 > -234> 2015-03-11 01:00:14.127005 7f41a1ba4880 10 _load_class log from /usr/lib64/rados-classes/libcls_log.so > -233> 2015-03-11 01:00:14.127213 7f41a1ba4880 1 <cls> cls/log/cls_log.cc:312: Loaded log class! > -232> 2015-03-11 01:00:14.127223 7f41a1ba4880 10 register_class log status 3 > -231> 2015-03-11 01:00:14.127225 7f41a1ba4880 10 register_cxx_method log.add flags 3 0x7f41923e1560 > -230> 2015-03-11 01:00:14.127228 7f41a1ba4880 10 register_cxx_method log.list flags 1 0x7f41923e2740 > -229> 2015-03-11 01:00:14.127231 7f41a1ba4880 10 register_cxx_method log.trim flags 3 0x7f41923e1f90 > -228> 2015-03-11 01:00:14.127234 7f41a1ba4880 10 register_cxx_method log.info flags 1 0x7f41923e1090 > -227> 2015-03-11 01:00:14.127238 7f41a1ba4880 10 _load_class log success > -226> 2015-03-11 01:00:14.127241 7f41a1ba4880 10 open_all_classes found rbd > -225> 2015-03-11 01:00:14.127244 7f41a1ba4880 10 _get_class adding new class name rbd 0x4c9e3a8 > -224> 2015-03-11 01:00:14.127247 7f41a1ba4880 10 _load_class rbd from /usr/lib64/rados-classes/libcls_rbd.so > -223> 2015-03-11 01:00:14.127735 7f41a1ba4880 10 register_class rbd status 3 > -222> 2015-03-11 01:00:14.127745 7f41a1ba4880 10 register_cxx_method rbd.create flags 3 0x7f41921cc5c0 > -221> 2015-03-11 01:00:14.127748 7f41a1ba4880 10 register_cxx_method rbd.get_features flags 1 0x7f41921d3190 > -220> 2015-03-11 01:00:14.127751 7f41a1ba4880 10 register_cxx_method rbd.get_size flags 1 0x7f41921d2de0 > -219> 2015-03-11 01:00:14.127754 7f41a1ba4880 10 register_cxx_method rbd.set_size flags 3 0x7f41921ca7e0 > -218> 2015-03-11 01:00:14.127758 7f41a1ba4880 10 register_cxx_method rbd.get_snapcontext flags 1 0x7f41921cdb40 > -217> 2015-03-11 01:00:14.127762 7f41a1ba4880 10 register_cxx_method rbd.get_object_prefix flags 1 0x7f41921c9770 > -216> 2015-03-11 01:00:14.127766 7f41a1ba4880 10 register_cxx_method rbd.get_snapshot_name flags 1 0x7f41921d1dd0 > -215> 2015-03-11 01:00:14.127770 7f41a1ba4880 10 register_cxx_method rbd.snapshot_add flags 3 0x7f41921d0830 > -214> 2015-03-11 01:00:14.127773 7f41a1ba4880 10 register_cxx_method rbd.snapshot_remove flags 3 0x7f41921d1b40 > -213> 2015-03-11 01:00:14.127776 7f41a1ba4880 10 register_cxx_method rbd.get_all_features flags 1 0x7f41921c78c0 > -212> 2015-03-11 01:00:14.127779 7f41a1ba4880 10 register_cxx_method rbd.copyup flags 3 0x7f41921c7940 > -211> 2015-03-11 01:00:14.127782 7f41a1ba4880 10 register_cxx_method rbd.get_parent flags 1 0x7f41921d2080 > -210> 2015-03-11 01:00:14.127786 7f41a1ba4880 10 register_cxx_method rbd.set_parent flags 3 0x7f41921ccce0 > -209> 2015-03-11 01:00:14.127789 7f41a1ba4880 10 register_cxx_method rbd.remove_parent flags 3 0x7f41921c9140 > -208> 2015-03-11 01:00:14.127792 7f41a1ba4880 10 register_cxx_method rbd.set_protection_status flags 3 0x7f41921d2770 > -207> 2015-03-11 01:00:14.127815 7f41a1ba4880 10 register_cxx_method rbd.get_protection_status flags 1 0x7f41921d2b60 > -206> 2015-03-11 01:00:14.127818 7f41a1ba4880 10 register_cxx_method rbd.get_stripe_unit_count flags 1 0x7f41921c9330 > -205> 2015-03-11 01:00:14.127822 7f41a1ba4880 10 register_cxx_method rbd.set_stripe_unit_count flags 3 0x7f41921cc070 > -204> 2015-03-11 01:00:14.127825 7f41a1ba4880 10 register_cxx_method rbd.get_flags flags 1 0x7f41921d2470 > -203> 2015-03-11 01:00:14.127828 7f41a1ba4880 10 register_cxx_method rbd.set_flags flags 3 0x7f41921cb370 > -202> 2015-03-11 01:00:14.127831 7f41a1ba4880 10 register_cxx_method rbd.add_child flags 3 0x7f41921cd790 > -201> 2015-03-11 01:00:14.127834 7f41a1ba4880 10 register_cxx_method rbd.remove_child flags 3 0x7f41921cdf50 > -200> 2015-03-11 01:00:14.127838 7f41a1ba4880 10 register_cxx_method rbd.get_children flags 1 0x7f41921cd4e0 > -199> 2015-03-11 01:00:14.127841 7f41a1ba4880 10 register_cxx_method rbd.get_id flags 1 0x7f41921ca580 > -198> 2015-03-11 01:00:14.127844 7f41a1ba4880 10 register_cxx_method rbd.set_id flags 3 0x7f41921cb6c0 > -197> 2015-03-11 01:00:14.127848 7f41a1ba4880 10 register_cxx_method rbd.dir_get_id flags 1 0x7f41921c9f40 > -196> 2015-03-11 01:00:14.127851 7f41a1ba4880 10 register_cxx_method rbd.dir_get_name flags 1 0x7f41921ca210 > -195> 2015-03-11 01:00:14.127855 7f41a1ba4880 10 register_cxx_method rbd.dir_list flags 1 0x7f41921cf9f0 > -194> 2015-03-11 01:00:14.127858 7f41a1ba4880 10 register_cxx_method rbd.dir_add_image flags 3 0x7f41921cf840 > -193> 2015-03-11 01:00:14.127862 7f41a1ba4880 10 register_cxx_method rbd.dir_remove_image flags 3 0x7f41921c9d60 > -192> 2015-03-11 01:00:14.127865 7f41a1ba4880 10 register_cxx_method rbd.dir_rename_image flags 3 0x7f41921cf660 > -191> 2015-03-11 01:00:14.127868 7f41a1ba4880 10 register_cxx_method rbd.object_map_load flags 1 0x7f41921ce5f0 > -190> 2015-03-11 01:00:14.127872 7f41a1ba4880 10 register_cxx_method rbd.object_map_resize flags 3 0x7f41921ceec0 > -189> 2015-03-11 01:00:14.127875 7f41a1ba4880 10 register_cxx_method rbd.object_map_update flags 3 0x7f41921ce720 > -188> 2015-03-11 01:00:14.127879 7f41a1ba4880 10 register_cxx_method rbd.snap_list flags 1 0x7f41921c8540 > -187> 2015-03-11 01:00:14.127882 7f41a1ba4880 10 register_cxx_method rbd.snap_add flags 3 0x7f41921cacc0 > -186> 2015-03-11 01:00:14.127885 7f41a1ba4880 10 register_cxx_method rbd.snap_remove flags 3 0x7f41921cb910 > -185> 2015-03-11 01:00:14.127891 7f41a1ba4880 10 _load_class rbd success > -184> 2015-03-11 01:00:14.127894 7f41a1ba4880 10 open_all_classes found refcount > -183> 2015-03-11 01:00:14.127897 7f41a1ba4880 10 _get_class adding new class name refcount 0x4c9e488 > -182> 2015-03-11 01:00:14.127900 7f41a1ba4880 10 _load_class refcount from /usr/lib64/rados-classes/libcls_refcount.so > -181> 2015-03-11 01:00:14.132905 7f41a1ba4880 1 <cls> cls/refcount/cls_refcount.cc:231: Loaded refcount class! > -180> 2015-03-11 01:00:14.132942 7f41a1ba4880 10 register_class refcount status 3 > -179> 2015-03-11 01:00:14.132946 7f41a1ba4880 10 register_cxx_method refcount.get flags 3 0x7f4191f32be0 > -178> 2015-03-11 01:00:14.132957 7f41a1ba4880 10 register_cxx_method refcount.put flags 3 0x7f4191f31b30 > -177> 2015-03-11 01:00:14.132960 7f41a1ba4880 10 register_cxx_method refcount.set flags 3 0x7f4191f32510 > -176> 2015-03-11 01:00:14.132963 7f41a1ba4880 10 register_cxx_method refcount.read flags 1 0x7f4191f31fb0 > -175> 2015-03-11 01:00:14.132970 7f41a1ba4880 10 _load_class refcount success > -174> 2015-03-11 01:00:14.132977 7f41a1ba4880 10 open_all_classes found replica_log > -173> 2015-03-11 01:00:14.132984 7f41a1ba4880 10 _get_class adding new class name replica_log 0x4c9e568 > -172> 2015-03-11 01:00:14.132987 7f41a1ba4880 10 _load_class replica_log from /usr/lib64/rados-classes/libcls_replica_log.so > -171> 2015-03-11 01:00:14.133259 7f41a1ba4880 1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class! > -170> 2015-03-11 01:00:14.133269 7f41a1ba4880 10 register_class replica_log status 3 > -169> 2015-03-11 01:00:14.133271 7f41a1ba4880 10 register_cxx_method replica_log.set flags 3 0x7f4191c7f920 > -168> 2015-03-11 01:00:14.133276 7f41a1ba4880 10 register_cxx_method replica_log.get flags 1 0x7f4191c809f0 > -167> 2015-03-11 01:00:14.133279 7f41a1ba4880 10 register_cxx_method replica_log.delete flags 3 0x7f4191c80400 > -166> 2015-03-11 01:00:14.133283 7f41a1ba4880 10 _load_class replica_log success > -165> 2015-03-11 01:00:14.133285 7f41a1ba4880 10 open_all_classes found rgw > -164> 2015-03-11 01:00:14.133288 7f41a1ba4880 10 _get_class adding new class name rgw 0x4c9e728 > -163> 2015-03-11 01:00:14.133291 7f41a1ba4880 10 _load_class rgw from /usr/lib64/rados-classes/libcls_rgw.so > -162> 2015-03-11 01:00:14.138677 7f41a1ba4880 1 <cls> cls/rgw/cls_rgw.cc:3046: Loaded rgw class! > -161> 2015-03-11 01:00:14.138712 7f41a1ba4880 10 register_class rgw status 3 > -160> 2015-03-11 01:00:14.138717 7f41a1ba4880 10 register_cxx_method rgw.bucket_init_index flags 3 0x7f41919c0860 > -159> 2015-03-11 01:00:14.138725 7f41a1ba4880 10 register_cxx_method rgw.bucket_set_tag_timeout flags 3 0x7f41919c4a40 > -158> 2015-03-11 01:00:14.138728 7f41a1ba4880 10 register_cxx_method rgw.bucket_list flags 1 0x7f41919c8160 > -157> 2015-03-11 01:00:14.138732 7f41a1ba4880 10 register_cxx_method rgw.bucket_check_index flags 1 0x7f41919ca300 > -156> 2015-03-11 01:00:14.138735 7f41a1ba4880 10 register_cxx_method rgw.bucket_rebuild_index flags 3 0x7f41919ca1a0 > -155> 2015-03-11 01:00:14.138740 7f41a1ba4880 10 register_cxx_method rgw.bucket_prepare_op flags 3 0x7f41919c5180 > -154> 2015-03-11 01:00:14.138745 7f41a1ba4880 10 register_cxx_method rgw.bucket_complete_op flags 3 0x7f41919c5980 > -153> 2015-03-11 01:00:14.138748 7f41a1ba4880 10 register_cxx_method rgw.bucket_link_olh flags 3 0x7f41919cc980 > -152> 2015-03-11 01:00:14.138751 7f41a1ba4880 10 register_cxx_method rgw.bucket_unlink_instance flags 3 0x7f41919cbb80 > -151> 2015-03-11 01:00:14.138754 7f41a1ba4880 10 register_cxx_method rgw.bucket_read_olh_log flags 1 0x7f41919cde40 > -150> 2015-03-11 01:00:14.138757 7f41a1ba4880 10 register_cxx_method rgw.bucket_trim_olh_log flags 3 0x7f41919cb2d0 > -149> 2015-03-11 01:00:14.138760 7f41a1ba4880 10 register_cxx_method rgw.bucket_clear_olh flags 3 0x7f41919cacd0 > -148> 2015-03-11 01:00:14.138763 7f41a1ba4880 10 register_cxx_method rgw.obj_remove flags 3 0x7f41919c3ff0 > -147> 2015-03-11 01:00:14.138771 7f41a1ba4880 10 register_cxx_method rgw.obj_check_attrs_prefix flags 1 0x7f41919c0b60 > -146> 2015-03-11 01:00:14.138774 7f41a1ba4880 10 register_cxx_method rgw.bi_get flags 1 0x7f41919bf050 > -145> 2015-03-11 01:00:14.138777 7f41a1ba4880 10 register_cxx_method rgw.bi_put flags 3 0x7f41919be700 > -144> 2015-03-11 01:00:14.138780 7f41a1ba4880 10 register_cxx_method rgw.bi_list flags 1 0x7f41919c6790 > -143> 2015-03-11 01:00:14.138784 7f41a1ba4880 10 register_cxx_method rgw.bi_log_list flags 1 0x7f41919c2180 > -142> 2015-03-11 01:00:14.138787 7f41a1ba4880 10 register_cxx_method rgw.bi_log_trim flags 3 0x7f41919c26e0 > -141> 2015-03-11 01:00:14.138791 7f41a1ba4880 10 register_cxx_method rgw.dir_suggest_changes flags 3 0x7f41919c9260 > -140> 2015-03-11 01:00:14.138794 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_add flags 3 0x7f41919ce6a0 > -139> 2015-03-11 01:00:14.138797 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_read flags 1 0x7f41919c31c0 > -138> 2015-03-11 01:00:14.138799 7f41a1ba4880 10 register_cxx_method rgw.user_usage_log_trim flags 3 0x7f41919c3950 > -137> 2015-03-11 01:00:14.138802 7f41a1ba4880 10 register_cxx_method rgw.gc_set_entry flags 3 0x7f41919c01b0 > -136> 2015-03-11 01:00:14.138806 7f41a1ba4880 10 register_cxx_method rgw.gc_defer_entry flags 3 0x7f41919bfe90 > -135> 2015-03-11 01:00:14.138809 7f41a1ba4880 10 register_cxx_method rgw.gc_list flags 1 0x7f41919c0f80 > -134> 2015-03-11 01:00:14.138812 7f41a1ba4880 10 register_cxx_method rgw.gc_remove flags 3 0x7f41919c0420 > -133> 2015-03-11 01:00:14.138815 7f41a1ba4880 10 _load_class rgw success > -132> 2015-03-11 01:00:14.138820 7f41a1ba4880 10 open_all_classes found statelog > -131> 2015-03-11 01:00:14.138826 7f41a1ba4880 10 _get_class adding new class name statelog 0x4c9e808 > -130> 2015-03-11 01:00:14.138830 7f41a1ba4880 10 _load_class statelog from /usr/lib64/rados-classes/libcls_statelog.so > -129> 2015-03-11 01:00:14.139131 7f41a1ba4880 1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class! > -128> 2015-03-11 01:00:14.139141 7f41a1ba4880 10 register_class statelog status 3 > -127> 2015-03-11 01:00:14.139143 7f41a1ba4880 10 register_cxx_method statelog.add flags 3 0x7f4191700bc0 > -126> 2015-03-11 01:00:14.139146 7f41a1ba4880 10 register_cxx_method statelog.list flags 1 0x7f4191700f40 > -125> 2015-03-11 01:00:14.139150 7f41a1ba4880 10 register_cxx_method statelog.remove flags 3 0x7f41916ffa80 > -124> 2015-03-11 01:00:14.139153 7f41a1ba4880 10 register_cxx_method statelog.check_state flags 1 0x7f41917002d0 > -123> 2015-03-11 01:00:14.139175 7f41a1ba4880 10 _load_class statelog success > -122> 2015-03-11 01:00:14.139178 7f41a1ba4880 10 open_all_classes found user > -121> 2015-03-11 01:00:14.139182 7f41a1ba4880 10 _get_class adding new class name user 0x4c9e8e8 > -120> 2015-03-11 01:00:14.139185 7f41a1ba4880 10 _load_class user from /usr/lib64/rados-classes/libcls_user.so > -119> 2015-03-11 01:00:14.139377 7f41a1ba4880 1 <cls> cls/user/cls_user.cc:367: Loaded user class! > -118> 2015-03-11 01:00:14.139386 7f41a1ba4880 10 register_class user status 3 > -117> 2015-03-11 01:00:14.139388 7f41a1ba4880 10 register_cxx_method user.set_buckets_info flags 3 0x7f41914f5180 > -116> 2015-03-11 01:00:14.139392 7f41a1ba4880 10 register_cxx_method user.complete_stats_sync flags 3 0x7f41914f4aa0 > -115> 2015-03-11 01:00:14.139394 7f41a1ba4880 10 register_cxx_method user.remove_bucket flags 3 0x7f41914f3d30 > -114> 2015-03-11 01:00:14.139397 7f41a1ba4880 10 register_cxx_method user.list_buckets flags 1 0x7f41914f64c0 > -113> 2015-03-11 01:00:14.139400 7f41a1ba4880 10 register_cxx_method user.get_header flags 1 0x7f41914f2d50 > -112> 2015-03-11 01:00:14.139403 7f41a1ba4880 10 _load_class user success > -111> 2015-03-11 01:00:14.139406 7f41a1ba4880 10 open_all_classes found version > -110> 2015-03-11 01:00:14.139409 7f41a1ba4880 10 _get_class adding new class name version 0x4c9e9c8 > -109> 2015-03-11 01:00:14.139412 7f41a1ba4880 10 _load_class version from /usr/lib64/rados-classes/libcls_version.so > -108> 2015-03-11 01:00:14.139604 7f41a1ba4880 1 <cls> cls/version/cls_version.cc:227: Loaded version class! > -107> 2015-03-11 01:00:14.139608 7f41a1ba4880 10 register_class version status 3 > -106> 2015-03-11 01:00:14.139611 7f41a1ba4880 10 register_cxx_method version.set flags 3 0x7f41912ebe60 > -105> 2015-03-11 01:00:14.139615 7f41a1ba4880 10 register_cxx_method version.inc flags 3 0x7f41912ecc80 > -104> 2015-03-11 01:00:14.139618 7f41a1ba4880 10 register_cxx_method version.inc_conds flags 3 0x7f41912ecc80 > -103> 2015-03-11 01:00:14.139621 7f41a1ba4880 10 register_cxx_method version.read flags 1 0x7f41912ebb60 > -102> 2015-03-11 01:00:14.139624 7f41a1ba4880 10 register_cxx_method version.check_conds flags 1 0x7f41912ec260 > -101> 2015-03-11 01:00:14.139627 7f41a1ba4880 10 _load_class version success > -100> 2015-03-11 01:00:14.139682 7f41a1ba4880 20 osd.3 0 get_map 2757 - loading and decoding 0x4b4cb40 > -99> 2015-03-11 01:00:14.139693 7f41a1ba4880 15 filestore(/var/lib/ceph/osd/ceph-3) read meta/a143428/osdmap.2757/0//-1 0~0 > -98> 2015-03-11 01:00:14.139804 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) FileStore::read meta/a143428/osdmap.2757/0//-1 0~3076/3076 > -97> 2015-03-11 01:00:14.139818 7f41a1ba4880 10 osd.3 0 add_map_bl 2757 3076 bytes > -96> 2015-03-11 01:00:14.139908 7f41a1ba4880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients > -95> 2015-03-11 01:00:14.139914 7f41a1ba4880 0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons > -94> 2015-03-11 01:00:14.139919 7f41a1ba4880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds > -93> 2015-03-11 01:00:14.139922 7f41a1ba4880 10 osd.3 2757 create_recoverystate_perf > -92> 2015-03-11 01:00:14.139939 7f41a1ba4880 0 osd.3 2757 load_pgs > -91> 2015-03-11 01:00:14.139942 7f41a1ba4880 10 filestore(/var/lib/ceph/osd/ceph-3) list_collections > -90> 2015-03-11 01:00:14.139981 7f41a1ba4880 10 osd.3 2757 load_pgs ignoring unrecognized meta > -89> 2015-03-11 01:00:14.139985 7f41a1ba4880 0 osd.3 2757 load_pgs opened 0 pgs > -88> 2015-03-11 01:00:14.139989 7f41a1ba4880 10 osd.3 2757 build_past_intervals_parallel nothing to build > -87> 2015-03-11 01:00:14.139992 7f41a1ba4880 2 osd.3 2757 superblock: i am osd.3 > -86> 2015-03-11 01:00:14.139995 7f41a1ba4880 10 osd.3 2757 create_logger > -85> 2015-03-11 01:00:14.140126 7f41a1ba4880 1 accepter.accepter.start > -84> 2015-03-11 01:00:14.140269 7f41a1ba4880 1 accepter.accepter.start > -83> 2015-03-11 01:00:14.140434 7f41a1ba4880 1 accepter.accepter.start > -82> 2015-03-11 01:00:14.140575 7f41a1ba4880 1 accepter.accepter.start > -81> 2015-03-11 01:00:14.140676 7f41a1ba4880 10 monclient(hunting): init > -80> 2015-03-11 01:00:14.140691 7f41a1ba4880 5 adding auth protocol: none > -79> 2015-03-11 01:00:14.140697 7f41a1ba4880 10 monclient(hunting): auth_supported 1 method none > -78> 2015-03-11 01:00:14.140803 7f41a1ba4880 10 log_channel(default) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info) > -77> 2015-03-11 01:00:14.140813 7f41a1ba4880 -1 osd.3 2757 log_to_monitors {default=true} > -76> 2015-03-11 01:00:14.141325 7f41a1ba4880 10 osd.3 2757 set_disk_tp_priority class priority -1 > -75> 2015-03-11 01:00:14.141461 7f41a1ba4880 10 monclient(hunting): renew_subs > -74> 2015-03-11 01:00:14.141466 7f41a1ba4880 10 monclient(hunting): _reopen_session rank -1 name > -73> 2015-03-11 01:00:14.141564 7f41a1ba4880 10 monclient(hunting): picked mon.a con 0x4cbe3c0 addr 192.168.0.10:6789/0 > -72> 2015-03-11 01:00:14.141594 7f41a1ba4880 10 monclient(hunting): _send_mon_message to mon.a at 192.168.0.10:6789/0 > -71> 2015-03-11 01:00:14.141599 7f41a1ba4880 1 -- 0.0.0.0:6800/2840 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x4b4cd80 con 0x4cbe3c0 > -70> 2015-03-11 01:00:14.141625 7f41a1ba4880 10 monclient(hunting): renew_subs > -69> 2015-03-11 01:00:14.142020 7f4180ac7700 20 osd.3 2757 update_osd_stat osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist []) > -68> 2015-03-11 01:00:14.142048 7f4180ac7700 5 osd.3 2757 heartbeat: osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist []) > -67> 2015-03-11 01:00:14.142221 7f417e2c2700 10 osd.3 2757 agent_entry start > -66> 2015-03-11 01:00:14.142241 7f417e2c2700 20 osd.3 2757 agent_entry empty queue > -65> 2015-03-11 01:00:14.158932 7f41a1ba0700 1 -- 192.168.0.250:6800/2840 learned my addr 192.168.0.250:6800/2840 > -64> 2015-03-11 01:00:14.159058 7f41a1ba0700 10 osd.3 2757 new session (outgoing) 0x4b92280 con=0x4b602c0 addr=192.168.0.250:6800/2840 > -63> 2015-03-11 01:00:14.159166 7f41a1ba0700 10 osd.3 2757 OSD::ms_get_authorizer type=mon > -62> 2015-03-11 01:00:14.173274 7f41912e8700 10 osd.3 2757 ms_handle_connect on mon > -61> 2015-03-11 01:00:14.173332 7f41912e8700 20 osd.3 2757 send_pg_stats > -60> 2015-03-11 01:00:14.173348 7f41912e8700 10 monclient(hunting): renew_subs > -59> 2015-03-11 01:00:14.175783 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x4b4f840 con 0x4cbe3c0 > -58> 2015-03-11 01:00:14.175871 7f41912e8700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 > -57> 2015-03-11 01:00:14.175889 7f41912e8700 10 monclient(hunting): got monmap 1, mon.a is now rank 0 > -56> 2015-03-11 01:00:14.175893 7f41912e8700 10 monclient(hunting): dump: > epoch 1 > fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57 > last_changed 2013-09-11 22:41:30.819687 > created 2013-09-11 22:41:30.819687 > 0: 192.168.0.10:6789/0 mon.a > > -55> 2015-03-11 01:00:14.175940 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (335930126 0 0) 0x4b4fa80 con 0x4cbe3c0 > -54> 2015-03-11 01:00:14.175985 7f41912e8700 10 monclient(hunting): my global_id is 14511547 > -53> 2015-03-11 01:00:14.175988 7f41912e8700 1 monclient(hunting): found mon.a > -52> 2015-03-11 01:00:14.175991 7f41912e8700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0 > -51> 2015-03-11 01:00:14.175997 7f41912e8700 1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x4b31a00 con 0x4cbe3c0 > -50> 2015-03-11 01:00:14.176035 7f41912e8700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0 > -49> 2015-03-11 01:00:14.176059 7f41912e8700 1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0,osdmap=2757}) v2 -- ?+0 0x4b32800 con 0x4cbe3c0 > -48> 2015-03-11 01:00:14.176077 7f41912e8700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2015-03-11 00:59:44.176076) > -47> 2015-03-11 01:00:14.176109 7f41a1ba4880 5 monclient: authenticate success, global_id 14511547 > -46> 2015-03-11 01:00:14.176147 7f41a1ba4880 10 osd.3 2757 ensuring pgs have consumed prior maps > -45> 2015-03-11 01:00:14.176170 7f41a1ba4880 7 osd.3 2757 consume_map version 2757 > -44> 2015-03-11 01:00:14.176187 7f41a1ba4880 0 osd.3 2757 done with init, starting boot process > -43> 2015-03-11 01:00:14.176191 7f41a1ba4880 10 osd.3 2757 start_boot - have maps 2757..2757 > -42> 2015-03-11 01:00:14.176197 7f41a1ba4880 10 monclient: get_version osdmap req 0x4bad1c0 > -41> 2015-03-11 01:00:14.176207 7f41a1ba4880 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0 > -40> 2015-03-11 01:00:14.176221 7f41a1ba4880 1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x4bc41e0 con 0x4cbe3c0 > -39> 2015-03-11 01:00:14.176628 7f41a1ba4880 5 asok(0x4b50000) register_command status hook 0x4b04290 > -38> 2015-03-11 01:00:14.176655 7f41a1ba4880 5 asok(0x4b50000) register_command flush_journal hook 0x4b04290 > -37> 2015-03-11 01:00:14.176666 7f41a1ba4880 5 asok(0x4b50000) register_command dump_ops_in_flight hook 0x4b04290 > -36> 2015-03-11 01:00:14.176686 7f41a1ba4880 5 asok(0x4b50000) register_command ops hook 0x4b04290 > -35> 2015-03-11 01:00:14.176705 7f41a1ba4880 5 asok(0x4b50000) register_command dump_historic_ops hook 0x4b04290 > -34> 2015-03-11 01:00:14.176723 7f41a1ba4880 5 asok(0x4b50000) register_command dump_op_pq_state hook 0x4b04290 > -33> 2015-03-11 01:00:14.176737 7f41a1ba4880 5 asok(0x4b50000) register_command dump_blacklist hook 0x4b04290 > -32> 2015-03-11 01:00:14.176746 7f41a1ba4880 5 asok(0x4b50000) register_command dump_watchers hook 0x4b04290 > -31> 2015-03-11 01:00:14.176760 7f41a1ba4880 5 asok(0x4b50000) register_command dump_reservations hook 0x4b04290 > -30> 2015-03-11 01:00:14.176770 7f41a1ba4880 5 asok(0x4b50000) register_command get_latest_osdmap hook 0x4b04290 > -29> 2015-03-11 01:00:14.176786 7f41a1ba4880 5 asok(0x4b50000) register_command setomapval hook 0x4bad260 > -28> 2015-03-11 01:00:14.176800 7f41a1ba4880 5 asok(0x4b50000) register_command rmomapkey hook 0x4bad260 > -27> 2015-03-11 01:00:14.176817 7f41a1ba4880 5 asok(0x4b50000) register_command setomapheader hook 0x4bad260 > -26> 2015-03-11 01:00:14.176834 7f41a1ba4880 5 asok(0x4b50000) register_command getomap hook 0x4bad260 > -25> 2015-03-11 01:00:14.176850 7f41a1ba4880 5 asok(0x4b50000) register_command truncobj hook 0x4bad260 > -24> 2015-03-11 01:00:14.176866 7f41a1ba4880 5 asok(0x4b50000) register_command injectdataerr hook 0x4bad260 > -23> 2015-03-11 01:00:14.176878 7f41a1ba4880 5 asok(0x4b50000) register_command injectmdataerr hook 0x4bad260 > -22> 2015-03-11 01:00:14.178539 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 3 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x4b4fcc0 con 0x4cbe3c0 > -21> 2015-03-11 01:00:14.178650 7f41912e8700 10 osd.3 2757 do_waiters -- start > -20> 2015-03-11 01:00:14.178658 7f41912e8700 10 osd.3 2757 do_waiters -- finish > -19> 2015-03-11 01:00:14.178661 7f41912e8700 20 osd.3 2757 _dispatch 0x4b4fcc0 osd_map(3277..3277 src has 2757..3277) v3 > -18> 2015-03-11 01:00:14.178888 7f41912e8700 3 osd.3 2757 handle_osd_map epochs [3277,3277], i have 2757, src has [2757,3277] > -17> 2015-03-11 01:00:14.178906 7f41912e8700 10 osd.3 2757 handle_osd_map message skips epochs 2758..3276 > -16> 2015-03-11 01:00:14.178914 7f41912e8700 10 monclient: renew_subs > -15> 2015-03-11 01:00:14.178922 7f41912e8700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0 > -14> 2015-03-11 01:00:14.178954 7f41912e8700 1 -- 192.168.0.250:6800/2840 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=2758}) v2 -- ?+0 0x4b31a00 con 0x4cbe3c0 > -13> 2015-03-11 01:00:14.178991 7f41912e8700 10 osd.3 2757 do_waiters -- start > -12> 2015-03-11 01:00:14.178996 7f41912e8700 10 osd.3 2757 do_waiters -- finish > -11> 2015-03-11 01:00:14.179012 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 4 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x4b4f840 con 0x4cbe3c0 > -10> 2015-03-11 01:00:14.179030 7f41912e8700 10 monclient: handle_monmap mon_map magic: 0 v1 > -9> 2015-03-11 01:00:14.179045 7f41912e8700 10 monclient: got monmap 1, mon.a is now rank 0 > -8> 2015-03-11 01:00:14.179051 7f41912e8700 10 monclient: dump: > epoch 1 > fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57 > last_changed 2013-09-11 22:41:30.819687 > created 2013-09-11 22:41:30.819687 > 0: 192.168.0.10:6789/0 mon.a > > -7> 2015-03-11 01:00:14.184757 7f41912e8700 1 -- 192.168.0.250:6800/2840 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(2757..2857 src has 2757..3277) v3 ==== 37648+0+0 (2648067016 0 0) 0x4b4fa80 con 0x4cbe3c0 > -6> 2015-03-11 01:00:14.184831 7f41912e8700 10 osd.3 2757 do_waiters -- start > -5> 2015-03-11 01:00:14.184838 7f41912e8700 10 osd.3 2757 do_waiters -- finish > -4> 2015-03-11 01:00:14.184841 7f41912e8700 20 osd.3 2757 _dispatch 0x4b4fa80 osd_map(2757..2857 src has 2757..3277) v3 > -3> 2015-03-11 01:00:14.184872 7f41912e8700 3 osd.3 2757 handle_osd_map epochs [2757,2857], i have 2757, src has [2757,3277] > -2> 2015-03-11 01:00:14.184884 7f41912e8700 10 osd.3 2757 handle_osd_map got inc map for epoch 2758 > -1> 2015-03-11 01:00:14.185229 7f41912e8700 10 osd.3 2757 handle_osd_map got inc map for epoch 2759 > 0> 2015-03-11 01:00:14.195165 7f41912e8700 -1 *** Caught signal (Aborted) ** > in thread 7f41912e8700 > > ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4) > 1: /usr/bin/ceph-osd() [0xac7cea] > 2: (()+0x10050) [0x7f41a0aaf050] > 3: (gsignal()+0x37) [0x7f419ef820f7] > 4: (abort()+0x13a) [0x7f419ef834ca] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f419f867fe5] > 6: (()+0x63186) [0x7f419f866186] > 7: (()+0x631b3) [0x7f419f8661b3] > 8: (()+0x633d2) [0x7f419f8663d2] > 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7] > 10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5] > 11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc] > 12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf] > 13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7] > 14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb] > 15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007] > 16: (DispatchQueue::entry()+0x649) [0xc6fe09] > 17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d] > 18: (()+0x83a4) [0x7f41a0aa73a4] > 19: (clone()+0x6d) [0x7f419f031a4d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 rbd_replay > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 20/20 osd > 0/ 5 optracker > 0/ 5 objclass > 20/20 filestore > 1/ 3 keyvaluestore > 1/ 3 journal > 1/ 1 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/10 civetweb > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > 0/ 0 refs > 1/ 5 xio > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-osd.3.log > --- end dump of recent events --- > > > -----Original Message----- > From: Samuel Just [mailto:sjust@xxxxxxxxxx] > Sent: Wednesday, 11 March 2015 12:59 AM > To: Malcolm Haak > Cc: ceph-users@xxxxxxxxxxxxxx > Subject: Re: Issues with fresh 0.93 OSD adding to existing cluster > > Can you reproduce this with > > debug osd = 20 > debug filestore = 20 > debug ms = 1 > > on the crashing osd? Also, what sha1 are the other osds and mons running? > -Sam > > ----- Original Message ----- > From: "Malcolm Haak" <malcolm@xxxxxxx> > To: ceph-users@xxxxxxxxxxxxxx > Sent: Tuesday, March 10, 2015 3:28:26 AM > Subject: Issues with fresh 0.93 OSD adding to existing cluster > > Hi all, > > I've just attempted to add a new node and OSD to an existing ceph cluster (it's a small one I use as a NAS at home, not like the big production ones I normally work on) and it seems to be throwing some odd errors... > > Just looking for where to poke it next... > > Log is below, > > It's a two node cluster with 3 osd's in node A and one osd in the new node (It's going to have more eventually and node one will be retired after node three gets added) And I've hit a weird snag. > > I was running 0.80 but I ran into the 'Invalid Command' bug on the new node so I opted to jump to the latest code with the required patches already. > > Please let me know what else you need.. > > This is the log content when attempting to start the new OSD: > > 2015-03-10 19:28:48.795318 7f0774108880 0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 10810 > 2015-03-10 19:28:48.817803 7f0774108880 0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) > 2015-03-10 19:28:48.866862 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work > 2015-03-10 19:28:48.866920 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option > 2015-03-10 19:28:48.905069 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) > 2015-03-10 19:28:48.905467 7f0774108880 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5 > 2015-03-10 19:28:49.077872 7f0774108880 0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled > 2015-03-10 19:28:49.078321 7f0774108880 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway > 2015-03-10 19:28:49.078328 7f0774108880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0 > 2015-03-10 19:28:49.079721 7f0774108880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0 > 2015-03-10 19:28:49.080948 7f0774108880 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello > 2015-03-10 19:28:49.094194 7f0774108880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients > 2015-03-10 19:28:49.094211 7f0774108880 0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons > 2015-03-10 19:28:49.094217 7f0774108880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds > 2015-03-10 19:28:49.094235 7f0774108880 0 osd.3 2757 load_pgs > 2015-03-10 19:28:49.094279 7f0774108880 0 osd.3 2757 load_pgs opened 0 pgs > 2015-03-10 19:28:49.095121 7f0774108880 -1 osd.3 2757 log_to_monitors {default=true} > 2015-03-10 19:28:49.134104 7f0774108880 0 osd.3 2757 done with init, starting boot process > 2015-03-10 19:28:49.149994 7f076384c700 -1 *** Caught signal (Aborted) ** in thread 7f076384c700 > > ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4) > 1: /usr/bin/ceph-osd() [0xac7cea] > 2: (()+0x10050) [0x7f0773013050] > 3: (gsignal()+0x37) [0x7f07714e60f7] > 4: (abort()+0x13a) [0x7f07714e74ca] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f0771dcbfe5] > 6: (()+0x63186) [0x7f0771dca186] > 7: (()+0x631b3) [0x7f0771dca1b3] > 8: (()+0x633d2) [0x7f0771dca3d2] > 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7] > 10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5] > 11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc] > 12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf] > 13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7] > 14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb] > 15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007] > 16: (DispatchQueue::entry()+0x649) [0xc6fe09] > 17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d] > 18: (()+0x83a4) [0x7f077300b3a4] > 19: (clone()+0x6d) [0x7f0771595a4d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > --- begin dump of recent events --- > -135> 2015-03-10 19:28:48.790490 7f0774108880 5 asok(0x4200000) register_command perfcounters_dump hook 0x41b4030 > -134> 2015-03-10 19:28:48.790565 7f0774108880 5 asok(0x4200000) register_command 1 hook 0x41b4030 > -133> 2015-03-10 19:28:48.790571 7f0774108880 5 asok(0x4200000) register_command perf dump hook 0x41b4030 > -132> 2015-03-10 19:28:48.790583 7f0774108880 5 asok(0x4200000) register_command perfcounters_schema hook 0x41b4030 > -131> 2015-03-10 19:28:48.790599 7f0774108880 5 asok(0x4200000) register_command 2 hook 0x41b4030 > -130> 2015-03-10 19:28:48.790605 7f0774108880 5 asok(0x4200000) register_command perf schema hook 0x41b4030 > -129> 2015-03-10 19:28:48.790635 7f0774108880 5 asok(0x4200000) register_command perf reset hook 0x41b4030 > -128> 2015-03-10 19:28:48.790646 7f0774108880 5 asok(0x4200000) register_command config show hook 0x41b4030 > -127> 2015-03-10 19:28:48.790652 7f0774108880 5 asok(0x4200000) register_command config set hook 0x41b4030 > -126> 2015-03-10 19:28:48.790660 7f0774108880 5 asok(0x4200000) register_command config get hook 0x41b4030 > -125> 2015-03-10 19:28:48.790672 7f0774108880 5 asok(0x4200000) register_command config diff hook 0x41b4030 > -124> 2015-03-10 19:28:48.790682 7f0774108880 5 asok(0x4200000) register_command log flush hook 0x41b4030 > -123> 2015-03-10 19:28:48.790689 7f0774108880 5 asok(0x4200000) register_command log dump hook 0x41b4030 > -122> 2015-03-10 19:28:48.790698 7f0774108880 5 asok(0x4200000) register_command log reopen hook 0x41b4030 > -121> 2015-03-10 19:28:48.795318 7f0774108880 0 ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4), process ceph-osd, pid 10810 > -120> 2015-03-10 19:28:48.796071 7f0774108880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/10810 need_addr=1 > -119> 2015-03-10 19:28:48.796123 7f0774108880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/10810 need_addr=1 > -118> 2015-03-10 19:28:48.796144 7f0774108880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/10810 need_addr=1 > -117> 2015-03-10 19:28:48.796160 7f0774108880 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/10810 need_addr=1 > -116> 2015-03-10 19:28:48.799414 7f0774108880 5 asok(0x4200000) init /var/run/ceph/ceph-osd.3.asok > -115> 2015-03-10 19:28:48.799446 7f0774108880 5 asok(0x4200000) bind_and_listen /var/run/ceph/ceph-osd.3.asok > -114> 2015-03-10 19:28:48.799644 7f0774108880 5 asok(0x4200000) register_command 0 hook 0x41b00c0 > -113> 2015-03-10 19:28:48.799659 7f0774108880 5 asok(0x4200000) register_command version hook 0x41b00c0 > -112> 2015-03-10 19:28:48.799665 7f0774108880 5 asok(0x4200000) register_command git_version hook 0x41b00c0 > -111> 2015-03-10 19:28:48.799671 7f0774108880 5 asok(0x4200000) register_command help hook 0x41b4110 > -110> 2015-03-10 19:28:48.799678 7f0774108880 5 asok(0x4200000) register_command get_command_descriptions hook 0x41b4120 > -109> 2015-03-10 19:28:48.799760 7f0774108880 10 monclient(hunting): build_initial_monmap > -108> 2015-03-10 19:28:48.799757 7f076f4e1700 5 asok(0x4200000) entry start > -107> 2015-03-10 19:28:48.816916 7f0774108880 5 adding auth protocol: none > -106> 2015-03-10 19:28:48.816948 7f0774108880 5 adding auth protocol: none > -105> 2015-03-10 19:28:48.817130 7f0774108880 5 asok(0x4200000) register_command objecter_requests hook 0x41b41b0 > -104> 2015-03-10 19:28:48.817273 7f0774108880 1 -- 0.0.0.0:6800/10810 messenger.start > -103> 2015-03-10 19:28:48.817334 7f0774108880 1 -- :/0 messenger.start > -102> 2015-03-10 19:28:48.817378 7f0774108880 1 -- 0.0.0.0:6803/10810 messenger.start > -101> 2015-03-10 19:28:48.817442 7f0774108880 1 -- 0.0.0.0:6802/10810 messenger.start > -100> 2015-03-10 19:28:48.817494 7f0774108880 1 -- 0.0.0.0:6801/10810 messenger.start > -99> 2015-03-10 19:28:48.817546 7f0774108880 1 -- :/0 messenger.start > -98> 2015-03-10 19:28:48.817666 7f0774108880 2 osd.3 0 mounting /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal > -97> 2015-03-10 19:28:48.817803 7f0774108880 0 filestore(/var/lib/ceph/osd/ceph-3) backend xfs (magic 0x58465342) > -96> 2015-03-10 19:28:48.866862 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is supported and appears to work > -95> 2015-03-10 19:28:48.866920 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option > -94> 2015-03-10 19:28:48.905069 7f0774108880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) > -93> 2015-03-10 19:28:48.905467 7f0774108880 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-3) detect_feature: extsize is supported and kernel 3.18.3-1-desktop >= 3.5 > -92> 2015-03-10 19:28:49.077872 7f0774108880 0 filestore(/var/lib/ceph/osd/ceph-3) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled > -91> 2015-03-10 19:28:49.078270 7f0774108880 2 journal open /var/lib/ceph/osd/ceph-3/journal fsid 05fb481f-68b3-4442-b7de-af4c867e18d5 fs_op_seq 2 > -90> 2015-03-10 19:28:49.078321 7f0774108880 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway > -89> 2015-03-10 19:28:49.078328 7f0774108880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0 > -88> 2015-03-10 19:28:49.078937 7f0774108880 2 journal read_entry 8192 : seq 3 193 bytes > -87> 2015-03-10 19:28:49.079000 7f0774108880 2 journal read_entry 8192 : seq 3 193 bytes > -86> 2015-03-10 19:28:49.079022 7f0774108880 3 journal journal_replay: applying op seq 3 > -85> 2015-03-10 19:28:49.079307 7f0774108880 3 journal journal_replay: r = 0, op_seq now 3 > -84> 2015-03-10 19:28:49.079367 7f0774108880 2 journal read_entry 12288 : seq 4 3829 bytes > -83> 2015-03-10 19:28:49.079390 7f0774108880 3 journal journal_replay: applying op seq 4 > -82> 2015-03-10 19:28:49.079667 7f0774108880 3 journal journal_replay: r = 0, op_seq now 4 > -81> 2015-03-10 19:28:49.079699 7f0774108880 2 journal No further valid entries found, journal is most likely valid > -80> 2015-03-10 19:28:49.079703 7f0774108880 3 journal journal_replay: end of journal, done. > -79> 2015-03-10 19:28:49.079721 7f0774108880 1 journal _open /var/lib/ceph/osd/ceph-3/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0 > -78> 2015-03-10 19:28:49.080265 7f0774108880 2 osd.3 0 boot > -77> 2015-03-10 19:28:49.080948 7f0774108880 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello > -76> 2015-03-10 19:28:49.081757 7f0774108880 1 <cls> cls/log/cls_log.cc:312: Loaded log class! > -75> 2015-03-10 19:28:49.087397 7f0774108880 1 <cls> cls/refcount/cls_refcount.cc:231: Loaded refcount class! > -74> 2015-03-10 19:28:49.087735 7f0774108880 1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class! > -73> 2015-03-10 19:28:49.093126 7f0774108880 1 <cls> cls/rgw/cls_rgw.cc:3046: Loaded rgw class! > -72> 2015-03-10 19:28:49.093484 7f0774108880 1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class! > -71> 2015-03-10 19:28:49.093723 7f0774108880 1 <cls> cls/user/cls_user.cc:367: Loaded user class! > -70> 2015-03-10 19:28:49.093924 7f0774108880 1 <cls> cls/version/cls_version.cc:227: Loaded version class! > -69> 2015-03-10 19:28:49.094194 7f0774108880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for clients > -68> 2015-03-10 19:28:49.094211 7f0774108880 0 osd.3 2757 crush map has features 33816576 was 8705, adjusting msgr requires for mons > -67> 2015-03-10 19:28:49.094217 7f0774108880 0 osd.3 2757 crush map has features 33816576, adjusting msgr requires for osds > -66> 2015-03-10 19:28:49.094235 7f0774108880 0 osd.3 2757 load_pgs > -65> 2015-03-10 19:28:49.094279 7f0774108880 0 osd.3 2757 load_pgs opened 0 pgs > -64> 2015-03-10 19:28:49.094285 7f0774108880 2 osd.3 2757 superblock: i am osd.3 > -63> 2015-03-10 19:28:49.094424 7f0774108880 1 accepter.accepter.start > -62> 2015-03-10 19:28:49.094573 7f0774108880 1 accepter.accepter.start > -61> 2015-03-10 19:28:49.094748 7f0774108880 1 accepter.accepter.start > -60> 2015-03-10 19:28:49.094864 7f0774108880 1 accepter.accepter.start > -59> 2015-03-10 19:28:49.094963 7f0774108880 10 monclient(hunting): init > -58> 2015-03-10 19:28:49.094980 7f0774108880 5 adding auth protocol: none > -57> 2015-03-10 19:28:49.095005 7f0774108880 10 monclient(hunting): auth_supported 1 method none > -56> 2015-03-10 19:28:49.095110 7f0774108880 10 log_channel(default) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info) > -55> 2015-03-10 19:28:49.095121 7f0774108880 -1 osd.3 2757 log_to_monitors {default=true} > -54> 2015-03-10 19:28:49.095899 7f0774108880 10 monclient(hunting): renew_subs > -53> 2015-03-10 19:28:49.095914 7f0774108880 10 monclient(hunting): _reopen_session rank -1 name > -52> 2015-03-10 19:28:49.096003 7f0774108880 10 monclient(hunting): picked mon.a con 0x43623c0 addr 192.168.0.10:6789/0 > -51> 2015-03-10 19:28:49.096086 7f0774108880 10 monclient(hunting): _send_mon_message to mon.a at 192.168.0.10:6789/0 > -50> 2015-03-10 19:28:49.096098 7f0774108880 1 -- 0.0.0.0:6800/10810 --> 192.168.0.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x41fcd80 con 0x43623c0 > -49> 2015-03-10 19:28:49.096132 7f0774108880 10 monclient(hunting): renew_subs > -48> 2015-03-10 19:28:49.096501 7f075302b700 5 osd.3 2757 heartbeat: osd_stat(1056 MB used, 1851 GB avail, 1852 GB total, peers []/[] op hist []) > -47> 2015-03-10 19:28:49.113808 7f0774104700 1 -- 192.168.0.250:6800/10810 learned my addr 192.168.0.250:6800/10810 > -46> 2015-03-10 19:28:49.131334 7f076384c700 10 monclient(hunting): renew_subs > -45> 2015-03-10 19:28:49.133513 7f076384c700 1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x41ff840 con 0x43623c0 > -44> 2015-03-10 19:28:49.133572 7f076384c700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 > -43> 2015-03-10 19:28:49.133587 7f076384c700 10 monclient(hunting): got monmap 1, mon.a is now rank 0 > -42> 2015-03-10 19:28:49.133597 7f076384c700 10 monclient(hunting): dump: > epoch 1 > fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57 > last_changed 2013-09-11 22:41:30.819687 > created 2013-09-11 22:41:30.819687 > 0: 192.168.0.10:6789/0 mon.a > > -41> 2015-03-10 19:28:49.133678 7f076384c700 1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 2 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (1541693634 0 0) 0x41ffa80 con 0x43623c0 > -40> 2015-03-10 19:28:49.133716 7f076384c700 10 monclient(hunting): my global_id is 14511543 > -39> 2015-03-10 19:28:49.133721 7f076384c700 1 monclient(hunting): found mon.a > -38> 2015-03-10 19:28:49.133803 7f076384c700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0 > -37> 2015-03-10 19:28:49.133820 7f076384c700 1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x41e1a00 con 0x43623c0 > -36> 2015-03-10 19:28:49.133906 7f076384c700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0 > -35> 2015-03-10 19:28:49.133925 7f076384c700 1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0,osdmap=2757}) v2 -- ?+0 0x41e2800 con 0x43623c0 > -34> 2015-03-10 19:28:49.134016 7f076384c700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2015-03-10 19:28:19.134014) > -33> 2015-03-10 19:28:49.134061 7f0774108880 5 monclient: authenticate success, global_id 14511543 > -32> 2015-03-10 19:28:49.134104 7f0774108880 0 osd.3 2757 done with init, starting boot process > -31> 2015-03-10 19:28:49.134118 7f0774108880 10 monclient: get_version osdmap req 0x4259160 > -30> 2015-03-10 19:28:49.134126 7f0774108880 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0 > -29> 2015-03-10 19:28:49.134136 7f0774108880 1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x42741e0 con 0x43623c0 > -28> 2015-03-10 19:28:49.134362 7f0774108880 5 asok(0x4200000) register_command status hook 0x41b4290 > -27> 2015-03-10 19:28:49.134388 7f0774108880 5 asok(0x4200000) register_command flush_journal hook 0x41b4290 > -26> 2015-03-10 19:28:49.134405 7f0774108880 5 asok(0x4200000) register_command dump_ops_in_flight hook 0x41b4290 > -25> 2015-03-10 19:28:49.134421 7f0774108880 5 asok(0x4200000) register_command ops hook 0x41b4290 > -24> 2015-03-10 19:28:49.134437 7f0774108880 5 asok(0x4200000) register_command dump_historic_ops hook 0x41b4290 > -23> 2015-03-10 19:28:49.134454 7f0774108880 5 asok(0x4200000) register_command dump_op_pq_state hook 0x41b4290 > -22> 2015-03-10 19:28:49.134469 7f0774108880 5 asok(0x4200000) register_command dump_blacklist hook 0x41b4290 > -21> 2015-03-10 19:28:49.134484 7f0774108880 5 asok(0x4200000) register_command dump_watchers hook 0x41b4290 > -20> 2015-03-10 19:28:49.134500 7f0774108880 5 asok(0x4200000) register_command dump_reservations hook 0x41b4290 > -19> 2015-03-10 19:28:49.134516 7f0774108880 5 asok(0x4200000) register_command get_latest_osdmap hook 0x41b4290 > -18> 2015-03-10 19:28:49.134532 7f0774108880 5 asok(0x4200000) register_command setomapval hook 0x4259200 > -17> 2015-03-10 19:28:49.134546 7f0774108880 5 asok(0x4200000) register_command rmomapkey hook 0x4259200 > -16> 2015-03-10 19:28:49.134563 7f0774108880 5 asok(0x4200000) register_command setomapheader hook 0x4259200 > -15> 2015-03-10 19:28:49.134580 7f0774108880 5 asok(0x4200000) register_command getomap hook 0x4259200 > -14> 2015-03-10 19:28:49.134595 7f0774108880 5 asok(0x4200000) register_command truncobj hook 0x4259200 > -13> 2015-03-10 19:28:49.134609 7f0774108880 5 asok(0x4200000) register_command injectdataerr hook 0x4259200 > -12> 2015-03-10 19:28:49.134624 7f0774108880 5 asok(0x4200000) register_command injectmdataerr hook 0x4259200 > -11> 2015-03-10 19:28:49.136325 7f076384c700 1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 3 ==== osd_map(3277..3277 src has 2757..3277) v3 ==== 5366+0+0 (3110999244 0 0) 0x41ffcc0 con 0x43623c0 > -10> 2015-03-10 19:28:49.136622 7f076384c700 3 osd.3 2757 handle_osd_map epochs [3277,3277], i have 2757, src has [2757,3277] > -9> 2015-03-10 19:28:49.136643 7f076384c700 10 monclient: renew_subs > -8> 2015-03-10 19:28:49.136652 7f076384c700 10 monclient: _send_mon_message to mon.a at 192.168.0.10:6789/0 > -7> 2015-03-10 19:28:49.136679 7f076384c700 1 -- 192.168.0.250:6800/10810 --> 192.168.0.10:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=2758}) v2 -- ?+0 0x41e1a00 con 0x43623c0 > -6> 2015-03-10 19:28:49.136943 7f076384c700 1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 4 ==== mon_map magic: 0 v1 ==== 191+0+0 (1112175541 0 0) 0x41ff840 con 0x43623c0 > -5> 2015-03-10 19:28:49.136994 7f076384c700 10 monclient: handle_monmap mon_map magic: 0 v1 > -4> 2015-03-10 19:28:49.137010 7f076384c700 10 monclient: got monmap 1, mon.a is now rank 0 > -3> 2015-03-10 19:28:49.137022 7f076384c700 10 monclient: dump: > epoch 1 > fsid 3bf19e54-97e2-4a5f-b811-7c3780cd5e57 > last_changed 2013-09-11 22:41:30.819687 > created 2013-09-11 22:41:30.819687 > 0: 192.168.0.10:6789/0 mon.a > > -2> 2015-03-10 19:28:49.142203 7f076384c700 1 -- 192.168.0.250:6800/10810 <== mon.0 192.168.0.10:6789/0 5 ==== osd_map(2757..2857 src has 2757..3277) v3 ==== 37648+0+0 (2648067016 0 0) 0x41ffa80 con 0x43623c0 > -1> 2015-03-10 19:28:49.142316 7f076384c700 3 osd.3 2757 handle_osd_map epochs [2757,2857], i have 2757, src has [2757,3277] > 0> 2015-03-10 19:28:49.149994 7f076384c700 -1 *** Caught signal (Aborted) ** in thread 7f076384c700 > > ceph version 0.93 (bebf8e9a830d998eeaab55f86bb256d4360dd3c4) > 1: /usr/bin/ceph-osd() [0xac7cea] > 2: (()+0x10050) [0x7f0773013050] > 3: (gsignal()+0x37) [0x7f07714e60f7] > 4: (abort()+0x13a) [0x7f07714e74ca] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f0771dcbfe5] > 6: (()+0x63186) [0x7f0771dca186] > 7: (()+0x631b3) [0x7f0771dca1b3] > 8: (()+0x633d2) [0x7f0771dca3d2] > 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x137) [0xc2cea7] > 10: (OSDMap::decode_classic(ceph::buffer::list::iterator&)+0x605) [0xb7b7b5] > 11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x8c) [0xb7bebc] > 12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0xb7dfbf] > 13: (OSD::handle_osd_map(MOSDMap*)+0xd37) [0x6cd9a7] > 14: (OSD::_dispatch(Message*)+0x3eb) [0x6d0afb] > 15: (OSD::ms_dispatch(Message*)+0x257) [0x6d1007] > 16: (DispatchQueue::entry()+0x649) [0xc6fe09] > 17: (DispatchQueue::DispatchThread::entry()+0xd) [0xb9dd7d] > 18: (()+0x83a4) [0x7f077300b3a4] > 19: (clone()+0x6d) [0x7f0771595a4d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 rbd_replay > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 0/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 keyvaluestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/10 civetweb > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > 0/ 0 refs > 1/ 5 xio > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-osd.3.log > --- end dump of recent events --- > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |