Hi,
we have multiple Ceph clusters. One is used as backend for OpenStack installation for developers - it's here we test Ceph upgrades before we upgrade prod Ceph clusters. The Ceph cluster is 4 nodes with 12 osds each running Ubuntu Trusty with latest 3.13 kernel.
This time when upgrading from 0.94.3 to 0.94.5 ceph-mons died during the upgrade a couple of times. One when we restarted the first monitor, of three, during the upgrade procedure and the second time when we ran 'ceph osd unset noout' in the end.
I thought this was a fluke during the upgrade, but ceph-mons seem to segfault fairly regular now, the day after the upgrade. Corefile doesn't get dumped, so I have only the log for this strange behaviour. The cluster has been following the upgrades from firefly to the current Hammer release and has worked flawless until now.
The cluster produces and work more or less as normal from the users viewpoint. Except we get segmentation faults in the logfile. Downgrading is a last resort that I rather not do.
What can cause these errors and how can I fix it is my question.
-Arnulf
The segfault look like this:
Oct 29 14:29:46 95z3zz1 ceph-mon: 0> 2015-10-29 14:29:46.297786 7f908e5af700 -1 *** Caught signal (Segmentation fault) **#012 in thread 7f908e5af700#012#012 ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)#012 1: /usr/bin/ceph-mon() [0x9adefa]#012 2: (()+0x10340) [0x7f90936b6340]#012 3: (std::_Rb_tree<std::string, std::pair<std::string const, std::string>, std::_Select1st<std::pair<std::string const, std::string> >, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >::find(std::string const&) const+0x25) [0x6518e5]#012 4: (get_str_map_key(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, std::string const&, std::string const*)+0x1e) [0x8a002e]#012 5: (LogMonitor::update_from_paxos(bool*)+0x87a) [0x6b0a5a]#012 6: (PaxosService::refresh(bool*)+0x19a) [0x60432a]#012 7: (Monitor::refresh_from_paxos(bool*)+0x1db) [0x5b03db]#012 8: (Paxos::do_refresh()+0
x2e) [0x5eea5e]#012 9: (Paxos::commit_finish()+0x569) [0x5fbf39]#012 10: (C_Committed::finish(int)+0x2b) [0x60038b]#012 11: (Context::complete(int)+0x9) [0x5d4d89]#012 12: (MonitorDBStore::C_DoTransaction::finish(int)+0x8c) [0x5ff4bc]#012 13: (Context::complete(int)+0x9) [0x5d4d89]#012 14: (Finisher::finisher_thread_entry()+0x158) [0x717e88]#012 15: (()+0x8182) [0x7f90936ae182]#012 16: (clone()+0x6d) [0x7f9091c1947d]#012 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Full log of event:
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.697177 7f3154801700 -1 *** Caught signal (Segmentation fault) **#012 in thread 7f3154801700#012#012 ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)#012 1: /usr/bin/ceph-mon() [0x9adefa]#012 2: (()+0x10340) [0x7f3159b63340]#012 3: (std::_Rb_tree<std::string, std::pair<std::string const, std::string>, std::_Select1st<std::pair<std::string const, std::string> >, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >::find(std::string const&) const+0x25) [0x6518e5]#012 4: (get_str_map_key(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, std::string const&, std::string const*)+0x1e) [0x8a002e]#012 5: (LogMonitor::update_from_paxos(bool*)+0x87a) [0x6b0a5a]#012 6: (PaxosService::refresh(bool*)+0x19a) [0x60432a]#012 7: (Monitor::refresh_from_paxos(bool*)+0x1db) [0x5b03db]#012 8: (Paxos::do_refresh()+0x2e) [0x
5eea5e]#012 9: (Paxos::commit_finish()+0x569) [0x5fbf39]#012 10: (C_Committed::finish(int)+0x2b) [0x60038b]#012 11: (Context::complete(int)+0x9) [0x5d4d89]#012 12: (MonitorDBStore::C_DoTransaction::finish(int)+0x8c) [0x5ff4bc]#012 13: (Context::complete(int)+0x9) [0x5d4d89]#012 14: (Finisher::finisher_thread_entry()+0x158) [0x717e88]#012 15: (()+0x8182) [0x7f3159b5b182]#012 16: (clone()+0x6d) [0x7f31580c647d]#012 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Oct 29 14:29:45 95z3zz1 ceph-mon: --- begin dump of recent events ---
Oct 29 14:29:45 95z3zz1 ceph-mon: -450> 2015-10-29 14:29:44.484656 7f315aa5d8c0 5 asok(0x4daa000) register_command perfcounters_dump hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -449> 2015-10-29 14:29:44.484677 7f315aa5d8c0 5 asok(0x4daa000) register_command 1 hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -448> 2015-10-29 14:29:44.484681 7f315aa5d8c0 5 asok(0x4daa000) register_command perf dump hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -447> 2015-10-29 14:29:44.484686 7f315aa5d8c0 5 asok(0x4daa000) register_command perfcounters_schema hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -446> 2015-10-29 14:29:44.484688 7f315aa5d8c0 5 asok(0x4daa000) register_command 2 hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -445> 2015-10-29 14:29:44.484690 7f315aa5d8c0 5 asok(0x4daa000) register_command perf schema hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -444> 2015-10-29 14:29:44.484692 7f315aa5d8c0 5 asok(0x4daa000) register_command perf reset hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -443> 2015-10-29 14:29:44.484694 7f315aa5d8c0 5 asok(0x4daa000) register_command config show hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -442> 2015-10-29 14:29:44.484697 7f315aa5d8c0 5 asok(0x4daa000) register_command config set hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -441> 2015-10-29 14:29:44.484698 7f315aa5d8c0 5 asok(0x4daa000) register_command config get hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -440> 2015-10-29 14:29:44.484700 7f315aa5d8c0 5 asok(0x4daa000) register_command config diff hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -439> 2015-10-29 14:29:44.484702 7f315aa5d8c0 5 asok(0x4daa000) register_command log flush hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -438> 2015-10-29 14:29:44.484704 7f315aa5d8c0 5 asok(0x4daa000) register_command log dump hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -437> 2015-10-29 14:29:44.484706 7f315aa5d8c0 5 asok(0x4daa000) register_command log reopen hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -436> 2015-10-29 14:29:44.486767 7f315aa5d8c0 0 ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43), process ceph-mon, pid 143413
Oct 29 14:29:45 95z3zz1 ceph-mon: -435> 2015-10-29 14:29:44.488255 7f315aa5d8c0 5 asok(0x4daa000) init /var/run/ceph/ceph-mon.95z3zz1.asok
Oct 29 14:29:45 95z3zz1 ceph-mon: -434> 2015-10-29 14:29:44.488271 7f315aa5d8c0 5 asok(0x4daa000) bind_and_listen /var/run/ceph/ceph-mon.95z3zz1.asok
Oct 29 14:29:45 95z3zz1 ceph-mon: -433> 2015-10-29 14:29:44.488320 7f315aa5d8c0 5 asok(0x4daa000) register_command 0 hook 0x4d2e0b8
Oct 29 14:29:45 95z3zz1 ceph-mon: -432> 2015-10-29 14:29:44.488327 7f315aa5d8c0 5 asok(0x4daa000) register_command version hook 0x4d2e0b8
Oct 29 14:29:45 95z3zz1 ceph-mon: -431> 2015-10-29 14:29:44.488332 7f315aa5d8c0 5 asok(0x4daa000) register_command git_version hook 0x4d2e0b8
Oct 29 14:29:45 95z3zz1 ceph-mon: -430> 2015-10-29 14:29:44.488336 7f315aa5d8c0 5 asok(0x4daa000) register_command help hook 0x4d32150
Oct 29 14:29:45 95z3zz1 ceph-mon: -429> 2015-10-29 14:29:44.488340 7f315aa5d8c0 5 asok(0x4daa000) register_command get_command_descriptions hook 0x4d32140
Oct 29 14:29:45 95z3zz1 ceph-mon: -428> 2015-10-29 14:29:44.488570 7f3155ddf700 5 asok(0x4daa000) entry start
Oct 29 14:29:45 95z3zz1 ceph-mon: -427> 2015-10-29 14:29:44.513886 7f315aa5d8c0 0 starting mon.95z3zz1 rank 0 at 157.249.160.11:6789/0 mon_data /var/lib/ceph/mon/ceph-95z3zz1 fsid 6a57f77c-a465-4fc6-b0a2-93c20371bab8
Oct 29 14:29:45 95z3zz1 ceph-mon: -426> 2015-10-29 14:29:44.513983 7f315aa5d8c0 1 -- 157.249.160.11:6789/0 learned my addr 157.249.160.11:6789/0
Oct 29 14:29:45 95z3zz1 ceph-mon: -425> 2015-10-29 14:29:44.513989 7f315aa5d8c0 1 accepter.accepter.bind my_inst.addr is 157.249.160.11:6789/0 need_addr=0
Oct 29 14:29:45 95z3zz1 ceph-mon: -424> 2015-10-29 14:29:44.514048 7f315aa5d8c0 5 adding auth protocol: cephx
Oct 29 14:29:45 95z3zz1 ceph-mon: -423> 2015-10-29 14:29:44.514055 7f315aa5d8c0 5 adding auth protocol: cephx
Oct 29 14:29:45 95z3zz1 ceph-mon: -422> 2015-10-29 14:29:44.514075 7f315aa5d8c0 10 log_channel(cluster) update_config to_monitors: true to_syslog: true syslog_facility: daemon prio: info)
Oct 29 14:29:45 95z3zz1 ceph-mon: -421> 2015-10-29 14:29:44.514081 7f315aa5d8c0 10 log_channel(audit) update_config to_monitors: true to_syslog: true syslog_facility: local0 prio: info)
Oct 29 14:29:45 95z3zz1 ceph-mon: -420> 2015-10-29 14:29:44.514144 7f315aa5d8c0 1 mon.95z3zz1@-1(probing) e7 preinit fsid 6a57f77c-a465-4fc6-b0a2-93c20371bab8
Oct 29 14:29:45 95z3zz1 ceph-mon: -419> 2015-10-29 14:29:44.514476 7f315aa5d8c0 1 mon.95z3zz1@-1(probing).paxosservice(pgmap 33661126..33661653) refresh upgraded, format 0 -> 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -418> 2015-10-29 14:29:44.514491 7f315aa5d8c0 1 mon.95z3zz1@-1(probing).pg v0 on_upgrade discarding in-core PGMap
Oct 29 14:29:45 95z3zz1 ceph-mon: -417> 2015-10-29 14:29:44.541117 7f3155ddf700 5 asok(0x4daa000) AdminSocket: request 'get_command_descriptions' '' to 0x4d32140 returned 1496 bytes
Oct 29 14:29:45 95z3zz1 ceph-mon: -416> 2015-10-29 14:29:44.626420 7f315aa5d8c0 4 mon.95z3zz1@-1(probing).mds e89 new map
Oct 29 14:29:45 95z3zz1 ceph-mon: -415> 2015-10-29 14:29:44.626431 7f315aa5d8c0 0 mon.95z3zz1@-1(probing).mds e89 print_map#012epoch#01189#012flags#0110#012created#0112014-07-31 11:09:25.038824#012modified#0112015-10-28 11:59:17.337799#012tableserver#0110#012root#0110#012session_timeout#01160#012session_autoclose#011300#012max_file_size#0111099511627776#012last_failure#0110#012last_failure_osd_epoch#01161286#012compat#011compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table}#012max_mds#0111#012in#0110#012up#011{0=15517917}#012failed#011#012stopped#011#012data_pools#0110#012metadata_pool#0111#012inline_data#011disabled#01215517917:#011157.249.160.107:6800/3998 '56z3zz1' mds.0.13 up:active seq 188
Oct 29 14:29:45 95z3zz1 ceph-mon: -414> 2015-10-29 14:29:44.626991 7f315aa5d8c0 0 mon.95z3zz1@-1(probing).osd e61385 crush map has features 1107558400, adjusting msgr requires
Oct 29 14:29:45 95z3zz1 ceph-mon: -413> 2015-10-29 14:29:44.627009 7f315aa5d8c0 0 mon.95z3zz1@-1(probing).osd e61385 crush map has features 1107558400, adjusting msgr requires
Oct 29 14:29:45 95z3zz1 ceph-mon: -412> 2015-10-29 14:29:44.627013 7f315aa5d8c0 0 mon.95z3zz1@-1(probing).osd e61385 crush map has features 1107558400, adjusting msgr requires
Oct 29 14:29:45 95z3zz1 ceph-mon: -411> 2015-10-29 14:29:44.627015 7f315aa5d8c0 0 mon.95z3zz1@-1(probing).osd e61385 crush map has features 1107558400, adjusting msgr requires
Oct 29 14:29:45 95z3zz1 ceph-mon: -410> 2015-10-29 14:29:44.627470 7f315aa5d8c0 1 mon.95z3zz1@-1(probing).paxosservice(auth 28285..28398) refresh upgraded, format 0 -> 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -409> 2015-10-29 14:29:44.627977 7f315aa5d8c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-95z3zz1/keyring
Oct 29 14:29:45 95z3zz1 ceph-mon: -408> 2015-10-29 14:29:44.627983 7f315aa5d8c0 5 asok(0x4daa000) register_command mon_status hook 0x5e1dbe0
Oct 29 14:29:45 95z3zz1 ceph-mon: -407> 2015-10-29 14:29:44.628001 7f315aa5d8c0 5 asok(0x4daa000) register_command quorum_status hook 0x5e1dbe0
Oct 29 14:29:45 95z3zz1 ceph-mon: -406> 2015-10-29 14:29:44.628005 7f315aa5d8c0 5 asok(0x4daa000) register_command sync_force hook 0x5e1dbe0
Oct 29 14:29:45 95z3zz1 ceph-mon: -405> 2015-10-29 14:29:44.628007 7f315aa5d8c0 5 asok(0x4daa000) register_command add_bootstrap_peer_hint hook 0x5e1dbe0
Oct 29 14:29:45 95z3zz1 ceph-mon: -404> 2015-10-29 14:29:44.628010 7f315aa5d8c0 5 asok(0x4daa000) register_command quorum enter hook 0x5e1dbe0
Oct 29 14:29:45 95z3zz1 ceph-mon: -403> 2015-10-29 14:29:44.628012 7f315aa5d8c0 5 asok(0x4daa000) register_command quorum exit hook 0x5e1dbe0
Oct 29 14:29:45 95z3zz1 ceph-mon: -402> 2015-10-29 14:29:44.628023 7f315aa5d8c0 1 -- 157.249.160.11:6789/0 messenger.start
Oct 29 14:29:45 95z3zz1 ceph-mon: -401> 2015-10-29 14:29:44.628058 7f315aa5d8c0 2 mon.95z3zz1@-1(probing) e7 init
Oct 29 14:29:45 95z3zz1 ceph-mon: -400> 2015-10-29 14:29:44.628118 7f315aa5d8c0 1 accepter.accepter.start
Oct 29 14:29:45 95z3zz1 ceph-mon: -399> 2015-10-29 14:29:44.628168 7f315aa5d8c0 0 mon.95z3zz1@-1(probing) e7 my rank is now 0 (was -1)
Oct 29 14:29:45 95z3zz1 ceph-mon: -398> 2015-10-29 14:29:44.628175 7f315aa5d8c0 1 -- 157.249.160.11:6789/0 mark_down_all
Oct 29 14:29:45 95z3zz1 ceph-mon: -397> 2015-10-29 14:29:44.628196 7f315aa5d8c0 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 -- mon_probe(probe 6a57f77c-a465-4fc6-b0a2-93c20371bab8 name 95z3zz1) v6 -- ?+0 0x4dbac80
Oct 29 14:29:45 95z3zz1 ceph-mon: -396> 2015-10-29 14:29:44.628303 7f315aa5d8c0 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 -- mon_probe(probe 6a57f77c-a465-4fc6-b0a2-93c20371bab8 name 95z3zz1) v6 -- ?+0 0x4dbaa00
Oct 29 14:29:45 95z3zz1 ceph-mon: -395> 2015-10-29 14:29:44.628313 7f314f84b700 1 -- 157.249.160.11:6789/0 >> :/0 pipe(0x5ee2000 sd=13 :6789 s=0 pgs=0 cs=0 l=0 c=0x4d54260).accept sd=13 157.249.160.106:43105/0
Oct 29 14:29:45 95z3zz1 ceph-mon: -394> 2015-10-29 14:29:44.628376 7f314f649700 1 -- 157.249.160.11:6789/0 >> :/0 pipe(0x5ef1000 sd=14 :6789 s=0 pgs=0 cs=0 l=0 c=0x4d543c0).accept sd=14 157.249.160.107:56748/0
Oct 29 14:29:45 95z3zz1 ceph-mon: -393> 2015-10-29 14:29:44.628435 7f314f447700 1 -- 157.249.160.11:6789/0 >> :/0 pipe(0x5eec000 sd=16 :6789 s=0 pgs=0 cs=0 l=0 c=0x4d547e0).accept sd=16 172.16.6.12:34085/0
Oct 29 14:29:45 95z3zz1 ceph-mon: -392> 2015-10-29 14:29:44.628696 7f314f84b700 0 -- 157.249.160.11:6789/0 >> 157.249.160.106:6789/0 pipe(0x5ee2000 sd=13 :6789 s=0 pgs=0 cs=0 l=0 c=0x4d54260).accept connect_seq 2 vs existing 0 state connecting
Oct 29 14:29:45 95z3zz1 ceph-mon: -391> 2015-10-29 14:29:44.628711 7f314f84b700 0 -- 157.249.160.11:6789/0 >> 157.249.160.106:6789/0 pipe(0x5ee2000 sd=13 :6789 s=0 pgs=0 cs=0 l=0 c=0x4d54260).accept we reset (peer sent cseq 2, 0x5ee7000.cseq = 0), sending RESETSESSION
Oct 29 14:29:45 95z3zz1 ceph-mon: -390> 2015-10-29 14:29:44.628723 7f314f649700 0 -- 157.249.160.11:6789/0 >> 157.249.160.107:6789/0 pipe(0x5ef1000 sd=14 :6789 s=0 pgs=0 cs=0 l=0 c=0x4d543c0).accept connect_seq 1 vs existing 0 state connecting
Oct 29 14:29:45 95z3zz1 ceph-mon: -389> 2015-10-29 14:29:44.628737 7f314f649700 0 -- 157.249.160.11:6789/0 >> 157.249.160.107:6789/0 pipe(0x5ef1000 sd=14 :6789 s=0 pgs=0 cs=0 l=0 c=0x4d543c0).accept we reset (peer sent cseq 1, 0x5efb000.cseq = 0), sending RESETSESSION
Oct 29 14:29:45 95z3zz1 ceph-mon: -388> 2015-10-29 14:29:44.629016 7f315104e700 1 -- 157.249.160.11:6789/0 <== client.4878761 172.16.6.12:0/2056508 1 ==== auth(proto 0 35 bytes epoch 7) v1 ==== 65+0+0 (3944302535 0 0) 0x4f8c900 con 0x4d547e0
Oct 29 14:29:45 95z3zz1 ceph-mon: -387> 2015-10-29 14:29:44.629055 7f315104e700 5 mon.95z3zz1@0(probing) e7 waitlisting message auth(proto 0 35 bytes epoch 7) v1
Oct 29 14:29:45 95z3zz1 ceph-mon: -386> 2015-10-29 14:29:44.629056 7f314f84b700 0 -- 157.249.160.11:6789/0 >> 157.249.160.106:6789/0 pipe(0x5ee2000 sd=13 :6789 s=0 pgs=0 cs=0 l=0 c=0x4d54260).accept connect_seq 0 vs existing 0 state connecting
Oct 29 14:29:45 95z3zz1 ceph-mon: -385> 2015-10-29 14:29:44.629076 7f314f649700 0 -- 157.249.160.11:6789/0 >> 157.249.160.107:6789/0 pipe(0x5ef1000 sd=14 :6789 s=0 pgs=0 cs=0 l=0 c=0x4d543c0).accept connect_seq 0 vs existing 0 state connecting
Oct 29 14:29:45 95z3zz1 ceph-mon: -384> 2015-10-29 14:29:44.629187 7f314f649700 2 -- 157.249.160.11:6789/0 >> 157.249.160.107:6789/0 pipe(0x5ef1000 sd=14 :6789 s=4 pgs=0 cs=0 l=0 c=0x4d543c0).fault (0) Success
Oct 29 14:29:45 95z3zz1 ceph-mon: -383> 2015-10-29 14:29:44.629247 7f314f84b700 2 -- 157.249.160.11:6789/0 >> 157.249.160.106:6789/0 pipe(0x5ee2000 sd=13 :6789 s=4 pgs=0 cs=0 l=0 c=0x4d54260).fault (0) Success
Oct 29 14:29:45 95z3zz1 ceph-mon: -382> 2015-10-29 14:29:44.629259 7f314f548700 2 -- 157.249.160.11:6789/0 >> 157.249.160.107:6789/0 pipe(0x5efb000 sd=17 :42956 s=1 pgs=0 cs=0 l=0 c=0x4d53b80). got newly_acked_seq 0 vs out_seq 0
Oct 29 14:29:45 95z3zz1 ceph-mon: -381> 2015-10-29 14:29:44.629322 7f314f74a700 2 -- 157.249.160.11:6789/0 >> 157.249.160.106:6789/0 pipe(0x5ee7000 sd=15 :57775 s=1 pgs=0 cs=0 l=0 c=0x4d53e40). got newly_acked_seq 0 vs out_seq 0
Oct 29 14:29:45 95z3zz1 ceph-mon: -380> 2015-10-29 14:29:44.629581 7f314f649700 0 -- 157.249.160.11:6789/0 >> 157.249.160.107:6789/0 pipe(0x5efb000 sd=17 :42956 s=2 pgs=3465 cs=1 l=0 c=0x4d53b80).reader missed message? skipped from seq 0 to 1506138999
Oct 29 14:29:45 95z3zz1 ceph-mon: -379> 2015-10-29 14:29:44.629607 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506138999 ==== mon_probe(reply 6a57f77c-a465-4fc6-b0a2-93c20371bab8 name 56z3zz1 quorum 0,1,2 paxos( fc 62203323 lc 62204013 )) v6 ==== 563+0+0 (1924762453 0 0) 0x4dbaa00 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -378> 2015-10-29 14:29:44.629635 7f315104e700 5 mon.95z3zz1@0(probing) e7 setting monitor caps on this connection
Oct 29 14:29:45 95z3zz1 ceph-mon: -377> 2015-10-29 14:29:44.629654 7f314f84b700 0 -- 157.249.160.11:6789/0 >> 157.249.160.106:6789/0 pipe(0x5ee7000 sd=15 :57775 s=2 pgs=2476 cs=1 l=0 c=0x4d53e40).reader missed message? skipped from seq 0 to 13999349
Oct 29 14:29:45 95z3zz1 ceph-mon: -376> 2015-10-29 14:29:44.629693 7f315104e700 0 log_channel(cluster) log [INF] : mon.95z3zz1 calling new monitor election
Oct 29 14:29:45 95z3zz1 ceph-mon: -375> 2015-10-29 14:29:44.629699 7f315104e700 0 log_channel(cluster) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -374> 2015-10-29 14:29:44.629725 7f315104e700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -373> 2015-10-29 14:29:44.629727 7f315104e700 10 log_client log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -372> 2015-10-29 14:29:44.629730 7f315104e700 10 log_client will send 2015-10-29 14:29:44.629699 mon.0 157.249.160.11:6789/0 1 : cluster [INF] mon.95z3zz1 calling new monitor election
Oct 29 14:29:45 95z3zz1 ceph-mon: -371> 2015-10-29 14:29:44.629745 7f315104e700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1 entries) v1 -- ?+0 0x4f8d8c0 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -370> 2015-10-29 14:29:44.629769 7f315104e700 5 mon.95z3zz1@0(electing).elector(0) start -- can i be leader?
Oct 29 14:29:45 95z3zz1 ceph-mon: -369> 2015-10-29 14:29:44.629922 7f315104e700 1 mon.95z3zz1@0(electing).elector(424) init, last seen epoch 424
Oct 29 14:29:45 95z3zz1 ceph-mon: -368> 2015-10-29 14:29:44.630487 7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 -- election(6a57f77c-a465-4fc6-b0a2-93c20371bab8 propose 425) v5 -- ?+0 0x4e11e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -367> 2015-10-29 14:29:44.630526 7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 -- election(6a57f77c-a465-4fc6-b0a2-93c20371bab8 propose 425) v5 -- ?+0 0x4e123c0
Oct 29 14:29:45 95z3zz1 ceph-mon: -366> 2015-10-29 14:29:44.630543 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999349 ==== mon_probe(reply 6a57f77c-a465-4fc6-b0a2-93c20371bab8 name c6z3zz1 quorum 0,1,2 paxos( fc 62203323 lc 62204013 )) v6 ==== 563+0+0 (534286656 0 0) 0x4dbac80 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -365> 2015-10-29 14:29:44.630555 7f315104e700 5 mon.95z3zz1@0(electing) e7 setting monitor caps on this connection
Oct 29 14:29:45 95z3zz1 ceph-mon: -364> 2015-10-29 14:29:44.630564 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8d8c0 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -363> 2015-10-29 14:29:44.630572 7f315104e700 5 mon.95z3zz1@0(electing) e7 setting monitor caps on this connection
Oct 29 14:29:45 95z3zz1 ceph-mon: -362> 2015-10-29 14:29:44.630581 7f315104e700 5 mon.95z3zz1@0(electing).paxos(paxos recovering c 62203323..62204013) is_readable = 0 - now=2015-10-29 14:29:44.630582 lease_expire=0.000000 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -361> 2015-10-29 14:29:44.630590 7f315104e700 5 mon.95z3zz1@0(electing).paxos(paxos recovering c 62203323..62204013) is_readable = 0 - now=2015-10-29 14:29:44.630590 lease_expire=0.000000 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -360> 2015-10-29 14:29:44.631261 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999350 ==== election(6a57f77c-a465-4fc6-b0a2-93c20371bab8 ack 425) v5 ==== 26181+0+0 (4009097990 0 0) 0x4e11e40 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -359> 2015-10-29 14:29:44.631282 7f315104e700 5 mon.95z3zz1@0(electing).elector(425) handle_ack from mon.1
Oct 29 14:29:45 95z3zz1 ceph-mon: -358> 2015-10-29 14:29:44.631289 7f315104e700 5 mon.95z3zz1@0(electing).elector(425) so far i have {0=37154696925806591,1=37154696925806591}
Oct 29 14:29:45 95z3zz1 ceph-mon: -357> 2015-10-29 14:29:44.631295 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139000 ==== election(6a57f77c-a465-4fc6-b0a2-93c20371bab8 ack 425) v5 ==== 26181+0+0 (4009097990 0 0) 0x4e123c0 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -356> 2015-10-29 14:29:44.631307 7f315104e700 5 mon.95z3zz1@0(electing).elector(425) handle_ack from mon.2
Oct 29 14:29:45 95z3zz1 ceph-mon: -355> 2015-10-29 14:29:44.631310 7f315104e700 5 mon.95z3zz1@0(electing).elector(425) so far i have {0=37154696925806591,1=37154696925806591,2=37154696925806591}
Oct 29 14:29:45 95z3zz1 ceph-mon: -354> 2015-10-29 14:29:44.631514 7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 -- election(6a57f77c-a465-4fc6-b0a2-93c20371bab8 victory 426) v5 -- ?+0 0x4e11e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -353> 2015-10-29 14:29:44.631533 7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 -- election(6a57f77c-a465-4fc6-b0a2-93c20371bab8 victory 426) v5 -- ?+0 0x4e12100
Oct 29 14:29:45 95z3zz1 ceph-mon: -352> 2015-10-29 14:29:44.631549 7f315104e700 0 log_channel(cluster) log [INF] : mon.95z3zz1@0 won leader election with quorum 0,1,2
Oct 29 14:29:45 95z3zz1 ceph-mon: -351> 2015-10-29 14:29:44.631552 7f315104e700 0 log_channel(cluster) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -350> 2015-10-29 14:29:44.631574 7f315104e700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -349> 2015-10-29 14:29:44.631575 7f315104e700 10 log_client log_queue is 2 last_log 2 sent 1 num 2 unsent 1 sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -348> 2015-10-29 14:29:44.631577 7f315104e700 10 log_client will send 2015-10-29 14:29:44.631552 mon.0 157.249.160.11:6789/0 2 : cluster [INF] mon.95z3zz1@0 won leader election with quorum 0,1,2
Oct 29 14:29:45 95z3zz1 ceph-mon: -347> 2015-10-29 14:29:44.631582 7f315104e700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1 entries) v1 -- ?+0 0x4f8c000 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -346> 2015-10-29 14:29:44.631797 7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 -- paxos(collect lc 62204013 fc 62203323 pn 21100 opn 0) v3 -- ?+0 0x4dbac80
Oct 29 14:29:45 95z3zz1 ceph-mon: -345> 2015-10-29 14:29:44.631811 7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 -- paxos(collect lc 62204013 fc 62203323 pn 21100 opn 0) v3 -- ?+0 0x4dbaa00
Oct 29 14:29:45 95z3zz1 ceph-mon: -344> 2015-10-29 14:29:44.631836 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos recovering c 62203323..62204013) is_readable = 0 - now=2015-10-29 14:29:44.631837 lease_expire=0.000000 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -343> 2015-10-29 14:29:44.631843 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos recovering c 62203323..62204013) is_readable = 0 - now=2015-10-29 14:29:44.631844 lease_expire=0.000000 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -342> 2015-10-29 14:29:44.631864 7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 -- time_check( ping e 426 r 1 ) v1 -- ?+0 0x4f8cd80
Oct 29 14:29:45 95z3zz1 ceph-mon: -341> 2015-10-29 14:29:44.631875 7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 -- time_check( ping e 426 r 1 ) v1 -- ?+0 0x4f8db00
Oct 29 14:29:45 95z3zz1 ceph-mon: -340> 2015-10-29 14:29:44.634897 7f315104e700 0 log_channel(cluster) log [INF] : HEALTH_WARN; noout flag(s) set
Oct 29 14:29:45 95z3zz1 ceph-mon: -339> 2015-10-29 14:29:44.634903 7f315104e700 0 log_channel(cluster) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -338> 2015-10-29 14:29:44.634927 7f315104e700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -337> 2015-10-29 14:29:44.634929 7f315104e700 10 log_client log_queue is 3 last_log 3 sent 2 num 3 unsent 1 sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -336> 2015-10-29 14:29:44.634931 7f315104e700 10 log_client will send 2015-10-29 14:29:44.634903 mon.0 157.249.160.11:6789/0 3 : cluster [INF] HEALTH_WARN; noout flag(s) set
Oct 29 14:29:45 95z3zz1 ceph-mon: -335> 2015-10-29 14:29:44.634938 7f315104e700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1 entries) v1 -- ?+0 0x4f8e1c0 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -334> 2015-10-29 14:29:44.634955 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8c000 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -333> 2015-10-29 14:29:44.634962 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos recovering c 62203323..62204013) is_readable = 0 - now=2015-10-29 14:29:44.634962 lease_expire=0.000000 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -332> 2015-10-29 14:29:44.634966 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos recovering c 62203323..62204013) is_readable = 0 - now=2015-10-29 14:29:44.634967 lease_expire=0.000000 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -331> 2015-10-29 14:29:44.634970 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999374 ==== paxos(last lc 62204013 fc 62203323 pn 21100 opn 0) v3 ==== 80+0+0 (1023772294 0 0) 0x5f96d00 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -330> 2015-10-29 14:29:44.635016 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139018 ==== paxos(last lc 62204013 fc 62203323 pn 21100 opn 0) v3 ==== 80+0+0 (3238039421 0 0) 0x5f93480 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -329> 2015-10-29 14:29:44.635043 7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 -- paxos(lease lc 62204013 fc 62203323 pn 0 opn 0) v3 -- ?+0 0x5f96d00
Oct 29 14:29:45 95z3zz1 ceph-mon: -328> 2015-10-29 14:29:44.635057 7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 -- paxos(lease lc 62204013 fc 62203323 pn 0 opn 0) v3 -- ?+0 0x5f66280
Oct 29 14:29:45 95z3zz1 ceph-mon: -327> 2015-10-29 14:29:44.635372 7f315104e700 0 log_channel(cluster) log [INF] : monmap e7: 3 mons at {56z3zz1=157.249.160.107:6789/0,95z3zz1=157.249.160.11:6789/0,c6z3zz1=157.249.160.106:6789/0}
Oct 29 14:29:45 95z3zz1 ceph-mon: -326> 2015-10-29 14:29:44.635376 7f315104e700 0 log_channel(cluster) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -325> 2015-10-29 14:29:44.635402 7f315104e700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -324> 2015-10-29 14:29:44.635404 7f315104e700 10 log_client log_queue is 4 last_log 4 sent 3 num 4 unsent 1 sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -323> 2015-10-29 14:29:44.635405 7f315104e700 10 log_client will send 2015-10-29 14:29:44.635376 mon.0 157.249.160.11:6789/0 4 : cluster [INF] monmap e7: 3 mons at {56z3zz1=157.249.160.107:6789/0,95z3zz1=157.249.160.11:6789/0,c6z3zz1=157.249.160.106:6789/0}
Oct 29 14:29:45 95z3zz1 ceph-mon: -322> 2015-10-29 14:29:44.635420 7f315104e700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1 entries) v1 -- ?+0 0x4f8d680 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -321> 2015-10-29 14:29:44.635463 7f315104e700 0 log_channel(cluster) log [INF] : pgmap v33661653: 21936 pgs: 21936 active+clean; 9824 GB data, 29683 GB used, 144 TB / 173 TB avail
Oct 29 14:29:45 95z3zz1 ceph-mon: -320> 2015-10-29 14:29:44.635466 7f315104e700 0 log_channel(cluster) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -319> 2015-10-29 14:29:44.635478 7f315104e700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -318> 2015-10-29 14:29:44.635479 7f315104e700 10 log_client log_queue is 5 last_log 5 sent 4 num 5 unsent 1 sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -317> 2015-10-29 14:29:44.635481 7f315104e700 10 log_client will send 2015-10-29 14:29:44.635466 mon.0 157.249.160.11:6789/0 5 : cluster [INF] pgmap v33661653: 21936 pgs: 21936 active+clean; 9824 GB data, 29683 GB used, 144 TB / 173 TB avail
Oct 29 14:29:45 95z3zz1 ceph-mon: -316> 2015-10-29 14:29:44.635486 7f315104e700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1 entries) v1 -- ?+0 0x4f8df80 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -315> 2015-10-29 14:29:44.635531 7f315104e700 0 log_channel(cluster) log [INF] : mdsmap e89: 1/1/1 up {0=56z3zz1=up:active}
Oct 29 14:29:45 95z3zz1 ceph-mon: -314> 2015-10-29 14:29:44.635535 7f315104e700 0 log_channel(cluster) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -313> 2015-10-29 14:29:44.635546 7f315104e700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -312> 2015-10-29 14:29:44.635548 7f315104e700 10 log_client log_queue is 6 last_log 6 sent 5 num 6 unsent 1 sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -311> 2015-10-29 14:29:44.635550 7f315104e700 10 log_client will send 2015-10-29 14:29:44.635534 mon.0 157.249.160.11:6789/0 6 : cluster [INF] mdsmap e89: 1/1/1 up {0=56z3zz1=up:active}
Oct 29 14:29:45 95z3zz1 ceph-mon: -310> 2015-10-29 14:29:44.635555 7f315104e700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1 entries) v1 -- ?+0 0x4f8dd40 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -309> 2015-10-29 14:29:44.635656 7f315104e700 0 log_channel(cluster) log [INF] : osdmap e61385: 48 osds: 48 up, 48 in
Oct 29 14:29:45 95z3zz1 ceph-mon: -308> 2015-10-29 14:29:44.635659 7f315104e700 0 log_channel(cluster) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -307> 2015-10-29 14:29:44.635672 7f315104e700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -306> 2015-10-29 14:29:44.635673 7f315104e700 10 log_client log_queue is 7 last_log 7 sent 6 num 7 unsent 1 sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -305> 2015-10-29 14:29:44.635675 7f315104e700 10 log_client will send 2015-10-29 14:29:44.635659 mon.0 157.249.160.11:6789/0 7 : cluster [INF] osdmap e61385: 48 osds: 48 up, 48 in
Oct 29 14:29:45 95z3zz1 ceph-mon: -304> 2015-10-29 14:29:44.635680 7f315104e700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1 entries) v1 -- ?+0 0x4f8d440 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -303> 2015-10-29 14:29:44.635736 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.635737 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -302> 2015-10-29 14:29:44.635755 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.635756 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -301> 2015-10-29 14:29:44.635791 7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.12:0/2056508 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x4f8e400 con 0x4d547e0
Oct 29 14:29:45 95z3zz1 ceph-mon: -300> 2015-10-29 14:29:44.635813 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.635814 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -299> 2015-10-29 14:29:44.635832 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8e1c0 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -298> 2015-10-29 14:29:44.635841 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.635842 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -297> 2015-10-29 14:29:44.635854 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999375 ==== time_check( pong e 426 r 1 ts 2015-10-29 14:29:44.629005 ) v1 ==== 36+0+0 (1025219847 0 0) 0x4f8cd80 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -296> 2015-10-29 14:29:44.635874 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139019 ==== time_check( pong e 426 r 1 ts 2015-10-29 14:29:44.631212 ) v1 ==== 36+0+0 (973715343 0 0) 0x4f8db00 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -295> 2015-10-29 14:29:44.635890 7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 -- time_check( report e 426 r 2 #skews 3 #latencies 3 ) v1 -- ?+0 0x4f8cd80
Oct 29 14:29:45 95z3zz1 ceph-mon: -294> 2015-10-29 14:29:44.635909 7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 -- time_check( report e 426 r 2 #skews 3 #latencies 3 ) v1 -- ?+0 0x4f8c900
Oct 29 14:29:45 95z3zz1 ceph-mon: -293> 2015-10-29 14:29:44.635920 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8d680 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -292> 2015-10-29 14:29:44.635926 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.635927 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -291> 2015-10-29 14:29:44.635937 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999376 ==== paxos(lease_ack lc 62204013 fc 62203323 pn 0 opn 0) v3 ==== 80+0+0 (1628804849 0 0) 0x5f96d00 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -290> 2015-10-29 14:29:44.635948 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139020 ==== paxos(lease_ack lc 62204013 fc 62203323 pn 0 opn 0) v3 ==== 80+0+0 (751612340 0 0) 0x5f66280 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -289> 2015-10-29 14:29:44.635958 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8df80 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -288> 2015-10-29 14:29:44.635966 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.635967 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -287> 2015-10-29 14:29:44.635977 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8dd40 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -286> 2015-10-29 14:29:44.635983 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.635983 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -285> 2015-10-29 14:29:44.635994 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8d440 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -284> 2015-10-29 14:29:44.635999 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636000 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -283> 2015-10-29 14:29:44.636009 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139001 ==== forward(pg_stats(3 pgs tid 19047 v 0) v1 caps allow profile osd tid 554 con_features 37154696925806591) to leader v3 ==== 2041+0+0 (154590633 0 0) 0x4e12100 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -282> 2015-10-29 14:29:44.636029 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636029 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -281> 2015-10-29 14:29:44.636050 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636051 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -280> 2015-10-29 14:29:44.636078 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999351 ==== forward(pg_stats(6 pgs tid 939 v 0) v1 caps allow profile osd tid 454 con_features 37154696925806591) to leader v3 ==== 3552+0+0 (2093592133 0 0) 0x4e11e40 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -279> 2015-10-29 14:29:44.636089 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636090 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -278> 2015-10-29 14:29:44.636102 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636103 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -277> 2015-10-29 14:29:44.636119 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139002 ==== forward(pg_stats(3 pgs tid 19188 v 0) v1 caps allow profile osd tid 555 con_features 37154696925806591) to leader v3 ==== 2041+0+0 (4008145650 0 0) 0x4e12ec0 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -276> 2015-10-29 14:29:44.636130 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636131 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -275> 2015-10-29 14:29:44.636141 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636142 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -274> 2015-10-29 14:29:44.636154 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999352 ==== forward(pg_stats(5 pgs tid 940 v 0) v1 caps allow profile osd tid 455 con_features 37154696925806591) to leader v3 ==== 3051+0+0 (29326344 0 0) 0x4e13440 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -273> 2015-10-29 14:29:44.636163 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636163 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -272> 2015-10-29 14:29:44.636175 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636176 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -271> 2015-10-29 14:29:44.636196 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139003 ==== forward(pg_stats(1 pgs tid 19164 v 0) v1 caps allow profile osd tid 556 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (219839371 0 0) 0x4e13c80 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -270> 2015-10-29 14:29:44.636207 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636208 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -269> 2015-10-29 14:29:44.636219 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636219 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -268> 2015-10-29 14:29:44.636231 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999353 ==== forward(pg_stats(3 pgs tid 19130 v 0) v1 caps allow profile osd tid 456 con_features 37154696925806591) to leader v3 ==== 2049+0+0 (667527532 0 0) 0x5f4a580 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -267> 2015-10-29 14:29:44.636242 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636243 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -266> 2015-10-29 14:29:44.636254 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636255 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -265> 2015-10-29 14:29:44.636268 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139004 ==== forward(pg_stats(6 pgs tid 18939 v 0) v1 caps allow profile osd tid 557 con_features 37154696925806591) to leader v3 ==== 3544+0+0 (4151017389 0 0) 0x4e13700 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -264> 2015-10-29 14:29:44.636278 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636279 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -263> 2015-10-29 14:29:44.636291 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636292 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -262> 2015-10-29 14:29:44.636308 7f315104e700 1 -- 157.249.160.11:6789/0 <== client.4878761 172.16.6.12:0/2056508 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (909137154 0 0) 0x4f8f600 con 0x4d547e0
Oct 29 14:29:45 95z3zz1 ceph-mon: -261> 2015-10-29 14:29:44.636316 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636316 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -260> 2015-10-29 14:29:44.636439 7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.12:0/2056508 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x4f8db00 con 0x4d547e0
Oct 29 14:29:45 95z3zz1 ceph-mon: -259> 2015-10-29 14:29:44.636457 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999354 ==== forward(pg_stats(2 pgs tid 19158 v 0) v1 caps allow profile osd tid 457 con_features 37154696925806591) to leader v3 ==== 1540+0+0 (1302266018 0 0) 0x5f4a000 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -258> 2015-10-29 14:29:44.636470 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636470 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -257> 2015-10-29 14:29:44.636484 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636484 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -256> 2015-10-29 14:29:44.636507 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139005 ==== forward(pg_stats(7 pgs tid 18875 v 0) v1 caps allow profile osd tid 558 con_features 37154696925806591) to leader v3 ==== 4045+0+0 (3611257738 0 0) 0x5f4adc0 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -255> 2015-10-29 14:29:44.636534 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636535 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -254> 2015-10-29 14:29:44.636546 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636546 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -253> 2015-10-29 14:29:44.636562 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999355 ==== forward(pg_stats(2 pgs tid 943 v 0) v1 caps allow profile osd tid 458 con_features 37154696925806591) to leader v3 ==== 1544+0+0 (1619202949 0 0) 0x5f4b8c0 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -252> 2015-10-29 14:29:44.636580 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636581 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -251> 2015-10-29 14:29:44.636593 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636594 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -250> 2015-10-29 14:29:44.636605 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139006 ==== forward(pg_stats(5 pgs tid 19001 v 0) v1 caps allow profile osd tid 559 con_features 37154696925806591) to leader v3 ==== 3043+0+0 (744702348 0 0) 0x5f4a840 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -249> 2015-10-29 14:29:44.636615 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636616 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -248> 2015-10-29 14:29:44.636626 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636626 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -247> 2015-10-29 14:29:44.636642 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999356 ==== forward(pg_stats(1 pgs tid 19101 v 0) v1 caps allow profile osd tid 459 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (1036056958 0 0) 0x5f4b340 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -246> 2015-10-29 14:29:44.636652 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636652 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -245> 2015-10-29 14:29:44.636663 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636664 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -244> 2015-10-29 14:29:44.636674 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139007 ==== forward(pg_stats(1 pgs tid 18881 v 0) v1 caps allow profile osd tid 560 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (3429570332 0 0) 0x5f4c680 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -243> 2015-10-29 14:29:44.636693 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636693 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -242> 2015-10-29 14:29:44.636705 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636705 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -241> 2015-10-29 14:29:44.636717 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999357 ==== forward(pg_stats(5 pgs tid 943 v 0) v1 caps allow profile osd tid 460 con_features 37154696925806591) to leader v3 ==== 3051+0+0 (1567813846 0 0) 0x5f4d440 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -240> 2015-10-29 14:29:44.636726 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636726 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -239> 2015-10-29 14:29:44.636738 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636738 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -238> 2015-10-29 14:29:44.636753 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139008 ==== forward(pg_stats(11 pgs tid 19000 v 0) v1 caps allow profile osd tid 561 con_features 37154696925806591) to leader v3 ==== 6049+0+0 (2318291987 0 0) 0x5f4c100 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -237> 2015-10-29 14:29:44.636763 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636763 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -236> 2015-10-29 14:29:44.636774 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636774 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -235> 2015-10-29 14:29:44.636801 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999358 ==== forward(pg_stats(2 pgs tid 19093 v 0) v1 caps allow profile osd tid 461 con_features 37154696925806591) to leader v3 ==== 1540+0+0 (3377069825 0 0) 0x5f4cec0 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -234> 2015-10-29 14:29:44.636812 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636813 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -233> 2015-10-29 14:29:44.636885 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636886 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -232> 2015-10-29 14:29:44.636901 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139009 ==== forward(pg_stats(1 pgs tid 19226 v 0) v1 caps allow profile osd tid 562 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (1339636414 0 0) 0x5f4dc80 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -231> 2015-10-29 14:29:44.636912 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636912 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -230> 2015-10-29 14:29:44.636924 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636924 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -229> 2015-10-29 14:29:44.636935 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999359 ==== forward(pg_stats(2 pgs tid 19095 v 0) v1 caps allow profile osd tid 462 con_features 37154696925806591) to leader v3 ==== 1540+0+0 (2108816789 0 0) 0x5f7cdc0 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -228> 2015-10-29 14:29:44.636944 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636945 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -227> 2015-10-29 14:29:44.636955 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636956 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -226> 2015-10-29 14:29:44.636968 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139010 ==== forward(pg_stats(9 pgs tid 19023 v 0) v1 caps allow profile osd tid 563 con_features 37154696925806591) to leader v3 ==== 5047+0+0 (4205152908 0 0) 0x5f7e100 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -225> 2015-10-29 14:29:44.636978 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636979 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -224> 2015-10-29 14:29:44.636989 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.636990 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -223> 2015-10-29 14:29:44.637010 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999360 ==== forward(pg_stats(8 pgs tid 19052 v 0) v1 caps allow profile osd tid 463 con_features 37154696925806591) to leader v3 ==== 4546+0+0 (1173158927 0 0) 0x5f7c840 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -222> 2015-10-29 14:29:44.637020 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637021 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -221> 2015-10-29 14:29:44.637031 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637031 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -220> 2015-10-29 14:29:44.637056 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139011 ==== forward(pg_stats(2 pgs tid 19096 v 0) v1 caps allow profile osd tid 564 con_features 37154696925806591) to leader v3 ==== 1540+0+0 (3933631993 0 0) 0x5f7db80 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -219> 2015-10-29 14:29:44.637067 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637067 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -218> 2015-10-29 14:29:44.637079 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637079 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -217> 2015-10-29 14:29:44.637090 7f315104e700 1 -- 157.249.160.11:6789/0 <== client.4878761 172.16.6.12:0/2056508 3 ==== mon_subscribe({monmap=8+,osdmap=61386}) v2 ==== 42+0+0 (2169229493 0 0) 0x6028000 con 0x4d547e0
Oct 29 14:29:45 95z3zz1 ceph-mon: -216> 2015-10-29 14:29:44.637110 7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.12:0/2056508 -- mon_subscribe_ack(300s) v1 -- ?+0 0x4f303c0 con 0x4d547e0
Oct 29 14:29:45 95z3zz1 ceph-mon: -215> 2015-10-29 14:29:44.637123 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999361 ==== forward(pg_stats(6 pgs tid 939 v 0) v1 caps allow profile osd tid 464 con_features 37154696925806591) to leader v3 ==== 3552+0+0 (1387593706 0 0) 0x5f7c2c0 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -214> 2015-10-29 14:29:44.637132 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637133 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -213> 2015-10-29 14:29:44.637144 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637145 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -212> 2015-10-29 14:29:44.637160 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139012 ==== forward(pg_stats(1 pgs tid 19135 v 0) v1 caps allow profile osd tid 565 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (534192359 0 0) 0x5f7d600 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -211> 2015-10-29 14:29:44.637170 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637170 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -210> 2015-10-29 14:29:44.637181 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637182 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -209> 2015-10-29 14:29:44.637192 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999362 ==== forward(pg_stats(1 pgs tid 18922 v 0) v1 caps allow profile osd tid 465 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (779363514 0 0) 0x5f7fc80 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -208> 2015-10-29 14:29:44.637202 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637202 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -207> 2015-10-29 14:29:44.637213 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637214 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -206> 2015-10-29 14:29:44.637224 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139013 ==== forward(pg_stats(3 pgs tid 18948 v 0) v1 caps allow profile osd tid 566 con_features 37154696925806591) to leader v3 ==== 2041+0+0 (2346298246 0 0) 0x5f7d080 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -205> 2015-10-29 14:29:44.637234 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637234 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -204> 2015-10-29 14:29:44.637245 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637245 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -203> 2015-10-29 14:29:44.637257 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999363 ==== forward(pg_stats(5 pgs tid 18985 v 0) v1 caps allow profile osd tid 466 con_features 37154696925806591) to leader v3 ==== 3051+0+0 (399880181 0 0) 0x5f89340 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -202> 2015-10-29 14:29:44.637266 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637266 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -201> 2015-10-29 14:29:44.637277 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637277 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -200> 2015-10-29 14:29:44.637293 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139014 ==== forward(pg_stats(1 pgs tid 18994 v 0) v1 caps allow profile osd tid 567 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (1768182538 0 0) 0x5f7f440 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -199> 2015-10-29 14:29:44.637302 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637303 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -198> 2015-10-29 14:29:44.637314 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637314 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -197> 2015-10-29 14:29:44.637324 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999364 ==== forward(pg_stats(2 pgs tid 942 v 0) v1 caps allow profile osd tid 467 con_features 37154696925806591) to leader v3 ==== 1544+0+0 (1147565120 0 0) 0x5f88dc0 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -196> 2015-10-29 14:29:44.637333 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637334 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -195> 2015-10-29 14:29:44.637344 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637345 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -194> 2015-10-29 14:29:44.637365 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139015 ==== forward(pg_stats(1 pgs tid 18954 v 0) v1 caps allow profile osd tid 568 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (1581327975 0 0) 0x5f7eec0 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -193> 2015-10-29 14:29:44.637376 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637376 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -192> 2015-10-29 14:29:44.637388 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637388 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -191> 2015-10-29 14:29:44.637398 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999365 ==== forward(pg_stats(9 pgs tid 18943 v 0) v1 caps allow profile osd tid 468 con_features 37154696925806591) to leader v3 ==== 5051+0+0 (64146959 0 0) 0x5f88840 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -190> 2015-10-29 14:29:44.637407 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637408 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -189> 2015-10-29 14:29:44.637427 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637428 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -188> 2015-10-29 14:29:44.637445 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139016 ==== forward(pg_stats(1 pgs tid 19179 v 0) v1 caps allow profile osd tid 569 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (4190925099 0 0) 0x5f7e940 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -187> 2015-10-29 14:29:44.637455 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637456 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -186> 2015-10-29 14:29:44.637467 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637467 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -185> 2015-10-29 14:29:44.637477 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999366 ==== forward(pg_stats(3 pgs tid 19165 v 0) v1 caps allow profile osd tid 469 con_features 37154696925806591) to leader v3 ==== 2069+0+0 (2844274865 0 0) 0x5f882c0 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -184> 2015-10-29 14:29:44.637495 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637496 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -183> 2015-10-29 14:29:44.637507 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637507 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -182> 2015-10-29 14:29:44.637519 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139017 ==== forward(pg_stats(4 pgs tid 18988 v 0) v1 caps allow profile osd tid 570 con_features 37154696925806591) to leader v3 ==== 2546+0+0 (1830822204 0 0) 0x5f7e3c0 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -181> 2015-10-29 14:29:44.637529 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637529 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -180> 2015-10-29 14:29:44.637541 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637541 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -179> 2015-10-29 14:29:44.637554 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999367 ==== forward(pg_stats(6 pgs tid 19221 v 0) v1 caps allow profile osd tid 470 con_features 37154696925806591) to leader v3 ==== 3544+0+0 (2898347798 0 0) 0x5f8ac00 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -178> 2015-10-29 14:29:44.637563 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637573 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -177> 2015-10-29 14:29:44.637585 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637585 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -176> 2015-10-29 14:29:44.637618 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999368 ==== forward(pg_stats(11 pgs tid 945 v 0) v1 caps allow profile osd tid 471 con_features 37154696925806591) to leader v3 ==== 6053+0+0 (2902556831 0 0) 0x5f8a680 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -175> 2015-10-29 14:29:44.637628 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637629 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -174> 2015-10-29 14:29:44.637640 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637640 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -173> 2015-10-29 14:29:44.637658 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999369 ==== forward(pg_stats(2 pgs tid 19072 v 0) v1 caps allow profile osd tid 472 con_features 37154696925806591) to leader v3 ==== 1548+0+0 (3189990860 0 0) 0x5f8a100 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -172> 2015-10-29 14:29:44.637667 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637668 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -171> 2015-10-29 14:29:44.637678 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637679 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -170> 2015-10-29 14:29:44.637690 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999370 ==== forward(pg_stats(2 pgs tid 943 v 0) v1 caps allow profile osd tid 473 con_features 37154696925806591) to leader v3 ==== 1544+0+0 (2273230390 0 0) 0x5f89b80 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -169> 2015-10-29 14:29:44.637699 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637700 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -168> 2015-10-29 14:29:44.637710 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637711 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -167> 2015-10-29 14:29:44.637721 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999371 ==== forward(pg_stats(5 pgs tid 945 v 0) v1 caps allow profile osd tid 474 con_features 37154696925806591) to leader v3 ==== 3051+0+0 (125037623 0 0) 0x5f89600 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -166> 2015-10-29 14:29:44.637739 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637740 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -165> 2015-10-29 14:29:44.637751 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637752 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -164> 2015-10-29 14:29:44.637765 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999372 ==== forward(pg_stats(7 pgs tid 19160 v 0) v1 caps allow profile osd tid 475 con_features 37154696925806591) to leader v3 ==== 4045+0+0 (1236860648 0 0) 0x5f9d600 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -163> 2015-10-29 14:29:44.637774 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637775 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -162> 2015-10-29 14:29:44.637786 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637786 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -161> 2015-10-29 14:29:44.637811 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999373 ==== forward(pg_stats(2 pgs tid 941 v 0) v1 caps allow profile osd tid 476 con_features 37154696925806591) to leader v3 ==== 1548+0+0 (4082189338 0 0) 0x5f9d080 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -160> 2015-10-29 14:29:44.637821 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637822 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -159> 2015-10-29 14:29:44.637832 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.637833 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -158> 2015-10-29 14:29:44.678540 7f314ea44700 1 -- 157.249.160.11:6789/0 >> :/0 pipe(0x5ef1000 sd=13 :6789 s=0 pgs=0 cs=0 l=0 c=0x603a940).accept sd=13 172.16.6.5:42554/0
Oct 29 14:29:45 95z3zz1 ceph-mon: -157> 2015-10-29 14:29:44.679435 7f315104e700 1 -- 157.249.160.11:6789/0 <== client.4399776 172.16.6.5:0/2080099 1 ==== auth(proto 0 35 bytes epoch 7) v1 ==== 65+0+0 (1271301283 0 0) 0x4f8e400 con 0x603a940
Oct 29 14:29:45 95z3zz1 ceph-mon: -156> 2015-10-29 14:29:44.679459 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.679461 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -155> 2015-10-29 14:29:44.679495 7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.5:0/2080099 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x4f8f600 con 0x603a940
Oct 29 14:29:45 95z3zz1 ceph-mon: -154> 2015-10-29 14:29:44.680046 7f315104e700 1 -- 157.249.160.11:6789/0 <== client.4399776 172.16.6.5:0/2080099 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (3085145404 0 0) 0x4f8f840 con 0x603a940
Oct 29 14:29:45 95z3zz1 ceph-mon: -153> 2015-10-29 14:29:44.680064 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.680064 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -152> 2015-10-29 14:29:44.680210 7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.5:0/2080099 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x4f8e400 con 0x603a940
Oct 29 14:29:45 95z3zz1 ceph-mon: -151> 2015-10-29 14:29:44.680803 7f315104e700 1 -- 157.249.160.11:6789/0 <== client.4399776 172.16.6.5:0/2080099 3 ==== mon_subscribe({monmap=8+,osdmap=61386}) v2 ==== 42+0+0 (2169229493 0 0) 0x602b200 con 0x603a940
Oct 29 14:29:45 95z3zz1 ceph-mon: -150> 2015-10-29 14:29:44.680827 7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.5:0/2080099 -- mon_subscribe_ack(300s) v1 -- ?+0 0x4f30000 con 0x603a940
Oct 29 14:29:45 95z3zz1 ceph-mon: -149> 2015-10-29 14:29:44.684170 7f314e842700 1 -- 157.249.160.11:6789/0 >> :/0 pipe(0x5ee2000 sd=14 :6789 s=0 pgs=0 cs=0 l=0 c=0x603a680).accept sd=14 172.16.6.4:51408/0
Oct 29 14:29:45 95z3zz1 ceph-mon: -148> 2015-10-29 14:29:44.684773 7f315104e700 1 -- 157.249.160.11:6789/0 <== client.1638129 172.16.6.4:0/1090975 1 ==== auth(proto 0 35 bytes epoch 7) v1 ==== 65+0+0 (2584606190 0 0) 0x4f8f600 con 0x603a680
Oct 29 14:29:45 95z3zz1 ceph-mon: -147> 2015-10-29 14:29:44.684790 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.684791 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -146> 2015-10-29 14:29:44.684825 7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.4:0/1090975 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x4f8f840 con 0x603a680
Oct 29 14:29:45 95z3zz1 ceph-mon: -145> 2015-10-29 14:29:44.685241 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999377 ==== forward(log(1 entries) v1 caps allow * tid 477 con_features 0) to leader v3 ==== 584+0+0 (1309431426 0 0) 0x5f9cb00 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -144> 2015-10-29 14:29:44.685263 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.685264 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -143> 2015-10-29 14:29:44.685325 7f315104e700 1 -- 157.249.160.11:6789/0 <== client.1638129 172.16.6.4:0/1090975 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (3893551484 0 0) 0x4f8fa80 con 0x603a680
Oct 29 14:29:45 95z3zz1 ceph-mon: -142> 2015-10-29 14:29:44.685336 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) is_readable = 1 - now=2015-10-29 14:29:44.685337 lease_expire=2015-10-29 14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -141> 2015-10-29 14:29:44.685459 7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.4:0/1090975 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x4f8f600 con 0x603a680
Oct 29 14:29:45 95z3zz1 ceph-mon: -140> 2015-10-29 14:29:44.685938 7f315184f700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013) queue_pending_finisher 0x5f0aaf0
Oct 29 14:29:45 95z3zz1 ceph-mon: -139> 2015-10-29 14:29:44.686171 7f315104e700 1 -- 157.249.160.11:6789/0 <== client.1638129 172.16.6.4:0/1090975 3 ==== mon_subscribe({monmap=8+,osdmap=61386}) v2 ==== 42+0+0 (2169229493 0 0) 0x602b400 con 0x603a680
Oct 29 14:29:45 95z3zz1 ceph-mon: -138> 2015-10-29 14:29:44.686376 7f315184f700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 -- paxos(begin lc 62204013 fc 0 pn 21100 opn 0) v3 -- ?+0 0x604f200
Oct 29 14:29:45 95z3zz1 ceph-mon: -137> 2015-10-29 14:29:44.686400 7f315184f700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 -- paxos(begin lc 62204013 fc 0 pn 21100 opn 0) v3 -- ?+0 0x604f700
Oct 29 14:29:45 95z3zz1 ceph-mon: -136> 2015-10-29 14:29:44.686434 7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.4:0/1090975 -- mon_subscribe_ack(300s) v1 -- ?+0 0x4f305a0 con 0x603a680
Oct 29 14:29:45 95z3zz1 ceph-mon: -135> 2015-10-29 14:29:44.687223 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999378 ==== paxos(accept lc 62204013 fc 0 pn 21100 opn 0) v3 ==== 80+0+0 (1988427842 0 0) 0x604f200 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -134> 2015-10-29 14:29:44.687599 7f315184f700 5 mon.95z3zz1@0(leader).paxos(paxos updating c 62203323..62204013) queue_pending_finisher 0x5f080b0
Oct 29 14:29:45 95z3zz1 ceph-mon: -133> 2015-10-29 14:29:44.687662 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139021 ==== paxos(accept lc 62204013 fc 0 pn 21100 opn 0) v3 ==== 80+0+0 (1732951130 0 0) 0x604f700 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -132> 2015-10-29 14:29:44.688677 7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 -- paxos(commit lc 62204014 fc 0 pn 21100 opn 0) v3 -- ?+0 0x604f980
Oct 29 14:29:45 95z3zz1 ceph-mon: -131> 2015-10-29 14:29:44.688697 7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 -- paxos(commit lc 62204014 fc 0 pn 21100 opn 0) v3 -- ?+0 0x604fc00
Oct 29 14:29:45 95z3zz1 ceph-mon: -130> 2015-10-29 14:29:44.689250 7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 -- paxos(lease lc 62204014 fc 62203323 pn 0 opn 0) v3 -- ?+0 0x60ea500
Oct 29 14:29:45 95z3zz1 ceph-mon: -129> 2015-10-29 14:29:44.689262 7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 -- paxos(lease lc 62204014 fc 62203323 pn 0 opn 0) v3 -- ?+0 0x60ea280
Oct 29 14:29:45 95z3zz1 ceph-mon: -128> 2015-10-29 14:29:44.689297 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(last 1) v1 -- ?+0 0x4f30780 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -127> 2015-10-29 14:29:44.689320 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(last 2) v1 -- ?+0 0x4f30b40 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -126> 2015-10-29 14:29:44.689326 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(last 3) v1 -- ?+0 0x4f30960 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -125> 2015-10-29 14:29:44.689335 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(last 4) v1 -- ?+0 0x4f310e0 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -124> 2015-10-29 14:29:44.689343 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(last 5) v1 -- ?+0 0x4f30f00 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -123> 2015-10-29 14:29:44.689336 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(last 1) v1 ==== 0+0+0 (0 0 0) 0x4f30780 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -122> 2015-10-29 14:29:44.689357 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(last 6) v1 -- ?+0 0x4f30d20 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -121> 2015-10-29 14:29:44.689366 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(last 7) v1 -- ?+0 0x4f31860 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -120> 2015-10-29 14:29:44.689374 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(log(last 454) v1 tid 477) v2 -- ?+0 0x60ea000 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -119> 2015-10-29 14:29:44.690195 7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 -- paxos(begin lc 62204014 fc 0 pn 21100 opn 0) v3 -- ?+0 0x5f96a80
Oct 29 14:29:45 95z3zz1 ceph-mon: -118> 2015-10-29 14:29:44.690229 7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 -- paxos(begin lc 62204014 fc 0 pn 21100 opn 0) v3 -- ?+0 0x60eaf00
Oct 29 14:29:45 95z3zz1 ceph-mon: -117> 2015-10-29 14:29:44.690371 7f315104e700 10 log_client handle_log_ack log(last 1) v1
Oct 29 14:29:45 95z3zz1 ceph-mon: -116> 2015-10-29 14:29:44.690380 7f315104e700 10 log_client logged 2015-10-29 14:29:44.629699 mon.0 157.249.160.11:6789/0 1 : cluster [INF] mon.95z3zz1 calling new monitor election
Oct 29 14:29:45 95z3zz1 ceph-mon: -115> 2015-10-29 14:29:44.690395 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(last 2) v1 ==== 0+0+0 (0 0 0) 0x4f30b40 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -114> 2015-10-29 14:29:44.690402 7f315104e700 10 log_client handle_log_ack log(last 2) v1
Oct 29 14:29:45 95z3zz1 ceph-mon: -113> 2015-10-29 14:29:44.690403 7f315104e700 10 log_client logged 2015-10-29 14:29:44.631552 mon.0 157.249.160.11:6789/0 2 : cluster [INF] mon.95z3zz1@0 won leader election with quorum 0,1,2
Oct 29 14:29:45 95z3zz1 ceph-mon: -112> 2015-10-29 14:29:44.690410 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999379 ==== paxos(lease_ack lc 62204014 fc 62203323 pn 0 opn 0) v3 ==== 80+0+0 (257771121 0 0) 0x60ea000 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -111> 2015-10-29 14:29:44.690425 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139022 ==== paxos(lease_ack lc 62204014 fc 62203323 pn 0 opn 0) v3 ==== 80+0+0 (3143862888 0 0) 0x60ea280 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -110> 2015-10-29 14:29:44.690434 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(last 3) v1 ==== 0+0+0 (0 0 0) 0x4f30960 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -109> 2015-10-29 14:29:44.690439 7f315104e700 10 log_client handle_log_ack log(last 3) v1
Oct 29 14:29:45 95z3zz1 ceph-mon: -108> 2015-10-29 14:29:44.690440 7f315104e700 10 log_client logged 2015-10-29 14:29:44.634903 mon.0 157.249.160.11:6789/0 3 : cluster [INF] HEALTH_WARN; noout flag(s) set
Oct 29 14:29:45 95z3zz1 ceph-mon: -107> 2015-10-29 14:29:44.690448 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(last 4) v1 ==== 0+0+0 (0 0 0) 0x4f310e0 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -106> 2015-10-29 14:29:44.690462 7f315104e700 10 log_client handle_log_ack log(last 4) v1
Oct 29 14:29:45 95z3zz1 ceph-mon: -105> 2015-10-29 14:29:44.690463 7f315104e700 10 log_client logged 2015-10-29 14:29:44.635376 mon.0 157.249.160.11:6789/0 4 : cluster [INF] monmap e7: 3 mons at {56z3zz1=157.249.160.107:6789/0,95z3zz1=157.249.160.11:6789/0,c6z3zz1=157.249.160.106:6789/0}
Oct 29 14:29:45 95z3zz1 ceph-mon: -104> 2015-10-29 14:29:44.690470 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(last 5) v1 ==== 0+0+0 (0 0 0) 0x4f30f00 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -103> 2015-10-29 14:29:44.690475 7f315104e700 10 log_client handle_log_ack log(last 5) v1
Oct 29 14:29:45 95z3zz1 ceph-mon: -102> 2015-10-29 14:29:44.690489 7f315104e700 10 log_client logged 2015-10-29 14:29:44.635466 mon.0 157.249.160.11:6789/0 5 : cluster [INF] pgmap v33661653: 21936 pgs: 21936 active+clean; 9824 GB data, 29683 GB used, 144 TB / 173 TB avail
Oct 29 14:29:45 95z3zz1 ceph-mon: -101> 2015-10-29 14:29:44.690496 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(last 6) v1 ==== 0+0+0 (0 0 0) 0x4f30d20 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -100> 2015-10-29 14:29:44.690502 7f315104e700 10 log_client handle_log_ack log(last 6) v1
Oct 29 14:29:45 95z3zz1 ceph-mon: -99> 2015-10-29 14:29:44.690503 7f315104e700 10 log_client logged 2015-10-29 14:29:44.635534 mon.0 157.249.160.11:6789/0 6 : cluster [INF] mdsmap e89: 1/1/1 up {0=56z3zz1=up:active}
Oct 29 14:29:45 95z3zz1 ceph-mon: -98> 2015-10-29 14:29:44.690509 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(last 7) v1 ==== 0+0+0 (0 0 0) 0x4f31860 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -97> 2015-10-29 14:29:44.690515 7f315104e700 10 log_client handle_log_ack log(last 7) v1
Oct 29 14:29:45 95z3zz1 ceph-mon: -96> 2015-10-29 14:29:44.690516 7f315104e700 10 log_client logged 2015-10-29 14:29:44.635659 mon.0 157.249.160.11:6789/0 7 : cluster [INF] osdmap e61385: 48 osds: 48 up, 48 in
Oct 29 14:29:45 95z3zz1 ceph-mon: -95> 2015-10-29 14:29:44.692160 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999380 ==== paxos(accept lc 62204014 fc 0 pn 21100 opn 0) v3 ==== 80+0+0 (2150592006 0 0) 0x5f96a80 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -94> 2015-10-29 14:29:44.692215 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139023 ==== paxos(accept lc 62204014 fc 0 pn 21100 opn 0) v3 ==== 80+0+0 (4160712019 0 0) 0x60eaf00 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -93> 2015-10-29 14:29:44.693301 7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 -- paxos(commit lc 62204015 fc 0 pn 21100 opn 0) v3 -- ?+0 0x60eaa00
Oct 29 14:29:45 95z3zz1 ceph-mon: -92> 2015-10-29 14:29:44.693340 7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 -- paxos(commit lc 62204015 fc 0 pn 21100 opn 0) v3 -- ?+0 0x60ea780
Oct 29 14:29:45 95z3zz1 ceph-mon: -91> 2015-10-29 14:29:44.696728 7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 -- paxos(lease lc 62204015 fc 62203323 pn 0 opn 0) v3 -- ?+0 0x60eb900
Oct 29 14:29:45 95z3zz1 ceph-mon: -90> 2015-10-29 14:29:44.696756 7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 -- paxos(lease lc 62204015 fc 62203323 pn 0 opn 0) v3 -- ?+0 0x60eb680
Oct 29 14:29:45 95z3zz1 ceph-mon: -89> 2015-10-29 14:29:44.696813 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(3 pgs tid 19047) v1 tid 554) v2 -- ?+0 0x5f67b80 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -88> 2015-10-29 14:29:44.696850 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(6 pgs tid 939) v1 tid 454) v2 -- ?+0 0x4dbaa00 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -87> 2015-10-29 14:29:44.696866 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(3 pgs tid 19188) v1 tid 555) v2 -- ?+0 0x4dbac80 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -86> 2015-10-29 14:29:44.696879 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(5 pgs tid 940) v1 tid 455) v2 -- ?+0 0x5f68800 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -85> 2015-10-29 14:29:44.696911 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(1 pgs tid 19164) v1 tid 556) v2 -- ?+0 0x4dbcf80 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -84> 2015-10-29 14:29:44.696929 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(3 pgs tid 19130) v1 tid 456) v2 -- ?+0 0x4dbd980 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -83> 2015-10-29 14:29:44.696942 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(6 pgs tid 18939) v1 tid 557) v2 -- ?+0 0x5f74c80 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -82> 2015-10-29 14:29:44.696956 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(2 pgs tid 19158) v1 tid 457) v2 -- ?+0 0x4dbd700 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -81> 2015-10-29 14:29:44.696967 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(7 pgs tid 18875) v1 tid 558) v2 -- ?+0 0x5f74500 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -80> 2015-10-29 14:29:44.696980 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(2 pgs tid 943) v1 tid 458) v2 -- ?+0 0x5f70280 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -79> 2015-10-29 14:29:44.697009 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(5 pgs tid 19001) v1 tid 559) v2 -- ?+0 0x5f74000 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -78> 2015-10-29 14:29:44.697024 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(1 pgs tid 19101) v1 tid 459) v2 -- ?+0 0x5f74f00 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -77> 2015-10-29 14:29:44.697035 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(1 pgs tid 18881) v1 tid 560) v2 -- ?+0 0x5f76a80 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -76> 2015-10-29 14:29:44.697059 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(5 pgs tid 943) v1 tid 460) v2 -- ?+0 0x5f77480 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -75> 2015-10-29 14:29:44.697073 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(11 pgs tid 19000) v1 tid 561) v2 -- ?+0 0x5f76800 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -74> 2015-10-29 14:29:44.697089 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(2 pgs tid 19093) v1 tid 461) v2 -- ?+0 0x5f77200 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -73> 2015-10-29 14:29:44.697102 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(1 pgs tid 19226) v1 tid 562) v2 -- ?+0 0x5f73c00 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -72> 2015-10-29 14:29:44.697123 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(2 pgs tid 19095) v1 tid 462) v2 -- ?+0 0x5f73480 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -71> 2015-10-29 14:29:44.697136 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(9 pgs tid 19023) v1 tid 563) v2 -- ?+0 0x5f81400 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -70> 2015-10-29 14:29:44.697149 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(8 pgs tid 19052) v1 tid 463) v2 -- ?+0 0x5f70a00 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -69> 2015-10-29 14:29:44.697161 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(2 pgs tid 19096) v1 tid 564) v2 -- ?+0 0x5f80f00 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -68> 2015-10-29 14:29:44.697183 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(6 pgs tid 939) v1 tid 464) v2 -- ?+0 0x5f82800 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -67> 2015-10-29 14:29:44.697197 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(1 pgs tid 19135) v1 tid 565) v2 -- ?+0 0x5f83980 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -66> 2015-10-29 14:29:44.697216 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(1 pgs tid 18922) v1 tid 465) v2 -- ?+0 0x5f82300 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -65> 2015-10-29 14:29:44.697228 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(3 pgs tid 18948) v1 tid 566) v2 -- ?+0 0x5f85680 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -64> 2015-10-29 14:29:44.697250 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(5 pgs tid 18985) v1 tid 466) v2 -- ?+0 0x5f82080 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -63> 2015-10-29 14:29:44.697265 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(1 pgs tid 18994) v1 tid 567) v2 -- ?+0 0x5f85400 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -62> 2015-10-29 14:29:44.697279 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(2 pgs tid 942) v1 tid 467) v2 -- ?+0 0x5f81900 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -61> 2015-10-29 14:29:44.697303 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(1 pgs tid 18954) v1 tid 568) v2 -- ?+0 0x5f84780 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -60> 2015-10-29 14:29:44.697319 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(9 pgs tid 18943) v1 tid 468) v2 -- ?+0 0x5f81680 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -59> 2015-10-29 14:29:44.697337 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(1 pgs tid 19179) v1 tid 569) v2 -- ?+0 0x5f81e00 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -58> 2015-10-29 14:29:44.697348 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(3 pgs tid 19165) v1 tid 469) v2 -- ?+0 0x5f87c00 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -57> 2015-10-29 14:29:44.697359 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 -- route(pg_stats_ack(4 pgs tid 18988) v1 tid 570) v2 -- ?+0 0x5f81b80 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -56> 2015-10-29 14:29:44.697371 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(6 pgs tid 19221) v1 tid 470) v2 -- ?+0 0x5f87980 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -55> 2015-10-29 14:29:44.697391 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(11 pgs tid 945) v1 tid 471) v2 -- ?+0 0x5f85e00 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -54> 2015-10-29 14:29:44.697418 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(2 pgs tid 19072) v1 tid 472) v2 -- ?+0 0x5f91400 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -53> 2015-10-29 14:29:44.697470 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(2 pgs tid 943) v1 tid 473) v2 -- ?+0 0x5f95b80 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -52> 2015-10-29 14:29:44.697511 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(5 pgs tid 945) v1 tid 474) v2 -- ?+0 0x5f95680 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -51> 2015-10-29 14:29:44.697565 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(7 pgs tid 19160) v1 tid 475) v2 -- ?+0 0x5f95180 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -50> 2015-10-29 14:29:44.697606 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 -- route(pg_stats_ack(2 pgs tid 941) v1 tid 476) v2 -- ?+0 0x5f94500 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -49> 2015-10-29 14:29:44.697678 7f3154801700 0 log_channel(cluster) log [INF] : pgmap v33661654: 21936 pgs: 21936 active+clean; 9824 GB data, 29683 GB used, 144 TB / 173 TB avail; 6259 kB/s rd, 39388 kB/s wr, 4361 op/s
Oct 29 14:29:45 95z3zz1 ceph-mon: -48> 2015-10-29 14:29:44.697683 7f3154801700 0 log_channel(cluster) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -47> 2015-10-29 14:29:44.697707 7f3154801700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -46> 2015-10-29 14:29:44.697709 7f3154801700 10 log_client log_queue is 1 last_log 8 sent 7 num 1 unsent 1 sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -45> 2015-10-29 14:29:44.697711 7f3154801700 10 log_client will send 2015-10-29 14:29:44.697682 mon.0 157.249.160.11:6789/0 8 : cluster [INF] pgmap v33661654: 21936 pgs: 21936 active+clean; 9824 GB data, 29683 GB used, 144 TB / 173 TB avail; 6259 kB/s rd, 39388 kB/s wr, 4361 op/s
Oct 29 14:29:45 95z3zz1 ceph-mon: -44> 2015-10-29 14:29:44.697716 7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1 entries) v1 -- ?+0 0x4f8df80 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -43> 2015-10-29 14:29:44.697774 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8df80 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -42> 2015-10-29 14:29:44.697790 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015) is_readable = 1 - now=2015-10-29 14:29:44.697791 lease_expire=2015-10-29 14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -41> 2015-10-29 14:29:44.698516 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999381 ==== paxos(lease_ack lc 62204015 fc 62203323 pn 0 opn 0) v3 ==== 80+0+0 (2218565783 0 0) 0x5f94500 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -40> 2015-10-29 14:29:44.698776 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139024 ==== paxos(lease_ack lc 62204015 fc 62203323 pn 0 opn 0) v3 ==== 80+0+0 (2124060710 0 0) 0x5f81b80 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -39> 2015-10-29 14:29:44.907157 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139025 ==== forward(log(1 entries) v1 caps allow * tid 571 con_features 0) to leader v3 ==== 584+0+0 (4179428545 0 0) 0x5f8b9c0 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -38> 2015-10-29 14:29:44.907187 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015) is_readable = 1 - now=2015-10-29 14:29:44.907188 lease_expire=2015-10-29 14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -37> 2015-10-29 14:29:45.074350 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999382 ==== forward(log(1 entries) v1 caps allow * tid 478 con_features 0) to leader v3 ==== 584+0+0 (48471394 0 0) 0x5f9c840 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -36> 2015-10-29 14:29:45.074382 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015) is_readable = 1 - now=2015-10-29 14:29:45.074384 lease_expire=2015-10-29 14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -35> 2015-10-29 14:29:45.303886 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999383 ==== forward(pg_stats(1 pgs tid 19053 v 0) v1 caps allow profile osd tid 479 con_features 37154696925806591) to leader v3 ==== 1041+0+0 (2232892023 0 0) 0x5f9c580 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -34> 2015-10-29 14:29:45.303921 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015) is_readable = 1 - now=2015-10-29 14:29:45.303922 lease_expire=2015-10-29 14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -33> 2015-10-29 14:29:45.303951 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015) is_readable = 1 - now=2015-10-29 14:29:45.303952 lease_expire=2015-10-29 14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -32> 2015-10-29 14:29:45.374985 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139026 ==== forward(pg_stats(2 pgs tid 18876 v 0) v1 caps allow profile osd tid 572 con_features 37154696925806591) to leader v3 ==== 1542+0+0 (1698272303 0 0) 0x5f8b700 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -31> 2015-10-29 14:29:45.375022 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015) is_readable = 1 - now=2015-10-29 14:29:45.375023 lease_expire=2015-10-29 14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -30> 2015-10-29 14:29:45.375046 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015) is_readable = 1 - now=2015-10-29 14:29:45.375046 lease_expire=2015-10-29 14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -29> 2015-10-29 14:29:45.591682 7f3155ddf700 5 asok(0x4daa000) AdminSocket: request 'get_command_descriptions' '' to 0x4d32140 returned 2098 bytes
Oct 29 14:29:45 95z3zz1 ceph-mon: -28> 2015-10-29 14:29:45.593975 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139027 ==== forward(pg_stats(2 pgs tid 19024 v 0) v1 caps allow profile osd tid 573 con_features 37154696925806591) to leader v3 ==== 1542+0+0 (2559589014 0 0) 0x5f8b180 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -27> 2015-10-29 14:29:45.594011 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015) is_readable = 1 - now=2015-10-29 14:29:45.594012 lease_expire=2015-10-29 14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -26> 2015-10-29 14:29:45.594038 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015) is_readable = 1 - now=2015-10-29 14:29:45.594039 lease_expire=2015-10-29 14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -25> 2015-10-29 14:29:45.596499 7f3155ddf700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
Oct 29 14:29:45 95z3zz1 ceph-mon: -24> 2015-10-29 14:29:45.596508 7f3155ddf700 0 log_channel(audit) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -23> 2015-10-29 14:29:45.596513 7f3155ddf700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -22> 2015-10-29 14:29:45.596514 7f3155ddf700 10 log_client log_queue is 2 last_log 9 sent 8 num 2 unsent 1 sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -21> 2015-10-29 14:29:45.596517 7f3155ddf700 10 log_client will send 2015-10-29 14:29:45.596508 mon.0 157.249.160.11:6789/0 9 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
Oct 29 14:29:45 95z3zz1 ceph-mon: -20> 2015-10-29 14:29:45.596545 7f3155ddf700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1 entries) v1 -- ?+0 0x545c480 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -19> 2015-10-29 14:29:45.596600 7f3155ddf700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
Oct 29 14:29:45 95z3zz1 ceph-mon: -18> 2015-10-29 14:29:45.596603 7f3155ddf700 0 log_channel(audit) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -17> 2015-10-29 14:29:45.596604 7f3155ddf700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -16> 2015-10-29 14:29:45.596605 7f3155ddf700 10 log_client log_queue is 3 last_log 10 sent 9 num 3 unsent 1 sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -15> 2015-10-29 14:29:45.596607 7f3155ddf700 10 log_client will send 2015-10-29 14:29:45.596603 mon.0 157.249.160.11:6789/0 10 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
Oct 29 14:29:45 95z3zz1 ceph-mon: -14> 2015-10-29 14:29:45.596613 7f3155ddf700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1 entries) v1 -- ?+0 0x4f8d8c0 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -13> 2015-10-29 14:29:45.596626 7f3155ddf700 5 asok(0x4daa000) AdminSocket: request 'mon_status' '' to 0x5e1dbe0 returned 855 bytes
Oct 29 14:29:45 95z3zz1 ceph-mon: -12> 2015-10-29 14:29:45.596652 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x545c480 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -11> 2015-10-29 14:29:45.596670 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015) is_readable = 1 - now=2015-10-29 14:29:45.596672 lease_expire=2015-10-29 14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -10> 2015-10-29 14:29:45.596706 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8d8c0 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -9> 2015-10-29 14:29:45.596711 7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015) is_readable = 1 - now=2015-10-29 14:29:45.596712 lease_expire=2015-10-29 14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -8> 2015-10-29 14:29:45.693414 7f315184f700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015) queue_pending_finisher 0x5f096b0
Oct 29 14:29:45 95z3zz1 ceph-mon: -7> 2015-10-29 14:29:45.693887 7f315184f700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 -- paxos(begin lc 62204015 fc 0 pn 21100 opn 0) v3 -- ?+0 0x604f480
Oct 29 14:29:45 95z3zz1 ceph-mon: -6> 2015-10-29 14:29:45.693918 7f315184f700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 -- paxos(begin lc 62204015 fc 0 pn 21100 opn 0) v3 -- ?+0 0x5f83200
Oct 29 14:29:45 95z3zz1 ceph-mon: -5> 2015-10-29 14:29:45.694012 7f315184f700 5 mon.95z3zz1@0(leader).paxos(paxos updating c 62203323..62204015) queue_pending_finisher 0x5f0aaf0
Oct 29 14:29:45 95z3zz1 ceph-mon: -4> 2015-10-29 14:29:45.694889 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0 1506139028 ==== paxos(accept lc 62204015 fc 0 pn 21100 opn 0) v3 ==== 80+0+0 (3498373181 0 0) 0x5f83200 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -3> 2015-10-29 14:29:45.694925 7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0 13999384 ==== paxos(accept lc 62204015 fc 0 pn 21100 opn 0) v3 ==== 80+0+0 (914064508 0 0) 0x604f480 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -2> 2015-10-29 14:29:45.695372 7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 -- paxos(commit lc 62204016 fc 0 pn 21100 opn 0) v3 -- ?+0 0x5f97200
Oct 29 14:29:45 95z3zz1 ceph-mon: -1> 2015-10-29 14:29:45.695392 7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 -- paxos(commit lc 62204016 fc 0 pn 21100 opn 0) v3 -- ?+0 0x5f96f80
Oct 29 14:29:45 95z3zz1 ceph-mon: 0> 2015-10-29 14:29:45.697177 7f3154801700 -1 *** Caught signal (Segmentation fault) **#012 in thread 7f3154801700#012#012 ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)#012 1: /usr/bin/ceph-mon() [0x9adefa]#012 2: (()+0x10340) [0x7f3159b63340]#012 3: (std::_Rb_tree<std::string, std::pair<std::string const, std::string>, std::_Select1st<std::pair<std::string const, std::string> >, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >::find(std::string const&) const+0x25) [0x6518e5]#012 4: (get_str_map_key(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, std::string const&, std::string const*)+0x1e) [0x8a002e]#012 5: (LogMonitor::update_from_paxos(bool*)+0x87a) [0x6b0a5a]#012 6: (PaxosService::refresh(bool*)+0x19a) [0x60432a]#012 7: (Monitor::refresh_from_paxos(bool*)+0x1db) [0x5b03db]#012 8: (Paxos::do_refresh()+0
x2e) [0x5eea5e]#012 9: (Paxos::commit_finish()+0x569) [0x5fbf39]#012 10: (C_Committed::finish(int)+0x2b) [0x60038b]#012 11: (Context::complete(int)+0x9) [0x5d4d89]#012 12: (MonitorDBStore::C_DoTransaction::finish(int)+0x8c) [0x5ff4bc]#012 13: (Context::complete(int)+0x9) [0x5d4d89]#012 14: (Finisher::finisher_thread_entry()+0x158) [0x717e88]#012 15: (()+0x8182) [0x7f3159b5b182]#012 16: (clone()+0x6d) [0x7f31580c647d]#012 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Oct 29 14:29:45 95z3zz1 ceph-mon: --- logging levels ---
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 none
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 1 lockdep
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 1 context
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 1 crush
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 mds
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 mds_balancer
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 mds_locker
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 mds_log
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 mds_log_expire
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 mds_migrator
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 1 buffer
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 1 timer
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 1 filer
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 1 striper
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 1 objecter
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 rados
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 rbd
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 rbd_replay
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 journaler
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 objectcacher
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 client
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 osd
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 optracker
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 objclass
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 3 filestore
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 3 keyvaluestore
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 3 journal
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 ms
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 mon
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/10 monc
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 paxos
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 tp
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 auth
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 crypto
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 1 finisher
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 heartbeatmap
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 perfcounter
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 rgw
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/10 civetweb
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 javaclient
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 asok
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 1 throttle
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 0 refs
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 xio
Oct 29 14:29:45 95z3zz1 ceph-mon: 99/99 (syslog threshold)
Oct 29 14:29:45 95z3zz1 ceph-mon: -1/-1 (stderr threshold)
Oct 29 14:29:45 95z3zz1 ceph-mon: max_recent 10000
Oct 29 14:29:45 95z3zz1 ceph-mon: max_new 1000
Oct 29 14:29:45 95z3zz1 ceph-mon: log_file /var/log/ceph/ceph-mon.95z3zz1.log
Oct 29 14:29:45 95z3zz1 ceph-mon: --- end dump of recent events ---
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.806299 7f31ad4e38c0 0 ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43), process ceph-mon, pid 143457
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.819073 7f31ad4e38c0 0 starting mon.95z3zz1 rank 0 at 157.249.160.11:6789/0 mon_data /var/lib/ceph/mon/ceph-95z3zz1 fsid 6a57f77c-a465-4fc6-b0a2-93c20371bab8
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.819332 7f31ad4e38c0 1 mon.95z3zz1@-1(probing) e7 preinit fsid 6a57f77c-a465-4fc6-b0a2-93c20371bab8
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.819670 7f31ad4e38c0 1 mon.95z3zz1@-1(probing).paxosservice(pgmap 33661126..33661654) refresh upgraded, format 0 -> 1
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.819682 7f31ad4e38c0 1 mon.95z3zz1@-1(probing).pg v0 on_upgrade discarding in-core PGMap
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.932544 7f31ad4e38c0 0 mon.95z3zz1@-1(probing).mds e89 print_map#012epoch#01189#012flags#0110#012created#0112014-07-31 11:09:25.038824#012modified#0112015-10-28 11:59:17.337799#012tableserver#0110#012root#0110#012session_timeout#01160#012session_autoclose#011300#012max_file_size#0111099511627776#012last_failure#0110#012last_failure_osd_epoch#01161286#012compat#011compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table}#012max_mds#0111#012in#0110#012up#011{0=15517917}#012failed#011#012stopped#011#012data_pools#0110#012metadata_pool#0111#012inline_data#011disabled#01215517917:#011157.249.160.107:6800/3998 '56z3zz1' mds.0.13 up:active seq 188
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.933110 7f31ad4e38c0 0 mon.95z3zz1@-1(probing).osd e61385 crush map has features 1107558400, adjusting msgr requires
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.933120 7f31ad4e38c0 0 mon.95z3zz1@-1(probing).osd e61385 crush map has features 1107558400, adjusting msgr requires
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.933126 7f31ad4e38c0 0 mon.95z3zz1@-1(probing).osd e61385 crush map has features 1107558400, adjusting msgr requires
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.933129 7f31ad4e38c0 0 mon.95z3zz1@-1(probing).osd e61385 crush map has features 1107558400, adjusting msgr requires
Oct 29 14:29:45 95z3zz1 ceph-mon: mon.0 157.249.160.11:6789/0 8 : pgmap v33661654: 21936 pgs: 21936 active+clean; 9824 GB data, 29683 GB used, 144 TB / 173 TB avail; 6259 kB/s rd, 39388 kB/s wr, 4361 op/s
--
Requests: servicedesk@xxxxxx
Personal: arnulf.heimsbakk@xxxxxx
Phone : +47 22 06 30 00/32 54
Confidential or sensitive, use one of
miniLock: T1zSJJA9dm7M7ECdmvYLe9jxmgGCGAifShcmJ787uDZ6q
PGP ID : AD21343A
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
