On Fri, Feb 10, 2017 at 7:04 PM, Luke Weber <luke.weber@xxxxxxxxx> wrote: > Hi Gregory, > > Thanks for the response. So you're at least partially right that I'm > shutting down 4/16 ods, and 1/3 mgr, and 1/3 mons, and 1/2 mds. However with > replication=2, and 2/3 mons still active I would expect that cluster could > recover from this. Also the other mds server doesn't have a mon or a manager > active on it, just the osds but I basically see the same behavior. After 8 > minutes on this one I gave up and just started the instance I shut down, and > the system quickly came back to life. > > mds_map is where things go into the stuck state @ 2017-02-11 01:56:40.934987 > > 2017-02-11 02:08:44.941573 is where I've rebooted things and they start to > come back to life. > > Using simple mds pair config as outlined from > http://docs.ceph.com/docs/master/cephfs/standby/ > > After a shutting down the mds server that contained the active mds: >> ceph mds stat > e496: 1/1/1 up {0=na9552=up:replay} >> ceph health > HEALTH_ERR 804 pgs are stuck inactive for more than 300 seconds; 804 pgs > degraded; 804 pgs stuck inactive; 804 pgs stuck unclean; 804 pgs undersized; > 2 requests are blocked > 32 sec; recovery 325255/1317940 objects degraded > (24.679%); mds cluster is degraded; 4/16 in osds are down; 1 mons down, > quorum 0,1 na9549,na9550 > [root@na9552 luke]# sudo ceph mds stat > e496: 1/1/1 up {0=na9552=up:replay} Okay, this has nothing to do with the MDS — if you've got inaccessible metadata it can't do anything. 804 PGs stuck inactive is bad and you'll need to figure out what's going on there. You'll want to figure out why; my first stab in the dark is that you've got your CRUSH map configured incorrectly and shut down all the replicas of some of your data, but the docs should be able to walk you through that. -Greg > > mds log from server that goes into replay mode for a very long time: > 2017-02-11 01:55:56.921779 7fa79fa2e700 10 mds.beacon.na9552 _send > up:standby-replay seq 15 > 2017-02-11 01:55:56.921817 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6789/0 -- mdsbeacon(814112/na9552 up:standby-replay seq 15 > v495) v7 -- 0x7fa7b78a56c0 con 0 > 2017-02-11 01:55:56.923325 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.2 172.20.1.137:6789/0 28 ==== mdsbeacon(814112/na9552 > up:standby-replay seq 15 v495) v7 ==== 131+0+0 (2676696853 0 0) > 0x7fa7b78a56c0 con 0x7fa7b2e35000 > 2017-02-11 01:55:56.923361 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:standby-replay seq 15 rtt 0.001559 > 2017-02-11 01:55:57.340512 7fa7a022f700 10 MDSInternalContextBase::complete: > N7MDSRank26C_MDS_StandbyReplayRestartE > 2017-02-11 01:55:57.340525 7fa7a022f700 4 mds.0.0 standby_replay_restart > (as standby) > 2017-02-11 01:55:57.340570 7fa7a022f700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.0:173 2.844f3494 200.00000000 > [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45e3180 con 0 > 2017-02-11 01:55:57.341287 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.2 172.20.1.133:6804/2623 58 ==== osd_op_reply(173 200.00000000 [read > 0~90] v0'0 uv1260 ondisk = 0) v7 ==== 132+0+90 (2300422887 0 1002082041) > 0x7fa7b45e2c00 con 0x7fa7b2efd000 > 2017-02-11 01:55:57.341458 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:174 2.9430acec > 200.00000613 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45f8b00 con 0 > 2017-02-11 01:55:57.341503 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:175 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45f8dc0 con 0 > 2017-02-11 01:55:57.342080 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 107 ==== osd_op_reply(174 200.00000613 > [stat] v0'0 uv554 ondisk = 0) v7 ==== 132+0+16 (981552572 0 1812931205) > 0x7fa7b45f8840 con 0x7fa7b2efe800 > 2017-02-11 01:55:57.342161 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 108 ==== osd_op_reply(175 200.00000614 > [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 > (2617829315 0 0) 0x7fa7b45f8840 con 0x7fa7b2efe800 > 2017-02-11 01:55:57.342239 7fa79da2a700 10 MDSIOContextBase::complete: > N7MDSRank32C_MDS_StandbyReplayRestartFinishE > 2017-02-11 01:55:57.342245 7fa79da2a700 10 mds.0.log standby_trim_segments > 2017-02-11 01:55:57.342247 7fa79da2a700 10 mds.0.log expire_pos=6490889269 > 2017-02-11 01:55:57.342250 7fa79da2a700 10 mds.0.log segment seq=3885173 > 6490889269~1297898 > 2017-02-11 01:55:57.342253 7fa79da2a700 10 mds.0.log won't remove, not > expired! > 2017-02-11 01:55:57.342257 7fa79da2a700 20 mds.0.log removed no segments! > 2017-02-11 01:55:57.342258 7fa79da2a700 2 mds.0.0 boot_start 2: replaying > mds log > 2017-02-11 01:55:57.342262 7fa79da2a700 10 mds.0.log replay - journal empty, > done. > 2017-02-11 01:55:57.342264 7fa79da2a700 7 mds.0.cache trim max=100000 > cur=33536 > 2017-02-11 01:55:57.342271 7fa79da2a700 10 MDSInternalContextBase::complete: > 15C_MDS_BootStart > 2017-02-11 01:55:57.342273 7fa79da2a700 1 mds.0.0 replay_done (as standby) > 2017-02-11 01:55:57.342275 7fa79da2a700 10 mds.0.0 setting replay timer > 2017-02-11 01:55:58.342382 7fa7a022f700 10 MDSInternalContextBase::complete: > N7MDSRank26C_MDS_StandbyReplayRestartE > 2017-02-11 01:55:58.342410 7fa7a022f700 4 mds.0.0 standby_replay_restart > (as standby) > 2017-02-11 01:55:58.342442 7fa7a022f700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.0:176 2.844f3494 200.00000000 > [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45e3440 con 0 > 2017-02-11 01:55:58.343084 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.2 172.20.1.133:6804/2623 59 ==== osd_op_reply(176 200.00000000 [read > 0~90] v0'0 uv1260 ondisk = 0) v7 ==== 132+0+90 (2300422887 0 1002082041) > 0x7fa7b45e3440 con 0x7fa7b2efd000 > 2017-02-11 01:55:58.343231 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:177 2.9430acec > 200.00000613 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45f9080 con 0 > 2017-02-11 01:55:58.343273 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:178 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45f9340 con 0 > 2017-02-11 01:55:58.343860 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 109 ==== osd_op_reply(177 200.00000613 > [stat] v0'0 uv554 ondisk = 0) v7 ==== 132+0+16 (981552572 0 1812931205) > 0x7fa7b45f8dc0 con 0x7fa7b2efe800 > 2017-02-11 01:55:58.343943 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 110 ==== osd_op_reply(178 200.00000614 > [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 > (2617829315 0 0) 0x7fa7b45f8dc0 con 0x7fa7b2efe800 > 2017-02-11 01:55:58.344019 7fa79da2a700 10 MDSIOContextBase::complete: > N7MDSRank32C_MDS_StandbyReplayRestartFinishE > 2017-02-11 01:55:58.344024 7fa79da2a700 10 mds.0.log standby_trim_segments > 2017-02-11 01:55:58.344026 7fa79da2a700 10 mds.0.log expire_pos=6490889269 > 2017-02-11 01:55:58.344028 7fa79da2a700 10 mds.0.log segment seq=3885173 > 6490889269~1297898 > 2017-02-11 01:55:58.344030 7fa79da2a700 10 mds.0.log won't remove, not > expired! > 2017-02-11 01:55:58.344032 7fa79da2a700 20 mds.0.log removed no segments! > 2017-02-11 01:55:58.344033 7fa79da2a700 2 mds.0.0 boot_start 2: replaying > mds log > 2017-02-11 01:55:58.344038 7fa79da2a700 10 mds.0.log replay - journal empty, > done. > 2017-02-11 01:55:58.344040 7fa79da2a700 7 mds.0.cache trim max=100000 > cur=33535 > 2017-02-11 01:55:58.344044 7fa79da2a700 10 MDSInternalContextBase::complete: > 15C_MDS_BootStart > 2017-02-11 01:55:58.344045 7fa79da2a700 1 mds.0.0 replay_done (as standby) > 2017-02-11 01:55:58.344047 7fa79da2a700 10 mds.0.0 setting replay timer > 2017-02-11 01:55:59.344149 7fa7a022f700 10 MDSInternalContextBase::complete: > N7MDSRank26C_MDS_StandbyReplayRestartE > 2017-02-11 01:55:59.344161 7fa7a022f700 4 mds.0.0 standby_replay_restart > (as standby) > 2017-02-11 01:55:59.344193 7fa7a022f700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.0:179 2.844f3494 200.00000000 > [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45e3700 con 0 > 2017-02-11 01:55:59.344799 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.2 172.20.1.133:6804/2623 60 ==== osd_op_reply(179 200.00000000 [read > 0~90] v0'0 uv1260 ondisk = 0) v7 ==== 132+0+90 (2300422887 0 1002082041) > 0x7fa7b45e3180 con 0x7fa7b2efd000 > 2017-02-11 01:55:59.344932 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:180 2.9430acec > 200.00000613 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45f9600 con 0 > 2017-02-11 01:55:59.344982 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:181 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45f98c0 con 0 > 2017-02-11 01:55:59.345575 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 111 ==== osd_op_reply(180 200.00000613 > [stat] v0'0 uv554 ondisk = 0) v7 ==== 132+0+16 (981552572 0 1812931205) > 0x7fa7b45f9340 con 0x7fa7b2efe800 > 2017-02-11 01:55:59.345653 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 112 ==== osd_op_reply(181 200.00000614 > [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 > (2617829315 0 0) 0x7fa7b45f9340 con 0x7fa7b2efe800 > 2017-02-11 01:55:59.345727 7fa79da2a700 10 MDSIOContextBase::complete: > N7MDSRank32C_MDS_StandbyReplayRestartFinishE > 2017-02-11 01:55:59.345732 7fa79da2a700 10 mds.0.log standby_trim_segments > 2017-02-11 01:55:59.345734 7fa79da2a700 10 mds.0.log expire_pos=6490889269 > 2017-02-11 01:55:59.345737 7fa79da2a700 10 mds.0.log segment seq=3885173 > 6490889269~1297898 > 2017-02-11 01:55:59.345740 7fa79da2a700 10 mds.0.log won't remove, not > expired! > 2017-02-11 01:55:59.345741 7fa79da2a700 20 mds.0.log removed no segments! > 2017-02-11 01:55:59.345743 7fa79da2a700 2 mds.0.0 boot_start 2: replaying > mds log > 2017-02-11 01:55:59.345745 7fa79da2a700 10 mds.0.log replay - journal empty, > done. > 2017-02-11 01:55:59.345747 7fa79da2a700 7 mds.0.cache trim max=100000 > cur=33534 > 2017-02-11 01:55:59.345753 7fa79da2a700 10 MDSInternalContextBase::complete: > 15C_MDS_BootStart > 2017-02-11 01:55:59.345755 7fa79da2a700 1 mds.0.0 replay_done (as standby) > 2017-02-11 01:55:59.345756 7fa79da2a700 10 mds.0.0 setting replay timer > 2017-02-11 01:56:00.345862 7fa7a022f700 10 MDSInternalContextBase::complete: > N7MDSRank26C_MDS_StandbyReplayRestartE > 2017-02-11 01:56:00.345873 7fa7a022f700 4 mds.0.0 standby_replay_restart > (as standby) > 2017-02-11 01:56:00.345902 7fa7a022f700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.0:182 2.844f3494 200.00000000 > [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45e39c0 con 0 > 2017-02-11 01:56:00.346461 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.2 172.20.1.133:6804/2623 61 ==== osd_op_reply(182 200.00000000 [read > 0~90] v0'0 uv1260 ondisk = 0) v7 ==== 132+0+90 (2300422887 0 1002082041) > 0x7fa7b45e39c0 con 0x7fa7b2efd000 > 2017-02-11 01:56:00.346606 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:183 2.9430acec > 200.00000613 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45f9b80 con 0 > 2017-02-11 01:56:00.346651 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:184 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45f9e40 con 0 > 2017-02-11 01:56:00.347124 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 113 ==== osd_op_reply(183 200.00000613 > [stat] v0'0 uv554 ondisk = 0) v7 ==== 132+0+16 (981552572 0 1812931205) > 0x7fa7b45f98c0 con 0x7fa7b2efe800 > 2017-02-11 01:56:00.347206 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 114 ==== osd_op_reply(184 200.00000614 > [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 > (2617829315 0 0) 0x7fa7b45f98c0 con 0x7fa7b2efe800 > 2017-02-11 01:56:00.347280 7fa79da2a700 10 MDSIOContextBase::complete: > N7MDSRank32C_MDS_StandbyReplayRestartFinishE > 2017-02-11 01:56:00.347285 7fa79da2a700 10 mds.0.log standby_trim_segments > 2017-02-11 01:56:00.347287 7fa79da2a700 10 mds.0.log expire_pos=6490889269 > 2017-02-11 01:56:00.347288 7fa79da2a700 10 mds.0.log segment seq=3885173 > 6490889269~1297898 > 2017-02-11 01:56:00.347291 7fa79da2a700 10 mds.0.log won't remove, not > expired! > 2017-02-11 01:56:00.347292 7fa79da2a700 20 mds.0.log removed no segments! > 2017-02-11 01:56:00.347294 7fa79da2a700 2 mds.0.0 boot_start 2: replaying > mds log > 2017-02-11 01:56:00.347298 7fa79da2a700 10 mds.0.log replay - journal empty, > done. > 2017-02-11 01:56:00.347301 7fa79da2a700 7 mds.0.cache trim max=100000 > cur=33533 > 2017-02-11 01:56:00.347306 7fa79da2a700 10 MDSInternalContextBase::complete: > 15C_MDS_BootStart > 2017-02-11 01:56:00.347307 7fa79da2a700 1 mds.0.0 replay_done (as standby) > 2017-02-11 01:56:00.347309 7fa79da2a700 10 mds.0.0 setting replay timer > 2017-02-11 01:56:00.921814 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:56:00.921857 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:56:00.921875 7fa79fa2e700 10 mds.beacon.na9552 _send > up:standby-replay seq 16 > 2017-02-11 01:56:00.921899 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6789/0 -- mdsbeacon(814112/na9552 up:standby-replay seq 16 > v495) v7 -- 0x7fa7b78a5a00 con 0 > 2017-02-11 01:56:00.923442 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.2 172.20.1.137:6789/0 29 ==== mdsbeacon(814112/na9552 > up:standby-replay seq 16 v495) v7 ==== 131+0+0 (2409888637 0 0) > 0x7fa7b78a5a00 con 0x7fa7b2e35000 > 2017-02-11 01:56:00.923474 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:standby-replay seq 16 rtt 0.001586 > 2017-02-11 01:56:00.925666 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b45e7cc0 con 0 > 2017-02-11 01:56:01.347409 7fa7a022f700 10 MDSInternalContextBase::complete: > N7MDSRank26C_MDS_StandbyReplayRestartE > 2017-02-11 01:56:01.347422 7fa7a022f700 4 mds.0.0 standby_replay_restart > (as standby) > 2017-02-11 01:56:01.347453 7fa7a022f700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.0:185 2.844f3494 200.00000000 > [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45e3c80 con 0 > 2017-02-11 01:56:01.348038 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.2 172.20.1.133:6804/2623 62 ==== osd_op_reply(185 200.00000000 [read > 0~90] v0'0 uv1260 ondisk = 0) v7 ==== 132+0+90 (2300422887 0 1002082041) > 0x7fa7b45e3700 con 0x7fa7b2efd000 > 2017-02-11 01:56:01.348181 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:186 2.9430acec > 200.00000613 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45fa100 con 0 > 2017-02-11 01:56:01.348244 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:187 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45fa3c0 con 0 > 2017-02-11 01:56:01.348826 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 115 ==== osd_op_reply(186 200.00000613 > [stat] v0'0 uv554 ondisk = 0) v7 ==== 132+0+16 (981552572 0 1812931205) > 0x7fa7b45f9e40 con 0x7fa7b2efe800 > 2017-02-11 01:56:01.348910 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 116 ==== osd_op_reply(187 200.00000614 > [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 > (2617829315 0 0) 0x7fa7b45f9e40 con 0x7fa7b2efe800 > 2017-02-11 01:56:01.348977 7fa79da2a700 10 MDSIOContextBase::complete: > N7MDSRank32C_MDS_StandbyReplayRestartFinishE > 2017-02-11 01:56:01.348982 7fa79da2a700 10 mds.0.log standby_trim_segments > 2017-02-11 01:56:01.348984 7fa79da2a700 10 mds.0.log expire_pos=6490889269 > 2017-02-11 01:56:01.348986 7fa79da2a700 10 mds.0.log segment seq=3885173 > 6490889269~1297898 > 2017-02-11 01:56:01.348989 7fa79da2a700 10 mds.0.log won't remove, not > expired! > 2017-02-11 01:56:01.348990 7fa79da2a700 20 mds.0.log removed no segments! > 2017-02-11 01:56:01.348992 7fa79da2a700 2 mds.0.0 boot_start 2: replaying > mds log > 2017-02-11 01:56:01.348994 7fa79da2a700 10 mds.0.log replay - journal empty, > done. > 2017-02-11 01:56:01.349015 7fa79da2a700 7 mds.0.cache trim max=100000 > cur=33532 > 2017-02-11 01:56:01.349024 7fa79da2a700 10 MDSInternalContextBase::complete: > 15C_MDS_BootStart > 2017-02-11 01:56:01.349026 7fa79da2a700 1 mds.0.0 replay_done (as standby) > 2017-02-11 01:56:01.349027 7fa79da2a700 10 mds.0.0 setting replay timer > 2017-02-11 01:56:02.349133 7fa7a022f700 10 MDSInternalContextBase::complete: > N7MDSRank26C_MDS_StandbyReplayRestartE > 2017-02-11 01:56:02.349144 7fa7a022f700 4 mds.0.0 standby_replay_restart > (as standby) > 2017-02-11 01:56:02.349193 7fa7a022f700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.0:188 2.844f3494 200.00000000 > [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45fa940 con 0 > 2017-02-11 01:56:02.349808 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.2 172.20.1.133:6804/2623 63 ==== osd_op_reply(188 200.00000000 [read > 0~90] v0'0 uv1260 ondisk = 0) v7 ==== 132+0+90 (2300422887 0 1002082041) > 0x7fa7b45fa940 con 0x7fa7b2efd000 > 2017-02-11 01:56:02.349955 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:189 2.9430acec > 200.00000613 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45fa680 con 0 > 2017-02-11 01:56:02.350015 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:190 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b4644000 con 0 > 2017-02-11 01:56:02.350629 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 117 ==== osd_op_reply(189 200.00000613 > [stat] v0'0 uv554 ondisk = 0) v7 ==== 132+0+16 (981552572 0 1812931205) > 0x7fa7b4644000 con 0x7fa7b2efe800 > 2017-02-11 01:56:02.350710 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 118 ==== osd_op_reply(190 200.00000614 > [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 > (2617829315 0 0) 0x7fa7b4644000 con 0x7fa7b2efe800 > 2017-02-11 01:56:02.350778 7fa79da2a700 10 MDSIOContextBase::complete: > N7MDSRank32C_MDS_StandbyReplayRestartFinishE > 2017-02-11 01:56:02.350783 7fa79da2a700 10 mds.0.log standby_trim_segments > 2017-02-11 01:56:02.350785 7fa79da2a700 10 mds.0.log expire_pos=6490889269 > 2017-02-11 01:56:02.350787 7fa79da2a700 10 mds.0.log segment seq=3885173 > 6490889269~1297898 > 2017-02-11 01:56:02.350790 7fa79da2a700 10 mds.0.log won't remove, not > expired! > 2017-02-11 01:56:02.350791 7fa79da2a700 20 mds.0.log removed no segments! > 2017-02-11 01:56:02.350793 7fa79da2a700 2 mds.0.0 boot_start 2: replaying > mds log > 2017-02-11 01:56:02.350796 7fa79da2a700 10 mds.0.log replay - journal empty, > done. > 2017-02-11 01:56:02.350798 7fa79da2a700 7 mds.0.cache trim max=100000 > cur=33531 > 2017-02-11 01:56:02.350802 7fa79da2a700 10 MDSInternalContextBase::complete: > 15C_MDS_BootStart > 2017-02-11 01:56:02.350804 7fa79da2a700 1 mds.0.0 replay_done (as standby) > 2017-02-11 01:56:02.350805 7fa79da2a700 10 mds.0.0 setting replay timer > 2017-02-11 01:56:03.350932 7fa7a022f700 10 MDSInternalContextBase::complete: > N7MDSRank26C_MDS_StandbyReplayRestartE > 2017-02-11 01:56:03.350943 7fa7a022f700 4 mds.0.0 standby_replay_restart > (as standby) > 2017-02-11 01:56:03.350974 7fa7a022f700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.0:191 2.844f3494 200.00000000 > [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45fac00 con 0 > 2017-02-11 01:56:03.351630 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.2 172.20.1.133:6804/2623 64 ==== osd_op_reply(191 200.00000000 [read > 0~90] v0'0 uv1260 ondisk = 0) v7 ==== 132+0+90 (2300422887 0 1002082041) > 0x7fa7b45fa940 con 0x7fa7b2efd000 > 2017-02-11 01:56:03.351775 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:192 2.9430acec > 200.00000613 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b46442c0 con 0 > 2017-02-11 01:56:03.351817 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:193 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b4644580 con 0 > 2017-02-11 01:56:03.352432 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 119 ==== osd_op_reply(192 200.00000613 > [stat] v0'0 uv554 ondisk = 0) v7 ==== 132+0+16 (981552572 0 1812931205) > 0x7fa7b4644000 con 0x7fa7b2efe800 > 2017-02-11 01:56:03.352525 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 120 ==== osd_op_reply(193 200.00000614 > [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 > (2617829315 0 0) 0x7fa7b4644000 con 0x7fa7b2efe800 > 2017-02-11 01:56:03.352594 7fa79da2a700 10 MDSIOContextBase::complete: > N7MDSRank32C_MDS_StandbyReplayRestartFinishE > 2017-02-11 01:56:03.352599 7fa79da2a700 10 mds.0.log standby_trim_segments > 2017-02-11 01:56:03.352601 7fa79da2a700 10 mds.0.log expire_pos=6490889269 > 2017-02-11 01:56:03.352603 7fa79da2a700 10 mds.0.log segment seq=3885173 > 6490889269~1297898 > 2017-02-11 01:56:03.352605 7fa79da2a700 10 mds.0.log won't remove, not > expired! > 2017-02-11 01:56:03.352609 7fa79da2a700 20 mds.0.log removed no segments! > 2017-02-11 01:56:03.352610 7fa79da2a700 2 mds.0.0 boot_start 2: replaying > mds log > 2017-02-11 01:56:03.352612 7fa79da2a700 10 mds.0.log replay - journal empty, > done. > 2017-02-11 01:56:03.352614 7fa79da2a700 7 mds.0.cache trim max=100000 > cur=33530 > 2017-02-11 01:56:03.352620 7fa79da2a700 10 MDSInternalContextBase::complete: > 15C_MDS_BootStart > 2017-02-11 01:56:03.352622 7fa79da2a700 1 mds.0.0 replay_done (as standby) > 2017-02-11 01:56:03.352623 7fa79da2a700 10 mds.0.0 setting replay timer > 2017-02-11 01:56:04.352756 7fa7a022f700 10 MDSInternalContextBase::complete: > N7MDSRank26C_MDS_StandbyReplayRestartE > 2017-02-11 01:56:04.352778 7fa7a022f700 4 mds.0.0 standby_replay_restart > (as standby) > 2017-02-11 01:56:04.352825 7fa7a022f700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.0:194 2.844f3494 200.00000000 > [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45faec0 con 0 > 2017-02-11 01:56:04.353491 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.2 172.20.1.133:6804/2623 65 ==== osd_op_reply(194 200.00000000 [read > 0~90] v0'0 uv1260 ondisk = 0) v7 ==== 132+0+90 (2300422887 0 1002082041) > 0x7fa7b45faec0 con 0x7fa7b2efd000 > 2017-02-11 01:56:04.353643 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:195 2.9430acec > 200.00000613 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b4644840 con 0 > 2017-02-11 01:56:04.353684 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:196 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b4644b00 con 0 > 2017-02-11 01:56:04.354250 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 121 ==== osd_op_reply(195 200.00000613 > [stat] v0'0 uv554 ondisk = 0) v7 ==== 132+0+16 (981552572 0 1812931205) > 0x7fa7b4644580 con 0x7fa7b2efe800 > 2017-02-11 01:56:04.354352 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 122 ==== osd_op_reply(196 200.00000614 > [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 > (2617829315 0 0) 0x7fa7b4644580 con 0x7fa7b2efe800 > 2017-02-11 01:56:04.354435 7fa79da2a700 10 MDSIOContextBase::complete: > N7MDSRank32C_MDS_StandbyReplayRestartFinishE > 2017-02-11 01:56:04.354440 7fa79da2a700 10 mds.0.log standby_trim_segments > 2017-02-11 01:56:04.354443 7fa79da2a700 10 mds.0.log expire_pos=6490889269 > 2017-02-11 01:56:04.354445 7fa79da2a700 10 mds.0.log segment seq=3885173 > 6490889269~1297898 > 2017-02-11 01:56:04.354447 7fa79da2a700 10 mds.0.log won't remove, not > expired! > 2017-02-11 01:56:04.354449 7fa79da2a700 20 mds.0.log removed no segments! > 2017-02-11 01:56:04.354451 7fa79da2a700 2 mds.0.0 boot_start 2: replaying > mds log > 2017-02-11 01:56:04.354455 7fa79da2a700 10 mds.0.log replay - journal empty, > done. > 2017-02-11 01:56:04.354457 7fa79da2a700 7 mds.0.cache trim max=100000 > cur=33529 > 2017-02-11 01:56:04.354463 7fa79da2a700 10 MDSInternalContextBase::complete: > 15C_MDS_BootStart > 2017-02-11 01:56:04.354465 7fa79da2a700 1 mds.0.0 replay_done (as standby) > 2017-02-11 01:56:04.354467 7fa79da2a700 10 mds.0.0 setting replay timer > 2017-02-11 01:56:04.921943 7fa79fa2e700 10 mds.beacon.na9552 _send > up:standby-replay seq 17 > 2017-02-11 01:56:04.921969 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6789/0 -- mdsbeacon(814112/na9552 up:standby-replay seq 17 > v495) v7 -- 0x7fa7b78a5d40 con 0 > 2017-02-11 01:56:04.923154 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.2 172.20.1.137:6789/0 30 ==== mdsbeacon(814112/na9552 > up:standby-replay seq 17 v495) v7 ==== 131+0+0 (3717486575 0 0) > 0x7fa7b78a5d40 con 0x7fa7b2e35000 > 2017-02-11 01:56:04.923202 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:standby-replay seq 17 rtt 0.001227 > 2017-02-11 01:56:05.354579 7fa7a022f700 10 MDSInternalContextBase::complete: > N7MDSRank26C_MDS_StandbyReplayRestartE > 2017-02-11 01:56:05.354594 7fa7a022f700 4 mds.0.0 standby_replay_restart > (as standby) > 2017-02-11 01:56:05.354657 7fa7a022f700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.0:197 2.844f3494 200.00000000 > [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b45fb180 con 0 > 2017-02-11 01:56:05.355422 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.2 172.20.1.133:6804/2623 66 ==== osd_op_reply(197 200.00000000 [read > 0~90] v0'0 uv1260 ondisk = 0) v7 ==== 132+0+90 (2300422887 0 1002082041) > 0x7fa7b45fac00 con 0x7fa7b2efd000 > 2017-02-11 01:56:05.355573 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:198 2.9430acec > 200.00000613 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b4644dc0 con 0 > 2017-02-11 01:56:05.355617 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:199 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1023) v7 -- > 0x7fa7b4645080 con 0 > 2017-02-11 01:56:05.356186 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 123 ==== osd_op_reply(198 200.00000613 > [stat] v0'0 uv554 ondisk = 0) v7 ==== 132+0+16 (981552572 0 1812931205) > 0x7fa7b4644b00 con 0x7fa7b2efe800 > 2017-02-11 01:56:05.356285 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 124 ==== osd_op_reply(199 200.00000614 > [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 > (2617829315 0 0) 0x7fa7b4644b00 con 0x7fa7b2efe800 > 2017-02-11 01:56:05.356346 7fa79da2a700 10 MDSIOContextBase::complete: > N7MDSRank32C_MDS_StandbyReplayRestartFinishE > 2017-02-11 01:56:05.356350 7fa79da2a700 10 mds.0.log standby_trim_segments > 2017-02-11 01:56:05.356352 7fa79da2a700 10 mds.0.log expire_pos=6490889269 > 2017-02-11 01:56:05.356354 7fa79da2a700 10 mds.0.log segment seq=3885173 > 6490889269~1297898 > 2017-02-11 01:56:05.356356 7fa79da2a700 10 mds.0.log won't remove, not > expired! > 2017-02-11 01:56:05.356359 7fa79da2a700 20 mds.0.log removed no segments! > 2017-02-11 01:56:05.356361 7fa79da2a700 2 mds.0.0 boot_start 2: replaying > mds log > 2017-02-11 01:56:05.356362 7fa79da2a700 10 mds.0.log replay - journal empty, > done. > 2017-02-11 01:56:05.356364 7fa79da2a700 7 mds.0.cache trim max=100000 > cur=33528 > 2017-02-11 01:56:05.356370 7fa79da2a700 10 MDSInternalContextBase::complete: > 15C_MDS_BootStart > 2017-02-11 01:56:05.356371 7fa79da2a700 1 mds.0.0 replay_done (as standby) > 2017-02-11 01:56:05.356372 7fa79da2a700 10 mds.0.0 setting replay timer > 2017-02-11 01:56:05.461915 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.2 172.20.1.137:6789/0 31 ==== osd_map(1024..1024 src has 501..1024) > v3 ==== 257+0+0 (3811605304 0 0) 0x7fa7b46406c0 con 0x7fa7b2e35000 > 2017-02-11 01:56:05.462016 7fa7a2a34700 7 mds.0.server operator(): full = 0 > epoch = 1024 > 2017-02-11 01:56:05.462039 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6789/0 -- mon_subscribe({osdmap=1025}) v2 -- 0x7fa7b2e67400 > con 0 > 2017-02-11 01:56:05.463221 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6789/0 conn(0x7fa7b2e35000 :-1 s=STATE_OPEN pgs=54 cs=1 > l=1).read_bulk peer close file descriptor 140 > 2017-02-11 01:56:05.463244 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6789/0 conn(0x7fa7b2e35000 :-1 s=STATE_OPEN pgs=54 cs=1 > l=1).read_until read failed > 2017-02-11 01:56:05.463251 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6789/0 conn(0x7fa7b2e35000 :-1 s=STATE_OPEN pgs=54 cs=1 > l=1).process read tag failed > 2017-02-11 01:56:05.463257 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6789/0 conn(0x7fa7b2e35000 :-1 s=STATE_OPEN pgs=54 cs=1 > l=1).fault on lossy channel, failing > 2017-02-11 01:56:05.463299 7fa7a2a34700 0 monclient: hunting for new mon > 2017-02-11 01:56:05.463289 7fa7a59ab700 1 -- 172.20.1.139:6800/4021830315 > reap_dead start > 2017-02-11 01:56:05.463305 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6789/0 conn(0x7fa7b2e35000 :-1 s=STATE_CLOSED pgs=54 cs=1 > l=1).mark_down > 2017-02-11 01:56:05.463357 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.135:6789/0 -- auth(proto 0 31 bytes epoch 6) v1 -- > 0x7fa7b46406c0 con 0 > 2017-02-11 01:56:05.921937 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:56:05.921986 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:56:05.925952 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b4643cc0 con 0 > 2017-02-11 01:56:06.356473 7fa7a022f700 10 MDSInternalContextBase::complete: > N7MDSRank26C_MDS_StandbyReplayRestartE > 2017-02-11 01:56:06.356486 7fa7a022f700 4 mds.0.0 standby_replay_restart > (as standby) > 2017-02-11 01:56:06.356529 7fa7a022f700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.0:200 2.844f3494 200.00000000 > [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b45fb440 con 0 > 2017-02-11 01:56:06.357093 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.2 172.20.1.133:6804/2623 67 ==== osd_op_reply(200 200.00000000 [read > 0~90] v0'0 uv1260 ondisk = 0) v7 ==== 132+0+90 (33992201 0 1002082041) > 0x7fa7b45fb440 con 0x7fa7b2efd000 > 2017-02-11 01:56:06.357253 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:201 2.9430acec > 200.00000613 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b4645340 con 0 > 2017-02-11 01:56:06.357302 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:202 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b4645600 con 0 > 2017-02-11 01:56:06.357917 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 125 ==== osd_op_reply(201 200.00000613 > [stat] v0'0 uv554 ondisk = 0) v7 ==== 132+0+16 (2979678546 0 1812931205) > 0x7fa7b4645080 con 0x7fa7b2efe800 > 2017-02-11 01:56:06.358017 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 126 ==== osd_op_reply(202 200.00000614 > [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 > (387179309 0 0) 0x7fa7b4645080 con 0x7fa7b2efe800 > 2017-02-11 01:56:06.358098 7fa79da2a700 10 MDSIOContextBase::complete: > N7MDSRank32C_MDS_StandbyReplayRestartFinishE > 2017-02-11 01:56:06.358103 7fa79da2a700 10 mds.0.log standby_trim_segments > 2017-02-11 01:56:06.358106 7fa79da2a700 10 mds.0.log expire_pos=6490889269 > 2017-02-11 01:56:06.358108 7fa79da2a700 10 mds.0.log segment seq=3885173 > 6490889269~1297898 > 2017-02-11 01:56:06.358110 7fa79da2a700 10 mds.0.log won't remove, not > expired! > 2017-02-11 01:56:06.358112 7fa79da2a700 20 mds.0.log removed no segments! > 2017-02-11 01:56:06.358114 7fa79da2a700 2 mds.0.0 boot_start 2: replaying > mds log > 2017-02-11 01:56:06.358118 7fa79da2a700 10 mds.0.log replay - journal empty, > done. > 2017-02-11 01:56:06.358121 7fa79da2a700 7 mds.0.cache trim max=100000 > cur=33527 > 2017-02-11 01:56:06.358127 7fa79da2a700 10 MDSInternalContextBase::complete: > 15C_MDS_BootStart > 2017-02-11 01:56:06.358129 7fa79da2a700 1 mds.0.0 replay_done (as standby) > 2017-02-11 01:56:06.358131 7fa79da2a700 10 mds.0.0 setting replay timer > 2017-02-11 01:56:07.358238 7fa7a022f700 10 MDSInternalContextBase::complete: > N7MDSRank26C_MDS_StandbyReplayRestartE > 2017-02-11 01:56:07.358254 7fa7a022f700 4 mds.0.0 standby_replay_restart > (as standby) > 2017-02-11 01:56:07.358298 7fa7a022f700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.0:203 2.844f3494 200.00000000 > [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b45fb700 con 0 > 2017-02-11 01:56:07.358974 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.2 172.20.1.133:6804/2623 68 ==== osd_op_reply(203 200.00000000 [read > 0~90] v0'0 uv1260 ondisk = 0) v7 ==== 132+0+90 (33992201 0 1002082041) > 0x7fa7b45fb180 con 0x7fa7b2efd000 > 2017-02-11 01:56:07.359165 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:204 2.9430acec > 200.00000613 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b46458c0 con 0 > 2017-02-11 01:56:07.359206 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:205 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b4645b80 con 0 > 2017-02-11 01:56:07.359712 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 127 ==== osd_op_reply(204 200.00000613 > [stat] v0'0 uv554 ondisk = 0) v7 ==== 132+0+16 (2979678546 0 1812931205) > 0x7fa7b4645600 con 0x7fa7b2efe800 > 2017-02-11 01:56:07.359821 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 128 ==== osd_op_reply(205 200.00000614 > [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 > (387179309 0 0) 0x7fa7b4645600 con 0x7fa7b2efe800 > 2017-02-11 01:56:07.359899 7fa79da2a700 10 MDSIOContextBase::complete: > N7MDSRank32C_MDS_StandbyReplayRestartFinishE > 2017-02-11 01:56:07.359904 7fa79da2a700 10 mds.0.log standby_trim_segments > 2017-02-11 01:56:07.359908 7fa79da2a700 10 mds.0.log expire_pos=6490889269 > 2017-02-11 01:56:07.359912 7fa79da2a700 10 mds.0.log segment seq=3885173 > 6490889269~1297898 > 2017-02-11 01:56:07.359914 7fa79da2a700 10 mds.0.log won't remove, not > expired! > 2017-02-11 01:56:07.359916 7fa79da2a700 20 mds.0.log removed no segments! > 2017-02-11 01:56:07.359918 7fa79da2a700 2 mds.0.0 boot_start 2: replaying > mds log > 2017-02-11 01:56:07.359920 7fa79da2a700 10 mds.0.log replay - journal empty, > done. > 2017-02-11 01:56:07.359925 7fa79da2a700 7 mds.0.cache trim max=100000 > cur=33526 > 2017-02-11 01:56:07.359932 7fa79da2a700 10 MDSInternalContextBase::complete: > 15C_MDS_BootStart > 2017-02-11 01:56:07.359934 7fa79da2a700 1 mds.0.0 replay_done (as standby) > 2017-02-11 01:56:07.359935 7fa79da2a700 10 mds.0.0 setting replay timer > 2017-02-11 01:56:08.360049 7fa7a022f700 10 MDSInternalContextBase::complete: > N7MDSRank26C_MDS_StandbyReplayRestartE > 2017-02-11 01:56:08.360062 7fa7a022f700 4 mds.0.0 standby_replay_restart > (as standby) > 2017-02-11 01:56:08.360092 7fa7a022f700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.0:206 2.844f3494 200.00000000 > [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b45fb9c0 con 0 > 2017-02-11 01:56:08.360779 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.2 172.20.1.133:6804/2623 69 ==== osd_op_reply(206 200.00000000 [read > 0~90] v0'0 uv1260 ondisk = 0) v7 ==== 132+0+90 (33992201 0 1002082041) > 0x7fa7b45fb9c0 con 0x7fa7b2efd000 > 2017-02-11 01:56:08.360924 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:207 2.9430acec > 200.00000613 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b4645e40 con 0 > 2017-02-11 01:56:08.360967 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:208 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b4646100 con 0 > 2017-02-11 01:56:08.361522 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 129 ==== osd_op_reply(207 200.00000613 > [stat] v0'0 uv554 ondisk = 0) v7 ==== 132+0+16 (2979678546 0 1812931205) > 0x7fa7b4645b80 con 0x7fa7b2efe800 > 2017-02-11 01:56:08.361686 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 130 ==== osd_op_reply(208 200.00000614 > [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 > (387179309 0 0) 0x7fa7b4645b80 con 0x7fa7b2efe800 > 2017-02-11 01:56:08.361763 7fa79da2a700 10 MDSIOContextBase::complete: > N7MDSRank32C_MDS_StandbyReplayRestartFinishE > 2017-02-11 01:56:08.361767 7fa79da2a700 10 mds.0.log standby_trim_segments > 2017-02-11 01:56:08.361769 7fa79da2a700 10 mds.0.log expire_pos=6490889269 > 2017-02-11 01:56:08.361771 7fa79da2a700 10 mds.0.log segment seq=3885173 > 6490889269~1297898 > 2017-02-11 01:56:08.361774 7fa79da2a700 10 mds.0.log won't remove, not > expired! > 2017-02-11 01:56:08.361775 7fa79da2a700 20 mds.0.log removed no segments! > 2017-02-11 01:56:08.361777 7fa79da2a700 2 mds.0.0 boot_start 2: replaying > mds log > 2017-02-11 01:56:08.361781 7fa79da2a700 10 mds.0.log replay - journal empty, > done. > 2017-02-11 01:56:08.361783 7fa79da2a700 7 mds.0.cache trim max=100000 > cur=33525 > 2017-02-11 01:56:08.361787 7fa79da2a700 10 MDSInternalContextBase::complete: > 15C_MDS_BootStart > 2017-02-11 01:56:08.361789 7fa79da2a700 1 mds.0.0 replay_done (as standby) > 2017-02-11 01:56:08.361790 7fa79da2a700 10 mds.0.0 setting replay timer > 2017-02-11 01:56:08.922044 7fa79fa2e700 10 mds.beacon.na9552 _send > up:standby-replay seq 18 > 2017-02-11 01:56:09.361901 7fa7a022f700 10 MDSInternalContextBase::complete: > N7MDSRank26C_MDS_StandbyReplayRestartE > 2017-02-11 01:56:09.361913 7fa7a022f700 4 mds.0.0 standby_replay_restart > (as standby) > 2017-02-11 01:56:09.361942 7fa7a022f700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.0:209 2.844f3494 200.00000000 > [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b45fbc80 con 0 > 2017-02-11 01:56:09.362515 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.2 172.20.1.133:6804/2623 70 ==== osd_op_reply(209 200.00000000 [read > 0~90] v0'0 uv1260 ondisk = 0) v7 ==== 132+0+90 (33992201 0 1002082041) > 0x7fa7b45fb700 con 0x7fa7b2efd000 > 2017-02-11 01:56:09.362657 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:210 2.9430acec > 200.00000613 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b46463c0 con 0 > 2017-02-11 01:56:09.362704 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:211 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b4646680 con 0 > 2017-02-11 01:56:09.363302 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 131 ==== osd_op_reply(210 200.00000613 > [stat] v0'0 uv554 ondisk = 0) v7 ==== 132+0+16 (2979678546 0 1812931205) > 0x7fa7b4646100 con 0x7fa7b2efe800 > 2017-02-11 01:56:09.363389 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 132 ==== osd_op_reply(211 200.00000614 > [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 > (387179309 0 0) 0x7fa7b4646100 con 0x7fa7b2efe800 > 2017-02-11 01:56:09.363484 7fa79da2a700 10 MDSIOContextBase::complete: > N7MDSRank32C_MDS_StandbyReplayRestartFinishE > 2017-02-11 01:56:09.363489 7fa79da2a700 10 mds.0.log standby_trim_segments > 2017-02-11 01:56:09.363491 7fa79da2a700 10 mds.0.log expire_pos=6490889269 > 2017-02-11 01:56:09.363493 7fa79da2a700 10 mds.0.log segment seq=3885173 > 6490889269~1297898 > 2017-02-11 01:56:09.363495 7fa79da2a700 10 mds.0.log won't remove, not > expired! > 2017-02-11 01:56:09.363497 7fa79da2a700 20 mds.0.log removed no segments! > 2017-02-11 01:56:09.363498 7fa79da2a700 2 mds.0.0 boot_start 2: replaying > mds log > 2017-02-11 01:56:09.363500 7fa79da2a700 10 mds.0.log replay - journal empty, > done. > 2017-02-11 01:56:09.363502 7fa79da2a700 7 mds.0.cache trim max=100000 > cur=33524 > 2017-02-11 01:56:09.363509 7fa79da2a700 10 MDSInternalContextBase::complete: > 15C_MDS_BootStart > 2017-02-11 01:56:09.363510 7fa79da2a700 1 mds.0.0 replay_done (as standby) > 2017-02-11 01:56:09.363515 7fa79da2a700 10 mds.0.0 setting replay timer > 2017-02-11 01:56:10.363621 7fa7a022f700 10 MDSInternalContextBase::complete: > N7MDSRank26C_MDS_StandbyReplayRestartE > 2017-02-11 01:56:10.363636 7fa7a022f700 4 mds.0.0 standby_replay_restart > (as standby) > 2017-02-11 01:56:10.363680 7fa7a022f700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.0:212 2.844f3494 200.00000000 > [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b4646c00 con 0 > 2017-02-11 01:56:10.364318 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.2 172.20.1.133:6804/2623 71 ==== osd_op_reply(212 200.00000000 [read > 0~90] v0'0 uv1260 ondisk = 0) v7 ==== 132+0+90 (33992201 0 1002082041) > 0x7fa7b4646c00 con 0x7fa7b2efd000 > 2017-02-11 01:56:10.364464 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:213 2.9430acec > 200.00000613 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b4646940 con 0 > 2017-02-11 01:56:10.364533 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:214 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b45d4000 con 0 > 2017-02-11 01:56:10.365102 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 133 ==== osd_op_reply(213 200.00000613 > [stat] v0'0 uv554 ondisk = 0) v7 ==== 132+0+16 (2979678546 0 1812931205) > 0x7fa7b45d4000 con 0x7fa7b2efe800 > 2017-02-11 01:56:10.365210 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6808/298193 134 ==== osd_op_reply(214 200.00000614 > [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 > (387179309 0 0) 0x7fa7b45d4000 con 0x7fa7b2efe800 > 2017-02-11 01:56:10.365282 7fa79da2a700 10 MDSIOContextBase::complete: > N7MDSRank32C_MDS_StandbyReplayRestartFinishE > 2017-02-11 01:56:10.365287 7fa79da2a700 10 mds.0.log standby_trim_segments > 2017-02-11 01:56:10.365288 7fa79da2a700 10 mds.0.log expire_pos=6490889269 > 2017-02-11 01:56:10.365311 7fa79da2a700 10 mds.0.log segment seq=3885173 > 6490889269~1297898 > 2017-02-11 01:56:10.365314 7fa79da2a700 10 mds.0.log won't remove, not > expired! > 2017-02-11 01:56:10.365316 7fa79da2a700 20 mds.0.log removed no segments! > 2017-02-11 01:56:10.365318 7fa79da2a700 2 mds.0.0 boot_start 2: replaying > mds log > 2017-02-11 01:56:10.365324 7fa79da2a700 10 mds.0.log replay - journal empty, > done. > 2017-02-11 01:56:10.365326 7fa79da2a700 7 mds.0.cache trim max=100000 > cur=33523 > 2017-02-11 01:56:10.365332 7fa79da2a700 10 MDSInternalContextBase::complete: > 15C_MDS_BootStart > 2017-02-11 01:56:10.365334 7fa79da2a700 1 mds.0.0 replay_done (as standby) > 2017-02-11 01:56:10.365336 7fa79da2a700 10 mds.0.0 setting replay timer > 2017-02-11 01:56:10.922046 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:56:10.922095 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:56:10.926288 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b4654000 con 0 > 2017-02-11 01:56:11.365441 7fa7a022f700 10 MDSInternalContextBase::complete: > N7MDSRank26C_MDS_StandbyReplayRestartE > 2017-02-11 01:56:11.365454 7fa7a022f700 4 mds.0.0 standby_replay_restart > (as standby) > 2017-02-11 01:56:11.365492 7fa7a022f700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.0:215 2.844f3494 200.00000000 > [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b4646ec0 con 0 > 2017-02-11 01:56:11.366107 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.2 172.20.1.133:6804/2623 72 ==== osd_op_reply(215 200.00000000 [read > 0~90] v0'0 uv1260 ondisk = 0) v7 ==== 132+0+90 (33992201 0 1002082041) > 0x7fa7b4646c00 con 0x7fa7b2efd000 > 2017-02-11 01:56:11.366244 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:216 2.9430acec > 200.00000613 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b45d42c0 con 0 > 2017-02-11 01:56:11.366283 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:217 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1024) v7 -- > 0x7fa7b45d4580 con 0 > 2017-02-11 01:56:12.648718 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6808/298193 conn(0x7fa7b2efe800 :-1 s=STATE_OPEN pgs=5 cs=1 > l=1).read_bulk peer close file descriptor 152 > 2017-02-11 01:56:12.648742 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6808/298193 conn(0x7fa7b2efe800 :-1 s=STATE_OPEN pgs=5 cs=1 > l=1).read_until read failed > 2017-02-11 01:56:12.648754 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6808/298193 conn(0x7fa7b2efe800 :-1 s=STATE_OPEN pgs=5 cs=1 > l=1).process read tag failed > 2017-02-11 01:56:12.648764 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6808/298193 conn(0x7fa7b2efe800 :-1 s=STATE_OPEN pgs=5 cs=1 > l=1).fault on lossy channel, failing > 2017-02-11 01:56:12.648859 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6808/298193 conn(0x7fa7b2efe800 :-1 s=STATE_CLOSED pgs=5 > cs=1 l=1).mark_down > 2017-02-11 01:56:12.648923 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:216 2.9430acec > 200.00000613 [stat] snapc 0=[] RETRY=1 > ack+retry+read+rwordered+known_if_redirected+full_force e1024) v7 -- > 0x7fa7bdbb6b00 con 0 > 2017-02-11 01:56:12.648959 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- osd_op(unknown.0.0:217 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] RETRY=1 > ack+retry+read+rwordered+known_if_redirected+full_force e1024) v7 -- > 0x7fa7bdbb6840 con 0 > 2017-02-11 01:56:12.649013 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6808/298193 conn(0x7fa7b2eed800 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:12.667486 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6801/298197 conn(0x7fa7b2f46000 :-1 s=STATE_OPEN pgs=4 cs=1 > l=1).read_bulk peer close file descriptor 157 > 2017-02-11 01:56:12.667510 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6801/298197 conn(0x7fa7b2f46000 :-1 s=STATE_OPEN pgs=4 cs=1 > l=1).read_until read failed > 2017-02-11 01:56:12.667517 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6801/298197 conn(0x7fa7b2f46000 :-1 s=STATE_OPEN pgs=4 cs=1 > l=1).process read tag failed > 2017-02-11 01:56:12.667524 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6801/298197 conn(0x7fa7b2f46000 :-1 s=STATE_OPEN pgs=4 cs=1 > l=1).fault on lossy channel, failing > 2017-02-11 01:56:12.667595 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6801/298197 conn(0x7fa7b2f46000 :-1 s=STATE_CLOSED pgs=4 > cs=1 l=1).mark_down > 2017-02-11 01:56:12.667724 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6801/298197 conn(0x7fa7b2efe800 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:12.668848 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6812/298187 conn(0x7fa7b2f2d800 :-1 s=STATE_OPEN pgs=4 cs=1 > l=1).read_bulk peer close file descriptor 154 > 2017-02-11 01:56:12.668872 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6812/298187 conn(0x7fa7b2f2d800 :-1 s=STATE_OPEN pgs=4 cs=1 > l=1).read_until read failed > 2017-02-11 01:56:12.668880 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6812/298187 conn(0x7fa7b2f2d800 :-1 s=STATE_OPEN pgs=4 cs=1 > l=1).process read tag failed > 2017-02-11 01:56:12.668888 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6812/298187 conn(0x7fa7b2f2d800 :-1 s=STATE_OPEN pgs=4 cs=1 > l=1).fault on lossy channel, failing > 2017-02-11 01:56:12.668975 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6812/298187 conn(0x7fa7b2f2d800 :-1 s=STATE_CLOSED pgs=4 > cs=1 l=1).mark_down > 2017-02-11 01:56:12.669108 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6812/298187 conn(0x7fa7b2f46000 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:12.849460 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6808/298193 conn(0x7fa7b2eed800 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:12.869534 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6801/298197 conn(0x7fa7b2efe800 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:12.869581 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6812/298187 conn(0x7fa7b2f46000 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:12.922140 7fa79fa2e700 10 mds.beacon.na9552 _send > up:standby-replay seq 19 > 2017-02-11 01:56:13.251572 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6808/298193 conn(0x7fa7b2eed800 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:13.270122 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6801/298197 conn(0x7fa7b2efe800 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:13.270183 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6812/298187 conn(0x7fa7b2f46000 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:13.918149 7fa7a1a32700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.135:6789/0 conn(0x7fa7b2efa000 :-1 s=STATE_OPEN pgs=100 cs=1 > l=1).mark_down > 2017-02-11 01:56:13.918275 7fa7a1a32700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6789/0 -- auth(proto 0 31 bytes epoch 6) v1 -- > 0x7fa7b4657600 con 0 > 2017-02-11 01:56:13.918361 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6789/0 conn(0x7fa7b7887800 :-1 s=STATE_CONNECTING_RE pgs=0 > cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:14.052608 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6808/298193 conn(0x7fa7b2eed800 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:14.070807 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6801/298197 conn(0x7fa7b2efe800 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:14.070849 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6812/298187 conn(0x7fa7b2f46000 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:14.119546 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6789/0 conn(0x7fa7b7887800 :-1 s=STATE_CONNECTING_RE pgs=0 > cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:14.520164 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6789/0 conn(0x7fa7b7887800 :-1 s=STATE_CONNECTING_RE pgs=0 > cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:15.321272 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6789/0 conn(0x7fa7b7887800 :-1 s=STATE_CONNECTING_RE pgs=0 > cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:15.653616 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6808/298193 conn(0x7fa7b2eed800 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:15.671828 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6801/298197 conn(0x7fa7b2efe800 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:15.671906 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6812/298187 conn(0x7fa7b2f46000 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:15.922123 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:56:15.922181 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:56:15.926601 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b4654240 con 0 > 2017-02-11 01:56:16.922251 7fa79fa2e700 10 mds.beacon.na9552 _send > up:standby-replay seq 20 > 2017-02-11 01:56:16.922417 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6789/0 conn(0x7fa7b7887800 :-1 s=STATE_CONNECTING_RE pgs=0 > cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:18.855490 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6808/298193 conn(0x7fa7b2eed800 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:18.872725 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6812/298187 conn(0x7fa7b2f46000 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:18.872775 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6801/298197 conn(0x7fa7b2efe800 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:20.123266 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6789/0 conn(0x7fa7b7887800 :-1 s=STATE_CONNECTING_RE pgs=0 > cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:20.922197 7fa7a022f700 5 mds.beacon.na9552 is_laggy > 16.000239 > 15 since last acked beacon > 2017-02-11 01:56:20.922238 7fa7a022f700 5 mds.0.0 tick bailing out since we > seem laggy > 2017-02-11 01:56:20.922362 7fa79fa2e700 10 mds.beacon.na9552 _send > up:standby-replay seq 21 > 2017-02-11 01:56:20.926890 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b4654480 con 0 > 2017-02-11 01:56:24.922445 7fa79fa2e700 10 mds.beacon.na9552 _send > up:standby-replay seq 22 > 2017-02-11 01:56:25.260820 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6808/298193 conn(0x7fa7b2eed800 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:25.278713 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6812/298187 conn(0x7fa7b2f46000 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:25.278770 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6801/298197 conn(0x7fa7b2efe800 :-1 s=STATE_CONNECTING_RE > pgs=0 cs=0 l=1)._process_connection reconnect failed > 2017-02-11 01:56:25.906252 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6808/298193 -- ping magic: 0 v1 -- 0x7fa7bdbbfa40 con 0 > 2017-02-11 01:56:25.918452 7fa7a1a32700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6789/0 conn(0x7fa7b7887800 :-1 s=STATE_CONNECTING pgs=0 cs=0 > l=1).mark_down > 2017-02-11 01:56:25.918527 7fa7a1a32700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- auth(proto 0 31 bytes epoch 6) v1 -- > 0x7fa7b4657600 con 0 > 2017-02-11 01:56:25.919504 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 > ==== 33+0+0 (4123088041 0 0) 0x7fa7b4657600 con 0x7fa7b788a800 > 2017-02-11 01:56:25.919635 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- auth(proto 2 128 bytes epoch 0) v1 -- > 0x7fa7b2d1ed00 con 0 > 2017-02-11 01:56:25.920182 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 > ==== 225+0+0 (2787284185 0 0) 0x7fa7b2d1ed00 con 0x7fa7b788a800 > 2017-02-11 01:56:25.920333 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- > mon_subscribe({mdsmap=496+,mgrmap=0+,monmap=7+,osdmap=1025}) v2 -- > 0x7fa7b2e67200 con 0 > 2017-02-11 01:56:25.920590 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 3 ==== mgrmap(e 101) v1 ==== 132+0+0 > (371604904 0 0) 0x7fa7b4641440 con 0x7fa7b788a800 > 2017-02-11 01:56:25.920855 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 4 ==== osd_map(1025..1026 src has 501..1026) > v3 ==== 23828+0+0 (2682139034 0 0) 0x7fa7b4641200 con 0x7fa7b788a800 > 2017-02-11 01:56:25.921666 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6801/298197 conn(0x7fa7b2efe800 :-1 s=STATE_CONNECTING pgs=0 > cs=0 l=1).mark_down > 2017-02-11 01:56:25.921716 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6808/298193 conn(0x7fa7b2eed800 :-1 s=STATE_CONNECTING pgs=0 > cs=0 l=1).mark_down > 2017-02-11 01:56:25.921732 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 >>> 172.20.1.137:6812/298187 conn(0x7fa7b2f46000 :-1 s=STATE_CONNECTING pgs=0 > cs=0 l=1).mark_down > 2017-02-11 01:56:25.921781 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.0:216 2.9430acec 200.00000613 > [stat] snapc 0=[] RETRY=2 > ack+retry+read+rwordered+known_if_redirected+full_force e1026) v7 -- > 0x7fa7bdbb6840 con 0 > 2017-02-11 01:56:25.921800 7fa7a59ab700 1 -- 172.20.1.139:6800/4021830315 > reap_dead start > 2017-02-11 01:56:25.921825 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- osd_op(unknown.0.0:217 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] RETRY=2 > ack+retry+read+rwordered+known_if_redirected+full_force e1026) v7 -- > 0x7fa7bdbb6b00 con 0 > 2017-02-11 01:56:25.921851 7fa7a2a34700 7 mds.0.server operator(): full = 0 > epoch = 1026 > 2017-02-11 01:56:25.921864 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mon_subscribe({osdmap=1027}) v2 -- 0x7fa7b2e69800 > con 0 > 2017-02-11 01:56:25.922271 7fa7a022f700 5 mds.beacon.na9552 is_laggy > 21.000315 > 15 since last acked beacon > 2017-02-11 01:56:25.922296 7fa7a022f700 5 mds.0.0 tick bailing out since we > seem laggy > 2017-02-11 01:56:25.927192 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46546c0 con 0 > 2017-02-11 01:56:28.922556 7fa79fa2e700 10 mds.beacon.na9552 _send > up:standby-replay seq 23 > 2017-02-11 01:56:28.922582 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:standby-replay seq 23 > v495) v7 -- 0x7fa7b78a6a40 con 0 > 2017-02-11 01:56:28.923098 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 5 ==== mdsbeacon(814112/na9552 > up:standby-replay seq 23 v495) v7 ==== 131+0+0 (885639794 0 0) > 0x7fa7b78a6a40 con 0x7fa7b788a800 > 2017-02-11 01:56:28.923132 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:standby-replay seq 23 rtt 0.000561 > 2017-02-11 01:56:28.923136 7fa7a2a34700 0 mds.beacon.na9552 > handle_mds_beacon no longer laggy > 2017-02-11 01:56:30.922442 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:56:30.922484 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:56:30.927481 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b4654900 con 0 > 2017-02-11 01:56:32.922663 7fa79fa2e700 10 mds.beacon.na9552 _send > up:standby-replay seq 24 > 2017-02-11 01:56:32.922690 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:standby-replay seq 24 > v495) v7 -- 0x7fa7b78a6d80 con 0 > 2017-02-11 01:56:32.923215 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 6 ==== mdsbeacon(814112/na9552 > up:standby-replay seq 24 v495) v7 ==== 131+0+0 (365326863 0 0) > 0x7fa7b78a6d80 con 0x7fa7b788a800 > 2017-02-11 01:56:32.923246 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:standby-replay seq 24 rtt 0.000568 > 2017-02-11 01:56:35.922551 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:56:35.922597 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:56:35.925655 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 7 ==== mgrmap(e 102) v1 ==== 100+0+0 > (807606552 0 0) 0x7fa7b4640fc0 con 0x7fa7b788a800 > 2017-02-11 01:56:35.927779 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b4654b40 con 0 > 2017-02-11 01:56:36.922733 7fa79fa2e700 10 mds.beacon.na9552 _send > up:standby-replay seq 25 > 2017-02-11 01:56:36.922758 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:standby-replay seq 25 > v495) v7 -- 0x7fa7b78a70c0 con 0 > 2017-02-11 01:56:36.923218 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 8 ==== mdsbeacon(814112/na9552 > up:standby-replay seq 25 v495) v7 ==== 131+0+0 (1207033501 0 0) > 0x7fa7b78a70c0 con 0x7fa7b788a800 > 2017-02-11 01:56:36.923249 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:standby-replay seq 25 rtt 0.000502 > 2017-02-11 01:56:40.906662 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbbf880 con 0 > 2017-02-11 01:56:40.906705 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc0680 con 0 > 2017-02-11 01:56:40.922653 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:56:40.922697 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:56:40.922836 7fa79fa2e700 10 mds.beacon.na9552 _send > up:standby-replay seq 26 > 2017-02-11 01:56:40.922867 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:standby-replay seq 26 > v495) v7 -- 0x7fa7b2ed5400 con 0 > 2017-02-11 01:56:40.923328 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 9 ==== mdsbeacon(814112/na9552 > up:standby-replay seq 26 v495) v7 ==== 131+0+0 (2980653867 0 0) > 0x7fa7b2ed5400 con 0x7fa7b788a800 > 2017-02-11 01:56:40.923360 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:standby-replay seq 26 rtt 0.000507 > 2017-02-11 01:56:40.928059 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b4654d80 con 0 > 2017-02-11 01:56:40.929127 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 10 ==== osd_map(1027..1027 src has 501..1027) > v3 ==== 257+0+0 (1436388920 0 0) 0x7fa7b4640d80 con 0x7fa7b788a800 > 2017-02-11 01:56:40.929221 7fa7a2a34700 7 mds.0.server operator(): full = 0 > epoch = 1027 > 2017-02-11 01:56:40.929247 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mon_subscribe({osdmap=1028}) v2 -- 0x7fa7b2e66800 > con 0 > 2017-02-11 01:56:40.934884 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 11 ==== mdsmap(e 496) v1 ==== 585+0+0 > (1092516299 0 0) 0x7fa7b7894000 con 0x7fa7b788a800 > 2017-02-11 01:56:40.934914 7fa7a2a34700 5 mds.na9552 handle_mds_map epoch > 496 from mon.0 > 2017-02-11 01:56:40.934948 7fa7a2a34700 10 mds.na9552 my compat > compat={},rocompat={},incompat={1=base v0.20,2=client writeable > ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds > uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline > data,8=file layout v2} > 2017-02-11 01:56:40.934957 7fa7a2a34700 10 mds.na9552 mdsmap compat > compat={},rocompat={},incompat={1=base v0.20,2=client writeable > ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds > uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} > 2017-02-11 01:56:40.934962 7fa7a2a34700 10 mds.na9552 peer mds gid 664254 > removed from map > 2017-02-11 01:56:40.934972 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > mark_down 172.20.1.137:6800/3184595269 -- connection dne > 2017-02-11 01:56:40.934978 7fa7a2a34700 10 mds.na9552 map says i am > 172.20.1.139:6800/4021830315 mds.0.496 state up:replay > 2017-02-11 01:56:40.934983 7fa7a2a34700 10 mds.na9552 handle_mds_map: > handling map as rank 0 > 2017-02-11 01:56:40.934985 7fa7a2a34700 1 mds.0.496 handle_mds_map i am now > mds.0.496 > 2017-02-11 01:56:40.934987 7fa7a2a34700 1 mds.0.496 handle_mds_map state > change up:standby-replay --> up:replay > 2017-02-11 01:56:40.934997 7fa7a2a34700 10 mds.beacon.na9552 set_want_state: > up:standby-replay -> up:replay > 2017-02-11 01:56:40.934999 7fa7a2a34700 10 mds.0.496 Monitor activated us! > Deactivating replay loop > 2017-02-11 01:56:40.935003 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > mark_down 172.20.1.137:6800/3184595269 -- connection dne > 2017-02-11 01:56:40.935010 7fa7a2a34700 5 mds.0.496 handle_mds_failure for > myself; not doing anything > 2017-02-11 01:56:44.922943 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 27 > 2017-02-11 01:56:44.922974 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 27 v496) v7 > -- 0x7fa7b78a6080 con 0 > 2017-02-11 01:56:44.923574 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 12 ==== mdsbeacon(814112/na9552 up:replay seq > 27 v496) v7 ==== 131+0+0 (705595652 0 0) 0x7fa7b78a6080 con 0x7fa7b788a800 > 2017-02-11 01:56:44.923607 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 27 rtt 0.000650 > 2017-02-11 01:56:45.906830 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc0840 con 0 > 2017-02-11 01:56:45.906862 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc0a00 con 0 > 2017-02-11 01:56:45.922746 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:56:45.922786 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:56:45.928344 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b4657600 con 0 > 2017-02-11 01:56:48.923051 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 28 > 2017-02-11 01:56:48.923076 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 28 v496) v7 > -- 0x7fa7b78a63c0 con 0 > 2017-02-11 01:56:48.923593 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 13 ==== mdsbeacon(814112/na9552 up:replay seq > 28 v496) v7 ==== 131+0+0 (2439297035 0 0) 0x7fa7b78a63c0 con 0x7fa7b788a800 > 2017-02-11 01:56:48.923628 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 28 rtt 0.000559 > 2017-02-11 01:56:50.906985 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc0bc0 con 0 > 2017-02-11 01:56:50.907034 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc0d80 con 0 > 2017-02-11 01:56:50.922844 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:56:50.922883 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:56:50.928611 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b4641440 con 0 > 2017-02-11 01:56:52.923151 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 29 > 2017-02-11 01:56:52.923177 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 29 v496) v7 > -- 0x7fa7b78a6700 con 0 > 2017-02-11 01:56:52.923661 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 14 ==== mdsbeacon(814112/na9552 up:replay seq > 29 v496) v7 ==== 131+0+0 (3277001881 0 0) 0x7fa7b78a6700 con 0x7fa7b788a800 > 2017-02-11 01:56:52.923695 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 29 rtt 0.000529 > 2017-02-11 01:56:55.907166 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc0f40 con 0 > 2017-02-11 01:56:55.907197 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc1100 con 0 > 2017-02-11 01:56:55.922955 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:56:55.922993 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:56:55.928894 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b2d1f180 con 0 > 2017-02-11 01:56:56.923254 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 30 > 2017-02-11 01:56:56.923279 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 30 v496) v7 > -- 0x7fa7b78a7400 con 0 > 2017-02-11 01:56:56.923776 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 15 ==== mdsbeacon(814112/na9552 up:replay seq > 30 v496) v7 ==== 131+0+0 (889970991 0 0) 0x7fa7b78a7400 con 0x7fa7b788a800 > 2017-02-11 01:56:56.923808 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 30 rtt 0.000541 > 2017-02-11 01:57:00.907323 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc12c0 con 0 > 2017-02-11 01:57:00.907356 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbbf880 con 0 > 2017-02-11 01:57:00.923051 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:57:00.923088 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:57:00.923314 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 31 > 2017-02-11 01:57:00.923340 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 31 v496) v7 > -- 0x7fa7b78a7740 con 0 > 2017-02-11 01:57:00.923796 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 16 ==== mdsbeacon(814112/na9552 up:replay seq > 31 v496) v7 ==== 131+0+0 (1732005309 0 0) 0x7fa7b78a7740 con 0x7fa7b788a800 > 2017-02-11 01:57:00.923828 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 31 rtt 0.000499 > 2017-02-11 01:57:00.929161 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b2d1d200 con 0 > 2017-02-11 01:57:04.923421 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 32 > 2017-02-11 01:57:04.923445 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 32 v496) v7 > -- 0x7fa7b78a7a80 con 0 > 2017-02-11 01:57:04.923963 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 17 ==== mdsbeacon(814112/na9552 up:replay seq > 32 v496) v7 ==== 131+0+0 (358242815 0 0) 0x7fa7b78a7a80 con 0x7fa7b788a800 > 2017-02-11 01:57:04.923997 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 32 rtt 0.000562 > 2017-02-11 01:57:05.907491 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc0680 con 0 > 2017-02-11 01:57:05.907523 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc0840 con 0 > 2017-02-11 01:57:05.923154 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:57:05.923191 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:57:05.929380 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b2d1cfc0 con 0 > 2017-02-11 01:57:08.923521 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 33 > 2017-02-11 01:57:08.923566 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 33 v496) v7 > -- 0x7fa7b45e0000 con 0 > 2017-02-11 01:57:08.924168 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 18 ==== mdsbeacon(814112/na9552 up:replay seq > 33 v496) v7 ==== 131+0+0 (1198388589 0 0) 0x7fa7b45e0000 con 0x7fa7b788a800 > 2017-02-11 01:57:08.924201 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 33 rtt 0.000666 > 2017-02-11 01:57:10.907662 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc0a00 con 0 > 2017-02-11 01:57:10.907702 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc0bc0 con 0 > 2017-02-11 01:57:10.923292 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:57:10.923337 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:57:10.929701 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b2d1d440 con 0 > 2017-02-11 01:57:12.923630 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 34 > 2017-02-11 01:57:12.923657 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 34 v496) v7 > -- 0x7fa7b45e0340 con 0 > 2017-02-11 01:57:12.924198 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 19 ==== mdsbeacon(814112/na9552 up:replay seq > 34 v496) v7 ==== 131+0+0 (2973040859 0 0) 0x7fa7b45e0340 con 0x7fa7b788a800 > 2017-02-11 01:57:12.924238 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 34 rtt 0.000593 > 2017-02-11 01:57:15.907865 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc0d80 con 0 > 2017-02-11 01:57:15.907897 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc0f40 con 0 > 2017-02-11 01:57:15.923417 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:57:15.923456 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:57:15.929948 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b2d1cb40 con 0 > 2017-02-11 01:57:16.923701 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 35 > 2017-02-11 01:57:16.923726 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 35 v496) v7 > -- 0x7fa7b45e0680 con 0 > 2017-02-11 01:57:16.924247 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 20 ==== mdsbeacon(814112/na9552 up:replay seq > 35 v496) v7 ==== 131+0+0 (3808603209 0 0) 0x7fa7b45e0680 con 0x7fa7b788a800 > 2017-02-11 01:57:16.924281 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 35 rtt 0.000566 > 2017-02-11 01:57:20.908061 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc1100 con 0 > 2017-02-11 01:57:20.908092 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc1480 con 0 > 2017-02-11 01:57:20.923507 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:57:20.923545 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:57:20.923799 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 36 > 2017-02-11 01:57:20.923831 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 36 v496) v7 > -- 0x7fa7b45e09c0 con 0 > 2017-02-11 01:57:20.924360 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 21 ==== mdsbeacon(814112/na9552 up:replay seq > 36 v496) v7 ==== 131+0+0 (1483251014 0 0) 0x7fa7b45e09c0 con 0x7fa7b788a800 > 2017-02-11 01:57:20.924406 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 36 rtt 0.000592 > 2017-02-11 01:57:20.930216 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b2d1ed00 con 0 > 2017-02-11 01:57:24.923866 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 37 > 2017-02-11 01:57:24.923890 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 37 v496) v7 > -- 0x7fa7b45e0d00 con 0 > 2017-02-11 01:57:24.924462 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 22 ==== mdsbeacon(814112/na9552 up:replay seq > 37 v496) v7 ==== 131+0+0 (174010836 0 0) 0x7fa7b45e0d00 con 0x7fa7b788a800 > 2017-02-11 01:57:24.924497 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 37 rtt 0.000616 > 2017-02-11 01:57:25.908242 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc1640 con 0 > 2017-02-11 01:57:25.908272 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc1800 con 0 > 2017-02-11 01:57:25.923609 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:57:25.923647 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:57:25.930524 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b2f2ad00 con 0 > 2017-02-11 01:57:28.923936 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 38 > 2017-02-11 01:57:28.923961 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 38 v496) v7 > -- 0x7fa7b45e1040 con 0 > 2017-02-11 01:57:28.924498 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 23 ==== mdsbeacon(814112/na9552 up:replay seq > 38 v496) v7 ==== 131+0+0 (4228367458 0 0) 0x7fa7b45e1040 con 0x7fa7b788a800 > 2017-02-11 01:57:28.924532 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 38 rtt 0.000582 > 2017-02-11 01:57:30.908422 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc12c0 con 0 > 2017-02-11 01:57:30.908454 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbbf880 con 0 > 2017-02-11 01:57:30.923680 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:57:30.923717 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:57:30.930820 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b2f29680 con 0 > 2017-02-11 01:57:32.924049 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 39 > 2017-02-11 01:57:32.924090 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 39 v496) v7 > -- 0x7fa7b45e1380 con 0 > 2017-02-11 01:57:32.924663 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 24 ==== mdsbeacon(814112/na9552 up:replay seq > 39 v496) v7 ==== 131+0+0 (2922408176 0 0) 0x7fa7b45e1380 con 0x7fa7b788a800 > 2017-02-11 01:57:32.924698 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 39 rtt 0.000628 > 2017-02-11 01:57:35.908601 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc0680 con 0 > 2017-02-11 01:57:35.908633 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc0840 con 0 > 2017-02-11 01:57:35.923801 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:57:35.923845 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:57:35.931121 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b2f29200 con 0 > 2017-02-11 01:57:36.924187 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 40 > 2017-02-11 01:57:36.924213 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 40 v496) v7 > -- 0x7fa7b45e16c0 con 0 > 2017-02-11 01:57:36.924791 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 25 ==== mdsbeacon(814112/na9552 up:replay seq > 40 v496) v7 ==== 131+0+0 (2403324045 0 0) 0x7fa7b45e16c0 con 0x7fa7b788a800 > 2017-02-11 01:57:36.924825 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 40 rtt 0.000623 > 2017-02-11 01:57:40.908776 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc0a00 con 0 > 2017-02-11 01:57:40.908806 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc0bc0 con 0 > 2017-02-11 01:57:40.923917 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:57:40.923955 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:57:40.924288 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 41 > 2017-02-11 01:57:40.924312 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 41 v496) v7 > -- 0x7fa7b45e1a00 con 0 > 2017-02-11 01:57:40.924878 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 26 ==== mdsbeacon(814112/na9552 up:replay seq > 41 v496) v7 ==== 131+0+0 (3708320799 0 0) 0x7fa7b45e1a00 con 0x7fa7b788a800 > 2017-02-11 01:57:40.924914 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 41 rtt 0.000612 > 2017-02-11 01:57:40.931439 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b2f29440 con 0 > 2017-02-11 01:57:44.924372 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 42 > 2017-02-11 01:57:44.924407 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 42 v496) v7 > -- 0x7fa7b45e1d40 con 0 > 2017-02-11 01:57:44.924926 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 27 ==== mdsbeacon(814112/na9552 up:replay seq > 42 v496) v7 ==== 131+0+0 (726698409 0 0) 0x7fa7b45e1d40 con 0x7fa7b788a800 > 2017-02-11 01:57:44.924960 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 42 rtt 0.000573 > 2017-02-11 01:57:45.908914 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc0d80 con 0 > 2017-02-11 01:57:45.908946 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc0f40 con 0 > 2017-02-11 01:57:45.924020 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:57:45.924059 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:57:45.931718 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b2f28b40 con 0 > 2017-02-11 01:57:48.924479 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 43 > 2017-02-11 01:57:48.924507 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 43 v496) v7 > -- 0x7fa7b45e2080 con 0 > 2017-02-11 01:57:48.925042 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 28 ==== mdsbeacon(814112/na9552 up:replay seq > 43 v496) v7 ==== 131+0+0 (2036802875 0 0) 0x7fa7b45e2080 con 0x7fa7b788a800 > 2017-02-11 01:57:48.925075 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 43 rtt 0.000581 > 2017-02-11 01:57:50.909077 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc1100 con 0 > 2017-02-11 01:57:50.909107 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc19c0 con 0 > 2017-02-11 01:57:50.924156 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:57:50.924196 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:57:50.932038 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b2f29f80 con 0 > 2017-02-11 01:57:52.924592 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 44 > 2017-02-11 01:57:52.924623 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 44 v496) v7 > -- 0x7fa7b45e23c0 con 0 > 2017-02-11 01:57:52.925218 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 29 ==== mdsbeacon(814112/na9552 up:replay seq > 44 v496) v7 ==== 131+0+0 (3255637044 0 0) 0x7fa7b45e23c0 con 0x7fa7b788a800 > 2017-02-11 01:57:52.925255 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 44 rtt 0.000642 > 2017-02-11 01:57:55.909253 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc1b80 con 0 > 2017-02-11 01:57:55.909293 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc1d40 con 0 > 2017-02-11 01:57:55.924296 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:57:55.924338 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:57:55.932362 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b2f29d40 con 0 > 2017-02-11 01:57:56.924700 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 45 > 2017-02-11 01:57:56.924729 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 45 v496) v7 > -- 0x7fa7b45e2700 con 0 > 2017-02-11 01:57:56.925326 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 30 ==== mdsbeacon(814112/na9552 up:replay seq > 45 v496) v7 ==== 131+0+0 (2419767462 0 0) 0x7fa7b45e2700 con 0x7fa7b788a800 > 2017-02-11 01:57:56.925362 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 45 rtt 0.000648 > 2017-02-11 01:58:00.909425 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc1480 con 0 > 2017-02-11 01:58:00.909455 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc1640 con 0 > 2017-02-11 01:58:00.924382 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:58:00.924446 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:58:00.924802 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 46 > 2017-02-11 01:58:00.924830 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 46 v496) v7 > -- 0x7fa7b45e2a40 con 0 > 2017-02-11 01:58:00.925351 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 31 ==== mdsbeacon(814112/na9552 up:replay seq > 46 v496) v7 ==== 131+0+0 (1717718288 0 0) 0x7fa7b45e2a40 con 0x7fa7b788a800 > 2017-02-11 01:58:00.925384 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 46 rtt 0.000568 > 2017-02-11 01:58:00.932644 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b2f28240 con 0 > 2017-02-11 01:58:04.924918 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 47 > 2017-02-11 01:58:04.924952 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 47 v496) v7 > -- 0x7fa7b45e2d80 con 0 > 2017-02-11 01:58:04.925585 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 32 ==== mdsbeacon(814112/na9552 up:replay seq > 47 v496) v7 ==== 131+0+0 (878043522 0 0) 0x7fa7b45e2d80 con 0x7fa7b788a800 > 2017-02-11 01:58:04.925621 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 47 rtt 0.000685 > 2017-02-11 01:58:05.909584 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc1800 con 0 > 2017-02-11 01:58:05.909630 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc12c0 con 0 > 2017-02-11 01:58:05.924484 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:58:05.924526 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:58:05.932933 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b2f286c0 con 0 > 2017-02-11 01:58:08.925047 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 48 > 2017-02-11 01:58:08.925081 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 48 v496) v7 > -- 0x7fa7b45e30c0 con 0 > 2017-02-11 01:58:08.925585 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 33 ==== mdsbeacon(814112/na9552 up:replay seq > 48 v496) v7 ==== 131+0+0 (612187626 0 0) 0x7fa7b45e30c0 con 0x7fa7b788a800 > 2017-02-11 01:58:08.925621 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 48 rtt 0.000554 > 2017-02-11 01:58:10.909752 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbbf880 con 0 > 2017-02-11 01:58:10.909784 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc0680 con 0 > 2017-02-11 01:58:10.924584 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:58:10.924626 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:58:10.933285 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46a2000 con 0 > 2017-02-11 01:58:12.925125 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 49 > 2017-02-11 01:58:12.925154 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 49 v496) v7 > -- 0x7fa7b45e3400 con 0 > 2017-02-11 01:58:12.925650 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 34 ==== mdsbeacon(814112/na9552 up:replay seq > 49 v496) v7 ==== 131+0+0 (1984598392 0 0) 0x7fa7b45e3400 con 0x7fa7b788a800 > 2017-02-11 01:58:12.925685 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 49 rtt 0.000543 > 2017-02-11 01:58:15.909883 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc0840 con 0 > 2017-02-11 01:58:15.909920 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc0a00 con 0 > 2017-02-11 01:58:15.924690 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:58:15.924736 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:58:15.933600 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46a2240 con 0 > 2017-02-11 01:58:16.925225 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 50 > 2017-02-11 01:58:16.925258 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 50 v496) v7 > -- 0x7fa7b45e3740 con 0 > 2017-02-11 01:58:16.925830 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 35 ==== mdsbeacon(814112/na9552 up:replay seq > 50 v496) v7 ==== 131+0+0 (2148689102 0 0) 0x7fa7b45e3740 con 0x7fa7b788a800 > 2017-02-11 01:58:16.925876 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 50 rtt 0.000635 > 2017-02-11 01:58:20.910068 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc0bc0 con 0 > 2017-02-11 01:58:20.910102 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc0f40 con 0 > 2017-02-11 01:58:20.924789 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:58:20.924828 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:58:20.925309 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 51 > 2017-02-11 01:58:20.925333 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 51 v496) v7 > -- 0x7fa7b45e3a80 con 0 > 2017-02-11 01:58:20.925814 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 36 ==== mdsbeacon(814112/na9552 up:replay seq > 51 v496) v7 ==== 131+0+0 (3525691484 0 0) 0x7fa7b45e3a80 con 0x7fa7b788a800 > 2017-02-11 01:58:20.925847 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 51 rtt 0.000524 > 2017-02-11 01:58:20.933883 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46a2480 con 0 > 2017-02-11 01:58:24.925419 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 52 > 2017-02-11 01:58:24.925487 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 52 v496) v7 > -- 0x7fa7b46aa000 con 0 > 2017-02-11 01:58:24.926131 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 37 ==== mdsbeacon(814112/na9552 up:replay seq > 52 v496) v7 ==== 131+0+0 (1766848851 0 0) 0x7fa7b46aa000 con 0x7fa7b788a800 > 2017-02-11 01:58:24.926167 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 52 rtt 0.000728 > 2017-02-11 01:58:25.910202 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc1100 con 0 > 2017-02-11 01:58:25.910234 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc19c0 con 0 > 2017-02-11 01:58:25.924939 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:58:25.924983 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:58:25.934194 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46a26c0 con 0 > 2017-02-11 01:58:28.925518 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 53 > 2017-02-11 01:58:28.925558 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 53 v496) v7 > -- 0x7fa7b46aa340 con 0 > 2017-02-11 01:58:28.926128 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 38 ==== mdsbeacon(814112/na9552 up:replay seq > 53 v496) v7 ==== 131+0+0 (997742017 0 0) 0x7fa7b46aa340 con 0x7fa7b788a800 > 2017-02-11 01:58:28.926164 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 53 rtt 0.000630 > 2017-02-11 01:58:30.910382 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc1b80 con 0 > 2017-02-11 01:58:30.910431 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc1d40 con 0 > 2017-02-11 01:58:30.925053 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:58:30.925093 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:58:30.934524 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46a2900 con 0 > 2017-02-11 01:58:32.925618 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 54 > 2017-02-11 01:58:32.925652 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 54 v496) v7 > -- 0x7fa7b46aa680 con 0 > 2017-02-11 01:58:32.926224 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 39 ==== mdsbeacon(814112/na9552 up:replay seq > 54 v496) v7 ==== 131+0+0 (3441467511 0 0) 0x7fa7b46aa680 con 0x7fa7b788a800 > 2017-02-11 01:58:32.926260 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 54 rtt 0.000619 > 2017-02-11 01:58:35.910569 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc1480 con 0 > 2017-02-11 01:58:35.910613 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc1640 con 0 > 2017-02-11 01:58:35.925173 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:58:35.925218 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:58:35.934838 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46a2b40 con 0 > 2017-02-11 01:58:36.925720 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 55 > 2017-02-11 01:58:36.925750 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 55 v496) v7 > -- 0x7fa7b46aa9c0 con 0 > 2017-02-11 01:58:36.926376 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 40 ==== mdsbeacon(814112/na9552 up:replay seq > 55 v496) v7 ==== 131+0+0 (2669087973 0 0) 0x7fa7b46aa9c0 con 0x7fa7b788a800 > 2017-02-11 01:58:36.926428 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 55 rtt 0.000691 > 2017-02-11 01:58:40.910742 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc1800 con 0 > 2017-02-11 01:58:40.910777 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc12c0 con 0 > 2017-02-11 01:58:40.925246 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:58:40.925290 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:58:40.925809 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 56 > 2017-02-11 01:58:40.925840 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 56 v496) v7 > -- 0x7fa7b46aad00 con 0 > 2017-02-11 01:58:40.926382 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 41 ==== mdsbeacon(814112/na9552 up:replay seq > 56 v496) v7 ==== 131+0+0 (3189290136 0 0) 0x7fa7b46aad00 con 0x7fa7b788a800 > 2017-02-11 01:58:40.926428 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 56 rtt 0.000602 > 2017-02-11 01:58:40.935139 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46a2d80 con 0 > 2017-02-11 01:58:44.925920 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 57 > 2017-02-11 01:58:44.925960 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 57 v496) v7 > -- 0x7fa7b46ab040 con 0 > 2017-02-11 01:58:44.926556 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 42 ==== mdsbeacon(814112/na9552 up:replay seq > 57 v496) v7 ==== 131+0+0 (3962509322 0 0) 0x7fa7b46ab040 con 0x7fa7b788a800 > 2017-02-11 01:58:44.926587 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 57 rtt 0.000647 > 2017-02-11 01:58:45.910880 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbbf880 con 0 > 2017-02-11 01:58:45.910913 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc0840 con 0 > 2017-02-11 01:58:45.925358 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:58:45.925428 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:58:45.935458 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46a2fc0 con 0 > 2017-02-11 01:58:48.926021 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 58 > 2017-02-11 01:58:48.926047 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 58 v496) v7 > -- 0x7fa7b46ab380 con 0 > 2017-02-11 01:58:48.926595 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 43 ==== mdsbeacon(814112/na9552 up:replay seq > 58 v496) v7 ==== 131+0+0 (444067260 0 0) 0x7fa7b46ab380 con 0x7fa7b788a800 > 2017-02-11 01:58:48.926629 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 58 rtt 0.000594 > 2017-02-11 01:58:50.911055 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc0a00 con 0 > 2017-02-11 01:58:50.911086 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc0bc0 con 0 > 2017-02-11 01:58:50.925460 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:58:50.925507 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:58:50.935785 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46a3200 con 0 > 2017-02-11 01:58:52.926132 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 59 > 2017-02-11 01:58:52.926157 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 59 v496) v7 > -- 0x7fa7b46ab6c0 con 0 > 2017-02-11 01:58:52.926675 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 44 ==== mdsbeacon(814112/na9552 up:replay seq > 59 v496) v7 ==== 131+0+0 (1212170542 0 0) 0x7fa7b46ab6c0 con 0x7fa7b788a800 > 2017-02-11 01:58:52.926709 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 59 rtt 0.000562 > 2017-02-11 01:58:55.911231 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc0f40 con 0 > 2017-02-11 01:58:55.911277 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc1100 con 0 > 2017-02-11 01:58:55.925558 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:58:55.925594 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:58:55.936098 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46a3440 con 0 > 2017-02-11 01:58:56.926243 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 60 > 2017-02-11 01:58:56.926270 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 60 v496) v7 > -- 0x7fa7b46aba00 con 0 > 2017-02-11 01:58:56.926820 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 45 ==== mdsbeacon(814112/na9552 up:replay seq > 60 v496) v7 ==== 131+0+0 (4079644705 0 0) 0x7fa7b46aba00 con 0x7fa7b788a800 > 2017-02-11 01:58:56.926856 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 60 rtt 0.000597 > 2017-02-11 01:59:00.911404 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc19c0 con 0 > 2017-02-11 01:59:00.911434 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc1b80 con 0 > 2017-02-11 01:59:00.925629 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:59:00.925667 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:59:00.926327 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 61 > 2017-02-11 01:59:00.926358 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 61 v496) v7 > -- 0x7fa7b46abd40 con 0 > 2017-02-11 01:59:00.926925 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 46 ==== mdsbeacon(814112/na9552 up:replay seq > 61 v496) v7 ==== 131+0+0 (2703088819 0 0) 0x7fa7b46abd40 con 0x7fa7b788a800 > 2017-02-11 01:59:00.926958 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 61 rtt 0.000611 > 2017-02-11 01:59:00.936377 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46a3680 con 0 > 2017-02-11 01:59:04.926443 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 62 > 2017-02-11 01:59:04.926476 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 62 v496) v7 > -- 0x7fa7b46ac080 con 0 > 2017-02-11 01:59:04.927081 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 47 ==== mdsbeacon(814112/na9552 up:replay seq > 62 v496) v7 ==== 131+0+0 (1464150277 0 0) 0x7fa7b46ac080 con 0x7fa7b788a800 > 2017-02-11 01:59:04.927114 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 62 rtt 0.000652 > 2017-02-11 01:59:05.911601 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc1d40 con 0 > 2017-02-11 01:59:05.911643 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc1480 con 0 > 2017-02-11 01:59:05.925758 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:59:05.925821 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:59:05.936697 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46a38c0 con 0 > 2017-02-11 01:59:08.926558 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 63 > 2017-02-11 01:59:08.926594 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 63 v496) v7 > -- 0x7fa7b46ac3c0 con 0 > 2017-02-11 01:59:08.927123 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 48 ==== mdsbeacon(814112/na9552 up:replay seq > 63 v496) v7 ==== 131+0+0 (91391383 0 0) 0x7fa7b46ac3c0 con 0x7fa7b788a800 > 2017-02-11 01:59:08.927160 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 63 rtt 0.000583 > 2017-02-11 01:59:10.911770 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc1640 con 0 > 2017-02-11 01:59:10.911800 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc1800 con 0 > 2017-02-11 01:59:10.925857 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:59:10.925900 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:59:10.937009 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46a3b00 con 0 > 2017-02-11 01:59:12.926663 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 64 > 2017-02-11 01:59:12.926688 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 64 v496) v7 > -- 0x7fa7b46ac700 con 0 > 2017-02-11 01:59:12.927267 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 49 ==== mdsbeacon(814112/na9552 up:replay seq > 64 v496) v7 ==== 131+0+0 (3012100481 0 0) 0x7fa7b46ac700 con 0x7fa7b788a800 > 2017-02-11 01:59:12.927301 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 64 rtt 0.000624 > 2017-02-11 01:59:15.911958 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc12c0 con 0 > 2017-02-11 01:59:15.911996 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc0680 con 0 > 2017-02-11 01:59:15.925975 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:59:15.926018 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:59:15.937328 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46a3d40 con 0 > 2017-02-11 01:59:16.926768 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 65 > 2017-02-11 01:59:16.926797 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 65 v496) v7 > -- 0x7fa7b46aca40 con 0 > 2017-02-11 01:59:16.927380 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 50 ==== mdsbeacon(814112/na9552 up:replay seq > 65 v496) v7 ==== 131+0+0 (3787369747 0 0) 0x7fa7b46aca40 con 0x7fa7b788a800 > 2017-02-11 01:59:16.927442 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 65 rtt 0.000658 > 2017-02-11 01:59:20.912127 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc0d80 con 0 > 2017-02-11 01:59:20.912159 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbbf880 con 0 > 2017-02-11 01:59:20.926071 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:59:20.926109 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:59:20.926874 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 66 > 2017-02-11 01:59:20.926904 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 66 v496) v7 > -- 0x7fa7b46acd80 con 0 > 2017-02-11 01:59:20.927484 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 51 ==== mdsbeacon(814112/na9552 up:replay seq > 66 v496) v7 ==== 131+0+0 (400971941 0 0) 0x7fa7b46acd80 con 0x7fa7b788a800 > 2017-02-11 01:59:20.927518 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 66 rtt 0.000625 > 2017-02-11 01:59:20.937614 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46a3f80 con 0 > 2017-02-11 01:59:24.926975 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 67 > 2017-02-11 01:59:24.927000 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 67 v496) v7 > -- 0x7fa7b46ad0c0 con 0 > 2017-02-11 01:59:24.927513 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 52 ==== mdsbeacon(814112/na9552 up:replay seq > 67 v496) v7 ==== 131+0+0 (1171387447 0 0) 0x7fa7b46ad0c0 con 0x7fa7b788a800 > 2017-02-11 01:59:24.927542 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 67 rtt 0.000553 > 2017-02-11 01:59:25.912280 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7bdbc0840 con 0 > 2017-02-11 01:59:25.912309 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7bdbc0a00 con 0 > 2017-02-11 01:59:25.926175 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0> > 2017-02-11 01:59:25.926213 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 01:59:25.937889 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b46a41c0 con 0 > 2017-02-11 01:59:28.927073 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 68 > 2017-02-11 01:59:28.927101 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 68 v496) v7 > -- 0x7fa7b46ad400 con 0 > 2017-02-11 01:59:28.927683 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 53 ==== mdsbeacon(814112/na9552 up:replay seq > 68 v496) v7 ==== 131+0+0 (4273718584 0 0) 0x7fa7b46ad400 con 0x7fa7b788a800 > 2017-02-11 01:59:28.927713 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 68 rtt 0.000626 > > > After starting the dead mds instance again almost immediately springs back > to life: > 2017-02-11 02:08:44.941573 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 207 > 2017-02-11 02:08:44.941602 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 207 v496) > v7 -- 0x7fa7b475e680 con 0 > 2017-02-11 02:08:44.942165 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 192 ==== mdsbeacon(814112/na9552 up:replay seq > 207 v496) v7 ==== 131+0+0 (508686245 0 0) 0x7fa7b475e680 con 0x7fa7b788a800 > 2017-02-11 02:08:44.942197 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 207 rtt 0.000608 > 2017-02-11 02:08:45.930998 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7b46f8d80 con 0 > 2017-02-11 02:08:45.931028 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7b46f6380 con 0 > 2017-02-11 02:08:45.937747 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0.04> > 2017-02-11 02:08:45.937787 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 02:08:45.971161 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b47541c0 con 0 > 2017-02-11 02:08:48.941678 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 208 > 2017-02-11 02:08:48.941707 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 208 v496) > v7 -- 0x7fa7b475e9c0 con 0 > 2017-02-11 02:08:48.942153 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 193 ==== mdsbeacon(814112/na9552 up:replay seq > 208 v496) v7 ==== 131+0+0 (242824141 0 0) 0x7fa7b475e9c0 con 0x7fa7b788a800 > 2017-02-11 02:08:48.942201 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 208 rtt 0.000486 > 2017-02-11 02:08:50.931149 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- ping magic: 0 v1 -- 0x7fa7b46f6540 con 0 > 2017-02-11 02:08:50.931180 7fa79ea2c700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- ping magic: 0 v1 -- 0x7fa7b46f92c0 con 0 > 2017-02-11 02:08:50.937847 7fa7a022f700 15 mds.0.bal get_load mdsload<[0,0 > 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0.03> > 2017-02-11 02:08:50.937887 7fa7a022f700 20 mds.beacon.na9552 0 slow request > found > 2017-02-11 02:08:50.971441 7fa7a0a30700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6816/3843 -- mgrreport(0 1734) v1 -- 0x7fa7b4754400 con 0 > 2017-02-11 02:08:51.365293 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 194 ==== mgrmap(e 103) v1 ==== 132+0+0 > (3330297727 0 0) 0x7fa7b7894240 con 0x7fa7b788a800 > 2017-02-11 02:08:52.378333 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 195 ==== osd_map(1028..1028 src has 501..1028) > v3 ==== 1038+0+0 (3680285544 0 0) 0x7fa7b7894480 con 0x7fa7b788a800 > 2017-02-11 02:08:52.378508 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6809/301672 -- osd_op(unknown.0.496:216 2.9430acec > 200.00000613 [stat] snapc 0=[] RETRY=3 > ack+retry+read+rwordered+known_if_redirected+full_force e1028) v7 -- > 0x7fa7b45d4000 con 0 > 2017-02-11 02:08:52.378563 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6809/301672 -- osd_op(unknown.0.496:217 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] RETRY=3 > ack+retry+read+rwordered+known_if_redirected+full_force e1028) v7 -- > 0x7fa7bdbb6580 con 0 > 2017-02-11 02:08:52.378609 7fa7a2a34700 7 mds.0.server operator(): full = 0 > epoch = 1028 > 2017-02-11 02:08:52.378639 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mon_subscribe({osdmap=1029}) v2 -- 0x7fa7b2e67a00 > con 0 > 2017-02-11 02:08:52.941786 7fa79fa2e700 10 mds.beacon.na9552 _send up:replay > seq 209 > 2017-02-11 02:08:52.941828 7fa79fa2e700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:replay seq 209 v496) > v7 -- 0x7fa7b475ed00 con 0 > 2017-02-11 02:08:52.942367 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 196 ==== mdsbeacon(814112/na9552 up:replay seq > 209 v496) v7 ==== 131+0+0 (1548656479 0 0) 0x7fa7b475ed00 con 0x7fa7b788a800 > 2017-02-11 02:08:52.942419 7fa7a2a34700 10 mds.beacon.na9552 > handle_mds_beacon up:replay seq 209 rtt 0.000613 > 2017-02-11 02:08:53.382776 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 197 ==== osd_map(1029..1029 src has 501..1029) > v3 ==== 17226+0+0 (3516612279 0 0) 0x7fa7b78946c0 con 0x7fa7b788a800 > 2017-02-11 02:08:53.383516 7fa7a2a34700 7 mds.0.server operator(): full = 0 > epoch = 1029 > 2017-02-11 02:08:53.383541 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mon_subscribe({osdmap=1030}) v2 -- 0x7fa7b2e68c00 > con 0 > 2017-02-11 02:08:53.442537 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6809/301672 1 ==== osd_op_reply(217 200.00000614 > [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 > (2272073290 0 0) 0x7fa7bdbb6580 con 0x7fa7b2ef0800 > 2017-02-11 02:08:53.453046 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6809/301672 2 ==== osd_op_reply(216 200.00000613 > [stat] v0'0 uv554 ondisk = 0) v7 ==== 132+0+16 (568638517 0 1812931205) > 0x7fa7bdbb6580 con 0x7fa7b2ef0800 > 2017-02-11 02:08:53.453119 7fa79da2a700 10 MDSIOContextBase::complete: > N7MDSRank32C_MDS_StandbyReplayRestartFinishE > 2017-02-11 02:08:53.453124 7fa79da2a700 10 mds.0.log standby_trim_segments > 2017-02-11 02:08:53.453127 7fa79da2a700 10 mds.0.log expire_pos=6490889269 > 2017-02-11 02:08:53.453129 7fa79da2a700 10 mds.0.log segment seq=3885173 > 6490889269~1297898 > 2017-02-11 02:08:53.453131 7fa79da2a700 10 mds.0.log won't remove, not > expired! > 2017-02-11 02:08:53.453132 7fa79da2a700 20 mds.0.log removed no segments! > 2017-02-11 02:08:53.453134 7fa79da2a700 2 mds.0.496 boot_start 2: replaying > mds log > 2017-02-11 02:08:53.453136 7fa79da2a700 10 mds.0.log replay - journal empty, > done. > 2017-02-11 02:08:53.453138 7fa79da2a700 7 mds.0.cache trim max=100000 > cur=33522 > 2017-02-11 02:08:53.453145 7fa79da2a700 10 MDSInternalContextBase::complete: > 15C_MDS_BootStart > 2017-02-11 02:08:53.453147 7fa79da2a700 1 mds.0.496 replay_done (as > standby) > 2017-02-11 02:08:53.453148 7fa79da2a700 10 mds.0.496 last replay pass was > as a standby; making final pass > 2017-02-11 02:08:53.453150 7fa79da2a700 1 mds.0.496 standby_replay_restart > (final takeover pass) > 2017-02-11 02:08:53.453180 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6804/2623 -- osd_op(unknown.0.496:218 2.844f3494 > 200.00000000 [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force > e1029) v7 -- 0x7fa7b45d4840 con 0 > 2017-02-11 02:08:53.453633 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.2 172.20.1.133:6804/2623 73 ==== osd_op_reply(218 200.00000000 [read > 0~90] v0'0 uv1260 ondisk = 0) v7 ==== 132+0+90 (2825467492 0 1002082041) > 0x7fa7b45d4840 con 0x7fa7b2efd000 > 2017-02-11 02:08:53.453713 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6809/301672 -- osd_op(unknown.0.496:219 2.9430acec > 200.00000613 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1029) v7 -- > 0x7fa7b45d4b00 con 0 > 2017-02-11 02:08:53.453760 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6809/301672 -- osd_op(unknown.0.496:220 2.ed704dd6 > 200.00000614 [stat] snapc 0=[] > ack+read+rwordered+known_if_redirected+full_force e1029) v7 -- > 0x7fa7b45d4dc0 con 0 > 2017-02-11 02:08:53.454677 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6809/301672 3 ==== osd_op_reply(219 200.00000613 > [stat] v0'0 uv554 ondisk = 0) v7 ==== 132+0+16 (469094719 0 1812931205) > 0x7fa7b45d4dc0 con 0x7fa7b2ef0800 > 2017-02-11 02:08:53.454725 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6809/301672 4 ==== osd_op_reply(220 200.00000614 > [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 > (3179051840 0 0) 0x7fa7b45d4dc0 con 0x7fa7b2ef0800 > 2017-02-11 02:08:53.454764 7fa79da2a700 10 MDSIOContextBase::complete: > N7MDSRank32C_MDS_StandbyReplayRestartFinishE > 2017-02-11 02:08:53.454770 7fa79da2a700 10 mds.0.log standby_trim_segments > 2017-02-11 02:08:53.454774 7fa79da2a700 10 mds.0.log expire_pos=6490889269 > 2017-02-11 02:08:53.454776 7fa79da2a700 10 mds.0.log segment seq=3885173 > 6490889269~1297898 > 2017-02-11 02:08:53.454779 7fa79da2a700 10 mds.0.log won't remove, not > expired! > 2017-02-11 02:08:53.454782 7fa79da2a700 20 mds.0.log removed no segments! > 2017-02-11 02:08:53.454784 7fa79da2a700 2 mds.0.496 boot_start 2: replaying > mds log > 2017-02-11 02:08:53.454786 7fa79da2a700 10 mds.0.log replay - journal empty, > done. > 2017-02-11 02:08:53.454788 7fa79da2a700 7 mds.0.cache trim max=100000 > cur=33521 > 2017-02-11 02:08:53.454792 7fa79da2a700 10 MDSInternalContextBase::complete: > 15C_MDS_BootStart > 2017-02-11 02:08:53.454794 7fa79da2a700 1 mds.0.496 replay_done > 2017-02-11 02:08:53.454796 7fa79da2a700 1 mds.0.496 making mds journal > writeable > 2017-02-11 02:08:53.454832 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6809/301672 -- osd_op(unknown.0.496:221 2.9430acec > 200.00000613 [zero 1794812~2399492] snapc 0=[] > ondisk+write+known_if_redirected+full_force e1029) v7 -- 0x7fa7b45d5080 con > 0 > 2017-02-11 02:08:53.454870 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6809/301672 -- osd_op(unknown.0.496:222 2.ed704dd6 > 200.00000614 [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force > e1029) v7 -- 0x7fa7b45d5340 con 0 > 2017-02-11 02:08:53.454932 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6808/311114 -- osd_op(unknown.0.496:223 2.f003e56f > 200.00000615 [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force > e1029) v7 -- 0x7fa7b45d5600 con 0 > 2017-02-11 02:08:53.454972 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.137:6809/301672 -- osd_op(unknown.0.496:224 2.52ec6e5 > 200.00000616 [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force > e1029) v7 -- 0x7fa7b45d58c0 con 0 > 2017-02-11 02:08:53.455023 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.139:6801/311106 -- osd_op(unknown.0.496:225 2.f45abaaf > 200.00000617 [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force > e1029) v7 -- 0x7fa7b45d5b80 con 0 > 2017-02-11 02:08:53.455083 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6800/1629 -- osd_op(unknown.0.496:226 2.fd991e4d > 200.00000618 [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force > e1029) v7 -- 0x7fa7b45d5e40 con 0 > 2017-02-11 02:08:53.455103 7fa79da2a700 2 mds.0.496 i am alone, moving to > state reconnect > 2017-02-11 02:08:53.455106 7fa79da2a700 3 mds.0.496 request_state > up:reconnect > 2017-02-11 02:08:53.455109 7fa79da2a700 10 mds.beacon.na9552 set_want_state: > up:replay -> up:reconnect > 2017-02-11 02:08:53.455122 7fa79da2a700 10 mds.beacon.na9552 _send > up:reconnect seq 210 > 2017-02-11 02:08:53.455148 7fa79da2a700 1 -- 172.20.1.139:6800/4021830315 > --> 172.20.1.133:6789/0 -- mdsbeacon(814112/na9552 up:reconnect seq 210 > v496) v7 -- 0x7fa7b468a000 con 0 > 2017-02-11 02:08:53.456595 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.12 172.20.1.139:6808/311114 1 ==== osd_op_reply(223 200.00000615 > [delete] v1029'389 uv379 ondisk = -2 ((2) No such file or directory)) v7 > ==== 132+0+0 (2170781617 0 0) 0x7fa7b45d5b80 con 0x7fa7b2eef000 > 2017-02-11 02:08:53.457147 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.0 172.20.1.133:6800/1629 1 ==== osd_op_reply(226 200.00000618 > [delete] v1029'226 uv216 ondisk = -2 ((2) No such file or directory)) v7 > ==== 132+0+0 (3878328486 0 0) 0x7fa7b45d5e40 con 0x7fa7b2efe800 > 2017-02-11 02:08:53.457557 7fa7a51aa700 1 -- 172.20.1.139:6800/4021830315 > <== osd.15 172.20.1.139:6801/311106 1 ==== osd_op_reply(225 200.00000617 > [delete] v1029'191 uv181 ondisk = -2 ((2) No such file or directory)) v7 > ==== 132+0+0 (562276898 0 0) 0x7fa7b45d5b80 con 0x7fa7b2efa000 > 2017-02-11 02:08:53.459907 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6809/301672 5 ==== osd_op_reply(224 200.00000616 > [delete] v1029'187 uv177 ondisk = -2 ((2) No such file or directory)) v7 > ==== 132+0+0 (1284066071 0 0) 0x7fa7b45d5e40 con 0x7fa7b2ef0800 > 2017-02-11 02:08:53.459976 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6809/301672 6 ==== osd_op_reply(221 200.00000613 > [zero 1794812~2399492] v1029'555 uv554 ondisk = 0) v7 ==== 132+0+0 > (409944788 0 0) 0x7fa7b45d5e40 con 0x7fa7b2ef0800 > 2017-02-11 02:08:53.460013 7fa7a49a9700 1 -- 172.20.1.139:6800/4021830315 > <== osd.10 172.20.1.137:6809/301672 7 ==== osd_op_reply(222 200.00000614 > [delete] v1029'586 uv576 ondisk = -2 ((2) No such file or directory)) v7 > ==== 132+0+0 (1227742960 0 0) 0x7fa7b45d5e40 con 0x7fa7b2ef0800 > 2017-02-11 02:08:54.470865 7fa7a2a34700 1 -- 172.20.1.139:6800/4021830315 > <== mon.0 172.20.1.133:6789/0 198 ==== mdsmap(e 498) v1 ==== 585+0+0 > (3687796942 0 0) 0x7fa7b4656400 con 0x7fa7b788a800 > > On Fri, Feb 10, 2017 at 1:46 PM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote: >> >> This is odd on several levels, and indeed a failover shouldn't take >> that long (unless you have a *lot* of metadata that needs to get >> loaded into memory, which you won't if running standby-replay). Are >> you sure that it's trying to connect to the other MDS, and not a >> monitor or OSD on the same host? >> If not, can you turn on debugging and reproduce? ("debug mds = 20", >> "debug ms = 1") >> -Greg >> >> On Wed, Feb 8, 2017 at 1:46 PM, Luke Weber <luke.weber@xxxxxxxxx> wrote: >> > Playing around with mds with a hot standby on kraken. When I fail out >> > the >> > active mds manually it switches correctly to the standby i.e. ceph mds >> > fail >> > <active-mds> >> > >> > Noticed that when I have two mds servers and I shutdown the active mds >> > server it takes 5 minutes for the standby relay to become active(Seems >> > it's >> > 20 retries at 15 seconds timeout to the previously active mds). I can't >> > fail >> > the active mds though as it's already been removed from the mds map, but >> > the >> > hot standby is stuck in replay mode for 5 minutes waiting for the active >> > before it gives up and becomes active. Curious if there's a preferred >> > way to >> > configure this behavior or force a failover in the event of unexpected >> > active failure. >> > >> > MSD log of standby becoming master: >> > >> > 2017-02-08 17:25:54.151002 7fa0a1502700 1 mds.0.0 replay_done (as >> > standby) >> > 2017-02-08 17:25:55.153022 7fa0a1502700 1 mds.0.0 replay_done (as >> > standby) >> > 2017-02-08 17:25:56.154928 7fa0a1502700 1 mds.0.0 replay_done (as >> > standby) >> > 2017-02-08 17:25:57.156771 7fa0a1502700 1 mds.0.0 replay_done (as >> > standby) >> > 2017-02-08 17:25:58.158700 7fa0a1502700 1 mds.0.0 replay_done (as >> > standby) >> > ----- Shutdown active mds (Start to see it reconnecting to active >> > server): >> > 2017-02-08 17:26:08.774979 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0baad6800 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:26:23.775456 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0baad5000 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > ----- 15 Second grace to get an mds map update (mds beacon grace=15) >> > 2017-02-08 17:26:25.003332 7fa0a650c700 1 mds.0.132 handle_mds_map i am >> > now >> > mds.0.132 >> > 2017-02-08 17:26:25.003340 7fa0a650c700 1 mds.0.132 handle_mds_map >> > state >> > change up:standby-replay --> up:replay >> > 2017-02-08 17:26:38.776036 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0baad3800 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:26:53.776916 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0baad6800 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:27:08.777962 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0baad5000 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:27:23.777884 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0b82d3800 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:27:38.778943 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0b82d2000 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:27:53.779926 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0b8316800 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:28:08.780927 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0baad6800 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:28:23.780909 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0baad5000 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:28:38.781947 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0b82d3800 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:28:53.782075 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0b82d2000 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:29:08.782916 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0b8315000 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:29:23.783476 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0b8315000 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:29:38.784445 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0baad6800 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:29:53.784934 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0baad5000 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:30:08.785959 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0b82d3800 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:30:23.786921 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0b82d2000 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:30:38.786923 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0baad6800 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:30:53.788035 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0baad5000 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > 2017-02-08 17:31:08.788730 7fa0a9483700 0 -- >> > 172.20.1.139:6800/255206595 >> >> > - conn(0x7fa0b8315000 :6800 s=STATE_ACCEPTING_WAIT_BANNER_ADDR pgs=0 >> > cs=0 >> > l=0).fault with nothing to send and in the half accept state just >> > closed >> > [2017-02-08 17:31:15.393349 7fa0a1502700 1 mds.0.132 replay_done (as >> > standby) >> > 2017-02-08 17:31:15.393353 7fa0a1502700 1 mds.0.132 >> > standby_replay_restart >> > (final takeover pass) >> > 2017-02-08 17:31:15.397825 7fa0a1502700 1 mds.0.132 replay_done >> > 2017-02-08 17:31:15.397832 7fa0a1502700 1 mds.0.132 making mds journal >> > writeable >> > 2017-02-08 17:31:16.163297 7fa0a650c700 1 mds.0.132 handle_mds_map i am >> > now >> > mds.0.132 >> > 2017-02-08 17:31:16.163303 7fa0a650c700 1 mds.0.132 handle_mds_map >> > state >> > change up:replay --> up:reconnect >> > 2017-02-08 17:31:16.163312 7fa0a650c700 1 mds.0.132 reconnect_start >> > 2017-02-08 17:31:16.163314 7fa0a650c700 1 mds.0.132 reopen_log >> > >> > _______________________________________________ >> > ceph-users mailing list >> > ceph-users@xxxxxxxxxxxxxx >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |