On 5/7/14 13:15 , Sergey Malinin wrote: > Is there anything unusual in dmesg at osd.5? Nothing in dmesg, but ceph-osd.5.log has plenty. I've attached the log after the restart. Logging levels are normal. What jumps out at me is: 2014-05-07 12:48:02.640164 7ff65d439700 -1 osd.5 38870 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:42.640163) 2014-05-07 12:48:02.640174 7ff65d439700 -1 osd.5 38870 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:42.640163) 2014-05-07 12:48:02.640180 7ff65d439700 -1 osd.5 38870 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:42.640163) 2014-05-07 12:48:02.640186 7ff65d439700 -1 osd.5 38870 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:42.640163) osd.5 is on host ceph0c. osd.8, osd.11, osd.12, and osd.13 are all on ceph1c (along with 4 other OSDs). Both the front and back network are working fine, and I can connect to osd.8 from host ceph0 just fine. These 4 OSDs are not reporting problems. The other OSDs that are flapping are osd.6 and osd.15. osd.15 says: 2014-05-07 13:25:44.626840 7fe312c9d700 -1 osd.15 38891 heartbeat_check: no reply from osd.5 since back 2014-05-07 13:10:01.239883 front 2014-05-07 13:10:01.239883 (cutoff 2014-05-07 13:25:24.626838) 2014-05-07 13:25:44.626849 7fe312c9d700 -1 osd.15 38891 heartbeat_check: no reply from osd.11 since back 2014-05-07 13:22:48.592121 front 2014-05-07 13:22:48.592121 (cutoff 2014-05-07 13:25:24.626838) osd.6 says: 2014-05-07 13:26:15.409217 7f35e50a7700 -1 osd.6 38891 heartbeat_check: no reply from osd.5 since back 2014-05-07 13:09:57.440713 front 2014-05-07 13:09:57.440713 (cutoff 2014-05-07 13:25:55.409216) 2014-05-07 13:26:15.409227 7f35e50a7700 -1 osd.6 38891 heartbeat_check: no reply from osd.11 since back 2014-05-07 13:22:50.353671 front 2014-05-07 13:22:50.353671 (cutoff 2014-05-07 13:25:55.409216) 2014-05-07 13:26:15.409235 7f35e50a7700 -1 osd.6 38891 heartbeat_check: no reply from osd.13 since back 2014-05-07 13:11:26.959761 front 2014-05-07 13:11:26.959761 (cutoff 2014-05-07 13:25:55.409216) 2014-05-07 13:26:15.409306 7f35e13e5700 0 -- 10.194.0.6:0/17586 >> 10.194.0.7:6803/19641 pipe(0x1c4d7500 sd=79 :56788 s=1 pgs=0 cs=0 l=1 c=0x1c646840).connect claims to be 10.194.0.7:6803/1019705 not 10.194.0.7:6803/19641 - wrong node! osd.11 and osd.13 have been kicked out for being unresponsive, but they don't have any heartbeat_check entries in their logs. -- *Craig Lewis* Senior Systems Engineer Office +1.714.602.1309 Email clewis at centraldesktop.com <mailto:clewis at centraldesktop.com> *Central Desktop. Work together in ways you never thought possible.* Connect with us Website <http://www.centraldesktop.com/> | Twitter <http://www.twitter.com/centraldesktop> | Facebook <http://www.facebook.com/CentralDesktop> | LinkedIn <http://www.linkedin.com/groups?gid=147417> | Blog <http://cdblog.centraldesktop.com/> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140507/fe06ada1/attachment.htm> -------------- next part -------------- 2014-05-07 12:47:26.122879 7ff66420a780 0 ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60), process ceph-osd, pid 17533 2014-05-07 12:47:26.135115 7ff66420a780 1 filestore(/var/lib/ceph/osd/ceph-5) mount detected xfs 2014-05-07 12:47:26.135125 7ff66420a780 1 filestore(/var/lib/ceph/osd/ceph-5) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs 2014-05-07 12:47:26.153025 7ff66420a780 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-5) detect_features: FIEMAP ioctl is supported and appears to work 2014-05-07 12:47:26.153046 7ff66420a780 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-5) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2014-05-07 12:47:26.158778 7ff66420a780 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-5) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2014-05-07 12:47:26.485796 7ff66420a780 0 filestore(/var/lib/ceph/osd/ceph-5) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2014-05-07 12:47:26.872001 7ff66420a780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-05-07 12:47:26.872015 7ff66420a780 1 journal _open /var/lib/ceph/osd/ceph-5/journal fd 20: 6442450944 bytes, block size 4096 bytes, directio = 1, aio = 0 2014-05-07 12:47:26.887133 7ff66420a780 1 journal _open /var/lib/ceph/osd/ceph-5/journal fd 20: 6442450944 bytes, block size 4096 bytes, directio = 1, aio = 0 2014-05-07 12:47:26.887586 7ff66420a780 1 journal close /var/lib/ceph/osd/ceph-5/journal 2014-05-07 12:47:26.888790 7ff66420a780 1 filestore(/var/lib/ceph/osd/ceph-5) mount detected xfs 2014-05-07 12:47:26.896080 7ff66420a780 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-5) detect_features: FIEMAP ioctl is supported and appears to work 2014-05-07 12:47:26.896095 7ff66420a780 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-5) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2014-05-07 12:47:26.901752 7ff66420a780 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-5) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2014-05-07 12:47:26.938873 7ff66420a780 0 filestore(/var/lib/ceph/osd/ceph-5) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2014-05-07 12:47:26.941408 7ff66420a780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-05-07 12:47:26.941424 7ff66420a780 1 journal _open /var/lib/ceph/osd/ceph-5/journal fd 21: 6442450944 bytes, block size 4096 bytes, directio = 1, aio = 0 2014-05-07 12:47:26.941485 7ff66420a780 1 journal _open /var/lib/ceph/osd/ceph-5/journal fd 21: 6442450944 bytes, block size 4096 bytes, directio = 1, aio = 0 2014-05-07 12:47:26.947110 7ff66420a780 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello 2014-05-07 12:47:26.957620 7ff66420a780 0 osd.5 38856 crush map has features 1073741824, adjusting msgr requires for clients 2014-05-07 12:47:26.957636 7ff66420a780 0 osd.5 38856 crush map has features 1073741824, adjusting msgr requires for osds 2014-05-07 12:47:38.737218 7ff63e94e700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/10004840 pipe(0x46bf500 sd=109 :0 s=1 pgs=0 cs=0 l=0 c=0x19b70840).fault 2014-05-07 12:47:38.737677 7ff63e84d700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6812/5140 pipe(0x46bf780 sd=111 :0 s=1 pgs=0 cs=0 l=0 c=0x19b70b00).fault with nothing to send, going to standby 2014-05-07 12:47:38.738363 7ff63e74c700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/4882 pipe(0x46bfc80 sd=112 :0 s=1 pgs=0 cs=0 l=0 c=0x19b70dc0).fault 2014-05-07 12:47:38.738610 7ff63e84d700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6812/5140 pipe(0x46bf780 sd=111 :0 s=1 pgs=0 cs=1 l=0 c=0x19b70b00).fault 2014-05-07 12:47:38.738818 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/4004869 pipe(0x1a0aa280 sd=113 :0 s=1 pgs=0 cs=0 l=0 c=0x1a0ab420).fault 2014-05-07 12:47:38.739812 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=119 :0 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).fault 2014-05-07 12:47:38.739865 7ff63e54a700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6810/6005098 pipe(0x46bfa00 sd=118 :0 s=1 pgs=0 cs=0 l=0 c=0x1a0abc60).fault 2014-05-07 12:47:39.004119 7ff63d622700 0 -- :/17533 >> 10.193.0.7:6813/4882 pipe(0x1a505780 sd=169 :0 s=1 pgs=0 cs=0 l=1 c=0x1a504840).fault 2014-05-07 12:47:39.004146 7ff63e84d700 0 -- :/17533 >> 10.193.0.7:6807/4004869 pipe(0x1a0aac80 sd=165 :0 s=1 pgs=0 cs=0 l=1 c=0x1a504000).fault 2014-05-07 12:47:39.004224 7ff63e54a700 0 -- :/17533 >> 10.194.0.7:6807/4004869 pipe(0x1a0aa000 sd=113 :0 s=1 pgs=0 cs=0 l=1 c=0x1a504160).fault 2014-05-07 12:47:39.004226 7ff63d824700 0 -- :/17533 >> 10.193.0.7:6801/10011084 pipe(0x1a505c80 sd=168 :0 s=1 pgs=0 cs=0 l=1 c=0x1a504b00).fault 2014-05-07 12:47:39.004200 7ff63db27700 0 -- :/17533 >> 10.194.0.7:6815/10004840 pipe(0x1a505500 sd=166 :0 s=1 pgs=0 cs=0 l=1 c=0x1a504580).fault 2014-05-07 12:47:39.004592 7ff63da26700 0 -- :/17533 >> 10.193.0.7:6816/10004840 pipe(0x1a505280 sd=167 :0 s=1 pgs=0 cs=0 l=1 c=0x1a504420).fault 2014-05-07 12:47:39.014515 7ff63d723700 0 -- :/17533 >> 10.194.0.7:6809/4882 pipe(0x1a505a00 sd=171 :0 s=1 pgs=0 cs=0 l=1 c=0x1a5049a0).fault 2014-05-07 12:47:39.014611 7ff63d925700 0 -- :/17533 >> 10.194.0.7:6801/10011084 pipe(0x1a505000 sd=170 :0 s=1 pgs=0 cs=0 l=1 c=0x1a5042c0).fault 2014-05-07 12:47:47.367208 7ff63cc18700 0 -- 10.194.0.6:6808/17533 >> :/0 pipe(0x1a77a780 sd=176 :6808 s=0 pgs=0 cs=0 l=0 c=0x1a72b9a0).accept failed to getpeername 107 Transport endpoint is not connected 2014-05-07 12:47:52.063527 7ff63c410700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.6:6806/17482 pipe(0x1a5ae780 sd=180 :6808 s=0 pgs=0 cs=0 l=0 c=0x1a72bb00).accept connect_seq 0 vs existing 0 state wait 2014-05-07 12:48:02.640164 7ff65d439700 -1 osd.5 38870 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:42.640163) 2014-05-07 12:48:02.640174 7ff65d439700 -1 osd.5 38870 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:42.640163) 2014-05-07 12:48:02.640180 7ff65d439700 -1 osd.5 38870 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:42.640163) 2014-05-07 12:48:02.640186 7ff65d439700 -1 osd.5 38870 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:42.640163) 2014-05-07 12:48:03.640344 7ff65d439700 -1 osd.5 38871 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:43.640343) 2014-05-07 12:48:03.640359 7ff65d439700 -1 osd.5 38871 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:43.640343) 2014-05-07 12:48:03.640365 7ff65d439700 -1 osd.5 38871 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:43.640343) 2014-05-07 12:48:03.640371 7ff65d439700 -1 osd.5 38871 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:43.640343) 2014-05-07 12:48:04.641274 7ff65d439700 -1 osd.5 38871 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:44.641272) 2014-05-07 12:48:04.641291 7ff65d439700 -1 osd.5 38871 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:44.641272) 2014-05-07 12:48:04.641296 7ff65d439700 -1 osd.5 38871 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:44.641272) 2014-05-07 12:48:04.641304 7ff65d439700 -1 osd.5 38871 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:44.641272) 2014-05-07 12:48:05.641437 7ff65d439700 -1 osd.5 38872 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:45.641436) 2014-05-07 12:48:05.641454 7ff65d439700 -1 osd.5 38872 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:45.641436) 2014-05-07 12:48:05.641460 7ff65d439700 -1 osd.5 38872 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:45.641436) 2014-05-07 12:48:05.641465 7ff65d439700 -1 osd.5 38872 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:45.641436) 2014-05-07 12:48:06.641626 7ff65d439700 -1 osd.5 38872 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:46.641625) 2014-05-07 12:48:06.641643 7ff65d439700 -1 osd.5 38872 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:46.641625) 2014-05-07 12:48:06.641650 7ff65d439700 -1 osd.5 38872 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:46.641625) 2014-05-07 12:48:06.641655 7ff65d439700 -1 osd.5 38872 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:46.641625) 2014-05-07 12:48:06.837994 7ff641365700 -1 osd.5 38872 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:46.837994) 2014-05-07 12:48:06.838006 7ff641365700 -1 osd.5 38872 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:46.837994) 2014-05-07 12:48:06.838010 7ff641365700 -1 osd.5 38872 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:46.837994) 2014-05-07 12:48:06.838029 7ff641365700 -1 osd.5 38872 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:46.837994) 2014-05-07 12:48:07.641787 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:47.641786) 2014-05-07 12:48:07.641800 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:47.641786) 2014-05-07 12:48:07.641806 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:47.641786) 2014-05-07 12:48:07.641811 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:47.641786) 2014-05-07 12:48:08.641913 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:48.641912) 2014-05-07 12:48:08.641927 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:48.641912) 2014-05-07 12:48:08.641932 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:48.641912) 2014-05-07 12:48:08.641937 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:48.641912) 2014-05-07 12:48:09.642452 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:49.642451) 2014-05-07 12:48:09.642464 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:49.642451) 2014-05-07 12:48:09.642471 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:49.642451) 2014-05-07 12:48:09.642477 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:49.642451) 2014-05-07 12:48:10.642568 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:50.642567) 2014-05-07 12:48:10.642577 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:50.642567) 2014-05-07 12:48:10.642582 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:50.642567) 2014-05-07 12:48:10.642587 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:50.642567) 2014-05-07 12:48:11.616728 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=26 :40223 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:11.617219 7ff63d824700 0 -- :/17533 >> 10.193.0.7:6801/10011084 pipe(0x1a505c80 sd=165 :45769 s=1 pgs=0 cs=0 l=1 c=0x1a504b00).connect claims to be 10.193.0.7:6801/19590 not 10.193.0.7:6801/10011084 - wrong node! 2014-05-07 12:48:11.617312 7ff63d925700 0 -- :/17533 >> 10.194.0.7:6801/10011084 pipe(0x1a505000 sd=65 :50646 s=1 pgs=0 cs=0 l=1 c=0x1a5042c0).connect claims to be 10.194.0.7:6801/19590 not 10.194.0.7:6801/10011084 - wrong node! 2014-05-07 12:48:11.642677 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:51.642675) 2014-05-07 12:48:11.642687 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:51.642675) 2014-05-07 12:48:11.642707 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:51.642675) 2014-05-07 12:48:11.642713 7ff65d439700 -1 osd.5 38873 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:51.642675) 2014-05-07 12:48:12.472556 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=26 :40371 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:12.479501 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=26 :40375 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:12.481463 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40389 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:12.491061 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40401 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:12.492066 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40403 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:12.498126 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40411 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:12.499653 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40413 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:12.505499 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40424 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:12.506422 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40425 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:12.511485 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40432 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:12.644198 7ff65d439700 -1 osd.5 38874 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:52.644103) 2014-05-07 12:48:12.644222 7ff65d439700 -1 osd.5 38874 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:52.644103) 2014-05-07 12:48:12.644256 7ff65d439700 -1 osd.5 38874 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:52.644103) 2014-05-07 12:48:12.644271 7ff65d439700 -1 osd.5 38874 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:52.644103) 2014-05-07 12:48:12.738624 7ff641365700 -1 osd.5 38874 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:52.738606) 2014-05-07 12:48:12.738646 7ff641365700 -1 osd.5 38874 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:52.738606) 2014-05-07 12:48:12.738655 7ff641365700 -1 osd.5 38874 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:52.738606) 2014-05-07 12:48:12.738663 7ff641365700 -1 osd.5 38874 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:52.738606) 2014-05-07 12:48:12.738979 7ff63d824700 0 -- 10.193.0.6:0/17533 >> 10.193.0.7:6801/10011084 pipe(0x1a505c80 sd=165 :46036 s=1 pgs=0 cs=0 l=1 c=0x1a504b00).connect claims to be 10.193.0.7:6801/19590 not 10.193.0.7:6801/10011084 - wrong node! 2014-05-07 12:48:12.738993 7ff63d925700 0 -- 10.193.0.6:0/17533 >> 10.194.0.7:6801/10011084 pipe(0x1a505000 sd=65 :50916 s=1 pgs=0 cs=0 l=1 c=0x1a5042c0).connect claims to be 10.194.0.7:6801/19590 not 10.194.0.7:6801/10011084 - wrong node! 2014-05-07 12:48:12.739167 7ff6391df700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.6:6804/17418 pipe(0x1a305780 sd=204 :6808 s=0 pgs=0 cs=0 l=0 c=0x1a3f7c60).accept connect_seq 0 vs existing 0 state connecting 2014-05-07 12:48:13.485769 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40474 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:13.487079 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40478 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:13.499404 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40495 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:13.502083 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40508 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:13.519323 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40511 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:13.644388 7ff65d439700 -1 osd.5 38875 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:53.644386) 2014-05-07 12:48:13.644404 7ff65d439700 -1 osd.5 38875 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:53.644386) 2014-05-07 12:48:13.644431 7ff65d439700 -1 osd.5 38875 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:53.644386) 2014-05-07 12:48:13.644440 7ff65d439700 -1 osd.5 38875 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:53.644386) 2014-05-07 12:48:14.644605 7ff65d439700 -1 osd.5 38875 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:54.644602) 2014-05-07 12:48:14.644627 7ff65d439700 -1 osd.5 38875 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:54.644602) 2014-05-07 12:48:14.644638 7ff65d439700 -1 osd.5 38875 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:54.644602) 2014-05-07 12:48:14.644646 7ff65d439700 -1 osd.5 38875 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:54.644602) 2014-05-07 12:48:15.039302 7ff641365700 -1 osd.5 38875 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:55.039300) 2014-05-07 12:48:15.039313 7ff641365700 -1 osd.5 38875 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:55.039300) 2014-05-07 12:48:15.039323 7ff641365700 -1 osd.5 38875 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:55.039300) 2014-05-07 12:48:15.039331 7ff641365700 -1 osd.5 38875 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:55.039300) 2014-05-07 12:48:15.039921 7ff63d925700 0 -- 10.193.0.6:0/17533 >> 10.194.0.7:6801/10011084 pipe(0x1a505000 sd=65 :50992 s=1 pgs=0 cs=0 l=1 c=0x1a5042c0).connect claims to be 10.194.0.7:6801/19590 not 10.194.0.7:6801/10011084 - wrong node! 2014-05-07 12:48:15.041458 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40531 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:15.041469 7ff63d824700 0 -- 10.193.0.6:0/17533 >> 10.193.0.7:6801/10011084 pipe(0x1a505c80 sd=29 :46112 s=1 pgs=0 cs=0 l=1 c=0x1a504b00).connect claims to be 10.193.0.7:6801/19590 not 10.193.0.7:6801/10011084 - wrong node! 2014-05-07 12:48:15.042349 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40532 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:15.042753 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40533 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:15.043456 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40534 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:15.043882 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40535 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:15.044620 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40537 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:15.045139 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40538 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:15.045544 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40539 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:15.046589 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40542 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:15.047032 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/10011084 pipe(0x1a0aaa00 sd=22 :40544 s=1 pgs=0 cs=0 l=0 c=0x1a0ab9a0).connect claims to be 10.194.0.7:6800/19590 not 10.194.0.7:6800/10011084 - wrong node! 2014-05-07 12:48:15.644928 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:55.644925) 2014-05-07 12:48:15.644942 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:55.644925) 2014-05-07 12:48:15.644951 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:55.644925) 2014-05-07 12:48:15.644957 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:55.644925) 2014-05-07 12:48:16.645122 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:56.645120) 2014-05-07 12:48:16.645136 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:56.645120) 2014-05-07 12:48:16.645141 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:56.645120) 2014-05-07 12:48:16.645164 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:56.645120) 2014-05-07 12:48:17.645419 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:57.645416) 2014-05-07 12:48:17.645432 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:57.645416) 2014-05-07 12:48:17.645438 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:57.645416) 2014-05-07 12:48:17.645443 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:57.645416) 2014-05-07 12:48:17.939954 7ff641365700 -1 osd.5 38876 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:57.939953) 2014-05-07 12:48:17.939973 7ff641365700 -1 osd.5 38876 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:57.939953) 2014-05-07 12:48:17.939986 7ff641365700 -1 osd.5 38876 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:57.939953) 2014-05-07 12:48:17.939994 7ff641365700 -1 osd.5 38876 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:57.939953) 2014-05-07 12:48:17.940350 7ff63d824700 0 -- 10.193.0.6:0/17533 >> 10.193.0.7:6801/10011084 pipe(0x1a505c80 sd=29 :46176 s=1 pgs=0 cs=0 l=1 c=0x1a504b00).connect claims to be 10.193.0.7:6801/19590 not 10.193.0.7:6801/10011084 - wrong node! 2014-05-07 12:48:17.940451 7ff63d925700 0 -- 10.193.0.6:0/17533 >> 10.194.0.7:6801/10011084 pipe(0x1a505000 sd=65 :51056 s=1 pgs=0 cs=0 l=1 c=0x1a5042c0).connect claims to be 10.194.0.7:6801/19590 not 10.194.0.7:6801/10011084 - wrong node! 2014-05-07 12:48:18.646266 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:58.646263) 2014-05-07 12:48:18.646279 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:58.646263) 2014-05-07 12:48:18.646284 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:58.646263) 2014-05-07 12:48:18.646290 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:58.646263) 2014-05-07 12:48:19.646432 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:59.646429) 2014-05-07 12:48:19.646445 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:59.646429) 2014-05-07 12:48:19.646451 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:59.646429) 2014-05-07 12:48:19.646459 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:47:59.646429) 2014-05-07 12:48:20.008366 7ff63e84d700 0 -- 10.193.0.6:0/17533 >> 10.193.0.7:6807/4004869 pipe(0x1a0aac80 sd=150 :35976 s=1 pgs=0 cs=0 l=1 c=0x1a504000).connect claims to be 10.193.0.7:6807/19705 not 10.193.0.7:6807/4004869 - wrong node! 2014-05-07 12:48:20.646634 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:00.646632) 2014-05-07 12:48:20.646644 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.11 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:00.646632) 2014-05-07 12:48:20.646658 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:00.646632) 2014-05-07 12:48:20.646664 7ff65d439700 -1 osd.5 38876 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:00.646632) 2014-05-07 12:48:21.440545 7ff641365700 -1 osd.5 38877 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:01.440544) 2014-05-07 12:48:21.440558 7ff641365700 -1 osd.5 38877 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:01.440544) 2014-05-07 12:48:21.440572 7ff641365700 -1 osd.5 38877 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:01.440544) 2014-05-07 12:48:21.440754 7ff63e84d700 0 -- 10.193.0.6:0/17533 >> 10.193.0.7:6807/4004869 pipe(0x1a0aac80 sd=22 :36426 s=1 pgs=0 cs=0 l=1 c=0x1a504000).connect claims to be 10.193.0.7:6807/19705 not 10.193.0.7:6807/4004869 - wrong node! 2014-05-07 12:48:21.646992 7ff65d439700 -1 osd.5 38877 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:01.646989) 2014-05-07 12:48:21.647009 7ff65d439700 -1 osd.5 38877 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:01.646989) 2014-05-07 12:48:21.647016 7ff65d439700 -1 osd.5 38877 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:01.646989) 2014-05-07 12:48:21.804783 7ff6389d7700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6804/19705 pipe(0x1a505c80 sd=167 :6808 s=0 pgs=0 cs=0 l=0 c=0x1a7d8580).accept connect_seq 1862 vs existing 0 state connecting 2014-05-07 12:48:22.541003 7ff641365700 -1 osd.5 38878 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:02.541002) 2014-05-07 12:48:22.541011 7ff641365700 -1 osd.5 38878 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:02.541002) 2014-05-07 12:48:22.541166 7ff63e84d700 0 -- 10.193.0.6:0/17533 >> 10.193.0.7:6807/4004869 pipe(0x1a0aac80 sd=22 :36447 s=1 pgs=0 cs=0 l=1 c=0x1a504000).connect claims to be 10.193.0.7:6807/19705 not 10.193.0.7:6807/4004869 - wrong node! 2014-05-07 12:48:22.647313 7ff65d439700 -1 osd.5 38878 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:02.647311) 2014-05-07 12:48:22.647322 7ff65d439700 -1 osd.5 38878 heartbeat_check: no reply from osd.12 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:02.647311) 2014-05-07 12:48:22.812794 7ff638cda700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6802/19641 pipe(0x1ab81780 sd=73 :6808 s=0 pgs=0 cs=0 l=0 c=0x1a189840).accept connect_seq 1416 vs existing 0 state connecting 2014-05-07 12:48:23.647602 7ff65d439700 -1 osd.5 38879 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:03.647600) 2014-05-07 12:48:24.648203 7ff65d439700 -1 osd.5 38880 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:04.648201) 2014-05-07 12:48:25.648460 7ff65d439700 -1 osd.5 38881 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:05.648458) 2014-05-07 12:48:26.641606 7ff641365700 -1 osd.5 38881 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:06.641604) 2014-05-07 12:48:26.641817 7ff63e84d700 0 -- 10.193.0.6:0/17533 >> 10.193.0.7:6807/4004869 pipe(0x1a0aac80 sd=22 :36504 s=1 pgs=0 cs=0 l=1 c=0x1a504000).connect claims to be 10.193.0.7:6807/19705 not 10.193.0.7:6807/4004869 - wrong node! 2014-05-07 12:48:26.741548 7ff65d439700 -1 osd.5 38882 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:06.741546) 2014-05-07 12:48:27.741665 7ff65d439700 -1 osd.5 38883 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:07.741662) 2014-05-07 12:48:27.742705 7ff641365700 -1 osd.5 38883 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-05-07 12:47:42.335591 (cutoff 2014-05-07 12:48:07.742704) 2014-05-07 12:48:27.743018 7ff63e84d700 0 -- 10.193.0.6:0/17533 >> 10.193.0.7:6807/4004869 pipe(0x1a0aac80 sd=22 :36520 s=1 pgs=0 cs=0 l=1 c=0x1a504000).connect claims to be 10.193.0.7:6807/19705 not 10.193.0.7:6807/4004869 - wrong node! 2014-05-07 13:03:05.077846 7ff63b804700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.6:6814/17691 pipe(0x1a305000 sd=185 :35751 s=2 pgs=436 cs=1 l=0 c=0x1a376dc0).fault with nothing to send, going to standby 2014-05-07 13:03:15.367239 7ff63c410700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.6:6806/17482 pipe(0x1a5ae780 sd=180 :6808 s=2 pgs=328 cs=1 l=0 c=0x1a72bdc0).fault with nothing to send, going to standby 2014-05-07 13:03:19.288862 7ff63c30f700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=129 :6808 s=2 pgs=19 cs=1 l=0 c=0x1a1179a0).fault with nothing to send, going to standby 2014-05-07 13:03:19.544701 7ff63c713700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.6:6800/17315 pipe(0x1a77a780 sd=178 :6808 s=2 pgs=374 cs=1 l=0 c=0x1a72bc60).fault with nothing to send, going to standby 2014-05-07 13:03:19.552387 7ff6389d7700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6804/19705 pipe(0x1a505c80 sd=167 :6808 s=2 pgs=6612 cs=1863 l=0 c=0x1a412420).fault with nothing to send, going to standby 2014-05-07 13:03:19.617385 7ff63b1fe700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.6:6802/17366 pipe(0x1a305a00 sd=166 :45746 s=2 pgs=14 cs=1 l=0 c=0x19b70000).fault with nothing to send, going to standby 2014-05-07 13:03:19.774292 7ff637cca700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6810/19972 pipe(0x1a131280 sd=28 :6808 s=2 pgs=5597 cs=1555 l=0 c=0x1a46edc0).fault with nothing to send, going to standby 2014-05-07 13:03:19.774371 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :6808 s=2 pgs=211 cs=1 l=0 c=0x1ab8a9a0).fault with nothing to send, going to standby 2014-05-07 13:03:19.774557 7ff62de2a700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a77ac80 sd=49 :6808 s=2 pgs=2278 cs=507 l=0 c=0x1ab8a840).fault with nothing to send, going to standby 2014-05-07 13:03:19.774736 7ff63b0fd700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/19590 pipe(0x1a305500 sd=62 :6808 s=2 pgs=2 cs=1 l=0 c=0x1a3f7b00).fault with nothing to send, going to standby 2014-05-07 13:03:20.021040 7ff63ba06700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.6:6812/17637 pipe(0x1a77aa00 sd=56 :50647 s=2 pgs=364 cs=1 l=0 c=0x19b70b00).fault with nothing to send, going to standby 2014-05-07 13:03:20.459671 7ff6391df700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.6:6804/17418 pipe(0x1a305780 sd=204 :6808 s=2 pgs=30 cs=1 l=0 c=0x1a495dc0).fault with nothing to send, going to standby 2014-05-07 13:03:27.049292 7ff638cda700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6802/19641 pipe(0x1ab81780 sd=73 :6808 s=2 pgs=5143 cs=1417 l=0 c=0x1a10c000).fault with nothing to send, going to standby 2014-05-07 13:03:54.963257 7ff6397e5700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6812/20058 pipe(0x1a5f3c80 sd=92 :6808 s=2 pgs=182 cs=1 l=0 c=0x1a46e2c0).fault with nothing to send, going to standby 2014-05-07 13:03:55.346249 7ff63cb17700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.6:6810/17586 pipe(0x1a77a500 sd=175 :42575 s=2 pgs=494 cs=1 l=0 c=0x1a72b840).fault with nothing to send, going to standby 2014-05-07 13:10:01.601412 7ff65d439700 0 log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 1291.719411 secs 2014-05-07 13:10:01.601420 7ff65d439700 0 log [WRN] : slow request 1291.719411 seconds old, received at 2014-05-07 12:48:29.881961: pg_query(40.788 epoch 38885) v2 currently wait for new map 2014-05-07 13:10:01.630545 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59207 s=2 pgs=7019 cs=1 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.632810 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59209 s=2 pgs=7020 cs=3 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.633516 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59210 s=2 pgs=7021 cs=5 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.634242 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59211 s=2 pgs=7022 cs=7 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.634882 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59212 s=2 pgs=7023 cs=9 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.637250 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43758 s=2 pgs=220 cs=3 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.637951 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43764 s=2 pgs=221 cs=5 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.638463 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59213 s=2 pgs=7024 cs=11 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.638749 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43765 s=2 pgs=222 cs=7 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.639259 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59216 s=2 pgs=7025 cs=13 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.639870 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43767 s=2 pgs=223 cs=9 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.640058 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59218 s=2 pgs=7026 cs=15 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.641815 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43769 s=2 pgs=224 cs=11 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.647364 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59220 s=2 pgs=7027 cs=17 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.647762 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43771 s=2 pgs=225 cs=13 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.648405 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59222 s=2 pgs=7028 cs=19 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.648792 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43773 s=2 pgs=226 cs=15 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.649986 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59224 s=2 pgs=7029 cs=21 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.650834 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59226 s=2 pgs=7030 cs=23 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.651656 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59227 s=2 pgs=7031 cs=25 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.652492 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59228 s=2 pgs=7032 cs=27 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.653261 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59229 s=2 pgs=7033 cs=29 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.653794 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43775 s=2 pgs=227 cs=17 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.654223 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59230 s=2 pgs=7034 cs=31 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.654620 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43781 s=2 pgs=228 cs=19 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.655448 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43783 s=2 pgs=229 cs=21 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.656180 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43784 s=2 pgs=230 cs=23 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.656972 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43785 s=2 pgs=231 cs=25 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.657709 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43786 s=2 pgs=232 cs=27 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.658394 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43787 s=2 pgs=233 cs=29 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.659263 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43788 s=2 pgs=234 cs=31 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.659489 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59232 s=2 pgs=7035 cs=33 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.660245 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59240 s=2 pgs=7036 cs=35 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.660952 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59241 s=2 pgs=7037 cs=37 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.661804 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59242 s=2 pgs=7038 cs=39 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.662537 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59243 s=2 pgs=7039 cs=41 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.663254 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59244 s=2 pgs=7040 cs=43 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.663982 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59245 s=2 pgs=7041 cs=45 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.664788 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59246 s=2 pgs=7042 cs=47 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.665610 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59247 s=2 pgs=7043 cs=49 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.670070 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43789 s=2 pgs=235 cs=33 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.670972 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43799 s=2 pgs=236 cs=35 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.671631 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59248 s=2 pgs=7044 cs=51 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.671830 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43800 s=2 pgs=237 cs=37 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.672424 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=141 :59251 s=2 pgs=7045 cs=53 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.673423 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43803 s=2 pgs=238 cs=39 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.673801 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59254 s=2 pgs=7046 cs=55 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.678118 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59256 s=2 pgs=7047 cs=57 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.678423 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43805 s=2 pgs=239 cs=41 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.679192 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43808 s=2 pgs=240 cs=43 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.679877 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43809 s=2 pgs=241 cs=45 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.680587 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43810 s=2 pgs=242 cs=47 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.681220 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43811 s=2 pgs=243 cs=49 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.684303 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59263 s=2 pgs=7048 cs=59 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.685021 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59264 s=2 pgs=7049 cs=61 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.685712 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59265 s=2 pgs=7050 cs=63 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.686557 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59266 s=2 pgs=7051 cs=65 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.687284 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59267 s=2 pgs=7052 cs=67 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.687978 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59268 s=2 pgs=7053 cs=69 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.688660 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59269 s=2 pgs=7054 cs=71 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.689370 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59270 s=2 pgs=7055 cs=73 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.689796 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43812 s=2 pgs=244 cs=51 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.690257 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59271 s=2 pgs=7056 cs=75 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.690955 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59272 s=2 pgs=7057 cs=77 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.691611 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59273 s=2 pgs=7058 cs=79 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.692291 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59274 s=2 pgs=7059 cs=81 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.693096 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59275 s=2 pgs=7060 cs=83 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.693903 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59276 s=2 pgs=7061 cs=85 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.694632 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59278 s=2 pgs=7062 cs=87 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.695375 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59279 s=2 pgs=7063 cs=89 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.696186 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59280 s=2 pgs=7064 cs=91 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.696865 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59281 s=2 pgs=7065 cs=93 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.697684 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59282 s=2 pgs=7066 cs=95 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.698346 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43827 s=2 pgs=245 cs=53 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.698460 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59283 s=2 pgs=7067 cs=97 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.699094 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43834 s=2 pgs=246 cs=55 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.699995 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43836 s=2 pgs=247 cs=57 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.700767 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43837 s=2 pgs=248 cs=59 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.701489 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43838 s=2 pgs=249 cs=61 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.702163 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43839 s=2 pgs=250 cs=63 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.702789 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43840 s=2 pgs=251 cs=65 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.703454 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43841 s=2 pgs=252 cs=67 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.704131 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43842 s=2 pgs=253 cs=69 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.704854 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43843 s=2 pgs=254 cs=71 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.705883 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43844 s=2 pgs=255 cs=73 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.706614 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43845 s=2 pgs=256 cs=75 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.707288 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43846 s=2 pgs=257 cs=77 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.710487 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43847 s=2 pgs=258 cs=79 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.710904 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59285 s=2 pgs=7068 cs=99 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.711271 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43848 s=2 pgs=259 cs=81 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.714269 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43850 s=2 pgs=260 cs=83 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.714478 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59299 s=2 pgs=7069 cs=101 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.715060 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43851 s=2 pgs=261 cs=85 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.715226 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59302 s=2 pgs=7070 cs=103 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.715842 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43853 s=2 pgs=262 cs=87 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.716305 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59304 s=2 pgs=7071 cs=105 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.716581 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43855 s=2 pgs=263 cs=89 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.717135 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59306 s=2 pgs=7072 cs=107 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.717454 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43857 s=2 pgs=264 cs=91 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.717876 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59308 s=2 pgs=7073 cs=109 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.718213 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43859 s=2 pgs=265 cs=93 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.718686 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59310 s=2 pgs=7074 cs=111 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.719191 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43861 s=2 pgs=266 cs=95 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.719459 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59312 s=2 pgs=7075 cs=113 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.720225 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59314 s=2 pgs=7076 cs=115 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.720439 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43863 s=2 pgs=267 cs=97 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.720920 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59315 s=2 pgs=7077 cs=117 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.722233 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59317 s=2 pgs=7078 cs=119 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.722722 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43866 s=2 pgs=268 cs=99 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.722937 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59318 s=2 pgs=7079 cs=121 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.723487 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43869 s=2 pgs=269 cs=101 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.724214 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43871 s=2 pgs=270 cs=103 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.724911 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43872 s=2 pgs=271 cs=105 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.725644 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43873 s=2 pgs=272 cs=107 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.726404 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43874 s=2 pgs=273 cs=109 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.727035 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43875 s=2 pgs=274 cs=111 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.727691 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43876 s=2 pgs=275 cs=113 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.728322 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43877 s=2 pgs=276 cs=115 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.728600 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59320 s=2 pgs=7080 cs=123 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.729449 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59329 s=2 pgs=7081 cs=125 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.730253 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59330 s=2 pgs=7082 cs=127 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.731070 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59331 s=2 pgs=7083 cs=129 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.731894 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59332 s=2 pgs=7084 cs=131 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.732718 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59333 s=2 pgs=7085 cs=133 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.733502 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59334 s=2 pgs=7086 cs=135 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.734312 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43878 s=2 pgs=277 cs=117 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.734505 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59335 s=2 pgs=7087 cs=137 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.735353 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59337 s=2 pgs=7088 cs=139 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.736177 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59338 s=2 pgs=7089 cs=141 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.737022 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59339 s=2 pgs=7090 cs=143 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.737999 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59340 s=2 pgs=7091 cs=145 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.738868 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59341 s=2 pgs=7092 cs=147 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.740536 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43886 s=2 pgs=278 cs=119 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.741044 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59342 s=2 pgs=7093 cs=149 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.741373 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43893 s=2 pgs=279 cs=121 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.741845 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59344 s=2 pgs=7094 cs=151 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.742515 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43895 s=2 pgs=280 cs=123 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.742836 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59346 s=2 pgs=7095 cs=153 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.743614 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59348 s=2 pgs=7096 cs=155 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.744389 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59349 s=2 pgs=7097 cs=157 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.745104 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59350 s=2 pgs=7098 cs=159 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.746060 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59351 s=2 pgs=7099 cs=161 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.746145 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43897 s=2 pgs=281 cs=125 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.747045 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43903 s=2 pgs=282 cs=127 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.747870 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59352 s=2 pgs=7100 cs=163 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.748286 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43904 s=2 pgs=283 cs=129 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.748870 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59355 s=2 pgs=7101 cs=165 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.749127 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43906 s=2 pgs=284 cs=131 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.749960 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59357 s=2 pgs=7102 cs=167 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.749992 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43908 s=2 pgs=285 cs=133 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.750724 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43910 s=2 pgs=286 cs=135 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.750952 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59359 s=2 pgs=7103 cs=169 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.751515 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43911 s=2 pgs=287 cs=137 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.751908 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59362 s=2 pgs=7104 cs=171 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.752458 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43913 s=2 pgs=288 cs=139 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.753177 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59364 s=2 pgs=7105 cs=173 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.753196 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43915 s=2 pgs=289 cs=141 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.753961 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43917 s=2 pgs=290 cs=143 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.754135 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59366 s=2 pgs=7106 cs=175 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.754643 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43918 s=2 pgs=291 cs=145 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.755320 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43920 s=2 pgs=292 cs=147 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.756001 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43921 s=2 pgs=293 cs=149 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.756227 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59369 s=2 pgs=7107 cs=177 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.756694 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43922 s=2 pgs=294 cs=151 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.757191 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59373 s=2 pgs=7108 cs=179 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.757249 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43924 s=2 pgs=295 cs=153 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.757931 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43926 s=2 pgs=296 cs=155 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.758148 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59375 s=2 pgs=7109 cs=181 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.758706 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43927 s=2 pgs=297 cs=157 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.758931 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59378 s=2 pgs=7110 cs=183 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.759491 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43929 s=2 pgs=298 cs=159 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.760059 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59380 s=2 pgs=7111 cs=185 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.760265 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43931 s=2 pgs=299 cs=161 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.760903 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59382 s=2 pgs=7112 cs=187 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.761594 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59383 s=2 pgs=7113 cs=189 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.762455 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59385 s=2 pgs=7114 cs=191 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.763564 7ff63aefb700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59386 s=2 pgs=7115 cs=193 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.763626 7ff63e84d700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43934 s=2 pgs=300 cs=163 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.766127 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59387 s=2 pgs=7116 cs=195 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.766797 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43938 s=2 pgs=301 cs=165 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.767036 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59389 s=2 pgs=7117 cs=197 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.767939 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59391 s=2 pgs=7118 cs=199 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.768654 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59392 s=2 pgs=7119 cs=201 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.769652 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59393 s=2 pgs=7120 cs=203 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.770007 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43940 s=2 pgs=302 cs=167 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.770908 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43946 s=2 pgs=303 cs=169 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.771019 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59395 s=2 pgs=7121 cs=205 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.771613 7ff63c30f700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=129 :49003 s=2 pgs=34 cs=3 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.771771 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59399 s=2 pgs=7122 cs=207 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.772033 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43948 s=2 pgs=304 cs=171 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.773158 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43952 s=2 pgs=305 cs=173 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.773378 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59401 s=2 pgs=7123 cs=209 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.773969 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43953 s=2 pgs=306 cs=175 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.774493 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59404 s=2 pgs=7124 cs=211 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.774854 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43955 s=2 pgs=307 cs=177 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.775263 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59406 s=2 pgs=7125 cs=213 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.775674 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43957 s=2 pgs=308 cs=179 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.776072 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59408 s=2 pgs=7126 cs=215 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.776511 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43959 s=2 pgs=309 cs=181 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.777051 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59410 s=2 pgs=7127 cs=217 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.777345 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43961 s=2 pgs=310 cs=183 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.778505 7ff63e64b700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59412 s=2 pgs=7128 cs=219 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.779151 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43963 s=2 pgs=311 cs=185 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.779557 7ff63aefb700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59414 s=2 pgs=7129 cs=221 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.779813 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43965 s=2 pgs=312 cs=187 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.780875 7ff63c30f700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=129 :49006 s=2 pgs=35 cs=5 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.781774 7ff639fed700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=129 :49030 s=2 pgs=36 cs=7 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.782369 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43967 s=2 pgs=313 cs=189 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.783022 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43975 s=2 pgs=314 cs=191 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.783653 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43976 s=2 pgs=315 cs=193 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.784291 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43977 s=2 pgs=316 cs=195 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.786109 7ff63c30f700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=118 :59416 s=2 pgs=7130 cs=223 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.786355 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43978 s=2 pgs=317 cs=197 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.787193 7ff63c30f700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=50 :59430 s=2 pgs=7131 cs=225 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.788063 7ff63c30f700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=50 :59431 s=2 pgs=7132 cs=227 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.788685 7ff63c30f700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=50 :59432 s=2 pgs=7133 cs=229 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.789589 7ff63c30f700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=50 :59433 s=2 pgs=7134 cs=231 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.790116 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=69 :49035 s=2 pgs=37 cs=9 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.791169 7ff63b0fd700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/19590 pipe(0x1a305500 sd=62 :41299 s=2 pgs=19 cs=3 l=0 c=0x1a3f7b00).fault, initiating reconnect 2014-05-07 13:10:01.791842 7ff63b0fd700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/19590 pipe(0x1a305500 sd=62 :41300 s=2 pgs=20 cs=5 l=0 c=0x1a3f7b00).fault, initiating reconnect 2014-05-07 13:10:01.792201 7ff639fed700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=69 :49035 s=1 pgs=37 cs=10 l=0 c=0x1a1179a0).fault 2014-05-07 13:10:01.792470 7ff63e84d700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43978 s=1 pgs=317 cs=198 l=0 c=0x1ab8a9a0).fault 2014-05-07 13:10:01.793092 7ff639fed700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=69 :49044 s=2 pgs=38 cs=11 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.792556 7ff63b0fd700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/19590 pipe(0x1a305500 sd=62 :41301 s=2 pgs=21 cs=7 l=0 c=0x1a3f7b00).fault, initiating reconnect 2014-05-07 13:10:01.794581 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=69 :49047 s=2 pgs=39 cs=13 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.795309 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=69 :49048 s=2 pgs=40 cs=15 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.795549 7ff63b0fd700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/19590 pipe(0x1a305500 sd=62 :41304 s=2 pgs=22 cs=9 l=0 c=0x1a3f7b00).fault, initiating reconnect 2014-05-07 13:10:01.796055 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=69 :49049 s=2 pgs=41 cs=17 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.796353 7ff63c30f700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=50 :59434 s=2 pgs=7135 cs=233 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.797132 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=69 :49051 s=2 pgs=42 cs=19 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.797929 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=69 :49053 s=2 pgs=43 cs=21 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.801790 7ff63b0fd700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/19590 pipe(0x1a305500 sd=62 :41308 s=2 pgs=23 cs=11 l=0 c=0x1a3f7b00).fault, initiating reconnect 2014-05-07 13:10:01.802077 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43989 s=2 pgs=318 cs=199 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.802135 7ff63e84d700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :43989 s=1 pgs=318 cs=200 l=0 c=0x1ab8a9a0).fault 2014-05-07 13:10:01.802296 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=69 :49054 s=2 pgs=44 cs=23 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.802520 7ff63c30f700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=50 :59446 s=2 pgs=7136 cs=235 l=0 c=0x1b7e9000).fault, initiating reconnect 2014-05-07 13:10:01.802710 7ff63b0fd700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6800/19590 pipe(0x1a305500 sd=62 :41313 s=2 pgs=24 cs=13 l=0 c=0x1a3f7b00).fault, initiating reconnect 2014-05-07 13:10:01.802995 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :44000 s=2 pgs=319 cs=201 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.803548 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=69 :49057 s=2 pgs=45 cs=25 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.803857 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :44003 s=2 pgs=320 cs=203 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.804655 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=69 :49060 s=2 pgs=46 cs=27 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.805044 7ff63acf9700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :44005 s=2 pgs=321 cs=205 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.805716 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=69 :49062 s=2 pgs=47 cs=29 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.805770 7ff63aefb700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6814/20160 pipe(0x1a327000 sd=50 :59446 s=1 pgs=7136 cs=236 l=0 c=0x1b7e9000).fault 2014-05-07 13:10:01.806032 7ff63e84d700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6806/19813 pipe(0x1a898500 sd=44 :44007 s=2 pgs=322 cs=207 l=0 c=0x1ab8a9a0).fault, initiating reconnect 2014-05-07 13:10:01.806831 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=63 :49065 s=2 pgs=48 cs=31 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.807858 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=63 :49066 s=2 pgs=49 cs=33 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.808643 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=63 :49067 s=2 pgs=50 cs=35 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.809431 7ff63e449700 0 -- 10.194.0.6:6808/17533 >> 10.194.0.7:6808/19885 pipe(0x1b418a00 sd=63 :49068 s=2 pgs=51 cs=37 l=0 c=0x1a1179a0).fault, initiating reconnect 2014-05-07 13:10:01.809947 7ff64a377700 0 log [WRN] : map e38904 wrongly marked me down 2014-05-07 13:10:08.392534 7ff6393e1700 0 -- 10.194.0.6:6816/1017533 >> 10.194.0.6:6800/17315 pipe(0x1c833500 sd=190 :6816 s=0 pgs=0 cs=0 l=0 c=0x1c47a580).accept connect_seq 0 vs existing 0 state connecting 2014-05-07 13:10:33.607169 7ff65d439700 0 log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 30.921614 secs 2014-05-07 13:10:33.607176 7ff65d439700 0 log [WRN] : slow request 30.921614 seconds old, received at 2014-05-07 13:10:02.685521: osd_op(client.68846698.0:1385343 gc.28 [call lock.lock] 32.4070ac6b RETRY=1 e38905) v4 currently reached pg 2014-05-07 13:11:03.611828 7ff65d439700 0 log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 60.926275 secs 2014-05-07 13:11:03.611835 7ff65d439700 0 log [WRN] : slow request 60.926275 seconds old, received at 2014-05-07 13:10:02.685521: osd_op(client.68846698.0:1385343 gc.28 [call lock.lock] 32.4070ac6b RETRY=1 e38905) v4 currently reached pg 2014-05-07 13:12:03.621927 7ff65d439700 0 log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 120.936380 secs 2014-05-07 13:12:03.621935 7ff65d439700 0 log [WRN] : slow request 120.936380 seconds old, received at 2014-05-07 13:10:02.685521: osd_op(client.68846698.0:1385343 gc.28 [call lock.lock] 32.4070ac6b RETRY=1 e38905) v4 currently reached pg 2014-05-07 13:14:02.838545 7ff65d439700 0 log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 240.152992 secs 2014-05-07 13:14:02.838556 7ff65d439700 0 log [WRN] : slow request 240.152992 seconds old, received at 2014-05-07 13:10:02.685521: osd_op(client.68846698.0:1385343 gc.28 [call lock.lock] 32.4070ac6b RETRY=1 e38905) v4 currently reached pg
 |