|
Sorry, I didn't mean it logged nothing, I just saw no clues that were apparent to me. Subsequent restart attempts log nothing.
> Date: Thu, 10 Oct 2013 23:04:05 +0200 > From: wido@xxxxxxxx > To: ceph-users@xxxxxxxxxxxxxx > CC: mike.otoole@xxxxxxxxxxx > Subject: Re: osds and gateway not coming up on restart > > On 10/10/2013 11:01 PM, Mike O'Toole wrote: > > > > I created them with ceph-deploy and there are no OSD entries in the > > ceph.conf. Trying to start them that way doesnt work. > > > > (bringing discussion back to the list) > > Are you sure there is no logging? Because there should be in /var/log/ceph > > Wido > > > > > > Date: Thu, 10 Oct 2013 22:57:29 +0200 > > > From: wido@xxxxxxxx > > > To: mike.otoole@xxxxxxxxxxx > > > Subject: Re: osds and gateway not coming up on restart > > > > > > On 10/10/2013 10:54 PM, Mike O'Toole wrote: > > > > I verified the OSDs were not running and I issued "sudo stop ceph-all" > > > > and "sudo start ceph-all" but nothing comes up. The OSDS are all on the > > > > same server. The file systems are xfs and I am able to mount them. > > > > > > Could you try starting them manually via: > > > > > > $ service ceph start osd.X > > > > > > where X is the OSD number of those three OSDs. > > > > > > If that doesn't work, check the logs of the OSDs why they aren't > > starting. > > > > > > I'm not so familiar with the upstart scripts from Ceph, but I think it > > > only starts the OSDs when they have been created via ceph-deploy thus > > > ceph-disk-prepare and ceph-disk-activate > > > > > > Wido > > > > > > > > > > > /dev/sdb1 931G 1.1G 930G 1% /data-1 > > > > /dev/sdb2 931G 1.1G 930G 1% /data-2 > > > > /dev/sdb3 931G 1.1G 930G 1% /data-3 > > > > > > > > Interestingly though they are empty. > > > > > > > > > Date: Thu, 10 Oct 2013 22:46:26 +0200 > > > > > From: wido@xxxxxxxx > > > > > To: ceph-users@xxxxxxxxxxxxxx > > > > > Subject: Re: osds and gateway not coming up on restart > > > > > > > > > > On 10/10/2013 10:43 PM, Mike O'Toole wrote: > > > > > > So I took a power hit today and after coming back up 3 of my osds > > > > and my > > > > > > radosgw are not coming back up. The logs show no clue as to > > what may > > > > > > have happened. > > > > > > > > > > > > When I manually try to restart the gateway I see the following in > > > > the logs: > > > > > > > > > > > > 2013-10-10 16:04:23.166046 7f8480d9a700 2 > > > > > > RGWDataChangesLog::ChangesRenewThread: start > > > > > > 2013-10-10 16:04:45.166193 7f8480d9a700 2 > > > > > > RGWDataChangesLog::ChangesRenewThread: start > > > > > > 2013-10-10 16:05:07.166335 7f8480d9a700 2 > > > > > > RGWDataChangesLog::ChangesRenewThread: start > > > > > > 2013-10-10 16:05:29.166501 7f8480d9a700 2 > > > > > > RGWDataChangesLog::ChangesRenewThread: start > > > > > > 2013-10-10 16:05:51.166638 7f8480d9a700 2 > > > > > > RGWDataChangesLog::ChangesRenewThread: start > > > > > > 2013-10-10 16:06:13.166762 7f8480d9a700 2 > > > > > > RGWDataChangesLog::ChangesRenewThread: start > > > > > > 2013-10-10 16:06:35.166914 7f8480d9a700 2 > > > > > > RGWDataChangesLog::ChangesRenewThread: start > > > > > > 2013-10-10 16:06:57.167055 7f8480d9a700 2 > > > > > > RGWDataChangesLog::ChangesRenewThread: start > > > > > > 2013-10-10 16:07:10.196475 7f848535c700 -1 Initialization timeout, > > > > > > failed to initialize > > > > > > > > > > > > and then the process dies. > > > > > > > > > > > > As for the OSDs, there is no logging. I try to manually start > > them and > > > > > > it reports they are already running all their are no OSD pids > > on that > > > > > > server. > > > > > > > > > > > > $ sudo start ceph-all > > > > > > start: Job is already running: ceph-all > > > > > > > > > > > > > > > > Can you verify the ceph-osd processes are actually there? > > > > > > > > > > > Any ideas where to look for more info on these two issues? I am > > running > > > > > > ceph 0.67.3. > > > > > > > > > > > > Cluster status : > > > > > > HEALTH_WARN 78 pgs down; 78 pgs peering; 78 pgs stuck inactive; > > 78 pgs > > > > > > stuck unclean; 16 requests are blocked > 32 sec; 1 osds have slow > > > > requests > > > > > > > > > > > > ceph osd stat > > > > > > e134: 18 osds: 15 up, 15 in > > > > > > > > > > > > > > > > I assume all the OSDs are on the same machine since you ran "start > > > > > ceph-all" on one node? > > > > > > > > > > Can you still manually mount the filesystems of those OSDs? Could > > it be > > > > > they got corrupted due to the powerfailure? btrfs? xfs? > > > > > > > > > > The RGW seems to block because 78 pgs are in peering and are > > inactive, > > > > > causing the RGW to be blocked from starting. > > > > > > > > > > Wido > > > > > > > > > > > Thanks, Mike > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > ceph-users mailing list > > > > > > ceph-users@xxxxxxxxxxxxxx > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > > > > > > > > > > > -- > > > > > Wido den Hollander > > > > > 42on B.V. > > > > > > > > > > Phone: +31 (0)20 700 9902 > > > > > Skype: contact42on > > > > > _______________________________________________ > > > > > ceph-users mailing list > > > > > ceph-users@xxxxxxxxxxxxxx > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > -- > > > Wido den Hollander > > > 42on B.V. > > > > > > Phone: +31 (0)20 700 9902 > > > Skype: contact42on > > > -- > Wido den Hollander > 42on B.V. > > Phone: +31 (0)20 700 9902 > Skype: contact42on |
2013-10-09 06:57:27.237841 7fa828907700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x1ecbb80 sd=201 :6801 s=0 pgs=0 cs=0 l=0 c=0x2b831e0).accept connect_seq 390 vs existing 389 state standby 2013-10-09 07:12:29.490774 7fa828907700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x1ecbb80 sd=201 :6801 s=2 pgs=260 cs=391 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 07:43:22.399031 7fa8277f6700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x1ecb900 sd=203 :6801 s=0 pgs=0 cs=0 l=0 c=0x267e9a0).accept connect_seq 392 vs existing 391 state standby 2013-10-09 07:58:23.115253 7fa8277f6700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x1ecb900 sd=203 :6801 s=2 pgs=261 cs=393 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 08:18:26.012287 7fa828a08700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b2d900 sd=201 :6801 s=0 pgs=0 cs=0 l=0 c=0x267e840).accept connect_seq 394 vs existing 393 state standby 2013-10-09 08:34:51.673760 7fa828a08700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b2d900 sd=201 :6801 s=2 pgs=263 cs=395 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 08:43:24.113278 7fa828907700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b2c500 sd=203 :6801 s=0 pgs=0 cs=0 l=0 c=0x267e6e0).accept connect_seq 396 vs existing 395 state standby 2013-10-09 08:58:26.865466 7fa828907700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b2c500 sd=203 :6801 s=2 pgs=264 cs=397 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 09:43:25.232734 7fa8277f6700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b2d400 sd=201 :6801 s=0 pgs=0 cs=0 l=0 c=0x267e580).accept connect_seq 398 vs existing 397 state standby 2013-10-09 09:58:26.279387 7fa8277f6700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b2d400 sd=201 :6801 s=2 pgs=265 cs=399 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 10:23:05.302841 7fa828a08700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b2ca00 sd=203 :6801 s=0 pgs=0 cs=0 l=0 c=0x1c22c60).accept connect_seq 400 vs existing 399 state standby 2013-10-09 10:38:19.412179 7fa828a08700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b2ca00 sd=203 :6801 s=2 pgs=266 cs=401 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 10:43:23.861744 7fa828907700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b2cf00 sd=201 :6801 s=0 pgs=0 cs=0 l=0 c=0x1c23080).accept connect_seq 402 vs existing 401 state standby 2013-10-09 10:58:24.215779 7fa828907700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b2cf00 sd=201 :6801 s=2 pgs=267 cs=403 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 11:43:26.495497 7fa8277f6700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b2cc80 sd=203 :6801 s=0 pgs=0 cs=0 l=0 c=0x1d6fa20).accept connect_seq 404 vs existing 403 state standby 2013-10-09 11:58:27.554059 7fa8277f6700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b2cc80 sd=203 :6801 s=2 pgs=268 cs=405 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 12:43:23.374159 7fa828a08700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b2c280 sd=201 :6801 s=0 pgs=0 cs=0 l=0 c=0x1d6f8c0).accept connect_seq 406 vs existing 405 state standby 2013-10-09 12:58:27.817563 7fa828a08700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b2c280 sd=201 :6801 s=2 pgs=269 cs=407 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 13:43:26.059864 7fa828907700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2bfc500 sd=203 :6801 s=0 pgs=0 cs=0 l=0 c=0x2920f20).accept connect_seq 408 vs existing 407 state standby 2013-10-09 13:58:28.262377 7fa828907700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2bfc500 sd=203 :6801 s=2 pgs=270 cs=409 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 14:43:27.086532 7fa8277f6700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2bfc780 sd=201 :6801 s=0 pgs=0 cs=0 l=0 c=0x1d6dce0).accept connect_seq 410 vs existing 409 state standby 2013-10-09 14:58:28.205522 7fa8277f6700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2bfc780 sd=201 :6801 s=2 pgs=271 cs=411 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 15:43:25.587151 7fa828a08700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2bfd180 sd=203 :6801 s=0 pgs=0 cs=0 l=0 c=0x1d6d760).accept connect_seq 412 vs existing 411 state standby 2013-10-09 15:58:27.804846 7fa828a08700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2bfd180 sd=203 :6801 s=2 pgs=272 cs=413 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 16:43:26.063994 7fa828907700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2bfcc80 sd=201 :6801 s=0 pgs=0 cs=0 l=0 c=0x1d6de40).accept connect_seq 414 vs existing 413 state standby 2013-10-09 16:58:28.610013 7fa828907700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2bfcc80 sd=201 :6801 s=2 pgs=273 cs=415 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 17:43:28.450606 7fa8277f6700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b28780 sd=203 :6801 s=0 pgs=0 cs=0 l=0 c=0x2821600).accept connect_seq 416 vs existing 415 state standby 2013-10-09 17:58:29.802289 7fa8277f6700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b28780 sd=203 :6801 s=2 pgs=274 cs=417 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 18:43:28.822235 7fa828a08700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x29de780 sd=201 :6801 s=0 pgs=0 cs=0 l=0 c=0x28206e0).accept connect_seq 418 vs existing 417 state standby 2013-10-09 18:58:30.066106 7fa828a08700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x29de780 sd=201 :6801 s=2 pgs=275 cs=419 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 19:43:27.541886 7fa828907700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x29dec80 sd=203 :6801 s=0 pgs=0 cs=0 l=0 c=0x1d64580).accept connect_seq 420 vs existing 419 state standby 2013-10-09 19:58:28.977324 7fa828907700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x29dec80 sd=203 :6801 s=2 pgs=276 cs=421 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 20:43:29.817823 7fa8277f6700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b29900 sd=201 :6801 s=0 pgs=0 cs=0 l=0 c=0x29a1600).accept connect_seq 422 vs existing 421 state standby 2013-10-09 20:58:30.782568 7fa8277f6700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b29900 sd=201 :6801 s=2 pgs=277 cs=423 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 21:43:28.342014 7fa828a08700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b29680 sd=203 :6801 s=0 pgs=0 cs=0 l=0 c=0x29a1760).accept connect_seq 424 vs existing 423 state standby 2013-10-09 21:58:30.077639 7fa828a08700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b29680 sd=203 :6801 s=2 pgs=278 cs=425 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 22:43:31.639452 7fa828907700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b28f00 sd=201 :6801 s=0 pgs=0 cs=0 l=0 c=0x29a18c0).accept connect_seq 426 vs existing 425 state standby 2013-10-09 22:58:32.387940 7fa828907700 0 -- 10.10.2.202:6801/7645 >> 10.10.2.205:6809/7643 pipe(0x2b28f00 sd=201 :6801 s=2 pgs=279 cs=427 l=0 c=0x29202c0).fault with nothing to send, going to standby 2013-10-09 23:02:45.612712 7fa82fc73700 -1 osd.6 85 *** Got signal Terminated *** 2013-10-10 15:17:57.433250 7f945c5d97c0 0 ceph version 0.67.3 (408cd61584c72c0d97b774b3d8f95c6b1b06341a), process ceph-osd, pid 15169 2013-10-10 15:17:57.442039 7f945c5d97c0 0 filestore(/var/lib/ceph/osd/ceph-6) limited size xattrs -- filestore_xattr_use_omap already enabled 2013-10-10 15:17:57.448058 7f945c5d97c0 0 filestore(/var/lib/ceph/osd/ceph-6) mount FIEMAP ioctl is supported and appears to work 2013-10-10 15:17:57.448078 7f945c5d97c0 0 filestore(/var/lib/ceph/osd/ceph-6) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option 2013-10-10 15:17:57.448374 7f945c5d97c0 0 filestore(/var/lib/ceph/osd/ceph-6) mount did NOT detect btrfs 2013-10-10 15:17:57.472095 7f945c5d97c0 0 filestore(/var/lib/ceph/osd/ceph-6) mount syncfs(2) syscall fully supported (by glibc and kernel) 2013-10-10 15:17:57.472337 7f945c5d97c0 0 filestore(/var/lib/ceph/osd/ceph-6) mount found snaps <> 2013-10-10 15:17:57.566882 7f945c5d97c0 0 filestore(/var/lib/ceph/osd/ceph-6) mount: enabling WRITEAHEAD journal mode: btrfs not detected 2013-10-10 15:17:57.567538 7f945c5d97c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2013-10-10 15:17:57.567562 7f945c5d97c0 1 journal _open /var/lib/ceph/osd/ceph-6/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0 2013-10-10 15:17:57.621974 7f945c5d97c0 1 journal _open /var/lib/ceph/osd/ceph-6/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0 2013-10-10 15:17:57.622802 7f945c5d97c0 1 journal close /var/lib/ceph/osd/ceph-6/journal 2013-10-10 15:17:57.628953 7f945c5d97c0 0 filestore(/var/lib/ceph/osd/ceph-6) limited size xattrs -- filestore_xattr_use_omap already enabled 2013-10-10 15:17:57.633583 7f945c5d97c0 0 filestore(/var/lib/ceph/osd/ceph-6) mount FIEMAP ioctl is supported and appears to work 2013-10-10 15:17:57.633599 7f945c5d97c0 0 filestore(/var/lib/ceph/osd/ceph-6) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option 2013-10-10 15:17:57.633860 7f945c5d97c0 0 filestore(/var/lib/ceph/osd/ceph-6) mount did NOT detect btrfs 2013-10-10 15:17:57.644249 7f945c5d97c0 0 filestore(/var/lib/ceph/osd/ceph-6) mount syncfs(2) syscall fully supported (by glibc and kernel) 2013-10-10 15:17:57.644389 7f945c5d97c0 0 filestore(/var/lib/ceph/osd/ceph-6) mount found snaps <> 2013-10-10 15:17:57.647813 7f945c5d97c0 0 filestore(/var/lib/ceph/osd/ceph-6) mount: enabling WRITEAHEAD journal mode: btrfs not detected 2013-10-10 15:17:57.648204 7f945c5d97c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2013-10-10 15:17:57.648221 7f945c5d97c0 1 journal _open /var/lib/ceph/osd/ceph-6/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0 2013-10-10 15:17:57.648389 7f945c5d97c0 1 journal _open /var/lib/ceph/osd/ceph-6/journal fd 19: 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0 2013-10-10 15:17:57.700010 7f9443f68700 0 -- 0.0.0.0:6809/15169 >> 10.10.2.201:6809/12043 pipe(0x1d1e780 sd=24 :0 s=1 pgs=0 cs=0 l=0 c=0x1d1b4a0).fault with nothing to send, going to standby 2013-10-10 15:17:58.441932 7f9443e67700 0 -- :/15169 >> 10.10.2.201:6811/12043 pipe(0x1d1e500 sd=26 :0 s=1 pgs=0 cs=0 l=1 c=0x1d45340).fault 2013-10-10 15:17:58.441994 7f9443d66700 0 -- :/15169 >> 10.10.2.201:6810/12043 pipe(0x1d1ef00 sd=25 :0 s=1 pgs=0 cs=0 l=1 c=0x1d1be40).fault 2013-10-10 15:17:58.443504 7f9441d4d700 0 -- 10.10.2.202:0/15169 >> 10.10.2.201:6807/11820 pipe(0x1dd1180 sd=64 :0 s=1 pgs=0 cs=0 l=1 c=0x1dd5a20).fault 2013-10-10 15:17:58.443515 7f9441e4e700 0 -- 10.10.2.202:0/15169 >> 10.10.2.201:6806/11820 pipe(0x1dd1400 sd=63 :0 s=1 pgs=0 cs=0 l=1 c=0x1dd5ce0).fault 2013-10-10 15:17:58.443598 7f9443f68700 0 -- 10.10.2.202:6809/15169 >> 10.10.2.201:6809/12043 pipe(0x1d1e780 sd=24 :0 s=1 pgs=0 cs=1 l=0 c=0x1d1b4a0).fault 2013-10-10 15:18:00.095468 7f9444069700 0 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=1 pgs=0 cs=0 l=0 c=0x1d1b340).fault 2013-10-10 15:18:00.095505 7f9442656700 0 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=60 :0 s=1 pgs=0 cs=0 l=1 c=0x1dd4c60).fault 2013-10-10 15:18:00.095543 7f9442252700 0 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=61 :0 s=1 pgs=0 cs=0 l=1 c=0x1dd49a0).fault 2013-10-10 15:18:00.095553 7f9441f4f700 0 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=62 :0 s=1 pgs=0 cs=0 l=1 c=0x1dd46e0).fault 2013-10-10 15:18:00.095566 7f9442757700 0 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=59 :0 s=1 pgs=0 cs=0 l=1 c=0x1dd4000).fault 2013-10-10 15:18:22.695517 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:02.695514) 2013-10-10 15:18:22.695534 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:02.695514) 2013-10-10 15:18:23.695827 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:03.695825) 2013-10-10 15:18:23.695843 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:03.695825) 2013-10-10 15:18:24.696049 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:04.696047) 2013-10-10 15:18:24.696064 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:04.696047) 2013-10-10 15:18:25.696211 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:05.696209) 2013-10-10 15:18:25.696225 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:05.696209) 2013-10-10 15:18:26.696370 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:06.696368) 2013-10-10 15:18:26.696384 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:06.696368) 2013-10-10 15:18:26.895915 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:06.895914) 2013-10-10 15:18:26.895925 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:06.895914) 2013-10-10 15:18:27.696526 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:07.696524) 2013-10-10 15:18:27.696540 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:07.696524) 2013-10-10 15:18:28.696730 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:08.696728) 2013-10-10 15:18:28.696746 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:08.696728) 2013-10-10 15:18:29.697019 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:09.697018) 2013-10-10 15:18:29.697034 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:09.697018) 2013-10-10 15:18:30.697185 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:10.697183) 2013-10-10 15:18:30.697214 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:10.697183) 2013-10-10 15:18:31.697354 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:11.697352) 2013-10-10 15:18:31.697368 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:11.697352) 2013-10-10 15:18:32.697514 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:12.697512) 2013-10-10 15:18:32.697528 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:12.697512) 2013-10-10 15:18:32.796380 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:12.796379) 2013-10-10 15:18:32.796389 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:12.796379) 2013-10-10 15:18:33.697670 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:13.697668) 2013-10-10 15:18:33.697684 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:13.697668) 2013-10-10 15:18:34.697911 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:14.697909) 2013-10-10 15:18:34.697925 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:14.697909) 2013-10-10 15:18:35.696760 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:15.696759) 2013-10-10 15:18:35.696772 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:15.696759) 2013-10-10 15:18:35.698082 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:15.698080) 2013-10-10 15:18:35.698094 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:15.698080) 2013-10-10 15:18:36.698229 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:16.698227) 2013-10-10 15:18:36.698242 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:16.698227) 2013-10-10 15:18:37.698408 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:17.698406) 2013-10-10 15:18:37.698434 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:17.698406) 2013-10-10 15:18:37.997113 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:17.997111) 2013-10-10 15:18:37.997124 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:17.997111) 2013-10-10 15:18:38.698582 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:18.698580) 2013-10-10 15:18:38.698596 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:18.698580) 2013-10-10 15:18:39.698841 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:19.698839) 2013-10-10 15:18:39.698855 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:19.698839) 2013-10-10 15:18:40.699011 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:20.699009) 2013-10-10 15:18:40.699053 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:20.699009) 2013-10-10 15:18:41.699225 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:21.699223) 2013-10-10 15:18:41.699238 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:21.699223) 2013-10-10 15:18:42.097543 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:22.097541) 2013-10-10 15:18:42.097555 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:22.097541) 2013-10-10 15:18:42.699379 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:22.699377) 2013-10-10 15:18:42.699393 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:22.699377) 2013-10-10 15:18:43.197895 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:23.197894) 2013-10-10 15:18:43.197913 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:23.197894) 2013-10-10 15:18:43.699589 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:23.699587) 2013-10-10 15:18:43.699603 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:23.699587) 2013-10-10 15:18:44.699807 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:24.699805) 2013-10-10 15:18:44.699822 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:24.699805) 2013-10-10 15:18:45.700018 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:25.700016) 2013-10-10 15:18:45.700032 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:25.700016) 2013-10-10 15:18:46.698276 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:26.698274) 2013-10-10 15:18:46.698294 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:26.698274) 2013-10-10 15:18:46.700163 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:26.700161) 2013-10-10 15:18:46.700177 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:26.700161) 2013-10-10 15:18:47.700315 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:27.700313) 2013-10-10 15:18:47.700329 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:27.700313) 2013-10-10 15:18:48.700517 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:28.700515) 2013-10-10 15:18:48.700531 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:28.700515) 2013-10-10 15:18:49.700718 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:29.700716) 2013-10-10 15:18:49.700732 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:29.700716) 2013-10-10 15:18:50.198658 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:30.198657) 2013-10-10 15:18:50.198682 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:30.198657) 2013-10-10 15:18:50.700874 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:30.700872) 2013-10-10 15:18:50.700888 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:30.700872) 2013-10-10 15:18:51.701084 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:31.701082) 2013-10-10 15:18:51.701101 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:31.701082) 2013-10-10 15:18:52.701279 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:32.701277) 2013-10-10 15:18:52.701293 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:32.701277) 2013-10-10 15:18:53.099133 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:33.099131) 2013-10-10 15:18:53.099159 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:33.099131) 2013-10-10 15:18:53.701486 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:33.701484) 2013-10-10 15:18:53.701500 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:33.701484) 2013-10-10 15:18:54.701755 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:34.701753) 2013-10-10 15:18:54.701770 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:34.701753) 2013-10-10 15:18:55.701887 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:35.701884) 2013-10-10 15:18:55.701900 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:35.701884) 2013-10-10 15:18:55.999578 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:35.999576) 2013-10-10 15:18:55.999600 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:35.999576) 2013-10-10 15:18:56.702061 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:36.702059) 2013-10-10 15:18:56.702075 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:36.702059) 2013-10-10 15:18:57.702203 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:37.702201) 2013-10-10 15:18:57.702217 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:37.702201) 2013-10-10 15:18:58.299937 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:38.299936) 2013-10-10 15:18:58.299950 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:38.299936) 2013-10-10 15:18:58.702371 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:38.702369) 2013-10-10 15:18:58.702384 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:38.702369) 2013-10-10 15:18:59.702602 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:39.702600) 2013-10-10 15:18:59.702632 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:39.702600) 2013-10-10 15:19:00.702770 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:40.702768) 2013-10-10 15:19:00.702784 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:40.702768) 2013-10-10 15:19:01.702931 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:41.702929) 2013-10-10 15:19:01.702945 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:41.702929) 2013-10-10 15:19:02.703144 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:42.703142) 2013-10-10 15:19:02.703157 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:42.703142) 2013-10-10 15:19:03.600359 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:43.600357) 2013-10-10 15:19:03.600368 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:43.600357) 2013-10-10 15:19:03.703352 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:43.703350) 2013-10-10 15:19:03.703366 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:43.703350) 2013-10-10 15:19:04.700783 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:44.700781) 2013-10-10 15:19:04.700807 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:44.700781) 2013-10-10 15:19:04.703526 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:44.703524) 2013-10-10 15:19:04.703540 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:44.703524) 2013-10-10 15:19:05.703731 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:45.703729) 2013-10-10 15:19:05.703746 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:45.703729) 2013-10-10 15:19:06.703940 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:46.703938) 2013-10-10 15:19:06.703955 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:46.703938) 2013-10-10 15:19:07.704093 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:47.704091) 2013-10-10 15:19:07.704107 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:47.704091) 2013-10-10 15:19:08.704302 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:48.704300) 2013-10-10 15:19:08.704315 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:48.704300) 2013-10-10 15:19:09.704530 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:49.704528) 2013-10-10 15:19:09.704558 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:49.704528) 2013-10-10 15:19:10.601192 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:50.601190) 2013-10-10 15:19:10.601201 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:50.601190) 2013-10-10 15:19:10.704704 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:50.704702) 2013-10-10 15:19:10.704718 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:50.704702) 2013-10-10 15:19:11.704888 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:51.704886) 2013-10-10 15:19:11.704902 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:51.704886) 2013-10-10 15:19:12.705099 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:52.705097) 2013-10-10 15:19:12.705113 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:52.705097) 2013-10-10 15:19:13.705258 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:53.705256) 2013-10-10 15:19:13.705272 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:53.705256) 2013-10-10 15:19:14.701580 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:54.701578) 2013-10-10 15:19:14.701593 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:54.701578) 2013-10-10 15:19:14.705482 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:54.705481) 2013-10-10 15:19:14.705499 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:54.705481) 2013-10-10 15:19:15.201944 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:55.201942) 2013-10-10 15:19:15.201956 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:55.201942) 2013-10-10 15:19:15.705641 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:55.705639) 2013-10-10 15:19:15.705656 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:55.705639) 2013-10-10 15:19:16.705804 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:56.705802) 2013-10-10 15:19:16.705832 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:56.705802) 2013-10-10 15:19:17.706027 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:57.706025) 2013-10-10 15:19:17.706041 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:57.706025) 2013-10-10 15:19:18.706189 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:58.706187) 2013-10-10 15:19:18.706203 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:58.706187) 2013-10-10 15:19:19.706407 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:59.706405) 2013-10-10 15:19:19.706422 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:18:59.706405) 2013-10-10 15:19:20.502387 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:00.502386) 2013-10-10 15:19:20.502429 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:00.502386) 2013-10-10 15:19:20.706611 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:00.706609) 2013-10-10 15:19:20.706625 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:00.706609) 2013-10-10 15:19:21.706766 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:01.706764) 2013-10-10 15:19:21.706780 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:01.706764) 2013-10-10 15:19:22.202813 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:02.202811) 2013-10-10 15:19:22.202832 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:02.202811) 2013-10-10 15:19:22.706921 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:02.706919) 2013-10-10 15:19:22.706935 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:02.706919) 2013-10-10 15:19:23.303208 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:03.303207) 2013-10-10 15:19:23.303217 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:03.303207) 2013-10-10 15:19:23.707091 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:03.707089) 2013-10-10 15:19:23.707105 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:03.707089) 2013-10-10 15:19:24.707305 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:04.707303) 2013-10-10 15:19:24.707319 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:04.707303) 2013-10-10 15:19:25.707515 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:05.707513) 2013-10-10 15:19:25.707529 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:05.707513) 2013-10-10 15:19:26.707674 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:06.707672) 2013-10-10 15:19:26.707688 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:06.707672) 2013-10-10 15:19:27.403593 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:07.403591) 2013-10-10 15:19:27.403615 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:07.403591) 2013-10-10 15:19:27.707832 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:07.707830) 2013-10-10 15:19:27.707845 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:07.707830) 2013-10-10 15:19:28.707991 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:08.707989) 2013-10-10 15:19:28.708005 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:08.707989) 2013-10-10 15:19:29.708260 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:09.708258) 2013-10-10 15:19:29.708275 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:09.708258) 2013-10-10 15:19:30.708472 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:10.708470) 2013-10-10 15:19:30.708486 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:10.708470) 2013-10-10 15:19:31.708631 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:11.708629) 2013-10-10 15:19:31.708644 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:11.708629) 2013-10-10 15:19:32.708817 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:12.708815) 2013-10-10 15:19:32.708830 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:12.708815) 2013-10-10 15:19:33.304039 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:13.304037) 2013-10-10 15:19:33.304070 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:13.304037) 2013-10-10 15:19:33.708974 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:13.708972) 2013-10-10 15:19:33.708987 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:13.708972) 2013-10-10 15:19:34.404521 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:14.404519) 2013-10-10 15:19:34.404548 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:14.404519) 2013-10-10 15:19:34.709197 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:14.709195) 2013-10-10 15:19:34.709212 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:14.709195) 2013-10-10 15:19:35.709350 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:15.709348) 2013-10-10 15:19:35.709364 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:15.709348) 2013-10-10 15:19:36.709556 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:16.709554) 2013-10-10 15:19:36.709569 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:16.709554) 2013-10-10 15:19:37.709708 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:17.709706) 2013-10-10 15:19:37.709721 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:17.709706) 2013-10-10 15:19:38.709866 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:18.709864) 2013-10-10 15:19:38.709881 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:18.709864) 2013-10-10 15:19:39.710123 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:19.710121) 2013-10-10 15:19:39.710137 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:19.710121) 2013-10-10 15:19:40.304962 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:20.304960) 2013-10-10 15:19:40.304976 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:20.304960) 2013-10-10 15:19:40.710330 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:20.710328) 2013-10-10 15:19:40.710344 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:20.710328) 2013-10-10 15:19:41.710489 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:21.710487) 2013-10-10 15:19:41.710503 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:21.710487) 2013-10-10 15:19:42.710639 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:22.710637) 2013-10-10 15:19:42.710653 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:22.710637) 2013-10-10 15:19:43.710796 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:23.710794) 2013-10-10 15:19:43.710824 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:23.710794) 2013-10-10 15:19:43.805369 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:23.805367) 2013-10-10 15:19:43.805395 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:23.805367) 2013-10-10 15:19:44.711032 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:24.711030) 2013-10-10 15:19:44.711045 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:24.711030) 2013-10-10 15:19:45.711255 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:25.711253) 2013-10-10 15:19:45.711268 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:25.711253) 2013-10-10 15:19:46.711408 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:26.711406) 2013-10-10 15:19:46.711422 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:26.711406) 2013-10-10 15:19:47.711565 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:27.711563) 2013-10-10 15:19:47.711579 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:27.711563) 2013-10-10 15:19:48.711770 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:28.711768) 2013-10-10 15:19:48.711784 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:28.711768) 2013-10-10 15:19:49.105784 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:29.105782) 2013-10-10 15:19:49.105804 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:29.105782) 2013-10-10 15:19:49.606194 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:29.606192) 2013-10-10 15:19:49.606219 7f944596d700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:29.606192) 2013-10-10 15:19:49.711972 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.13 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:29.711970) 2013-10-10 15:19:49.711985 7f9455fab700 -1 osd.6 112 heartbeat_check: no reply from osd.14 ever on either front or back, first ping sent 2013-10-10 15:18:02.392990 (cutoff 2013-10-10 15:19:29.711970) 2013-10-10 15:19:49.804606 7f9443c65700 -1 osd.6 112 *** Got signal Terminated *** 2013-10-10 15:19:49.806149 7f9443c65700 -1 osd.6 112 shutdown 2013-10-10 15:19:49.806313 7f9443c65700 20 osd.6 112 kicking pg 2.31 2013-10-10 15:19:49.806331 7f9443c65700 30 osd.6 pg_epoch: 112 pg[2.31( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=66-107/6 inactive NOTIFY] lock 2013-10-10 15:19:49.806372 7f9443c65700 10 osd.6 pg_epoch: 112 pg[2.31( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=66-107/6 inactive NOTIFY] on_shutdown 2013-10-10 15:19:49.806414 7f9443c65700 10 osd.6 pg_epoch: 112 pg[2.31( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=66-107/6 inactive NOTIFY] clear_primary_state 2013-10-10 15:19:49.806440 7f9443c65700 10 osd.6 pg_epoch: 112 pg[2.31( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=66-107/6 inactive NOTIFY] cancel_recovery 2013-10-10 15:19:49.806461 7f9443c65700 10 osd.6 pg_epoch: 112 pg[2.31( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=66-107/6 inactive NOTIFY] clear_recovery_state 2013-10-10 15:19:49.806485 7f9443c65700 20 osd.6 112 kicking pg 1.32 2013-10-10 15:19:49.806509 7f9443c65700 30 osd.6 pg_epoch: 112 pg[1.32( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=54-107/7 inactive NOTIFY] lock 2013-10-10 15:19:49.806531 7f9443c65700 10 osd.6 pg_epoch: 112 pg[1.32( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=54-107/7 inactive NOTIFY] on_shutdown 2013-10-10 15:19:49.806549 7f9443c65700 10 osd.6 pg_epoch: 112 pg[1.32( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=54-107/7 inactive NOTIFY] clear_primary_state 2013-10-10 15:19:49.806562 7f9443c65700 10 osd.6 pg_epoch: 112 pg[1.32( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=54-107/7 inactive NOTIFY] cancel_recovery 2013-10-10 15:19:49.806574 7f9443c65700 10 osd.6 pg_epoch: 112 pg[1.32( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=54-107/7 inactive NOTIFY] clear_recovery_state 2013-10-10 15:19:49.806588 7f9443c65700 20 osd.6 112 kicking pg 0.33 2013-10-10 15:19:49.806591 7f9443c65700 30 osd.6 pg_epoch: 112 pg[0.33( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=54-107/7 inactive NOTIFY] lock 2013-10-10 15:19:49.806604 7f9443c65700 10 osd.6 pg_epoch: 112 pg[0.33( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=54-107/7 inactive NOTIFY] on_shutdown 2013-10-10 15:19:49.806617 7f9443c65700 10 osd.6 pg_epoch: 112 pg[0.33( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=54-107/7 inactive NOTIFY] clear_primary_state 2013-10-10 15:19:49.806629 7f9443c65700 10 osd.6 pg_epoch: 112 pg[0.33( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=54-107/7 inactive NOTIFY] cancel_recovery 2013-10-10 15:19:49.806640 7f9443c65700 10 osd.6 pg_epoch: 112 pg[0.33( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=54-107/7 inactive NOTIFY] clear_recovery_state 2013-10-10 15:19:49.806655 7f9443c65700 20 osd.6 112 kicking pg 2.19 2013-10-10 15:19:49.806661 7f9443c65700 30 osd.6 pg_epoch: 112 pg[2.19( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.806677 7f9443c65700 10 osd.6 pg_epoch: 112 pg[2.19( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] on_shutdown 2013-10-10 15:19:49.806692 7f9443c65700 10 osd.6 pg_epoch: 112 pg[2.19( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] clear_primary_state 2013-10-10 15:19:49.806706 7f9443c65700 10 osd.6 pg_epoch: 112 pg[2.19( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] cancel_recovery 2013-10-10 15:19:49.806717 7f9443c65700 10 osd.6 pg_epoch: 112 pg[2.19( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] clear_recovery_state 2013-10-10 15:19:49.806730 7f9443c65700 20 osd.6 112 kicking pg 1.1a 2013-10-10 15:19:49.806734 7f9443c65700 30 osd.6 pg_epoch: 112 pg[1.1a( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.806746 7f9443c65700 10 osd.6 pg_epoch: 112 pg[1.1a( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] on_shutdown 2013-10-10 15:19:49.806760 7f9443c65700 10 osd.6 pg_epoch: 112 pg[1.1a( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] clear_primary_state 2013-10-10 15:19:49.806773 7f9443c65700 10 osd.6 pg_epoch: 112 pg[1.1a( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] cancel_recovery 2013-10-10 15:19:49.806784 7f9443c65700 10 osd.6 pg_epoch: 112 pg[1.1a( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] clear_recovery_state 2013-10-10 15:19:49.806796 7f9443c65700 20 osd.6 112 kicking pg 0.1b 2013-10-10 15:19:49.806799 7f9443c65700 30 osd.6 pg_epoch: 112 pg[0.1b( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.806811 7f9443c65700 10 osd.6 pg_epoch: 112 pg[0.1b( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] on_shutdown 2013-10-10 15:19:49.806826 7f9443c65700 10 osd.6 pg_epoch: 112 pg[0.1b( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] clear_primary_state 2013-10-10 15:19:49.806838 7f9443c65700 10 osd.6 pg_epoch: 112 pg[0.1b( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] cancel_recovery 2013-10-10 15:19:49.806849 7f9443c65700 10 osd.6 pg_epoch: 112 pg[0.1b( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] clear_recovery_state 2013-10-10 15:19:49.806862 7f9443c65700 20 osd.6 112 kicking pg 6.0 2013-10-10 15:19:49.806865 7f9443c65700 30 osd.6 pg_epoch: 112 pg[6.0( v 85'1263 (0'0,85'1263] local-les=81 n=3 ec=28 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.806883 7f9443c65700 10 osd.6 pg_epoch: 112 pg[6.0( v 85'1263 (0'0,85'1263] local-les=81 n=3 ec=28 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] on_shutdown 2013-10-10 15:19:49.806902 7f9443c65700 10 osd.6 pg_epoch: 112 pg[6.0( v 85'1263 (0'0,85'1263] local-les=81 n=3 ec=28 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] clear_primary_state 2013-10-10 15:19:49.806917 7f9443c65700 10 osd.6 pg_epoch: 112 pg[6.0( v 85'1263 (0'0,85'1263] local-les=81 n=3 ec=28 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] cancel_recovery 2013-10-10 15:19:49.806931 7f9443c65700 10 osd.6 pg_epoch: 112 pg[6.0( v 85'1263 (0'0,85'1263] local-les=81 n=3 ec=28 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] clear_recovery_state 2013-10-10 15:19:49.806947 7f9443c65700 20 osd.6 112 kicking pg 4.2 2013-10-10 15:19:49.806950 7f9443c65700 30 osd.6 pg_epoch: 112 pg[4.2( empty local-les=81 n=0 ec=25 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.806962 7f9443c65700 10 osd.6 pg_epoch: 112 pg[4.2( empty local-les=81 n=0 ec=25 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] on_shutdown 2013-10-10 15:19:49.806976 7f9443c65700 10 osd.6 pg_epoch: 112 pg[4.2( empty local-les=81 n=0 ec=25 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] clear_primary_state 2013-10-10 15:19:49.807030 7f9443c65700 10 osd.6 pg_epoch: 112 pg[4.2( empty local-les=81 n=0 ec=25 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] cancel_recovery 2013-10-10 15:19:49.807051 7f9443c65700 10 osd.6 pg_epoch: 112 pg[4.2( empty local-les=81 n=0 ec=25 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] clear_recovery_state 2013-10-10 15:19:49.807073 7f9443c65700 20 osd.6 112 kicking pg 2.4 2013-10-10 15:19:49.807079 7f9443c65700 30 osd.6 pg_epoch: 112 pg[2.4( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.807102 7f9443c65700 10 osd.6 pg_epoch: 112 pg[2.4( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] on_shutdown 2013-10-10 15:19:49.807130 7f9443c65700 10 osd.6 pg_epoch: 112 pg[2.4( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] clear_primary_state 2013-10-10 15:19:49.807148 7f9443c65700 10 osd.6 pg_epoch: 112 pg[2.4( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] cancel_recovery 2013-10-10 15:19:49.807169 7f9443c65700 10 osd.6 pg_epoch: 112 pg[2.4( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] clear_recovery_state 2013-10-10 15:19:49.807184 7f9443c65700 20 osd.6 112 kicking pg 0.6 2013-10-10 15:19:49.807187 7f9443c65700 30 osd.6 pg_epoch: 112 pg[0.6( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.807199 7f9443c65700 10 osd.6 pg_epoch: 112 pg[0.6( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] on_shutdown 2013-10-10 15:19:49.807215 7f9443c65700 10 osd.6 pg_epoch: 112 pg[0.6( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] clear_primary_state 2013-10-10 15:19:49.807228 7f9443c65700 10 osd.6 pg_epoch: 112 pg[0.6( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] cancel_recovery 2013-10-10 15:19:49.807239 7f9443c65700 10 osd.6 pg_epoch: 112 pg[0.6( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] clear_recovery_state 2013-10-10 15:19:49.807252 7f9443c65700 20 osd.6 112 kicking pg 5.1 2013-10-10 15:19:49.807255 7f9443c65700 30 osd.6 pg_epoch: 112 pg[5.1( v 85'3 (0'0,85'3] local-les=81 n=3 ec=27 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.807270 7f9443c65700 10 osd.6 pg_epoch: 112 pg[5.1( v 85'3 (0'0,85'3] local-les=81 n=3 ec=27 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] on_shutdown 2013-10-10 15:19:49.807286 7f9443c65700 10 osd.6 pg_epoch: 112 pg[5.1( v 85'3 (0'0,85'3] local-les=81 n=3 ec=27 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] clear_primary_state 2013-10-10 15:19:49.807302 7f9443c65700 10 osd.6 pg_epoch: 112 pg[5.1( v 85'3 (0'0,85'3] local-les=81 n=3 ec=27 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] cancel_recovery 2013-10-10 15:19:49.807316 7f9443c65700 10 osd.6 pg_epoch: 112 pg[5.1( v 85'3 (0'0,85'3] local-les=81 n=3 ec=27 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] clear_recovery_state 2013-10-10 15:19:49.807330 7f9443c65700 20 osd.6 112 kicking pg 1.5 2013-10-10 15:19:49.807333 7f9443c65700 30 osd.6 pg_epoch: 112 pg[1.5( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.807345 7f9443c65700 10 osd.6 pg_epoch: 112 pg[1.5( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] on_shutdown 2013-10-10 15:19:49.807359 7f9443c65700 10 osd.6 pg_epoch: 112 pg[1.5( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] clear_primary_state 2013-10-10 15:19:49.807372 7f9443c65700 10 osd.6 pg_epoch: 112 pg[1.5( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] cancel_recovery 2013-10-10 15:19:49.807383 7f9443c65700 10 osd.6 pg_epoch: 112 pg[1.5( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] clear_recovery_state 2013-10-10 15:19:49.807395 7f9443c65700 20 osd.6 112 kicking pg 3.3 2013-10-10 15:19:49.807397 7f9443c65700 30 osd.6 pg_epoch: 112 pg[3.3( v 35'1 (0'0,35'1] local-les=81 n=1 ec=23 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.807412 7f9443c65700 10 osd.6 pg_epoch: 112 pg[3.3( v 35'1 (0'0,35'1] local-les=81 n=1 ec=23 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] on_shutdown 2013-10-10 15:19:49.807428 7f9443c65700 10 osd.6 pg_epoch: 112 pg[3.3( v 35'1 (0'0,35'1] local-les=81 n=1 ec=23 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] clear_primary_state 2013-10-10 15:19:49.807443 7f9443c65700 10 osd.6 pg_epoch: 112 pg[3.3( v 35'1 (0'0,35'1] local-les=81 n=1 ec=23 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] cancel_recovery 2013-10-10 15:19:49.807457 7f9443c65700 10 osd.6 pg_epoch: 112 pg[3.3( v 35'1 (0'0,35'1] local-les=81 n=1 ec=23 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] clear_recovery_state 2013-10-10 15:19:49.807472 7f9443c65700 20 osd.6 112 kicking pg 0.0 2013-10-10 15:19:49.807474 7f9443c65700 30 osd.6 pg_epoch: 112 pg[0.0( empty local-les=61 n=0 ec=1 les/c 61/61 110/110/108) [7] r=-1 lpr=110 pi=4-109/16 inactive NOTIFY] lock 2013-10-10 15:19:49.807486 7f9443c65700 10 osd.6 pg_epoch: 112 pg[0.0( empty local-les=61 n=0 ec=1 les/c 61/61 110/110/108) [7] r=-1 lpr=110 pi=4-109/16 inactive NOTIFY] on_shutdown 2013-10-10 15:19:49.807499 7f9443c65700 10 osd.6 pg_epoch: 112 pg[0.0( empty local-les=61 n=0 ec=1 les/c 61/61 110/110/108) [7] r=-1 lpr=110 pi=4-109/16 inactive NOTIFY] clear_primary_state 2013-10-10 15:19:49.807512 7f9443c65700 10 osd.6 pg_epoch: 112 pg[0.0( empty local-les=61 n=0 ec=1 les/c 61/61 110/110/108) [7] r=-1 lpr=110 pi=4-109/16 inactive NOTIFY] cancel_recovery 2013-10-10 15:19:49.807523 7f9443c65700 10 osd.6 pg_epoch: 112 pg[0.0( empty local-les=61 n=0 ec=1 les/c 61/61 110/110/108) [7] r=-1 lpr=110 pi=4-109/16 inactive NOTIFY] clear_recovery_state 2013-10-10 15:19:49.807787 7f9443c65700 10 osd.6 112 recovery tp stopped 2013-10-10 15:19:49.807948 7f9443c65700 10 osd.6 112 op tp stopped 2013-10-10 15:19:49.808141 7f9443c65700 10 osd.6 112 command tp stopped 2013-10-10 15:19:49.808344 7f9443c65700 10 osd.6 112 disk tp paused (new), kicking all pgs 2013-10-10 15:19:49.808352 7f9443c65700 10 osd.6 112 reset_heartbeat_peers 2013-10-10 15:19:49.808356 7f9443c65700 1 -- 10.10.2.202:0/15169 mark_down 0x1d1b8c0 -- 0x1dd0500 2013-10-10 15:19:49.808369 7f9443c65700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6806/15053 pipe(0x1dd0500 sd=53 :41795 s=2 pgs=2 cs=1 l=1 c=0x1d1b8c0).unregister_pipe 2013-10-10 15:19:49.808383 7f9443c65700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6806/15053 pipe(0x1dd0500 sd=53 :41795 s=2 pgs=2 cs=1 l=1 c=0x1d1b8c0).stop 2013-10-10 15:19:49.808439 7f9443c65700 1 -- 10.10.2.202:0/15169 mark_down 0x1d45600 -- 0x1dd0280 2013-10-10 15:19:49.808453 7f9443c65700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6807/15053 pipe(0x1dd0280 sd=54 :44343 s=2 pgs=2 cs=1 l=1 c=0x1d45600).unregister_pipe 2013-10-10 15:19:49.808439 7f9442151700 2 -- 10.10.2.202:0/15169 >> 10.10.2.202:6806/15053 pipe(0x1dd0500 sd=53 :41795 s=4 pgs=2 cs=1 l=1 c=0x1d1b8c0).reader couldn't read tag, Success 2013-10-10 15:19:49.808473 7f9443c65700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6807/15053 pipe(0x1dd0280 sd=54 :44343 s=2 pgs=2 cs=1 l=1 c=0x1d45600).stop 2013-10-10 15:19:49.808474 7f9442151700 2 -- 10.10.2.202:0/15169 >> 10.10.2.202:6806/15053 pipe(0x1dd0500 sd=53 :41795 s=4 pgs=2 cs=1 l=1 c=0x1d1b8c0).fault 0: Success 2013-10-10 15:19:49.808491 7f9442151700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6806/15053 pipe(0x1dd0500 sd=53 :41795 s=4 pgs=2 cs=1 l=1 c=0x1d1b8c0).fault already closed|closing 2013-10-10 15:19:49.808509 7f9442151700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6806/15053 pipe(0x1dd0500 sd=53 :41795 s=4 pgs=2 cs=1 l=1 c=0x1d1b8c0).reader done 2013-10-10 15:19:49.808529 7f9443c65700 1 -- 10.10.2.202:0/15169 mark_down 0x1d45e40 -- 0x1dd0000 2013-10-10 15:19:49.808540 7f9443c65700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6802/14946 pipe(0x1dd0000 sd=55 :53767 s=2 pgs=2 cs=1 l=1 c=0x1d45e40).unregister_pipe 2013-10-10 15:19:49.808559 7f9443c65700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6802/14946 pipe(0x1dd0000 sd=55 :53767 s=2 pgs=2 cs=1 l=1 c=0x1d45e40).stop 2013-10-10 15:19:49.808608 7f9443c65700 1 -- 10.10.2.202:0/15169 mark_down 0x1d45b80 -- 0x1dd0f00 2013-10-10 15:19:49.808618 7f9443c65700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6803/14946 pipe(0x1dd0f00 sd=56 :55829 s=2 pgs=2 cs=1 l=1 c=0x1d45b80).unregister_pipe 2013-10-10 15:19:49.808629 7f9443c65700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6803/14946 pipe(0x1dd0f00 sd=56 :55829 s=2 pgs=2 cs=1 l=1 c=0x1d45b80).stop 2013-10-10 15:19:49.808614 7f9442b5b700 20 -- 10.10.2.202:0/15169 >> 10.10.2.202:6802/14946 pipe(0x1dd0000 sd=55 :53767 s=4 pgs=2 cs=1 l=1 c=0x1d45e40).writer finishing 2013-10-10 15:19:49.808647 7f9442b5b700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6802/14946 pipe(0x1dd0000 sd=55 :53767 s=4 pgs=2 cs=1 l=1 c=0x1d45e40).writer done 2013-10-10 15:19:49.808684 7f9443c65700 1 -- 10.10.2.202:0/15169 mark_down 0x1dd4000 -- 0x1dd0780 2013-10-10 15:19:49.808696 7f9443c65700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=1 pgs=0 cs=0 l=1 c=0x1dd4000).unregister_pipe 2013-10-10 15:19:49.808670 7f9442454700 2 -- 10.10.2.202:0/15169 >> 10.10.2.202:6802/14946 pipe(0x1dd0000 sd=55 :53767 s=4 pgs=2 cs=1 l=1 c=0x1d45e40).reader couldn't read tag, Success 2013-10-10 15:19:49.808710 7f9443c65700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=1 pgs=0 cs=0 l=1 c=0x1dd4000).stop 2013-10-10 15:19:49.808717 7f9442454700 2 -- 10.10.2.202:0/15169 >> 10.10.2.202:6802/14946 pipe(0x1dd0000 sd=55 :53767 s=4 pgs=2 cs=1 l=1 c=0x1d45e40).fault 0: Success 2013-10-10 15:19:49.808697 7f9442a5a700 20 -- 10.10.2.202:0/15169 >> 10.10.2.202:6803/14946 pipe(0x1dd0f00 sd=56 :55829 s=4 pgs=2 cs=1 l=1 c=0x1d45b80).writer finishing 2013-10-10 15:19:49.808730 7f9442454700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6802/14946 pipe(0x1dd0000 sd=55 :53767 s=4 pgs=2 cs=1 l=1 c=0x1d45e40).fault already closed|closing 2013-10-10 15:19:49.808732 7f9443c65700 1 -- 10.10.2.202:0/15169 mark_down 0x1dd4c60 -- 0x1dd1b80 2013-10-10 15:19:49.808733 7f9442a5a700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6803/14946 pipe(0x1dd0f00 sd=56 :55829 s=4 pgs=2 cs=1 l=1 c=0x1d45b80).writer done 2013-10-10 15:19:49.808744 7f9443c65700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=1 pgs=0 cs=0 l=1 c=0x1dd4c60).unregister_pipe 2013-10-10 15:19:49.808744 7f9442454700 10 -- 10.10.2.202:0/15169 queue_reap 0x1dd0000 2013-10-10 15:19:49.808739 7f9442757700 2 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000).connect error 10.10.2.204:6806/7474, 104: Connection reset by peer 2013-10-10 15:19:49.808762 7f9443c65700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=1 pgs=0 cs=0 l=1 c=0x1dd4c60).stop 2013-10-10 15:19:49.808777 7f9442757700 3 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000).connect fault, but state = closed != connecting, stopping 2013-10-10 15:19:49.808799 7f9443c65700 1 -- 10.10.2.202:0/15169 mark_down 0x1dd49a0 -- 0x1dd1900 2013-10-10 15:19:49.808766 7f9442555700 2 -- 10.10.2.202:0/15169 >> 10.10.2.202:6803/14946 pipe(0x1dd0f00 sd=56 :55829 s=4 pgs=2 cs=1 l=1 c=0x1d45b80).reader couldn't read tag, Success 2013-10-10 15:19:49.808794 7f9442757700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000).writer finishing 2013-10-10 15:19:49.808813 7f9442757700 10 -- 10.10.2.202:0/15169 queue_reap 0x1dd0780 2013-10-10 15:19:49.808809 7f9443c65700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=1 pgs=0 cs=0 l=1 c=0x1dd49a0).unregister_pipe 2013-10-10 15:19:49.808811 7f9442555700 2 -- 10.10.2.202:0/15169 >> 10.10.2.202:6803/14946 pipe(0x1dd0f00 sd=56 :55829 s=4 pgs=2 cs=1 l=1 c=0x1d45b80).fault 0: Success 2013-10-10 15:19:49.808824 7f9443c65700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=1 pgs=0 cs=0 l=1 c=0x1dd49a0).stop 2013-10-10 15:19:49.808807 7f9442656700 2 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60).connect error 10.10.2.204:6807/7474, 104: Connection reset by peer 2013-10-10 15:19:49.808831 7f9442555700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6803/14946 pipe(0x1dd0f00 sd=56 :55829 s=4 pgs=2 cs=1 l=1 c=0x1d45b80).fault already closed|closing 2013-10-10 15:19:49.808850 7f9442555700 10 -- 10.10.2.202:0/15169 queue_reap 0x1dd0f00 2013-10-10 15:19:49.808841 7f9442656700 3 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60).connect fault, but state = closed != connecting, stopping 2013-10-10 15:19:49.808852 7f9443c65700 1 -- 10.10.2.202:0/15169 mark_down 0x1dd46e0 -- 0x1dd1680 2013-10-10 15:19:49.808832 7f9442d5d700 20 -- 10.10.2.202:0/15169 >> 10.10.2.202:6807/15053 pipe(0x1dd0280 sd=54 :44343 s=4 pgs=2 cs=1 l=1 c=0x1d45600).writer finishing 2013-10-10 15:19:49.808865 7f9443c65700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=1 pgs=0 cs=0 l=1 c=0x1dd46e0).unregister_pipe 2013-10-10 15:19:49.808860 7f9442656700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60).writer finishing 2013-10-10 15:19:49.808871 7f9442d5d700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6807/15053 pipe(0x1dd0280 sd=54 :44343 s=4 pgs=2 cs=1 l=1 c=0x1d45600).writer done 2013-10-10 15:19:49.808878 7f9443c65700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=1 pgs=0 cs=0 l=1 c=0x1dd46e0).stop 2013-10-10 15:19:49.808882 7f9442656700 10 -- 10.10.2.202:0/15169 queue_reap 0x1dd1b80 2013-10-10 15:19:49.808861 7f9442252700 2 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0).connect error 10.10.2.204:6810/7795, 104: Connection reset by peer 2013-10-10 15:19:49.808911 7f9441f4f700 2 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0).connect error 10.10.2.204:6811/7795, 104: Connection reset by peer 2013-10-10 15:19:49.808949 7f9441f4f700 3 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0).connect fault, but state = closed != connecting, stopping 2013-10-10 15:19:49.808943 7f9442353700 2 -- 10.10.2.202:0/15169 >> 10.10.2.202:6807/15053 pipe(0x1dd0280 sd=54 :44343 s=4 pgs=2 cs=1 l=1 c=0x1d45600).reader couldn't read tag, Success 2013-10-10 15:19:49.808971 7f9441f4f700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0).writer finishing 2013-10-10 15:19:49.808996 7f9441f4f700 10 -- 10.10.2.202:0/15169 queue_reap 0x1dd1680 2013-10-10 15:19:49.809000 7f9443c65700 10 osd.6 112 noting clean unmount in epoch 112 2013-10-10 15:19:49.808976 7f9442252700 3 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0).connect fault, but state = closed != connecting, stopping 2013-10-10 15:19:49.809000 7f9442757700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000).writer done 2013-10-10 15:19:49.809010 7f9441f4f700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0).writer done 2013-10-10 15:19:49.809014 7f9442252700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0).writer finishing 2013-10-10 15:19:49.809012 7f9443c65700 10 osd.6 112 write_superblock sb(66679100-960f-47ec-8c91-c22696ee8b8e osd.6 335f8fbf-2c92-4717-85df-f27ad5488fcc e112 [1,112] lci=[108,112]) 2013-10-10 15:19:49.809021 7f9442555700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6803/14946 pipe(0x1dd0f00 sd=56 :55829 s=4 pgs=2 cs=1 l=1 c=0x1d45b80).reader done 2013-10-10 15:19:49.809034 7f9442252700 10 -- 10.10.2.202:0/15169 queue_reap 0x1dd1900 2013-10-10 15:19:49.809049 7f9442252700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0).writer done 2013-10-10 15:19:49.809066 7f9443c65700 5 filestore(/var/lib/ceph/osd/ceph-6) queue_transactions existing osr(default 0x1a00af8)/0x1a00af8 2013-10-10 15:19:49.809087 7f9443c65700 10 journal op_submit_start 2487 2013-10-10 15:19:49.809090 7f9443c65700 5 filestore(/var/lib/ceph/osd/ceph-6) queue_transactions (writeahead) 2487 0x7f9443c64980 2013-10-10 15:19:49.808980 7f9442353700 2 -- 10.10.2.202:0/15169 >> 10.10.2.202:6807/15053 pipe(0x1dd0280 sd=54 :44343 s=4 pgs=2 cs=1 l=1 c=0x1d45600).fault 0: Success 2013-10-10 15:19:49.809095 7f9443c65700 10 journal op_journal_transactions 2487 0x7f9443c64980 2013-10-10 15:19:49.809103 7f9443c65700 5 journal submit_entry seq 2487 len 479 (0x1eef020) 2013-10-10 15:19:49.809098 7f9442353700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6807/15053 pipe(0x1dd0280 sd=54 :44343 s=4 pgs=2 cs=1 l=1 c=0x1d45600).fault already closed|closing 2013-10-10 15:19:49.809110 7f9443c65700 30 journal XXX throttle take 479 2013-10-10 15:19:49.809115 7f9442353700 10 -- 10.10.2.202:0/15169 queue_reap 0x1dd0280 2013-10-10 15:19:49.809126 7f9442353700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6807/15053 pipe(0x1dd0280 sd=54 :44343 s=4 pgs=2 cs=1 l=1 c=0x1d45600).reader done 2013-10-10 15:19:49.809131 7f9443c65700 10 journal op_submit_finish 2487 2013-10-10 15:19:49.809137 7f94547a8700 20 journal write_thread_entry woke up 2013-10-10 15:19:49.808933 7f9442e5e700 20 -- 10.10.2.202:0/15169 >> 10.10.2.202:6806/15053 pipe(0x1dd0500 sd=53 :41795 s=4 pgs=2 cs=1 l=1 c=0x1d1b8c0).writer finishing 2013-10-10 15:19:49.809149 7f94547a8700 10 journal room 1073737727 max_size 1073741824 pos 11493376 header.start 11493376 top 4096 2013-10-10 15:19:49.809150 7f9442e5e700 10 -- 10.10.2.202:0/15169 queue_reap 0x1dd0500 2013-10-10 15:19:49.809160 7f94547a8700 10 journal check_for_full at 11493376 : 4096 < 1073737727 2013-10-10 15:19:49.809163 7f9442e5e700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6806/15053 pipe(0x1dd0500 sd=53 :41795 s=4 pgs=2 cs=1 l=1 c=0x1d1b8c0).writer done 2013-10-10 15:19:49.809165 7f94547a8700 15 journal prepare_single_write 1 will write 11493376 : seq 2487 len 479 -> 4096 (head 40 pre_pad 0 ebl 479 post_pad 3537 tail 40) (ebl alignment -1) 2013-10-10 15:19:49.809172 7f9442656700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60).writer done 2013-10-10 15:19:49.809192 7f94547a8700 20 journal prepare_multi_write queue_pos now 11497472 2013-10-10 15:19:49.809214 7f94547a8700 15 journal do_write writing 11493376~4096 + header 2013-10-10 15:19:49.809302 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper 2013-10-10 15:19:49.809330 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper reaping pipe 0x1dd0780 10.10.2.204:6806/7474 2013-10-10 15:19:49.809338 7f94537a6700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000).discard_queue 2013-10-10 15:19:49.809361 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1ef0c40 2013-10-10 15:19:49.809372 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1ef16c0 2013-10-10 15:19:49.809382 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1df8e00 2013-10-10 15:19:49.809392 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1df9a40 2013-10-10 15:19:49.809400 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1efc8c0 2013-10-10 15:19:49.809409 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1ddc000 2013-10-10 15:19:49.809418 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f10fc0 2013-10-10 15:19:49.809427 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f101c0 2013-10-10 15:19:49.809435 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f11dc0 2013-10-10 15:19:49.809444 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1ec3dc0 2013-10-10 15:19:49.809453 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f116c0 2013-10-10 15:19:49.809462 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f13180 2013-10-10 15:19:49.809470 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1ef0fc0 2013-10-10 15:19:49.809478 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1efc1c0 2013-10-10 15:19:49.809487 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f128c0 2013-10-10 15:19:49.809496 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1efcc40 2013-10-10 15:19:49.809505 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f13a40 2013-10-10 15:19:49.809513 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f43180 2013-10-10 15:19:49.809522 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f3e8c0 2013-10-10 15:19:49.809530 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f40c40 2013-10-10 15:19:49.809539 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f436c0 2013-10-10 15:19:49.809547 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1efce00 2013-10-10 15:19:49.809556 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f43500 2013-10-10 15:19:49.809539 7f9442f5f700 2 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/15053 pipe(0x1d1fb80 sd=52 :6810 s=2 pgs=2 cs=1 l=1 c=0x1d44160).reader couldn't read tag, Success 2013-10-10 15:19:49.809564 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f12e00 2013-10-10 15:19:49.809573 7f9442f5f700 2 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/15053 pipe(0x1d1fb80 sd=52 :6810 s=2 pgs=2 cs=1 l=1 c=0x1d44160).fault 0: Success 2013-10-10 15:19:49.809578 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f67c00 2013-10-10 15:19:49.809595 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f66e00 2013-10-10 15:19:49.809616 7f9442f5f700 10 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/15053 pipe(0x1d1fb80 sd=52 :6810 s=2 pgs=2 cs=1 l=1 c=0x1d44160).fault on lossy channel, failing 2013-10-10 15:19:49.809611 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f66000 2013-10-10 15:19:49.809629 7f9442f5f700 10 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/15053 pipe(0x1d1fb80 sd=52 :6810 s=2 pgs=2 cs=1 l=1 c=0x1d44160).stop 2013-10-10 15:19:49.809631 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f40380 2013-10-10 15:19:49.809641 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f3fc00 2013-10-10 15:19:49.809611 7f9443060700 2 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/15053 pipe(0x1d1f900 sd=51 :6811 s=2 pgs=1 cs=1 l=1 c=0x1d44580).reader couldn't read tag, Success 2013-10-10 15:19:49.809650 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f69c00 2013-10-10 15:19:49.809658 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f3e1c0 2013-10-10 15:19:49.809647 7f9442f5f700 10 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/15053 pipe(0x1d1fb80 sd=52 :6810 s=4 pgs=2 cs=1 l=1 c=0x1d44160).unregister_pipe 2013-10-10 15:19:49.809666 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f87dc0 2013-10-10 15:19:49.809675 7f9442f5f700 10 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/15053 pipe(0x1d1fb80 sd=52 :6810 s=4 pgs=2 cs=1 l=1 c=0x1d44160).discard_queue 2013-10-10 15:19:49.809677 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000). discard 0x1f86fc0 2013-10-10 15:19:49.809677 7f9443060700 2 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/15053 pipe(0x1d1f900 sd=51 :6811 s=2 pgs=1 cs=1 l=1 c=0x1d44580).fault 0: Success 2013-10-10 15:19:49.809699 7f94537a6700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6806/7474 pipe(0x1dd0780 sd=58 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4000).unregister_pipe - not registered 2013-10-10 15:19:49.809702 7f9442f5f700 10 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/15053 pipe(0x1d1fb80 sd=52 :6810 s=4 pgs=2 cs=1 l=1 c=0x1d44160).reader done 2013-10-10 15:19:49.809734 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper reaped pipe 0x1dd0780 10.10.2.204:6806/7474 2013-10-10 15:19:49.809752 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper deleted pipe 0x1dd0780 2013-10-10 15:19:49.809762 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper reaping pipe 0x1dd1680 10.10.2.204:6811/7795 2013-10-10 15:19:49.809751 7f9443060700 10 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/15053 pipe(0x1d1f900 sd=51 :6811 s=2 pgs=1 cs=1 l=1 c=0x1d44580).fault on lossy channel, failing 2013-10-10 15:19:49.809733 7f9442c5c700 20 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/15053 pipe(0x1d1fb80 sd=52 :6810 s=4 pgs=2 cs=1 l=1 c=0x1d44160).writer finishing 2013-10-10 15:19:49.809775 7f94537a6700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0).discard_queue 2013-10-10 15:19:49.809786 7f9443060700 10 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/15053 pipe(0x1d1f900 sd=51 :6811 s=2 pgs=1 cs=1 l=1 c=0x1d44580).stop 2013-10-10 15:19:49.809792 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1ef1340 2013-10-10 15:19:49.809794 7f9442c5c700 10 -- 10.10.2.202:6810/15169 queue_reap 0x1d1fb80 2013-10-10 15:19:49.809786 7f9443464700 2 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f180 sd=49 :6810 s=2 pgs=25 cs=1 l=1 c=0x1d44000).reader couldn't read tag, Success 2013-10-10 15:19:49.809810 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1df8540 2013-10-10 15:19:49.809821 7f9442c5c700 10 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/15053 pipe(0x1d1fb80 sd=52 :6810 s=4 pgs=2 cs=1 l=1 c=0x1d44160).writer done 2013-10-10 15:19:49.809827 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1ef01c0 2013-10-10 15:19:49.809823 7f9443060700 10 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/15053 pipe(0x1d1f900 sd=51 :6811 s=4 pgs=1 cs=1 l=1 c=0x1d44580).unregister_pipe 2013-10-10 15:19:49.809838 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1df9500 2013-10-10 15:19:49.809833 7f9443464700 2 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f180 sd=49 :6810 s=2 pgs=25 cs=1 l=1 c=0x1d44000).fault 0: Success 2013-10-10 15:19:49.809848 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1ef08c0 2013-10-10 15:19:49.809846 7f9443060700 10 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/15053 pipe(0x1d1f900 sd=51 :6811 s=4 pgs=1 cs=1 l=1 c=0x1d44580).discard_queue 2013-10-10 15:19:49.809857 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1df8fc0 2013-10-10 15:19:49.809865 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f10a80 2013-10-10 15:19:49.809874 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f13340 2013-10-10 15:19:49.809882 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1efcfc0 2013-10-10 15:19:49.809842 7f9456fad700 10 -- 10.10.2.202:6810/15169 reaper 2013-10-10 15:19:49.809890 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f10540 2013-10-10 15:19:49.809887 7f9443060700 10 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/15053 pipe(0x1d1f900 sd=51 :6811 s=4 pgs=1 cs=1 l=1 c=0x1d44580).reader done 2013-10-10 15:19:49.809891 7f9443464700 10 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f180 sd=49 :6810 s=2 pgs=25 cs=1 l=1 c=0x1d44000).fault on lossy channel, failing 2013-10-10 15:19:49.809885 7f9443363700 2 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f400 sd=50 :6811 s=2 pgs=26 cs=1 l=1 c=0x1d44420).reader couldn't read tag, Success 2013-10-10 15:19:49.809904 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f13dc0 2013-10-10 15:19:49.809896 7f9456fad700 10 -- 10.10.2.202:6810/15169 reaper reaping pipe 0x1d1fb80 10.10.2.202:0/15053 2013-10-10 15:19:49.809915 7f9443464700 10 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f180 sd=49 :6810 s=2 pgs=25 cs=1 l=1 c=0x1d44000).stop 2013-10-10 15:19:49.809921 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f27c00 2013-10-10 15:19:49.809893 7f9442050700 20 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/15053 pipe(0x1d1f900 sd=51 :6811 s=4 pgs=1 cs=1 l=1 c=0x1d44580).writer finishing 2013-10-10 15:19:49.809927 7f9443363700 2 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f400 sd=50 :6811 s=2 pgs=26 cs=1 l=1 c=0x1d44420).fault 0: Success 2013-10-10 15:19:49.809927 7f9456fad700 10 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/15053 pipe(0x1d1fb80 sd=52 :6810 s=4 pgs=2 cs=1 l=1 c=0x1d44160).discard_queue 2013-10-10 15:19:49.809943 7f9442050700 10 -- 10.10.2.202:6811/15169 queue_reap 0x1d1f900 2013-10-10 15:19:49.809939 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f12000 2013-10-10 15:19:49.809930 7f9443161700 20 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f180 sd=49 :6810 s=4 pgs=25 cs=1 l=1 c=0x1d44000).writer finishing 2013-10-10 15:19:49.809961 7f9443161700 10 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f180 sd=49 :6810 s=4 pgs=25 cs=1 l=1 c=0x1d44000).writer done 2013-10-10 15:19:49.809955 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1efda40 2013-10-10 15:19:49.809962 7f9442050700 10 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/15053 pipe(0x1d1f900 sd=51 :6811 s=4 pgs=1 cs=1 l=1 c=0x1d44580).writer done 2013-10-10 15:19:49.809954 7f9456fad700 10 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/15053 pipe(0x1d1fb80 sd=52 :6810 s=4 pgs=2 cs=1 l=1 c=0x1d44160).unregister_pipe - not registered 2013-10-10 15:19:49.809979 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f12540 2013-10-10 15:19:49.809995 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1efc000 2013-10-10 15:19:49.809988 7f9443363700 10 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f400 sd=50 :6811 s=2 pgs=26 cs=1 l=1 c=0x1d44420).fault on lossy channel, failing 2013-10-10 15:19:49.810005 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f27a40 2013-10-10 15:19:49.810014 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f42c40 2013-10-10 15:19:49.810009 7f9456fad700 10 -- 10.10.2.202:6810/15169 reaper reaped pipe 0x1d1fb80 10.10.2.202:0/15053 2013-10-10 15:19:49.810025 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f27180 2013-10-10 15:19:49.810007 7f9443363700 10 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f400 sd=50 :6811 s=2 pgs=26 cs=1 l=1 c=0x1d44420).stop 2013-10-10 15:19:49.810039 7f9456fad700 10 -- 10.10.2.202:6810/15169 reaper deleted pipe 0x1d1fb80 2013-10-10 15:19:49.810049 7f9456fad700 10 -- 10.10.2.202:6810/15169 reaper done 2013-10-10 15:19:49.810042 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f408c0 2013-10-10 15:19:49.810061 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f26c40 2013-10-10 15:19:49.810078 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f26380 2013-10-10 15:19:49.810077 7f9453fa7700 10 -- 10.10.2.202:6811/15169 reaper 2013-10-10 15:19:49.810078 7f9443464700 10 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f180 sd=49 :6810 s=4 pgs=25 cs=1 l=1 c=0x1d44000).unregister_pipe 2013-10-10 15:19:49.810093 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1efd6c0 2013-10-10 15:19:49.810103 7f9453fa7700 10 -- 10.10.2.202:6811/15169 reaper reaping pipe 0x1d1f900 10.10.2.202:0/15053 2013-10-10 15:19:49.810109 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f12380 2013-10-10 15:19:49.810108 7f9443464700 10 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f180 sd=49 :6810 s=4 pgs=25 cs=1 l=1 c=0x1d44000).discard_queue 2013-10-10 15:19:49.810127 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f676c0 2013-10-10 15:19:49.810117 7f9453fa7700 10 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/15053 pipe(0x1d1f900 sd=51 :6811 s=4 pgs=1 cs=1 l=1 c=0x1d44580).discard_queue 2013-10-10 15:19:49.810140 7f9443464700 10 -- 10.10.2.202:6810/15169 queue_reap 0x1d1f180 2013-10-10 15:19:49.810146 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f668c0 2013-10-10 15:19:49.810154 7f9443464700 10 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f180 sd=49 :6810 s=4 pgs=25 cs=1 l=1 c=0x1d44000).reader done 2013-10-10 15:19:49.810149 7f9453fa7700 10 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/15053 pipe(0x1d1f900 sd=51 :6811 s=4 pgs=1 cs=1 l=1 c=0x1d44580).unregister_pipe - not registered 2013-10-10 15:19:49.810162 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f41a40 2013-10-10 15:19:49.810177 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f27880 2013-10-10 15:19:49.810186 7f9453fa7700 10 -- 10.10.2.202:6811/15169 reaper reaped pipe 0x1d1f900 10.10.2.202:0/15053 2013-10-10 15:19:49.810193 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f3f340 2013-10-10 15:19:49.810203 7f9453fa7700 10 -- 10.10.2.202:6811/15169 reaper deleted pipe 0x1d1f900 2013-10-10 15:19:49.810208 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f66380 2013-10-10 15:19:49.810213 7f9453fa7700 10 -- 10.10.2.202:6811/15169 reaper done 2013-10-10 15:19:49.810222 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f696c0 2013-10-10 15:19:49.810209 7f9443262700 20 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f400 sd=50 :6811 s=4 pgs=26 cs=1 l=1 c=0x1d44420).writer finishing 2013-10-10 15:19:49.810235 7f9456fad700 10 -- 10.10.2.202:6810/15169 reaper 2013-10-10 15:19:49.810243 7f9443262700 10 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f400 sd=50 :6811 s=4 pgs=26 cs=1 l=1 c=0x1d44420).writer done 2013-10-10 15:19:49.810241 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f87880 2013-10-10 15:19:49.810246 7f9456fad700 10 -- 10.10.2.202:6810/15169 reaper reaping pipe 0x1d1f180 10.10.2.202:0/14946 2013-10-10 15:19:49.810258 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0). discard 0x1f86a80 2013-10-10 15:19:49.810260 7f9456fad700 10 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f180 sd=49 :6810 s=4 pgs=25 cs=1 l=1 c=0x1d44000).discard_queue 2013-10-10 15:19:49.810280 7f94537a6700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6811/7795 pipe(0x1dd1680 sd=59 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd46e0).unregister_pipe - not registered 2013-10-10 15:19:49.810279 7f9456fad700 10 -- 10.10.2.202:6810/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f180 sd=49 :6810 s=4 pgs=25 cs=1 l=1 c=0x1d44000).unregister_pipe - not registered 2013-10-10 15:19:49.810299 7f9443363700 10 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f400 sd=50 :6811 s=4 pgs=26 cs=1 l=1 c=0x1d44420).unregister_pipe 2013-10-10 15:19:49.810314 7f9443363700 10 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f400 sd=50 :6811 s=4 pgs=26 cs=1 l=1 c=0x1d44420).discard_queue 2013-10-10 15:19:49.810316 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper reaped pipe 0x1dd1680 10.10.2.204:6811/7795 2013-10-10 15:19:49.810332 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper deleted pipe 0x1dd1680 2013-10-10 15:19:49.810344 7f9443363700 10 -- 10.10.2.202:6811/15169 queue_reap 0x1d1f400 2013-10-10 15:19:49.810347 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper reaping pipe 0x1dd0f00 10.10.2.202:6803/14946 2013-10-10 15:19:49.810355 7f9456fad700 10 -- 10.10.2.202:6810/15169 reaper reaped pipe 0x1d1f180 10.10.2.202:0/14946 2013-10-10 15:19:49.810361 7f94537a6700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6803/14946 pipe(0x1dd0f00 sd=56 :55829 s=4 pgs=2 cs=1 l=1 c=0x1d45b80).discard_queue 2013-10-10 15:19:49.810363 7f9443363700 10 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f400 sd=50 :6811 s=4 pgs=26 cs=1 l=1 c=0x1d44420).reader done 2013-10-10 15:19:49.810373 7f9453fa7700 10 -- 10.10.2.202:6811/15169 reaper 2013-10-10 15:19:49.810377 7f9456fad700 10 -- 10.10.2.202:6810/15169 reaper deleted pipe 0x1d1f180 2013-10-10 15:19:49.810377 7f94537a6700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6803/14946 pipe(0x1dd0f00 sd=56 :55829 s=4 pgs=2 cs=1 l=1 c=0x1d45b80).unregister_pipe - not registered 2013-10-10 15:19:49.810388 7f9456fad700 10 -- 10.10.2.202:6810/15169 reaper done 2013-10-10 15:19:49.810389 7f9453fa7700 10 -- 10.10.2.202:6811/15169 reaper reaping pipe 0x1d1f400 10.10.2.202:0/14946 2013-10-10 15:19:49.810398 7f9453fa7700 10 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f400 sd=50 :6811 s=4 pgs=26 cs=1 l=1 c=0x1d44420).discard_queue 2013-10-10 15:19:49.810412 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper reaped pipe 0x1dd0f00 10.10.2.202:6803/14946 2013-10-10 15:19:49.810426 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper deleted pipe 0x1dd0f00 2013-10-10 15:19:49.810414 7f9453fa7700 10 -- 10.10.2.202:6811/15169 >> 10.10.2.202:0/14946 pipe(0x1d1f400 sd=50 :6811 s=4 pgs=26 cs=1 l=1 c=0x1d44420).unregister_pipe - not registered 2013-10-10 15:19:49.810433 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper reaping pipe 0x1dd1900 10.10.2.204:6810/7795 2013-10-10 15:19:49.810444 7f94537a6700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0).discard_queue 2013-10-10 15:19:49.810457 7f9453fa7700 10 -- 10.10.2.202:6811/15169 reaper reaped pipe 0x1d1f400 10.10.2.202:0/14946 2013-10-10 15:19:49.810458 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1ef1500 2013-10-10 15:19:49.810473 7f9453fa7700 10 -- 10.10.2.202:6811/15169 reaper deleted pipe 0x1d1f400 2013-10-10 15:19:49.810474 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1df8700 2013-10-10 15:19:49.810482 7f9453fa7700 10 -- 10.10.2.202:6811/15169 reaper done 2013-10-10 15:19:49.810491 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1ef0380 2013-10-10 15:19:49.810506 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1df96c0 2013-10-10 15:19:49.810521 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1ef0700 2013-10-10 15:19:49.810536 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1df8a80 2013-10-10 15:19:49.810553 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f10c40 2013-10-10 15:19:49.810571 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f13500 2013-10-10 15:19:49.810586 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1efca80 2013-10-10 15:19:49.810602 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f10700 2013-10-10 15:19:49.810616 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1ddd340 2013-10-10 15:19:49.810633 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f27dc0 2013-10-10 15:19:49.810649 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f12700 2013-10-10 15:19:49.810664 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1efd880 2013-10-10 15:19:49.810680 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f12a80 2013-10-10 15:19:49.810695 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1efc540 2013-10-10 15:19:49.810710 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f136c0 2013-10-10 15:19:49.810726 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f42e00 2013-10-10 15:19:49.810743 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f26000 2013-10-10 15:19:49.810760 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f40700 2013-10-10 15:19:49.810775 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f428c0 2013-10-10 15:19:49.810793 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1ef1a40 2013-10-10 15:19:49.810807 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f42540 2013-10-10 15:19:49.810821 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f13c00 2013-10-10 15:19:49.810838 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f67880 2013-10-10 15:19:49.810854 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f66a80 2013-10-10 15:19:49.810868 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f41340 2013-10-10 15:19:49.810884 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f26a80 2013-10-10 15:19:49.810899 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f3e700 2013-10-10 15:19:49.810914 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f69880 2013-10-10 15:19:49.810928 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f68540 2013-10-10 15:19:49.810943 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f87a40 2013-10-10 15:19:49.810959 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0). discard 0x1f86c40 2013-10-10 15:19:49.811000 7f94537a6700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6810/7795 pipe(0x1dd1900 sd=57 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd49a0).unregister_pipe - not registered 2013-10-10 15:19:49.811030 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper reaped pipe 0x1dd1900 10.10.2.204:6810/7795 2013-10-10 15:19:49.811047 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper deleted pipe 0x1dd1900 2013-10-10 15:19:49.811056 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper reaping pipe 0x1dd0280 10.10.2.202:6807/15053 2013-10-10 15:19:49.811068 7f94537a6700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6807/15053 pipe(0x1dd0280 sd=54 :44343 s=4 pgs=2 cs=1 l=1 c=0x1d45600).discard_queue 2013-10-10 15:19:49.811085 7f94537a6700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6807/15053 pipe(0x1dd0280 sd=54 :44343 s=4 pgs=2 cs=1 l=1 c=0x1d45600).unregister_pipe - not registered 2013-10-10 15:19:49.811117 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper reaped pipe 0x1dd0280 10.10.2.202:6807/15053 2013-10-10 15:19:49.811134 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper deleted pipe 0x1dd0280 2013-10-10 15:19:49.811140 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper reaping pipe 0x1dd0500 10.10.2.202:6806/15053 2013-10-10 15:19:49.811150 7f94537a6700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6806/15053 pipe(0x1dd0500 sd=53 :41795 s=4 pgs=2 cs=1 l=1 c=0x1d1b8c0).discard_queue 2013-10-10 15:19:49.811165 7f94537a6700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6806/15053 pipe(0x1dd0500 sd=53 :41795 s=4 pgs=2 cs=1 l=1 c=0x1d1b8c0).unregister_pipe - not registered 2013-10-10 15:19:49.811220 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper reaped pipe 0x1dd0500 10.10.2.202:6806/15053 2013-10-10 15:19:49.811238 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper deleted pipe 0x1dd0500 2013-10-10 15:19:49.811246 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper reaping pipe 0x1dd1b80 10.10.2.204:6807/7474 2013-10-10 15:19:49.811256 7f94537a6700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60).discard_queue 2013-10-10 15:19:49.811270 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1ef0a80 2013-10-10 15:19:49.811285 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1df88c0 2013-10-10 15:19:49.811300 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1df8c40 2013-10-10 15:19:49.811316 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1df9880 2013-10-10 15:19:49.811331 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1efc380 2013-10-10 15:19:49.811348 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1df8380 2013-10-10 15:19:49.811364 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f10e00 2013-10-10 15:19:49.811379 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f10000 2013-10-10 15:19:49.811392 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f11880 2013-10-10 15:19:49.811409 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f108c0 2013-10-10 15:19:49.811424 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f11c00 2013-10-10 15:19:49.811438 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f12fc0 2013-10-10 15:19:49.811454 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1ef1180 2013-10-10 15:19:49.811468 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1efd180 2013-10-10 15:19:49.811483 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f12c40 2013-10-10 15:19:49.811499 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1efc700 2013-10-10 15:19:49.811514 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f13880 2013-10-10 15:19:49.811528 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f42fc0 2013-10-10 15:19:49.811543 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1ef1880 2013-10-10 15:19:49.811558 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f40a80 2013-10-10 15:19:49.811575 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f42700 2013-10-10 15:19:49.811591 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f11340 2013-10-10 15:19:49.811605 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f43340 2013-10-10 15:19:49.811622 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f121c0 2013-10-10 15:19:49.811638 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f67a40 2013-10-10 15:19:49.811653 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f66c40 2013-10-10 15:19:49.811668 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f41180 2013-10-10 15:19:49.811676 7f94547a8700 20 journal do_write latency 0.002455 2013-10-10 15:19:49.811685 7f94547a8700 20 journal do_write queueing finishers through seq 2487 2013-10-10 15:19:49.811683 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f26fc0 2013-10-10 15:19:49.811691 7f94547a8700 10 journal queue_completions_thru seq 2487 queueing seq 2487 0x1eef020 lat 0.002572 2013-10-10 15:19:49.811699 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f3f880 2013-10-10 15:19:49.811717 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f69a40 2013-10-10 15:19:49.811728 7f94547a8700 5 journal put_throttle finished 1 ops and 479 bytes, now 0 ops and 0 bytes 2013-10-10 15:19:49.811736 7f94547a8700 20 journal write_thread_entry going to sleep 2013-10-10 15:19:49.811731 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f68380 2013-10-10 15:19:49.811748 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f87c00 2013-10-10 15:19:49.811745 7f9451fa3700 5 filestore(/var/lib/ceph/osd/ceph-6) _journaled_ahead 0x1ec6d20 seq 2487 osr(default 0x1a00af8) 0x7f9443c64980 2013-10-10 15:19:49.811764 7f94537a6700 20 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60). discard 0x1f86e00 2013-10-10 15:19:49.811769 7f9451fa3700 5 filestore(/var/lib/ceph/osd/ceph-6) queue_op 0x1ec6d20 seq 2487 osr(default 0x1a00af8) 473 bytes (queue has 1 ops and 473 bytes) 2013-10-10 15:19:49.811778 7f94537a6700 10 -- 10.10.2.202:0/15169 >> 10.10.2.204:6807/7474 pipe(0x1dd1b80 sd=26 :0 s=4 pgs=0 cs=0 l=1 c=0x1dd4c60).unregister_pipe - not registered 2013-10-10 15:19:49.811799 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper reaped pipe 0x1dd1b80 10.10.2.204:6807/7474 2013-10-10 15:19:49.811808 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper deleted pipe 0x1dd1b80 2013-10-10 15:19:49.811813 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper done 2013-10-10 15:19:49.811807 7f9450fa1700 10 journal op_apply_start 2487 open_ops 0 -> 1 2013-10-10 15:19:49.811824 7f9450fa1700 5 filestore(/var/lib/ceph/osd/ceph-6) _do_op 0x1ec6d20 seq 2487 osr(default 0x1a00af8)/0x1a00af8 start 2013-10-10 15:19:49.811837 7f9450fa1700 10 filestore(/var/lib/ceph/osd/ceph-6) _do_transaction on 0x7f9443c64980 2013-10-10 15:19:49.811854 7f9450fa1700 15 filestore(/var/lib/ceph/osd/ceph-6) write meta/23c2fcde/osd_superblock/0//-1 0~354 2013-10-10 15:19:49.811839 7f9442454700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6802/14946 pipe(0x1dd0000 sd=55 :53767 s=4 pgs=2 cs=1 l=1 c=0x1d45e40).reader done 2013-10-10 15:19:49.811857 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper 2013-10-10 15:19:49.811869 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper reaping pipe 0x1dd0000 10.10.2.202:6802/14946 2013-10-10 15:19:49.811879 7f94537a6700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6802/14946 pipe(0x1dd0000 sd=55 :53767 s=4 pgs=2 cs=1 l=1 c=0x1d45e40).discard_queue 2013-10-10 15:19:49.811894 7f94537a6700 10 -- 10.10.2.202:0/15169 >> 10.10.2.202:6802/14946 pipe(0x1dd0000 sd=55 :53767 s=4 pgs=2 cs=1 l=1 c=0x1d45e40).unregister_pipe - not registered 2013-10-10 15:19:49.811925 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper reaped pipe 0x1dd0000 10.10.2.202:6802/14946 2013-10-10 15:19:49.811939 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper deleted pipe 0x1dd0000 2013-10-10 15:19:49.811943 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper done 2013-10-10 15:19:49.811960 7f9450fa1700 10 filestore(/var/lib/ceph/osd/ceph-6) write meta/23c2fcde/osd_superblock/0//-1 0~354 = 354 2013-10-10 15:19:49.811977 7f9450fa1700 10 journal op_apply_finish 2487 open_ops 1 -> 0, max_applied_seq 2486 -> 2487 2013-10-10 15:19:49.811985 7f9450fa1700 10 filestore(/var/lib/ceph/osd/ceph-6) _do_op 0x1ec6d20 seq 2487 r = 0, finisher 0x1eefa70 0 2013-10-10 15:19:49.812000 7f9450fa1700 10 filestore(/var/lib/ceph/osd/ceph-6) _finish_op 0x1ec6d20 seq 2487 osr(default 0x1a00af8)/0x1a00af8 2013-10-10 15:19:49.812042 7f9443c65700 10 osd.6 112 syncing store 2013-10-10 15:19:49.812052 7f9443c65700 10 filestore(/var/lib/ceph/osd/ceph-6) flush 2013-10-10 15:19:49.812055 7f9443c65700 10 journal waiting for completions to empty 2013-10-10 15:19:49.812058 7f9443c65700 10 journal flush waiting for finisher 2013-10-10 15:19:49.812061 7f9443c65700 10 journal flush done 2013-10-10 15:19:49.812062 7f9443c65700 10 filestore(/var/lib/ceph/osd/ceph-6) flush draining ondisk finisher 2013-10-10 15:19:49.812065 7f9443c65700 10 filestore(/var/lib/ceph/osd/ceph-6) _flush_op_queue draining op tp 2013-10-10 15:19:49.812068 7f9443c65700 10 filestore(/var/lib/ceph/osd/ceph-6) _flush_op_queue waiting for apply finisher 2013-10-10 15:19:49.812070 7f9443c65700 10 filestore(/var/lib/ceph/osd/ceph-6) flush complete 2013-10-10 15:19:49.812080 7f9443c65700 10 filestore(/var/lib/ceph/osd/ceph-6) start_sync 2013-10-10 15:19:49.812082 7f9443c65700 10 filestore(/var/lib/ceph/osd/ceph-6) sync waiting 2013-10-10 15:19:49.812103 7f9454fa9700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry woke after 2.156475 2013-10-10 15:19:49.812121 7f9454fa9700 10 journal commit_start max_applied_seq 2487, open_ops 0 2013-10-10 15:19:49.812127 7f9454fa9700 10 journal commit_start blocked, all open_ops have completed 2013-10-10 15:19:49.812143 7f9454fa9700 10 journal commit_start committing 2487, still blocked 2013-10-10 15:19:49.812148 7f9454fa9700 10 journal commit_start 2013-10-10 15:19:49.812164 7f9454fa9700 15 filestore(/var/lib/ceph/osd/ceph-6) sync_entry committing 2487 sync_epoch 2 2013-10-10 15:19:49.812173 7f9454fa9700 10 journal commit_started committing 2487, unblocking 2013-10-10 15:19:49.812184 7f9454fa9700 15 filestore(/var/lib/ceph/osd/ceph-6) sync_entry doing a full sync (syncfs(2) if possible) 2013-10-10 15:19:49.827501 7f9454fa9700 10 filestore(/var/lib/ceph/osd/ceph-6) sync_entry commit took 0.015347, interval was 2.171878 2013-10-10 15:19:49.827550 7f9454fa9700 10 journal commit_finish thru 2487 2013-10-10 15:19:49.827557 7f9454fa9700 5 journal committed_thru 2487 (last_committed_seq 2486) 2013-10-10 15:19:49.827564 7f9454fa9700 10 journal header: block_size 4096 alignment 4096 max_size 1073741824 2013-10-10 15:19:49.827567 7f9454fa9700 10 journal header: start 11497472 2013-10-10 15:19:49.827570 7f9454fa9700 10 journal write_pos 11497472 2013-10-10 15:19:49.827572 7f9454fa9700 10 journal committed_thru done 2013-10-10 15:19:49.827589 7f9454fa9700 15 filestore(/var/lib/ceph/osd/ceph-6) sync_entry committed to op_seq 2487 2013-10-10 15:19:49.827614 7f9454fa9700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry waiting for max_interval 5.000000 2013-10-10 15:19:49.827626 7f9443c65700 10 filestore(/var/lib/ceph/osd/ceph-6) sync done 2013-10-10 15:19:49.827635 7f9443c65700 5 filestore(/var/lib/ceph/osd/ceph-6) umount /var/lib/ceph/osd/ceph-6 2013-10-10 15:19:49.827666 7f9454fa9700 20 filestore(/var/lib/ceph/osd/ceph-6) sync_entry force_sync set 2013-10-10 15:19:49.827676 7f9454fa9700 10 journal commit_start max_applied_seq 2487, open_ops 0 2013-10-10 15:19:49.827681 7f9454fa9700 10 journal commit_start blocked, all open_ops have completed 2013-10-10 15:19:49.827685 7f9454fa9700 10 journal commit_start nothing to do 2013-10-10 15:19:49.827690 7f9454fa9700 10 journal commit_start 2013-10-10 15:19:49.828101 7f9443c65700 10 journal journal_stop 2013-10-10 15:19:49.828273 7f9443c65700 1 journal close /var/lib/ceph/osd/ceph-6/journal 2013-10-10 15:19:49.828300 7f94547a8700 20 journal write_thread_entry woke up 2013-10-10 15:19:49.828310 7f94547a8700 10 journal write_thread_entry finish 2013-10-10 15:19:49.828309 7f94527a4700 10 journal write_finish_thread_entry exit 2013-10-10 15:19:49.829583 7f9443c65700 10 osd.6 112 Store synced 2013-10-10 15:19:49.829609 7f9443c65700 20 osd.6 112 kicking pg 2.31 2013-10-10 15:19:49.829615 7f9443c65700 30 osd.6 pg_epoch: 112 pg[2.31( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=66-107/6 inactive NOTIFY] lock 2013-10-10 15:19:49.829702 7f9443c65700 20 osd.6 112 kicking pg 1.32 2013-10-10 15:19:49.829706 7f9443c65700 30 osd.6 pg_epoch: 112 pg[1.32( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=54-107/7 inactive NOTIFY] lock 2013-10-10 15:19:49.829737 7f9443c65700 20 osd.6 112 kicking pg 0.33 2013-10-10 15:19:49.829741 7f9443c65700 30 osd.6 pg_epoch: 112 pg[0.33( empty local-les=71 n=0 ec=1 les/c 71/71 108/108/70) [14,6] r=1 lpr=108 pi=54-107/7 inactive NOTIFY] lock 2013-10-10 15:19:49.829769 7f9443c65700 20 osd.6 112 kicking pg 2.19 2013-10-10 15:19:49.829772 7f9443c65700 30 osd.6 pg_epoch: 112 pg[2.19( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.829810 7f9443c65700 20 osd.6 112 kicking pg 1.1a 2013-10-10 15:19:49.829814 7f9443c65700 30 osd.6 pg_epoch: 112 pg[1.1a( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.829845 7f9443c65700 20 osd.6 112 kicking pg 0.1b 2013-10-10 15:19:49.829849 7f9443c65700 30 osd.6 pg_epoch: 112 pg[0.1b( empty local-les=50 n=0 ec=1 les/c 50/50 110/110/110) [6] r=0 lpr=110 pi=15-109/6 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.829876 7f9443c65700 20 osd.6 112 kicking pg 6.0 2013-10-10 15:19:49.829880 7f9443c65700 30 osd.6 pg_epoch: 112 pg[6.0( v 85'1263 (0'0,85'1263] local-les=81 n=3 ec=28 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.830179 7f9443c65700 20 osd.6 112 kicking pg 4.2 2013-10-10 15:19:49.830186 7f9443c65700 30 osd.6 pg_epoch: 112 pg[4.2( empty local-les=81 n=0 ec=25 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.830222 7f9443c65700 20 osd.6 112 kicking pg 2.4 2013-10-10 15:19:49.830226 7f9443c65700 30 osd.6 pg_epoch: 112 pg[2.4( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.830257 7f9443c65700 20 osd.6 112 kicking pg 0.6 2013-10-10 15:19:49.830262 7f9443c65700 30 osd.6 pg_epoch: 112 pg[0.6( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.830292 7f9443c65700 20 osd.6 112 kicking pg 5.1 2013-10-10 15:19:49.830296 7f9443c65700 30 osd.6 pg_epoch: 112 pg[5.1( v 85'3 (0'0,85'3] local-les=81 n=3 ec=27 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.830332 7f9443c65700 20 osd.6 112 kicking pg 1.5 2013-10-10 15:19:49.830336 7f9443c65700 30 osd.6 pg_epoch: 112 pg[1.5( empty local-les=81 n=0 ec=1 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.830365 7f9443c65700 20 osd.6 112 kicking pg 3.3 2013-10-10 15:19:49.830368 7f9443c65700 30 osd.6 pg_epoch: 112 pg[3.3( v 35'1 (0'0,35'1] local-les=81 n=1 ec=23 les/c 81/81 110/110/108) [6] r=0 lpr=110 pi=64-109/11 lcod 0'0 mlcod 0'0 down+peering] lock 2013-10-10 15:19:49.830402 7f9443c65700 20 osd.6 112 kicking pg 0.0 2013-10-10 15:19:49.830406 7f9443c65700 30 osd.6 pg_epoch: 112 pg[0.0( empty local-les=61 n=0 ec=1 les/c 61/61 110/110/108) [7] r=-1 lpr=110 pi=4-109/16 inactive NOTIFY] lock 2013-10-10 15:19:49.830609 7f9443c65700 1 -- 10.10.2.202:6808/15169 mark_down 0x1d1a2c0 -- 0x1d1e000 2013-10-10 15:19:49.830635 7f9443c65700 10 -- 10.10.2.202:6808/15169 >> 10.10.2.51:6789/0 pipe(0x1d1e000 sd=22 :34196 s=2 pgs=251 cs=1 l=1 c=0x1d1a2c0).unregister_pipe 2013-10-10 15:19:49.830651 7f9443c65700 10 -- 10.10.2.202:6808/15169 >> 10.10.2.51:6789/0 pipe(0x1d1e000 sd=22 :34196 s=2 pgs=251 cs=1 l=1 c=0x1d1a2c0).stop 2013-10-10 15:19:49.830712 7f945c538700 20 -- 10.10.2.202:6808/15169 >> 10.10.2.51:6789/0 pipe(0x1d1e000 sd=22 :34196 s=4 pgs=251 cs=1 l=1 c=0x1d1a2c0).writer finishing 2013-10-10 15:19:49.830754 7f945c538700 10 -- 10.10.2.202:6808/15169 >> 10.10.2.51:6789/0 pipe(0x1d1e000 sd=22 :34196 s=4 pgs=251 cs=1 l=1 c=0x1d1a2c0).writer done 2013-10-10 15:19:49.830758 7f944416a700 2 -- 10.10.2.202:6808/15169 >> 10.10.2.51:6789/0 pipe(0x1d1e000 sd=22 :34196 s=4 pgs=251 cs=1 l=1 c=0x1d1a2c0).reader couldn't read tag, Success 2013-10-10 15:19:49.830805 7f944416a700 2 -- 10.10.2.202:6808/15169 >> 10.10.2.51:6789/0 pipe(0x1d1e000 sd=22 :34196 s=4 pgs=251 cs=1 l=1 c=0x1d1a2c0).fault 0: Success 2013-10-10 15:19:49.830820 7f944416a700 10 -- 10.10.2.202:6808/15169 >> 10.10.2.51:6789/0 pipe(0x1d1e000 sd=22 :34196 s=4 pgs=251 cs=1 l=1 c=0x1d1a2c0).fault already closed|closing 2013-10-10 15:19:49.830841 7f944416a700 10 -- 10.10.2.202:6808/15169 queue_reap 0x1d1e000 2013-10-10 15:19:49.830855 7f944416a700 10 -- 10.10.2.202:6808/15169 >> 10.10.2.51:6789/0 pipe(0x1d1e000 sd=22 :34196 s=4 pgs=251 cs=1 l=1 c=0x1d1a2c0).reader done 2013-10-10 15:19:49.830867 7f9452fa5700 10 -- 10.10.2.202:6808/15169 reaper 2013-10-10 15:19:49.830937 7f9452fa5700 10 -- 10.10.2.202:6808/15169 reaper reaping pipe 0x1d1e000 10.10.2.51:6789/0 2013-10-10 15:19:49.830947 7f9452fa5700 10 -- 10.10.2.202:6808/15169 >> 10.10.2.51:6789/0 pipe(0x1d1e000 sd=22 :34196 s=4 pgs=251 cs=1 l=1 c=0x1d1a2c0).discard_queue 2013-10-10 15:19:49.830963 7f9452fa5700 10 -- 10.10.2.202:6808/15169 >> 10.10.2.51:6789/0 pipe(0x1d1e000 sd=22 :34196 s=4 pgs=251 cs=1 l=1 c=0x1d1a2c0).unregister_pipe - not registered 2013-10-10 15:19:49.831006 7f9452fa5700 10 -- 10.10.2.202:6808/15169 reaper reaped pipe 0x1d1e000 10.10.2.51:6789/0 2013-10-10 15:19:49.831021 7f9452fa5700 10 -- 10.10.2.202:6808/15169 reaper deleted pipe 0x1d1e000 2013-10-10 15:19:49.831027 7f9452fa5700 10 -- 10.10.2.202:6808/15169 reaper done 2013-10-10 15:19:49.831384 7f9443c65700 10 -- 10.10.2.202:6808/15169 shutdown 10.10.2.202:6808/15169 2013-10-10 15:19:49.831403 7f9443c65700 1 -- 10.10.2.202:6808/15169 mark_down_all 2013-10-10 15:19:49.831415 7f9443c65700 10 -- 10.10.2.202:6809/15169 shutdown 10.10.2.202:6809/15169 2013-10-10 15:19:49.831421 7f9443c65700 1 -- 10.10.2.202:6809/15169 mark_down_all 2013-10-10 15:19:49.831425 7f9443c65700 5 -- 10.10.2.202:6809/15169 mark_down_all 10.10.2.204:6809/7795 0x1d1e280 2013-10-10 15:19:49.831433 7f9443c65700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=1 pgs=0 cs=0 l=0 c=0x1d1b340).unregister_pipe - not registered 2013-10-10 15:19:49.831446 7f9443c65700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=1 pgs=0 cs=0 l=0 c=0x1d1b340).stop 2013-10-10 15:19:49.831490 7f9443c65700 5 -- 10.10.2.202:6809/15169 mark_down_all 10.10.2.202:6801/14946 0x1d1ef00 2013-10-10 15:19:49.831498 7f9443c65700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6801/14946 pipe(0x1d1ef00 sd=62 :42886 s=2 pgs=26 cs=1 l=0 c=0x1d1b4a0).unregister_pipe - not registered 2013-10-10 15:19:49.831493 7f944c97b700 1 osd.6 0 ms_handle_reset con 0x1d1b340 session 0 2013-10-10 15:19:49.831508 7f9443c65700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6801/14946 pipe(0x1d1ef00 sd=62 :42886 s=2 pgs=26 cs=1 l=0 c=0x1d1b4a0).stop 2013-10-10 15:19:49.831505 7f9444069700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340).fault done waiting or woke up 2013-10-10 15:19:49.831536 7f9444069700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340).writer finishing 2013-10-10 15:19:49.831556 7f9444069700 10 -- 10.10.2.202:6809/15169 queue_reap 0x1d1e280 2013-10-10 15:19:49.831583 7f944c97b700 1 osd.6 0 ms_handle_reset con 0x1d1b4a0 session 0 2013-10-10 15:19:49.831566 7f9442858700 2 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6801/14946 pipe(0x1d1ef00 sd=62 :42886 s=4 pgs=26 cs=1 l=0 c=0x1d1b4a0).reader couldn't read tag, Success 2013-10-10 15:19:49.831603 7f9443c65700 5 -- 10.10.2.202:6809/15169 mark_down_all 10.10.2.202:6805/15053 0x1e04500 2013-10-10 15:19:49.831607 7f9442858700 2 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6801/14946 pipe(0x1d1ef00 sd=62 :42886 s=4 pgs=26 cs=1 l=0 c=0x1d1b4a0).fault 0: Success 2013-10-10 15:19:49.831623 7f9443c65700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6805/15053 pipe(0x1e04500 sd=73 :46606 s=2 pgs=14 cs=1 l=0 c=0x1e06160).unregister_pipe - not registered 2013-10-10 15:19:49.831634 7f9442858700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6801/14946 pipe(0x1d1ef00 sd=62 :42886 s=4 pgs=26 cs=1 l=0 c=0x1d1b4a0).fault already closed|closing 2013-10-10 15:19:49.831628 7f945c5d97c0 10 -- 10.10.2.202:6808/15169 wait: dispatch queue is stopped 2013-10-10 15:19:49.831642 7f9443c65700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6805/15053 pipe(0x1e04500 sd=73 :46606 s=2 pgs=14 cs=1 l=0 c=0x1e06160).stop 2013-10-10 15:19:49.831654 7f945c5d97c0 20 -- 10.10.2.202:6808/15169 wait: stopping accepter thread 2013-10-10 15:19:49.831665 7f945c5d97c0 10 accepter.stop accepter 2013-10-10 15:19:49.831658 7f9442858700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6801/14946 pipe(0x1d1ef00 sd=62 :42886 s=4 pgs=26 cs=1 l=0 c=0x1d1b4a0).reader done 2013-10-10 15:19:49.831664 7f9442959700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6801/14946 pipe(0x1d1ef00 sd=62 :42886 s=4 pgs=26 cs=1 l=0 c=0x1d1b4a0).writer finishing 2013-10-10 15:19:49.831701 7f9442959700 10 -- 10.10.2.202:6809/15169 queue_reap 0x1d1ef00 2013-10-10 15:19:49.831700 7f944d17c700 20 accepter.accepter poll got 1 2013-10-10 15:19:49.831718 7f944d17c700 20 accepter.accepter closing 2013-10-10 15:19:49.831717 7f944c97b700 1 osd.6 0 ms_handle_reset con 0x1e06160 session 0 2013-10-10 15:19:49.831705 7f9441a4a700 2 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6805/15053 pipe(0x1e04500 sd=73 :46606 s=4 pgs=14 cs=1 l=0 c=0x1e06160).reader couldn't read tag, Success 2013-10-10 15:19:49.831734 7f944d17c700 10 accepter.accepter stopping 2013-10-10 15:19:49.831723 7f9443c65700 10 -- 10.10.2.202:0/15169 shutdown 10.10.2.202:0/15169 2013-10-10 15:19:49.831722 7f9442959700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6801/14946 pipe(0x1d1ef00 sd=62 :42886 s=4 pgs=26 cs=1 l=0 c=0x1d1b4a0).writer done 2013-10-10 15:19:49.831732 7f9441a4a700 2 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6805/15053 pipe(0x1e04500 sd=73 :46606 s=4 pgs=14 cs=1 l=0 c=0x1e06160).fault 0: Success 2013-10-10 15:19:49.831727 7f9444069700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340).writer done 2013-10-10 15:19:49.831742 7f9443c65700 1 -- 10.10.2.202:0/15169 mark_down_all 2013-10-10 15:19:49.831746 7f9441a4a700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6805/15053 pipe(0x1e04500 sd=73 :46606 s=4 pgs=14 cs=1 l=0 c=0x1e06160).fault already closed|closing 2013-10-10 15:19:49.831758 7f9443c65700 10 -- 10.10.2.202:6811/15169 shutdown 10.10.2.202:6811/15169 2013-10-10 15:19:49.831763 7f9441a4a700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6805/15053 pipe(0x1e04500 sd=73 :46606 s=4 pgs=14 cs=1 l=0 c=0x1e06160).reader done 2013-10-10 15:19:49.831772 7f9443c65700 1 -- 10.10.2.202:6811/15169 mark_down_all 2013-10-10 15:19:49.831745 7f94567ac700 10 -- 10.10.2.202:6809/15169 reaper 2013-10-10 15:19:49.831773 7f9441b4b700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6805/15053 pipe(0x1e04500 sd=73 :46606 s=4 pgs=14 cs=1 l=0 c=0x1e06160).writer finishing 2013-10-10 15:19:49.831783 7f94567ac700 10 -- 10.10.2.202:6809/15169 reaper reaping pipe 0x1d1ef00 10.10.2.202:6801/14946 2013-10-10 15:19:49.831790 7f9443c65700 10 -- 10.10.2.202:6810/15169 shutdown 10.10.2.202:6810/15169 2013-10-10 15:19:49.831801 7f9441b4b700 10 -- 10.10.2.202:6809/15169 queue_reap 0x1e04500 2013-10-10 15:19:49.831803 7f9443c65700 1 -- 10.10.2.202:6810/15169 mark_down_all 2013-10-10 15:19:49.831798 7f94567ac700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6801/14946 pipe(0x1d1ef00 sd=62 :42886 s=4 pgs=26 cs=1 l=0 c=0x1d1b4a0).discard_queue 2013-10-10 15:19:49.831821 7f94567ac700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6801/14946 pipe(0x1d1ef00 sd=62 :42886 s=4 pgs=26 cs=1 l=0 c=0x1d1b4a0).unregister_pipe - not registered 2013-10-10 15:19:49.831886 7f945c5d97c0 20 -- 10.10.2.202:6808/15169 wait: stopped accepter thread 2013-10-10 15:19:49.831902 7f945c5d97c0 20 -- 10.10.2.202:6808/15169 wait: stopping reaper thread 2013-10-10 15:19:49.831922 7f9452fa5700 10 -- 10.10.2.202:6808/15169 reaper_entry done 2013-10-10 15:19:49.831921 7f94567ac700 10 -- 10.10.2.202:6809/15169 reaper reaped pipe 0x1d1ef00 10.10.2.202:6801/14946 2013-10-10 15:19:49.831944 7f94567ac700 10 -- 10.10.2.202:6809/15169 reaper deleted pipe 0x1d1ef00 2013-10-10 15:19:49.831953 7f94567ac700 10 -- 10.10.2.202:6809/15169 reaper reaping pipe 0x1d1e280 10.10.2.204:6809/7795 2013-10-10 15:19:49.831976 7f945c5d97c0 20 -- 10.10.2.202:6808/15169 wait: stopped reaper thread 2013-10-10 15:19:49.831965 7f94567ac700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340).discard_queue 2013-10-10 15:19:49.831987 7f945c5d97c0 10 -- 10.10.2.202:6808/15169 wait: closing pipes 2013-10-10 15:19:49.831993 7f945c5d97c0 10 -- 10.10.2.202:6808/15169 reaper 2013-10-10 15:19:49.831999 7f945c5d97c0 10 -- 10.10.2.202:6808/15169 reaper done 2013-10-10 15:19:49.831989 7f94567ac700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340). discard 0x1ddc380 2013-10-10 15:19:49.832003 7f945c5d97c0 10 -- 10.10.2.202:6808/15169 wait: waiting for pipes to close 2013-10-10 15:19:49.832009 7f945c5d97c0 10 -- 10.10.2.202:6808/15169 wait: done. 2013-10-10 15:19:49.832012 7f945c5d97c0 1 -- 10.10.2.202:6808/15169 shutdown complete. 2013-10-10 15:19:49.832017 7f945c5d97c0 10 -- 10.10.2.202:0/15169 wait: waiting for dispatch queue 2013-10-10 15:19:49.832017 7f94567ac700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340). discard 0x1ddc700 2013-10-10 15:19:49.832044 7f945c5d97c0 10 -- 10.10.2.202:0/15169 wait: dispatch queue is stopped 2013-10-10 15:19:49.832052 7f945c5d97c0 20 -- 10.10.2.202:0/15169 wait: stopping reaper thread 2013-10-10 15:19:49.832044 7f94567ac700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340). discard 0x1ddcc40 2013-10-10 15:19:49.832068 7f94537a6700 10 -- 10.10.2.202:0/15169 reaper_entry done 2013-10-10 15:19:49.832074 7f94567ac700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340). discard 0x1ddce00 2013-10-10 15:19:49.832097 7f94567ac700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340). discard 0x1ec2700 2013-10-10 15:19:49.832122 7f94567ac700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340). discard 0x1ec2540 2013-10-10 15:19:49.832159 7f945c5d97c0 20 -- 10.10.2.202:0/15169 wait: stopped reaper thread 2013-10-10 15:19:49.832169 7f945c5d97c0 10 -- 10.10.2.202:0/15169 wait: closing pipes 2013-10-10 15:19:49.832158 7f94567ac700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340). discard 0x1ec2380 2013-10-10 15:19:49.832173 7f945c5d97c0 10 -- 10.10.2.202:0/15169 reaper 2013-10-10 15:19:49.832181 7f945c5d97c0 10 -- 10.10.2.202:0/15169 reaper done 2013-10-10 15:19:49.832184 7f945c5d97c0 10 -- 10.10.2.202:0/15169 wait: waiting for pipes to close 2013-10-10 15:19:49.832188 7f945c5d97c0 10 -- 10.10.2.202:0/15169 wait: done. 2013-10-10 15:19:49.832194 7f945c5d97c0 1 -- 10.10.2.202:0/15169 shutdown complete. 2013-10-10 15:19:49.832198 7f945c5d97c0 10 -- 10.10.2.202:6811/15169 wait: waiting for dispatch queue 2013-10-10 15:19:49.832186 7f94567ac700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340). discard 0x1ec21c0 2013-10-10 15:19:49.832216 7f945c5d97c0 10 -- 10.10.2.202:6811/15169 wait: dispatch queue is stopped 2013-10-10 15:19:49.832224 7f945c5d97c0 20 -- 10.10.2.202:6811/15169 wait: stopping accepter thread 2013-10-10 15:19:49.832228 7f945c5d97c0 10 accepter.stop accepter 2013-10-10 15:19:49.832219 7f94567ac700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340). discard 0x1ec2000 2013-10-10 15:19:49.832248 7f944a977700 20 accepter.accepter poll got 1 2013-10-10 15:19:49.832258 7f944a977700 20 accepter.accepter closing 2013-10-10 15:19:49.832245 7f94567ac700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340). discard 0x1ddddc0 2013-10-10 15:19:49.832277 7f944a977700 10 accepter.accepter stopping 2013-10-10 15:19:49.832269 7f94567ac700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340). discard 0x1ddc8c0 2013-10-10 15:19:49.832289 7f94567ac700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340). discard 0x1ddca80 2013-10-10 15:19:49.832313 7f94567ac700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340). discard 0x1ddd180 2013-10-10 15:19:49.832345 7f945c5d97c0 20 -- 10.10.2.202:6811/15169 wait: stopped accepter thread 2013-10-10 15:19:49.832355 7f945c5d97c0 20 -- 10.10.2.202:6811/15169 wait: stopping reaper thread 2013-10-10 15:19:49.832344 7f94567ac700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340). discard 0x1ddc540 2013-10-10 15:19:49.832375 7f94567ac700 20 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340). discard 0x1ddcfc0 2013-10-10 15:19:49.832379 7f9453fa7700 10 -- 10.10.2.202:6811/15169 reaper_entry done 2013-10-10 15:19:49.832403 7f94567ac700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.204:6809/7795 pipe(0x1d1e280 sd=23 :0 s=4 pgs=0 cs=0 l=0 c=0x1d1b340).unregister_pipe - not registered 2013-10-10 15:19:49.832456 7f945c5d97c0 20 -- 10.10.2.202:6811/15169 wait: stopped reaper thread 2013-10-10 15:19:49.832466 7f945c5d97c0 10 -- 10.10.2.202:6811/15169 wait: closing pipes 2013-10-10 15:19:49.832470 7f945c5d97c0 10 -- 10.10.2.202:6811/15169 reaper 2013-10-10 15:19:49.832474 7f945c5d97c0 10 -- 10.10.2.202:6811/15169 reaper done 2013-10-10 15:19:49.832477 7f945c5d97c0 10 -- 10.10.2.202:6811/15169 wait: waiting for pipes to close 2013-10-10 15:19:49.832481 7f945c5d97c0 10 -- 10.10.2.202:6811/15169 wait: done. 2013-10-10 15:19:49.832489 7f945c5d97c0 1 -- 10.10.2.202:6811/15169 shutdown complete. 2013-10-10 15:19:49.832492 7f945c5d97c0 10 -- 10.10.2.202:6810/15169 wait: waiting for dispatch queue 2013-10-10 15:19:49.832512 7f945c5d97c0 10 -- 10.10.2.202:6810/15169 wait: dispatch queue is stopped 2013-10-10 15:19:49.832519 7f945c5d97c0 20 -- 10.10.2.202:6810/15169 wait: stopping accepter thread 2013-10-10 15:19:49.832524 7f945c5d97c0 10 accepter.stop accepter 2013-10-10 15:19:49.832514 7f94567ac700 10 -- 10.10.2.202:6809/15169 reaper reaped pipe 0x1d1e280 10.10.2.204:6809/7795 2013-10-10 15:19:49.832542 7f94567ac700 10 -- 10.10.2.202:6809/15169 reaper deleted pipe 0x1d1e280 2013-10-10 15:19:49.832544 7f9449975700 20 accepter.accepter poll got 1 2013-10-10 15:19:49.832555 7f9449975700 20 accepter.accepter closing 2013-10-10 15:19:49.832550 7f94567ac700 10 -- 10.10.2.202:6809/15169 reaper done 2013-10-10 15:19:49.832567 7f9449975700 10 accepter.accepter stopping 2013-10-10 15:19:49.832588 7f9441b4b700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6805/15053 pipe(0x1e04500 sd=73 :46606 s=4 pgs=14 cs=1 l=0 c=0x1e06160).writer done 2013-10-10 15:19:49.832599 7f94567ac700 10 -- 10.10.2.202:6809/15169 reaper 2013-10-10 15:19:49.832613 7f945c5d97c0 20 -- 10.10.2.202:6810/15169 wait: stopped accepter thread 2013-10-10 15:19:49.832622 7f945c5d97c0 20 -- 10.10.2.202:6810/15169 wait: stopping reaper thread 2013-10-10 15:19:49.832618 7f94567ac700 10 -- 10.10.2.202:6809/15169 reaper reaping pipe 0x1e04500 10.10.2.202:6805/15053 2013-10-10 15:19:49.832641 7f9456fad700 10 -- 10.10.2.202:6810/15169 reaper_entry done 2013-10-10 15:19:49.832636 7f94567ac700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6805/15053 pipe(0x1e04500 sd=73 :46606 s=4 pgs=14 cs=1 l=0 c=0x1e06160).discard_queue 2013-10-10 15:19:49.832657 7f94567ac700 10 -- 10.10.2.202:6809/15169 >> 10.10.2.202:6805/15053 pipe(0x1e04500 sd=73 :46606 s=4 pgs=14 cs=1 l=0 c=0x1e06160).unregister_pipe - not registered 2013-10-10 15:19:49.832706 7f945c5d97c0 20 -- 10.10.2.202:6810/15169 wait: stopped reaper thread 2013-10-10 15:19:49.832716 7f945c5d97c0 10 -- 10.10.2.202:6810/15169 wait: closing pipes 2013-10-10 15:19:49.832720 7f945c5d97c0 10 -- 10.10.2.202:6810/15169 reaper 2013-10-10 15:19:49.832723 7f945c5d97c0 10 -- 10.10.2.202:6810/15169 reaper done 2013-10-10 15:19:49.832727 7f945c5d97c0 10 -- 10.10.2.202:6810/15169 wait: waiting for pipes to close 2013-10-10 15:19:49.832731 7f945c5d97c0 10 -- 10.10.2.202:6810/15169 wait: done. 2013-10-10 15:19:49.832734 7f945c5d97c0 1 -- 10.10.2.202:6810/15169 shutdown complete. 2013-10-10 15:19:49.832750 7f94567ac700 10 -- 10.10.2.202:6809/15169 reaper reaped pipe 0x1e04500 10.10.2.202:6805/15053 2013-10-10 15:19:49.832776 7f94567ac700 10 -- 10.10.2.202:6809/15169 reaper deleted pipe 0x1e04500 2013-10-10 15:19:49.832785 7f94567ac700 10 -- 10.10.2.202:6809/15169 reaper done 2013-10-10 15:19:49.832805 7f945c5d97c0 10 -- 10.10.2.202:6809/15169 wait: waiting for dispatch queue 2013-10-10 15:19:49.832829 7f945c5d97c0 10 -- 10.10.2.202:6809/15169 wait: dispatch queue is stopped 2013-10-10 15:19:49.832837 7f945c5d97c0 20 -- 10.10.2.202:6809/15169 wait: stopping accepter thread 2013-10-10 15:19:49.832841 7f945c5d97c0 10 accepter.stop accepter 2013-10-10 15:19:49.832864 7f944c17a700 20 accepter.accepter poll got 1 2013-10-10 15:19:49.832874 7f944c17a700 20 accepter.accepter closing 2013-10-10 15:19:49.832891 7f944c17a700 10 accepter.accepter stopping 2013-10-10 15:19:49.832956 7f945c5d97c0 20 -- 10.10.2.202:6809/15169 wait: stopped accepter thread 2013-10-10 15:19:49.832969 7f945c5d97c0 20 -- 10.10.2.202:6809/15169 wait: stopping reaper thread 2013-10-10 15:19:49.832995 7f94567ac700 10 -- 10.10.2.202:6809/15169 reaper_entry done 2013-10-10 15:19:49.833072 7f945c5d97c0 20 -- 10.10.2.202:6809/15169 wait: stopped reaper thread 2013-10-10 15:19:49.833082 7f945c5d97c0 10 -- 10.10.2.202:6809/15169 wait: closing pipes 2013-10-10 15:19:49.833086 7f945c5d97c0 10 -- 10.10.2.202:6809/15169 reaper 2013-10-10 15:19:49.833090 7f945c5d97c0 10 -- 10.10.2.202:6809/15169 reaper done 2013-10-10 15:19:49.833093 7f945c5d97c0 10 -- 10.10.2.202:6809/15169 wait: waiting for pipes to close 2013-10-10 15:19:49.833097 7f945c5d97c0 10 -- 10.10.2.202:6809/15169 wait: done. 2013-10-10 15:19:49.833101 7f945c5d97c0 1 -- 10.10.2.202:6809/15169 shutdown complete.
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
