OK, I've had a look. Haven't been able to take a proper look at the network yet but here's what I've gathered on other fronts so far: * Marking either osd.595 or osd.7 out results in this: $ ceph health detail | grep -v stuck | grep 1.323 pg 1.323 is remapped+peering, acting [2147483647,1391,240,127,937,362,267,320,7,634,716] The only way to fix this is to restart 595 and 1391 a couple times. Then you get a proper set with 595(0) and a peering state as opposed to remapped+peering. * I have looked through the PG mappings and ** PG 1.323 is the only PG which has both 595 and 7 in its acting set. ** there are 218 PGs which have OSDs that live on both the hosts that 595 and 7 live on Given the above information I'm very inclined to think it's a network issue. If it were I'd expect at least another PG that requires the same network path to work to be failing. As it stands this persists after: * having restarted all OSDs in the acting set with osd_find_best_info_ignore_history_les = true * having restarted both hosts that the OSDs failing to talk to each other live on * marking either OSD out and allowing recovery to finish Also worth noting that after multiple restarts, osd.595 is still not responding to `ceph tell osd.595` and `ceph pg 1.323 query`. Anybody have any clue as to what this might be? George ________________________________________ From: ceph-users [ceph-users-bounces@xxxxxxxxxxxxxx] on behalf of george.vasilakakos@xxxxxxxxxx [george.vasilakakos@xxxxxxxxxx] Sent: 08 February 2017 18:32 To: gfarnum@xxxxxxxxxx Cc: ceph-users@xxxxxxxxxxxxxx Subject: Re: PG stuck peering after host reboot Hey Greg, Thanks for your quick responses. I have to leave the office now but I'll look deeper into it tomorrow to try and understand what's the cause of this. I'll try to find other peerings between these two hosts and check those OSDs' logs for potential anomalies. I'll also have a look at any potential configuration changes that might have affected the host post-reboot. I'll be back here with more info once I have it tomorrow. Thanks again! George ________________________________________ From: Gregory Farnum [gfarnum@xxxxxxxxxx] Sent: 08 February 2017 18:29 To: Vasilakakos, George (STFC,RAL,SC) Cc: Ceph Users Subject: Re: PG stuck peering after host reboot On Wed, Feb 8, 2017 at 10:25 AM, <george.vasilakakos@xxxxxxxxxx> wrote: > Hi Greg, > >> Yes, "bad crc" indicates that the checksums on an incoming message did >> not match what was provided — ie, the message got corrupted. You >> shouldn't try and fix that by playing around with the peering settings >> as it's not a peering bug. >> Unless there's a bug in the messaging layer causing this (very >> unlikely), you have bad hardware or a bad network configuration >> (people occasionally talk about MTU settings?). Fix that and things >> will work; don't and the only software tweaks you could apply are more >> likely to result in lost data than a happy cluster. >> -Greg > > > I thought of the network initially but I didn't observe packet loss between the two hosts and neither host is having trouble talking to the rest of its peers. It's these two OSDs that can't talk to each other so I figured it's not likely to be a network issue. Network monitoring does show virtually non-existent inbound traffic over those links compared to the other ports on the switch but no other peerings fail. > > Is there something you can suggest to do to drill down deeper? Sadly no. It being a single route is indeed weird and hopefully somebody with more networking background can suggest a cause. :) > Also, am I correct in assuming that I can pull one of these OSDs from the cluster as a last resort to cause a remapping to a different to potentially give this a quick/temp fix and get the cluster serving I/O properly again? I'd expect so! _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |