The trick with debugging heartbeat errors is to grep back through the log to find the last thing the affected thread was doing, e.g. is 0x7f5affe72700 stuck in messaging, writing to the disk, reading through the omap, etc..
I agree this doesn't look to be network related, but if you want to rule it out you should set debug_ms=1.
Last week we upgraded a 1200 osd cluster from firefly to 0.94.5 and similarly started getting slow requests. To make a long story short, our issue turned out to be sendmsg blocking (very rarely), probably due to an ancient el6 kernel (these osd servers had ~800 days' uptime). The signature of this was 900s of slow requests, then an ms log showing "initiating reconnect". Until we got the kernel upgraded everywhere, we used a workaround of ms tcp read timeout = 60.
So, check your kernels, and upgrade if they're ancient. Latest el6 kernels work for us.
Otherwise, those huge osd leveldb's don't look right. (Unless you're using tons and tons of omap...) And it kinda reminds me of the other problem we hit during the hammer upgrade was a return of ever growing mon leveldb's. The solution to that was to recreate the mons one by one. Perhaps you've hit something similar with the OSDs.
Good luck,
Dan
On 30 Nov 2015 20:56, "Tom Christensen" <pavera@xxxxxxxxx> wrote:
>
> We recently upgraded to 0.94.3 from firefly and now for the last week have had intermittent slow requests and flapping OSDs. We have been unable to nail down the cause, but its feeling like it may be related to our osdmaps not getting deleted properly. Most of our osds are now storing over 100GB of data in the meta directory, almost all of that is historical osd maps going back over 7 days old.
>
> We did do a small cluster change (We added 35 OSDs to a 1445 OSD cluster), the rebalance took about 36 hours, and it completed 10 days ago. Since that time the cluster has been HEALTH_OK and all pgs have been active+clean except for when we have an OSD flap.
>
> When the OSDs flap they do not crash and restart, they just go unresponsive for 1-3 minutes, and then come back alive all on their own. They get marked down by peers, and cause some peering and then they just come back rejoin the cluster and continue on their merry way.
>
> We see a bunch of this in the logs while the OSD is catatonic:
>
> Nov 30 11:23:38 osd-10 ceph-osd: 2015-11-30 11:22:32.143166 7f5b03679700 1 heartbeat_map is_healthy 'OSD::osd_tp thread 0x7f5affe72700' had timed out after 15
>
> Nov 30 11:23:38 osd-10 ceph-osd: 2015-11-30 11:22:32.143176 7f5b03679700 10 osd.1191 1203850 internal heartbeat not healthy, dropping ping request
>
> Nov 30 11:23:38 osd-10 ceph-osd: 2015-11-30 11:22:32.143210 7f5b04e7c700 1 heartbeat_map is_healthy 'OSD::osd_tp thread 0x7f5affe72700' had timed out after 15
>
> Nov 30 11:23:38 osd-10 ceph-osd: 2015-11-30 11:22:32.143218 7f5b04e7c700 10 osd.1191 1203850 internal heartbeat not healthy, dropping ping request
>
> Nov 30 11:23:38 osd-10 ceph-osd: 2015-11-30 11:22:32.143288 7f5b03679700 1 heartbeat_map is_healthy 'OSD::osd_tp thread 0x7f5affe72700' had timed out after 15
>
> Nov 30 11:23:38 osd-10 ceph-osd: 2015-11-30 11:22:32.143293 7f5b03679700 10 osd.1191 1203850 internal heartbeat not healthy, dropping ping request
>
>
> I have a chunk of logs at debug 20/5, not sure if I should have done just 20... It's pretty hard to catch, we have to basically see the slow requests and get debug logging set in about a 5-10 second window before the OSD stops responding to the admin socket...
>
> As networking is almost always the cause of flapping OSDs we have tested the network quite extensively. It hasn't changed physically since before the hammer upgrade, and was performing well. We have done large amounts of ping tests and have not seen a single dropped packet between osd nodes or between osd nodes and mons.
>
> I don't see any error packets or drops on switches either.
>
> Ideas?
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
