Hello, to quote Sherlock Holmes: "Data, data, data. I cannot make bricks without clay." That the number of blocked requests is varying is indeed interesting, but I presume you're more interested in fixing this than dissecting this particular tidbit? If so... Start with the basics, all relevant software version, a description of your cluster, full outputs of "ceph osd tree" and "ceph -s", etc. The same 2 OSDs are affected, anything peculiar going on in their logs? How about their SMART status? Are they being deep-scrubbed (logs above) or otherwise busy (atop, iostat)? You may find something in the performance counters, blocked requests section, see: http://ceph.com/docs/v0.69/dev/perf_counters/ Lastly, the most likely fix will be restarting the affected OSDs. See also: https://www.mail-archive.com/ceph-users@xxxxxxxxxxxxxx/msg15410.html Christian On Mon, 13 Jul 2015 22:38:57 +0000 Deneau, Tom wrote: > I have a cluster where over the weekend something happened and > successive calls to ceph health detail show things like below. What does > it mean when the number of blocked requests goes up and down like this? > Some clients are still running successfully. > > -- Tom Deneau, AMD > > > > HEALTH_WARN 20 requests are blocked > 32 sec; 2 osds have slow requests > 20 ops are blocked > 536871 sec > 2 ops are blocked > 536871 sec on osd.5 > 18 ops are blocked > 536871 sec on osd.7 > 2 osds have slow requests > > HEALTH_WARN 4 requests are blocked > 32 sec; 2 osds have slow requests > 4 ops are blocked > 536871 sec > 2 ops are blocked > 536871 sec on osd.5 > 2 ops are blocked > 536871 sec on osd.7 > 2 osds have slow requests > > HEALTH_WARN 27 requests are blocked > 32 sec; 2 osds have slow requests > 27 ops are blocked > 536871 sec > 2 ops are blocked > 536871 sec on osd.5 > 25 ops are blocked > 536871 sec on osd.7 > 2 osds have slow requests > > HEALTH_WARN 34 requests are blocked > 32 sec; 2 osds have slow requests > 34 ops are blocked > 536871 sec > 9 ops are blocked > 536871 sec on osd.5 > 25 ops are blocked > 536871 sec on osd.7 > 2 osds have slow requests > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Fusion Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |