Hello, Hello, On Tue, 24 May 2016 15:32:02 -0700 Gregory Farnum wrote: > On Tue, May 24, 2016 at 2:16 PM, Heath Albritton <halbritt@xxxxxxxx> > wrote: > > Having some problems with my cluster. Wondering if I could get some > > troubleshooting tips: > > > > Running hammer 0.94.5. Small cluster with cache tiering. 3 spinning > > nodes and 3 SSD nodes. > > > > Lots of blocked ops. OSDs are consuming the entirety of the system > > memory (128GB) and then falling over. Lots of blocked ops, slow > > requests. Seeing logs like this: > > > > 2016-05-24 19:30:09.288941 7f63c126b700 1 heartbeat_map is_healthy > > 'FileStore::op_tp thread 0x7f63cb3cd700' had timed out after 60 > > 2016-05-24 19:30:09.503712 7f63c5273700 0 log_channel(cluster) log > > [WRN] : map e7779 wrongly marked me down > > 2016-05-24 19:30:11.190178 7f63cabcc700 0 -- > > 10.164.245.22:6831/5013886 submit_message MOSDPGPushReply(9.10d 7762 > > [PushReplyOp(3110010d/rbd_data.9647882ae8944a.00000000000026e7/head//9)]) > > v2 remote, 10.164.245.23:6821/3028423, failed lossy con, dropping > > message 0xfc21e00 > > 2016-05-24 19:30:22.832381 7f63bca62700 -1 osd.23 7780 > > lsb_release_parse - failed to call lsb_release binary with error: (12) > > Cannot allocate memory > > > > Eventually the OSD fails. Cluster is in an unhealthy state. > > > > I can set noup, restart the OSDs and get them on the current map, but > > once I put them back into the cluster, they eventually fail. > > What's the full output of "ceph -s"? It sort of sounds like you're > just overloading your spinning OSDs with too many ops. Cache tiering > is often less helpful than people think it is, and in some > circumstances it can actively hurt throughput; you might be running > into that. > Greg, that's an abbreviated re-post of his original message 6 hours earlier titled "blocked ops". And in that he gave more details, as in 12 HDD based OSDs per node in the backing pool. So while you're right about cache-tiers not being an universal cure and the need to configure/size/understand them correctly, I think the elephant in the room here is that 12 OSDs manage to consume 128GB of RAM. And that's just beyond odd. As for Heath, we do indeed need more data as in: a) How busy are your HDD nodes? (atop, iostat). Any particular HDDs/OSDs standing out, as in being slower/busier for a prolonged time? b) No SSD journals for the spinners right? c) The memory exhaustion is purely caused by the OSD processes (atop)? All of them equally or are there particular large ones? Christian -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |