|
Wow I feel like we got off easy. We're on our second generation of hardware running Jewel:
32 Nodes
35 8TB spinners w/collocated journals
128GB Mem
Dual E5-2620
Dual 10G nics w/jumbo frames
14.04 on 4.2 kernel
5 RGWs - these are the only clients
This was designed as "warm" object storage with performance secondary to price, hence the lack of flash for journals. Still we've gotten perfectly adequate performance and had nearly 0 problems with it. Jumbo frames, turning off swap and upping
some ulimits are pretty much the only tuning we've had to do. We've gone through recovery from removing 2 nodes with no problems.
Aaron
Hi,
when you say "XFS crashes" is it actual crashes, or is it "soft" allocation failures ?
We were "experimenting" with dense nodes, as in 31 x 8TB spinning rust, NVMe journal, 256GB RAM, 40 GigE, and ended up on the wrong end of how linux-xfs reclaims memory.
Our symptoms were packet drops (I forget which direction) and latencies jumping up into the second range when packets were't dropped, which caused osd's to be marked as non-responsive by the cluster, and sometimes this spiraled out of hand. Essentially,
the XFS reclaim were something along the lines of, for each xfs filesystem, try to reclaim without "big fat lock", if that fails, grab "big fat lock" and reclaim. The drives were hideously slow, sadness ensued. In the end, we ended up systemtap'ing the xfs
driver into not grabbing the big fat lock, hoping that it could do reclaim on one of the other XFS filesystems without "big fat lock". It's so far worked really well for us, and our current plan is to leave the ugly hack in place until we feel bluestore is
the way to go.
-KJ
_______________________________________________
Ceph-large mailing list
Ceph-large@xxxxxxxxxxxxxx
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ceph.com_listinfo.cgi_ceph-2Dlarge-2Dceph.com&d=DwICAg&c=Tpa2GKmmYSmpYS4baANxQwQYqA0vwGXwkJOPBegaiTs&r=5nKer5huNDFQXjYpOR4o_7t5CRI8wb5Vb_v1pBywbYw&m=YdsjMWuTddcgt4W1SKq6Lan0v6fSzsVTsGq8p2Env2A&s=tyT4PzkLTGGR5Z1rp-tQSBMSmGs5MnElR7AaBo8-6C4&e=
CONFIDENTIALITY NOTICE
This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution
or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you.
|
