Way back in the Hammerish days we build our first gen of large object store with 60 osds per host. When we tried to deploy with erasure coded pools, it fell on its face under load from all the context switching to handle intra-osd traffic. We were forced to fall back to simple replication to reduce the number of pgs and hence connections in the mix. My understanding is that since then the async messenger should have mitigated a lot of what we saw then. Unfortunately we've also moved to our second gen hardware where we went down to 35 osds per node, so I don't have good data on more recent releases. We are working towards re-purposing some of the first gen hardware and will certainly be attempting EC again on it with Luminous or later. Hopefully in the next month or so we'd have something to report. > On Apr 9, 2018, at 12:06 PM, Sage Weil <sweil@xxxxxxxxxx> wrote: > > Hi, > > I'm wondering if anybody running large clusters has seen pressure or > problems with the large number of TCP streams between OSDs. Is this a > problem? Intuitively it sort of seems like it shouldn't be (10s of OSDs x > ~100 peers -> 1000s of TCP streams on the backend doesn't seem like much, > especially compared to the number of *client* TCP connections). But I > want to check and see what you guys see... > > I'm asking because the idea of multiplexing multiple logical OSDs on > the same host running in a single process across a single stream has come > up as an important change for supporting RDMA for large clusters, where > there is some limitation in the number of streams you can have (I think > because of the reserved memory you need for each stream?). I'm > questioning whether this is an important goal since RDMA seems less > relevant/important and I don't think there is a similar problem on the TCP > side... > > Thanks! > sage > _______________________________________________ > Ceph-large mailing list > Ceph-large@xxxxxxxxxxxxxx > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ceph.com_listinfo.cgi_ceph-2Dlarge-2Dceph.com&d=DwICAg&c=Tpa2GKmmYSmpYS4baANxQwQYqA0vwGXwkJOPBegaiTs&r=5nKer5huNDFQXjYpOR4o_7t5CRI8wb5Vb_v1pBywbYw&m=uuArdOx00PPkI7uCUDV20QPNlUGI3yGTcm2mWG2-GgQ&s=BVV3lOShbcnLwmGFwQEfmPYkSgocgttD-9O-rBt30oc&e= CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you. _______________________________________________ Ceph-large mailing list Ceph-large@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-large-ceph.com
