On Wed, Aug 27, 2014 at 4:15 PM, Sage Weil <sweil at redhat.com> wrote: > On Wed, 27 Aug 2014, Robert LeBlanc wrote: > > I'm looking for a way to prioritize the heartbeat traffic higher than the > > storage and replication traffic. I would like to keep the ceph.conf as > > simple as possible by not adding the individual osd IP addresses and > ports, > > but it looks like the listening ports are pretty random. I'd like to use > > iptables to mark the packet with DSCP, then our network team can mark it > > with COS. I'm having a hard time figuring out what in the payload I can > > match to say that it is a heartbeat packet. > > What would be best way for us to mark which sockets are heartbeat related? > Is there some setsockopt() type call we should be using, or should we > perhaps use a different port range for heartbeat traffic? > I'm not really sure setting up a separate port range is the correct answer, but it could be a very simple implementation, something like ports 6900+ and would be very simple itpables rules. Based on my packet capture, it seems that heartbeat traffic is less than 150 bytes and the payload always starts with 0x08, I'm just not sure how true that is. I could build an iptables rule off of that. > We have some concerns that under high network congestion scenarios, > > heartbeats will be lost, marking OSDs down, triggering replication, > causing > > more traffic causing a compounding to the congestion, marking more OSDs > down > > until the entire cluster falls apart. Our intention is that we can try to > > deliver heartbeat packets first with the hope that the system will know > that > > OSDs are up, just busy. Is our reasoning sound in this regard? > > It is reasonable. We do need to be careful, however. The heartbeats are > attempting to measure whether the OSDs are able to effectively > communicate. Every time we try to give heartbeats a different level of > service, we are diminishing their ability to measure that reality. For > example, if your QoS starved or nearly starved non-heartbeat traffic the > OSDs wouldn't be able to tell. > >From what I understand the heartbeat traffic is passed over both the public and cluster networks now, so congestion on one network would not necessarily cause an OSD to be marked down. Giving heartbeats higher priority will actually confirm that OSDs can talk (albeit a bit slower) instead of incorrectly assuming that they are dead and need to be taken over. sage > Thanks, Robert LeBlanc -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140827/c66e0a8f/attachment.htm>
 |