On Thu, 17 Jul 2014, Quenten Grasso wrote: > Hi Sage & List > > I understand this is probably a hard question to answer. > > I mentioned previously our cluster is co-located MON?s on OSD servers, which > are R515?s w/ 1 x AMD 6 Core processor & 11 3TB OSD?s w/ dual 10GBE. > > When our cluster is doing these busy operations and IO has stopped as in my > case, I mentioned earlier running/setting tuneable to optimal or heavy > recovery > > operations is there a way to ensure our IO doesn?t get completely > blocked/stopped/frozen in our vms? > > Could it be as simple as putting all 3 of our mon servers on baremetal > ?w/ssd?s? (I recall reading somewhere that a mon disk was doing several > thousand IOPS during a recovery operation) > > I assume putting just one on baremetal won?t help because our mon?s will only > ever be as fast as our slowest mon server? I don't think this is related to where the mons are (most likely). The big question for me is whether IO is getting completely blocked, or just slowed enough that the VMs are all timing out. What slow request messages did you see during the rebalance? What were the op latencies? It's possible there is a bug here, but it's also possible the cluster is just operating close enough to capacity that the additional rebalancing work pushes it into a place where it can't keep up and the IO latencies are too high. Or that we just have more work to do prioritizing requests.. but it's hard to say without more info. sage
 |