On Thu, 14 Aug 2014, Andrei Mikhailovsky wrote: > Hi guys, > > Could someone from the ceph team please comment on running osd cache pool on > the hypervisors? Is this a good idea, or will it create a lot of performance > issues? It doesn't sound like an especially good idea. In general you want the cache pool to be significantly faster than the base pool (think PCI attached flash). And there won't be any particular affinity to the host where the VM consuming the sotrage happens to be, so I don't think there is a reason to put the flash in the hypervisor nodes unless there simply isn't anywhere else to put them. Probably what you're after is a client-side write-thru cache? There is some ongoing work to build this into qemu and possibly librbd, but nothing is ready yet that I know of. sage > > Anyone in the ceph community that has done this? Any results to share? > > Many thanks > > Andrei > > ____________________________________________________________________________ > From: "Robert van Leeuwen" <Robert.vanLeeuwen at spilgames.com> > To: "Andrei Mikhailovsky" <andrei at arhont.com> > Cc: ceph-users at lists.ceph.com > Sent: Thursday, 14 August, 2014 9:31:24 AM > Subject: RE: cache pools on hypervisor servers > > > Personally I am not worried too much about the hypervisor - > hypervisor traffic as I am using a dedicated infiniband network for > storage. > > It is not used for the guest to guest or the internet traffic or > anything else. I would like to decrease or at least smooth out the > traffic peaks between the hypervisors and the SAS/SATA osd storage > servers. > > I guess the ssd cache pool would enable me to do that as the > eviction rate should be more structured compared to the random io > writes that guest vms generate. Sounds reasonable > > >>I'm very interested in the effect of caching pools in combination > with running VMs on them so I'd be happy to hear what you find ;) > > I will give it a try and share back the results when we get the ssd > kit. > Excellent, looking forward to it. > > > >> As a side note: Running OSDs on hypervisors would not be my > preferred choice since hypervisor load might impact Ceph performance. > > Do you think it is not a good idea even if you have a lot of cores > on the hypervisors? > > Like 24 or 32 per host server? > > According to my monitoring, our osd servers are not that stressed > and generally have over 50% of free cpu power. > > The number of cores do not really matter if they are all busy ;) > I honestly do not know how Ceph behaves when it is CPU starved but I > guess it might not be pretty. > Since your whole environment will be crumbling down if your storage > becomes unavailable it is not a risk I would take lightly. > > Cheers, > Robert van Leeuwen > > > > > > >
 |