I run a converged openstack / ceph cluster with 14 1U nodes. Each has 1 SSD (os / journals), 3 1TB spinners (1 OSD each), 16 HT cores, 10Gb NICs for ceph network, and 72GB of RAM. I configure openstack to leave 3GB of RAM unused on each node for OSD / OS overhead. All the VMs are backed by ceph volumes and things generally work very well. I would prefer a dedicated storage layer simply because it seems more "right", but I can't say that any of the common concerns of using this kind of setup have come up for me. Aside from shaving off that 3GB of RAM, my deployment isn't any more complex than a split stack deployment would be. After running like this for the better part of a year, I would have a hard time honestly making a real business case for the extra hardware a split stack cluster would require.
QH
On Thu, Mar 26, 2015 at 6:57 AM, Mark Nelson <mnelson@xxxxxxxxxx> wrote:
It's kind of a philosophical question. Technically there's nothing that prevents you from putting ceph and the hypervisor on the same boxes. It's a question of whether or not potential cost savings are worth increased risk of failure and contention. You can minimize those things through various means (cgroups, ristricting NUMA nodes, etc). What is more difficult is isolating disk IO contention (say if you want local SSDs for VMs), memory bus and QPI contention, network contention, etc. If the VMs are working really hard you can restrict them to their own socket, and you can even restrict memory usage to the local socket, but what about remote socket network or disk IO? (you will almost certainly want these things on the ceph socket) I wonder as well about increased risk of hardware failure with the increased load, but I don't have any statistics.
I'm guessing if you spent enough time at it you could make it work relatively well, but at least personally I question how beneficial it really is after all of that. If you are going for cost savings, I suspect efficient compute and storage node designs will be nearly as good with much less complexity.
Mark
On 03/26/2015 07:11 AM, Wido den Hollander wrote:
On 26-03-15 12:04, Stefan Priebe - Profihost AG wrote:_______________________________________________
Hi Wido,
Am 26.03.2015 um 11:59 schrieb Wido den Hollander:
On 26-03-15 11:52, Stefan Priebe - Profihost AG wrote:
Hi,
in the past i rwad pretty often that it's not a good idea to run ceph
and qemu / the hypervisors on the same nodes.
But why is this a bad idea? You save space and can better use the
ressources you have in the nodes anyway.
Memory pressure during recovery *might* become a problem. If you make
sure that you don't allocate more then let's say 50% for the guests it
could work.
mhm sure? I've never seen problems like that. Currently i ran each ceph
node with 64GB of memory and each hypervisor node with around 512GB to
1TB RAM while having 48 cores.
Yes, it can happen. You have machines with enough memory, but if you
overprovision the machines it can happen.
Using cgroups you could also prevent that the OSDs eat up all memory or CPU.Never seen an OSD doing so crazy things.
Again, it really depends on the available memory and CPU. If you buy big
machines for this purpose it probably won't be a problem.
Stefan
So technically it could work, but memorey and CPU pressure is something
which might give you problems.
Stefan
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
