On Wed, Nov 12, 2014 at 5:30 PM, Haomai Wang <haomaiwang@xxxxxxxxx> wrote: > Actually, our production cluster(up to ten) all are that ceph-osd ran > on compute-node(KVM). > > The primary action is that you need to constrain the cpu and memory. > For example, you can alloc a ceph cpu-set and memory group, let > ceph-osd run with it within limited cores and memory. > > The another risk is the network. Because compute-node and ceph-osd > shared the same kernel network stack, it exists some risks that VM may > ran out of network resources such as conntracker in netfilter > framework. > > On Wed, Nov 12, 2014 at 10:23 PM, Mark Nelson <mark.nelson@xxxxxxxxxxx> wrote: >> Technically there's no reason it shouldn't work, but it does complicate >> things. Probably the biggest worry would be that if something bad happens >> on the compute side (say it goes nuts with network or memory transfers) it >> could slow things down enough that OSDs start failing heartbeat checks >> causing ceph to go into recovery and maybe cause a vicious cycle of >> nastiness. >> >> You can mitigate some of this with cgroups and try to dedicate specific >> sockets and memory banks to Ceph/Compute, but we haven't done a lot of >> testing yet afaik. >> >> Mark >> >> >> On 11/12/2014 07:45 AM, Pieter Koorts wrote: >>> >>> Hi, >>> >>> A while back on a blog I saw mentioned that Ceph should not be run on >>> compute nodes and in the general sense should be on dedicated hardware. >>> Does this really still apply? >>> >>> An example, if you have nodes comprised of >>> >>> 16+ cores >>> 256GB+ RAM >>> Dual 10GBE Network >>> 2+8 OSD (SSD log + HDD store) >>> >>> I understand that Ceph can use a lot of IO and CPU in some cases but if >>> the nodes are powerful enough does it not make it an option to run >>> compute and storage on the same hardware to either increase density of >>> compute or save money on additional hardware? >>> >>> What are the reasons for not running Ceph on the Compute nodes. >>> >>> Thanks >>> >>> Pieter >>> >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > Best Regards, > > Wheat > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com Yes, the essential part is a resource management, which can neither be dynamic or static. In Flops we had implemented dynamic resource control which allows to pack VMs and OSDs more densely than static cg-based jails can allow (and it requires deep orchestration modifications for every open source cloud orchestrator, unfortunately). As long as you are able to manage strong traffic isolation for storage and vm segment, there are absolutely no problem (it can be static limits from linux-qos or tricky flow management for OpenFlow, depends on what your orchestration allows). The possibility of putting together compute and storage roles without significant impact to performance characteristics was one of key features which led our selection to Ceph as a storage backend three years ago. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |