Hi,
Thanks for the replies. Likely will not choose this method but wanted to make sure that it was a good technical reason rather than just a "best practice". I did not quite think of "conntracker" at the time so this is a good one to consider.
Thanks
Pieter
On 12 November 2014 14:30, Haomai Wang <haomaiwang@xxxxxxxxx> wrote:
Actually, our production cluster(up to ten) all are that ceph-osd ran
on compute-node(KVM).
The primary action is that you need to constrain the cpu and memory.
For example, you can alloc a ceph cpu-set and memory group, let
ceph-osd run with it within limited cores and memory.
The another risk is the network. Because compute-node and ceph-osd
shared the same kernel network stack, it exists some risks that VM may
ran out of network resources such as conntracker in netfilter
framework.
--
On Wed, Nov 12, 2014 at 10:23 PM, Mark Nelson <mark.nelson@xxxxxxxxxxx> wrote:
> Technically there's no reason it shouldn't work, but it does complicate
> things. Probably the biggest worry would be that if something bad happens
> on the compute side (say it goes nuts with network or memory transfers) it
> could slow things down enough that OSDs start failing heartbeat checks
> causing ceph to go into recovery and maybe cause a vicious cycle of
> nastiness.
>
> You can mitigate some of this with cgroups and try to dedicate specific
> sockets and memory banks to Ceph/Compute, but we haven't done a lot of
> testing yet afaik.
>
> Mark
>
>
> On 11/12/2014 07:45 AM, Pieter Koorts wrote:
>>
>> Hi,
>>
>> A while back on a blog I saw mentioned that Ceph should not be run on
>> compute nodes and in the general sense should be on dedicated hardware.
>> Does this really still apply?
>>
>> An example, if you have nodes comprised of
>>
>> 16+ cores
>> 256GB+ RAM
>> Dual 10GBE Network
>> 2+8 OSD (SSD log + HDD store)
>>
>> I understand that Ceph can use a lot of IO and CPU in some cases but if
>> the nodes are powerful enough does it not make it an option to run
>> compute and storage on the same hardware to either increase density of
>> compute or save money on additional hardware?
>>
>> What are the reasons for not running Ceph on the Compute nodes.
>>
>> Thanks
>>
>> Pieter
>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Best Regards,
Wheat
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
