Re: Ceph and Compute on same hardware?

[gaoxingxing <itxx00@xxxxxxxxx>]

I think you may also consider risk like kernel crashes etc,since storage and compute node are sharing the same box.

---

Date: Wed, 12 Nov 2014 14:51:47 +0000
From: pieter.koorts@xxxxxx
To: ceph-users@xxxxxxxxxxxxxx
Subject: Re: [ceph-users] Ceph and Compute on same hardware?

Hi,

Thanks for the replies. Likely will not choose this method but wanted to make sure that it was a good technical reason rather than just a "best practice". I did not quite think of "conntracker" at the time so this is a good one to consider.

Thanks

Pieter

On 12 November 2014 14:30, Haomai Wang <haomaiwang@xxxxxxxxx> wrote:

Actually, our production cluster(up to ten) all are that ceph-osd ran
on compute-node(KVM).

The primary action is that you need to constrain the cpu and memory.
For example, you can alloc a ceph cpu-set and memory group, let
ceph-osd run with it within limited cores and memory.

The another risk is the network. Because compute-node and ceph-osd
shared the same kernel network stack, it exists some risks that VM may
ran out of network resources such as conntracker in netfilter
framework.

On Wed, Nov 12, 2014 at 10:23 PM, Mark Nelson <mark.nelson@xxxxxxxxxxx> wrote:
> Technically there's no reason it shouldn't work, but it does complicate
> things.  Probably the biggest worry would be that if something bad happens
> on the compute side (say it goes nuts with network or memory transfers) it
> could slow things down enough that OSDs start failing heartbeat checks
> causing ceph to go into recovery and maybe cause a vicious cycle of
> nastiness.
>
> You can mitigate some of this with cgroups and try to dedicate specific
> sockets and memory banks to Ceph/Compute, but we haven't done a lot of
> testing yet afaik.
>
> Mark
>
>
> On 11/12/2014 07:45 AM, Pieter Koorts wrote:
>>
>> Hi,
>>
>> A while back on a blog I saw mentioned that Ceph should not be run on
>> compute nodes and in the general sense should be on dedicated hardware.
>> Does this really still apply?
>>
>> An example, if you have nodes comprised of
>>
>> 16+ cores
>> 256GB+ RAM
>> Dual 10GBE Network
>> 2+8 OSD (SSD log + HDD store)
>>
>> I understand that Ceph can use a lot of IO and CPU in some cases but if
>> the nodes are powerful enough does it not make it an option to run
>> compute and storage on the same hardware to either increase density of
>> compute or save money on additional hardware?
>>
>> What are the reasons for not running Ceph on the Compute nodes.
>>
>> Thanks
>>
>> Pieter
>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

--
Best Regards,

Wheat

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com