On Fri, Jan 2, 2015 at 9:59 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote: > I think it's just for service isolation that people recommend splitting > them. The only technical issue I can think of is that you don't want to put > kernel clients on the same OS as an OSD (due to deadlock scenarios under > memory pressure and writeback). In addition to the true deadlocks that come from kernel clients, there are other kind of "catch 22" resource contention issues that can emerge in userspace. In theory that we shouldn't see deadlock per-se in the userspace processes: they should eventually either achieve progress or get killed, but in a system with timeouts we can get nasty cycles when OSDs don't respond fast enough, therefore OSD map updates occur, therefore mons consume more resources, therefore co-hosted OSDs respond even more slowly, generating more map updates, while some other OSDs come back online, and they replay, which slows down the mons again... The solution to the above problem is to resource-isolate the watchers (mons) from the watched (osds). The simplest way to achieve resource isolation is separate hosts, but of course if you have a trusted way of doing that within a host that's fine too. Things like cgroups are very interesting, or virtualisation/containerisation where you can allocate hard limits on resources to each VM. However, simply putting services in VMs/containers won't necessarily help if the containers/VMs are going to contend for resources the same way that processes would. John _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |