Re: Multi-Tenancy: Network Isolation

[Deepak Naidu <dnaidu@xxxxxxxxxx>]

Thanks much Vlad and Dave for suggestions appreciate it !

--

Deepak

On May 29, 2017, at 1:04 AM, Дробышевский, Владимир <vlad@xxxxxxxxxx> wrote:

Hi, Deepak!

  The easiest way I can imagine is to use multiple VLANs, put all ceph hosts ports into every VLAN and use a wider subnet. For example, you can set 192.168.0.0/16 for the public ceph network, use 192.168.0.1-254 IPs for ceph hosts, 192.168.1.1-254/16 IPs for the first tenant, 192.168.2.1-254/16 for the second and so on. You'll have to be sure that no ceph hosts have any routing facilities running and then get a number of isolated L2 networks with the common part. Actually it's not a good way and lead to many errors (your tenants must carefully use provided IPs and do not cross with other IPs spaces despite of the /16 bitmask).

  An another option is - like David said - L3 routed network. In this case you will probably face with network bandwidth problems: all your traffic will go through one interface. But if your switches have L3 functionality you can route packets there. And again, the problem would be in bandwidth: usually switches doesn't have a lot of power and routed bandwidth leaves a lot to desire.

  And the craziest one :-). It just a theory, never tried this in production and even in a lab.

  As with previous options you go with multiple per-tenant VLANs and ceph hosts ports in all of these VLANs. 

  You need to choose a different network for public interfaces, for ex., 10.0.0.0/24. Then set loopback interface on each ceph host and attach a single unique IP to it, like 10.0.0.1/32, 10.0.0.2/32 and so on. Enable IP forwarding and start RIP routing daemon on each ceph host. Setup and configure ceph, use attached IP as MON IP.

  Create ceph VLAN with all ceph hosts and set a common network IP subnet (for ex, 172.16.0.0/24), attach IP from this network to every ceph host. Check that you can reach any of the public (loopback) IPs from any ceph host.

  Now create multiple per-tenant VLANs and put ceph hosts ports into every one. Set isolated subnets for your tenant's networks, for example, 192.168.0.0/23, use 192.168.0.x IPs as the additional addresses for the ceph hosts, 192.168.1.x as tenant network. Start RIP routing daemon on every tenant host. Check that you can reach every ceph public IPs (10.0.0.x/32).

  I would also configure RIP daemon to advertise only 10.0.0.x/32 network on each ceph host and set RIP daemon on passive mode on client hosts. It's better to configure firewall on ceph hosts as well to prevent extra-subnets communications.

  In theory it should work but can't say much on how stable would it be.

Best regards,

Vladimir

2017-05-26 20:36 GMT+05:00 Deepak Naidu <dnaidu@xxxxxxxxxx>:

Hi Vlad,

Thanks for chiming in.

>>It's not clear what you want to achieve from the ceph point of view?
Multiple tenancy. We will have multiple tenants from different isolated subnet/network accessing single ceph cluster which can support multiple tenants. The only problem I see with ceph in a physical env setup is I cannot isolate public networks , example mon,mds for multiple subnet/network/tenants.

>>For example, for the network isolation you can use managed switches, set different VLANs and put ceph hosts to the every VLAN.
Yes we have managed switches with VLAN. And if I add for example 2x public interferences on Net1(subnet 192.168.1.0/24) and Net2(subnet 192.168.2.0/24) how does the ceph.conf look like. How does my mon and MDS server config look like, that's the challenge/question.

>>But it's a shoot in the dark as I don't know what exactly you need. For example, what services (block storage, object storage, API etc) you want to offer to your tenants and so on

CephFS and Object. I am familiar on how to get the ceph storage part "tenant friendly", it's just the network part I need to isolate.

--
Deepak

> On May 26, 2017, at 12:03 AM, Дробышевский, Владимир <vlad@xxxxxxxxxx> wrote:
>
>   It's not clear what you want to achieve from the ceph point of view? For example, for the network isolation you can use managed switches, set different VLANs and put ceph hosts to the every VLAN. But it's a shoot in the dark as I don't know what exactly you need. For example, what services (block storage, object storage, API etc) you want to offer to your tenants and so on
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

--

С уважением,
Дробышевский Владимир
Компания "АйТи Город"
+7 343 2222192

ИТ-консалтинг
Поставка проектов "под ключ"
Аутсорсинг ИТ-услуг
Аутсорсинг ИТ-инфраструктуры

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com