I work for Cisco.
We have a 200TB cluster (108 OSDs on 12 OSD Nodes) and use the cluster for both OpenStack and VMware deployments.
We are using iSCSI now, but it really would be much better if VMware did support RBD natively.
We present a 1-2TB Volume that is shared between 4-8 ESXi hosts.
I have been looking for an optimal solution for a few years now, and I have finally found something that works pretty well:
We are installing FreeNAS on a KVM hypervisor and passing through rbd volumes as disks on a SCSI bus. We are able to add volumes dynamically (no need to reboot FreeNAS to recognize new drives). In FreeNAS, we are passing the disks through directly as iscsi targets, we are not putting the disks into a ZFS volume.
The biggest benefit to this is that VMware really likes the FreeBSD target and all VAAI stuff works reliably. We also get the benefit of the stability of rbd in QEMU client.
My next step is to create a redundant KVM host with a redundant FreeNAS VM and see how iscsi multipath works with the ESXi hosts.
We have tried many different things and have run into all the same issues as others have posted on this list. The general theme seems to be that most (all?) Linux iSCSI Target software and Linux NFS solutions are not very good. The BSD OS's (FreeBSD, Solaris derivatives, etc.) do these things a lot better, but typically lack Ceph support as well as having poor HW compatibility (compared to Linux).
Our goal has always been to replace FC SAN with something comparable in performance, reliability and redundancy.
Again, the best thing in the world would be for ESXi to mount rbd volumes natively using librbd. I'm not sure if VMware is interested in this though.
Jake
On Wednesday, October 5, 2016, Patrick McGarry <pmcgarry@xxxxxxxxxx> wrote:
Hey guys,
Starting to buckle down a bit in looking at how we can better set up
Ceph for VMWare integration, but I need a little info/help from you
folks.
If you currently are using Ceph+VMWare, or are exploring the option,
I'd like some simple info from you:
1) Company
2) Current deployment size
3) Expected deployment growth
4) Integration method (or desired method) ex: iscsi, native, etc
Just casting the net so we know who is interested and might want to
help us shape and/or test things in the future if we can make it
better. Thanks.
--
Best Regards,
Patrick McGarry
Director Ceph Community || Red Hat
http://ceph.com || http://community.redhat.com
@scuttlemonkey || @ceph
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
