I started down that path and got so deep that I couldn't even find where I went in. I couldn't make heads or tails out of what would or wouldn't work.
We didn't need multiple hosts accessing a single datastore, so on the client side I just have a single VM guest running on each ESXi hosts, with the cephfs filesystem mounted on it(via a 10Gb connection to the ceph environment), and then exported via NFS on a host-only network, and mounted on the host.
Not quite as redundant as it could be, but good enough for our usage. I'm seeing ~500MB/s speeds going to a 4-node cluster with 5x1TB 7200rpm drives. I tried it first, in a similar config, except using LIO to export an RBD device via iSCSI, still on the local host network. Write performance was good, but read performance was only around 120MB/s. I didn't do much troubleshooting, just tried NFS after that and was happy with it.
On Tue, Jun 6, 2017 at 2:33 AM, Adrian Saul <Adrian.Saul@xxxxxxxxxxxxxxxxx> wrote:
> > Early usage will be CephFS, exported via NFS and mounted on ESXi 5.5
> > and
> > 6.0 hosts(migrating from a VMWare environment), later to transition to
> > qemu/kvm/libvirt using native RBD mapping. I tested iscsi using lio
> > and saw much worse performance with the first cluster, so it seems
> > this may be the better way, but I'm open to other suggestions.
> >
> I've never seen any ultimate solution to providing HA iSCSI on top of Ceph,
> though other people here have made significant efforts.
In our tests our best results were with SCST - also because it provided proper ALUA support at the time. I ended up developing my own pacemaker cluster resources to manage the SCST orchestration and ALUA failover. In our model we have a pacemaker cluster in front being an RBD client presenting LUNs/NFS out to VMware (NFS), Solaris and Hyper-V (iSCSI). We are using CephFS over NFS but performance has been poor, even using it just for VMware templates. We are on an earlier version of Jewel so its possibly some later versions may improve CephFS for that but I have not had time to test it.
We have been running a small production/POC for over 18 months on that setup, and gone live into a much larger setup in the last 6 months based on that model. It's not without its issues, but most of that is a lack of test resources to be able to shake out some of the client compatibility and failover shortfalls we have.
Confidentiality: This email and any attachments are confidential and may be subject to copyright, legal or some other professional privilege. They are intended solely for the attention and use of the named addressee(s). They may only be copied, distributed or disclosed with the consent of the copyright owner. If you have received this email by mistake or by breach of the confidentiality clause, please notify the sender immediately by return email and delete or destroy all copies of the email. Any confidentiality, privilege or copyright is not waived or lost because this email has been sent to you by mistake.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
