On Fri, Jun 29, 2018 at 8:08 PM Nick Fisk <nick@xxxxxxxxxx> wrote:
>
> This is for us peeps using Ceph with VMWare.
>
>
>
> My current favoured solution for consuming Ceph in VMWare is via RBD’s formatted with XFS and exported via NFS to ESXi. This seems to perform better than iSCSI+VMFS which seems to not play nicely with Ceph’s PG contention issues particularly if working with thin provisioned VMDK’s.
>
>
>
> I’ve still been noticing some performance issues however, mainly noticeable when doing any form of storage migrations. This is largely due to the way vSphere transfers VM’s in 64KB IO’s at a QD of 32. vSphere does this so Arrays with QOS can balance the IO easier than if larger IO’s were submitted. However Ceph’s PG locking means that only one or two of these IO’s can happen at a time, seriously lowering throughput. Typically you won’t be able to push more than 20-25MB/s during a storage migration
>
>
>
> There is also another issue in that the IO needed for the XFS journal on the RBD, can cause contention and effectively also means every NFS write IO sends 2 down to Ceph. This can have an impact on latency as well. Due to possible PG contention caused by the XFS journal updates when multiple IO’s are in flight, you normally end up making more and more RBD’s to try and spread the load. This normally means you end up having to do storage migrations…..you can see where I’m getting at here.
>
>
>
> I’ve been thinking for a while that CephFS works around a lot of these limitations.
>
>
>
> 1. It supports fancy striping, so should mean there is less per object contention
Hi Nick,
Fancy striping is supported since 4.17. I think its primary use case
is small sequential I/Os, so not sure if it is going to help much, but
it might be worth doing some benchmarking.
Thanks,
Ilya
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
