For NFS (e.g., as implemented by NFS-ganesha), the situation is also
quite stupid.
Without high availability (HA), it works (that is, until you update
NFS-Ganesha version), but corporate architects won't let you deploy
any system without HA, because, in their view, non-HA systems are not
production-ready by definition. (And BTW, the current NVMe-oF gateway
also has no multipath and thus no viable HA)
With an attempt to set up HA for NFS, you'll get at least the
following showstoppers:
For NFS v4.1:
* VMware refuses to work until the manual admin intervention if it
sees any change in the "owner" and "scope" fields of the EXCHANGE_ID
message between the previous and the current NFS connection.
* NFS-Ganesha sets both fields from the hostname by default, and the
patch that makes these fields configurable is "quite recent" (in
version 4.3). This is important, as otherwise, every NFS server
fail-over would trip off VMware, thus defeating the point of a
high-availability setup.
* There is a regression in NFS-Ganesha that manifests as a deadlock
(easily triggerable even without Ceph by running xfstests), which is
critical, because systemd cannot restart deadlocked services.
Unfortunately, the last NFS-Ganesha version before the regression
(4.0.8) does not contain the patch that allows manipulating the
"owner" and "scope" fields.
* Cephadm-based deployments do not set these configuration options anyway.
* If you would like to use the "rados_cluster" NFSv4 recovery backend
(used for grace periods), you need to be extra careful with various
"server names" also because they are used to decide whether to end the
grace period. If the recovery backend has seen two server names
(corresponding to two NFS-Ganesha instances, for scale-out), then both
must be up for the grace period to end. If there is only one server
name, you are allowed to run only one instance. If you want high
availability together with scale-out, you need to be able to schedule
two NFS-Ganesha instances (with names like a and b, not corresponding
to the names of hosts where they run) on two out of three available
servers. Orchestrators do not do this, you need to implement this on
your own.
For NFS v3:
* NFS-Ganesha opens files and acquires MDS locks just in case, to make
sure that another client cannot modify them while the original client
might have cached something.
* If NFS-Ganesha crashes or a server reboots, then the other
NFS-Ganesha, brought up to replace the original one, will also stumble
upon these locks, as the MDS recognizes it as a different client.
Result: it waits until the locks time out, which is too long
(minutes!), as the guest OS in VMware would then time out its storage.
* To avoid the problem mentioned above and to get seamless fail-over,
the replacement instance of NFS-Ganesha must present itself as the
same client (i.e., as the same fake hostname) to the MDS, but no known
orchestrators facilitate this.
Conclusion: please use iSCSI or sacrifice HA, as there are no working
alternatives yet.
On Fri, Jun 28, 2024 at 1:31 AM Anthony D'Atri <anthony.datri@xxxxxxxxx> wrote:
There are folks actively working on this gateway and there's a Slack channel. I
haven't used it myself yet.
My understanding is that ESXi supports NFS. Some people have had good success
mounting KRBD volumes on a gateway system or VM and re-exporting via NFS.
On Jun 27, 2024, at 09:01, Drew Weaver <drew.weaver@xxxxxxxxxx> wrote:
Howdy,
I recently saw that Ceph has a gateway which allows VMWare ESXi to connect to
RBD.
We had another gateway like this awhile back the ISCSI gateway.
The ISCSI gateway ended up being... let's say problematic.
Is there any reason to believe that NVMeOF will also end up on the floor and has
anyone that uses VMWare extensively evaluated it's viability?
Just curious!
Thanks,
-Drew
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
--
Alexander Patrakov
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
