On 3/11/20 11:16 PM, Seth Galitzer wrote:
I have a hybrid environment and need to share with both Linux and
Windows clients. For my previous iterations of file storage, I
exported nfs and samba shares directly from my monolithic file server.
All Linux clients used nfs and all Windows clients used samba. Now
that I've switched to ceph, things are a bit more complicated. I built
a gateway to export nfs and samba as needed, and connect that as a
client to my ceph cluster.
After having file locking problems with kernel nfs, I made the switch
to nfs-ganesha, which has helped immensely. For Linux clients that
have high I/O needs, like desktops and some web servers, I connect to
ceph directly for those shares. For all other Linux needs, I use nfs
from the gateway. For all Windows clients (desktops and a small number
of servers), I use samba exported from the gateway.
Since my ceph cluster went live in August, I have had some kind of
strange (to me) error at least once a week, almost always related to
the gateway client. Last night, it was MDS_CLIENT_OLDEST_TID. Since
we're on Spring Break at my university and not very busy, I decided to
unmount/remount the ceph share, requiring stopping nfs and samba
services. Stopping nfs-ganesha took a while, but it finally completed
with no complaints from the ceph cluster. Stopping samba took longer
and gave me MDS_SLOW_REQUEST and MDS_CLIENT_LATE_RELEASE on the mds.
It finally finished, and I was able to unmount/remount the ceph share
and that finally cleared all the errors.
This is leading me to believe that samba on the gateway and all the
clients attaching to that is putting a strain on the connection back
to ceph. Which finally brings me to my question: is there a better way
to export samba to my clients using the ceph back end? Or is this as
good as it gets and I just have to put up with the seemingly frequent
errors? I can live with the errors and have been able to handle them
so far, but I know people who have much bigger clusters and many more
clients than me (by an order of magnitude) and don't see nearly as
many errors as I do. Which is why I'm trying to figure out what is
special about my setup.
All my ceph nodes are running latest nautilus on Centos 7 (I just
updated last week to 14.2.8), as is the gateway host. I'm mounting
ceph directly on the gateway (by way of the kernel using cephfs, not
rados/rbd) to a single mount point and exporting from there.
My searches so far have not turned up anything extraordinarily useful,
so I'm asking for some guidance here. Any advice is welcome.
You can connect to your cluster directly from userland, without kernel.
Use Samba vfs_ceph for this.
k
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
