Re: Ceph native clients

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

On 10/26/24 18:45, Tim Holloway wrote:
On the whole, I prefer to use NFS for my clients to use Ceph
filesystem. It has the advantage that NFS client/mount is practically
guaranteed to be pre-installed on all my client systems.

On the other hand, there are downsides. NFS (Ceph/NFS-Ganesha) has been
known to be cranky on my network when it comes to serving resources on
occasion, and should the NFS host go down, the NFS client cannot
fallback to the alternate NFS host without assistance. Which I should
probably add, since I run keepalive anyway, but that's another matter.

My issue, then is that if I do a native/FUSE mount of Ceph FS on my
desktop it hurts Ceph. The desktop hibernates when not in use and that
causes the network communications between it and the Ceph servers to
break until it wakes up again. In the mean time, I get "laggy OSD"
errors.

In extreme cases, where the desktop doesn't wake up clean and requires
a hardware reset, I think it even ends up with orphan connections,
since at one point I recall having to forcible terminate close to 20
connections.

Ideally, then, since systemd is handling hibernation, there should be a
hook in the Ceph client to ensure it and the Ceph servers are at peace
with each other while it's down. Probably better reporting on the Ceph
admin side as to the hostnames of clients with connection issues, but
that's just icing.


Do not use a network file system with hibernating machines. It will not work reliably.

Both NFS and especially CephFS can delegate responsibility to the clients. In case of NFSv4 these are the delegates, in case of CephFS it are cephfs capabilities. So the filesystem needs to have a back channel to the clients in case capabilities / delegates have to be revoked.

This won't work if the client is not reachable. Access to the files/directory protected by a capability might not be possible depending on the type of capability granted. E.g. in case of a write capability read access might be stalled; in case of a read capability any other client requesting write access will be blocked.

To make things worse, the cephfs capabilities also cover cached files. So even files not actively used by a client might be blocked....

A hook might reduce the impact of the problem, but it won't resolve it. At least the mountpoint itself will be as active capability on the client. This is the cap list of a client without any active file access and after dropping the caches:


# cat caps
total        8193
avail        8192
used        1
reserved    0
min        8192

ino              mds  issued           implemented
--------------------------------------------------
0x1                0  pAsLsXsFs        pAsLsXsFs


Waiters:
--------
tgid         ino                need             want
-----------------------------------------------------

(see /sys/kernel/debug/ceph/<fs id>.<client id>/caps)


TL;DR: do not use cephfs with hibernating clients.


We had the same problem with our desktops a while ago, and decided to switch to a NFS re-export of the CephFS filesystem. This has proven to be much more reliable in case of hibernation. But as you already mentioned NFS also has other problems...


Regards,

Burkhard Linke

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux