Hi,
On 10/26/24 18:45, Tim Holloway wrote:
On the whole, I prefer to use NFS for my clients to use Ceph
filesystem. It has the advantage that NFS client/mount is practically
guaranteed to be pre-installed on all my client systems.
On the other hand, there are downsides. NFS (Ceph/NFS-Ganesha) has been
known to be cranky on my network when it comes to serving resources on
occasion, and should the NFS host go down, the NFS client cannot
fallback to the alternate NFS host without assistance. Which I should
probably add, since I run keepalive anyway, but that's another matter.
My issue, then is that if I do a native/FUSE mount of Ceph FS on my
desktop it hurts Ceph. The desktop hibernates when not in use and that
causes the network communications between it and the Ceph servers to
break until it wakes up again. In the mean time, I get "laggy OSD"
errors.
In extreme cases, where the desktop doesn't wake up clean and requires
a hardware reset, I think it even ends up with orphan connections,
since at one point I recall having to forcible terminate close to 20
connections.
Ideally, then, since systemd is handling hibernation, there should be a
hook in the Ceph client to ensure it and the Ceph servers are at peace
with each other while it's down. Probably better reporting on the Ceph
admin side as to the hostnames of clients with connection issues, but
that's just icing.
Do not use a network file system with hibernating machines. It will not
work reliably.
Both NFS and especially CephFS can delegate responsibility to the
clients. In case of NFSv4 these are the delegates, in case of CephFS it
are cephfs capabilities. So the filesystem needs to have a back channel
to the clients in case capabilities / delegates have to be revoked.
This won't work if the client is not reachable. Access to the
files/directory protected by a capability might not be possible
depending on the type of capability granted. E.g. in case of a write
capability read access might be stalled; in case of a read capability
any other client requesting write access will be blocked.
To make things worse, the cephfs capabilities also cover cached files.
So even files not actively used by a client might be blocked....
A hook might reduce the impact of the problem, but it won't resolve it.
At least the mountpoint itself will be as active capability on the
client. This is the cap list of a client without any active file access
and after dropping the caches:
# cat caps
total 8193
avail 8192
used 1
reserved 0
min 8192
ino mds issued implemented
--------------------------------------------------
0x1 0 pAsLsXsFs pAsLsXsFs
Waiters:
--------
tgid ino need want
-----------------------------------------------------
(see /sys/kernel/debug/ceph/<fs id>.<client id>/caps)
TL;DR: do not use cephfs with hibernating clients.
We had the same problem with our desktops a while ago, and decided to
switch to a NFS re-export of the CephFS filesystem. This has proven to
be much more reliable in case of hibernation. But as you already
mentioned NFS also has other problems...
Regards,
Burkhard Linke
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
