That's unfortunate. For one, it says "some filesystems are more equal
than others".
Back in ancient days, when dinosaurs used time-sharing, you could mount
a remote fileystem at login and logout, whether explicit or timeout
would unmount it. But virtually nobody time-shares anymore, but lots of
people hibernate, either because they want to be more energy efficient
or to prolong the life of a battery.
And unlike a login, awaking from hibernation doesn't run a login script.
Offhand, I'm not entirely sure if you can easily add any sort of wakeup
script at the user level.
My system turns off the office fan when it goes to sleep. The timeout
interval is long enough that I'm probably no longer in the room at that
point. In theory I could handle unmount/mount there, but I'm uncertain
that I have the full range of system services at that level, and, as you
pointed out, there might be open files that would expect to remain open
when the machine came back up.
Hence my wish that the ceph client and servers - especially linked OSD's
- could "make their peace". which is to say quiesce their I/O,
disconnect from the servers and possibly have Ceph mark the session as
hibernating in order to come back up faster/more intelligently.
Because it's a bloody royal PITA to have to manually mount Ceph every
time I need to edit or run provisioning or use any of the other myriad
systems that I keep in Ceph.
Tim
On 10/28/24 04:54, Burkhard Linke wrote:
Hi,
On 10/26/24 18:45, Tim Holloway wrote:
On the whole, I prefer to use NFS for my clients to use Ceph
filesystem. It has the advantage that NFS client/mount is practically
guaranteed to be pre-installed on all my client systems.
On the other hand, there are downsides. NFS (Ceph/NFS-Ganesha) has been
known to be cranky on my network when it comes to serving resources on
occasion, and should the NFS host go down, the NFS client cannot
fallback to the alternate NFS host without assistance. Which I should
probably add, since I run keepalive anyway, but that's another matter.
My issue, then is that if I do a native/FUSE mount of Ceph FS on my
desktop it hurts Ceph. The desktop hibernates when not in use and that
causes the network communications between it and the Ceph servers to
break until it wakes up again. In the mean time, I get "laggy OSD"
errors.
In extreme cases, where the desktop doesn't wake up clean and requires
a hardware reset, I think it even ends up with orphan connections,
since at one point I recall having to forcible terminate close to 20
connections.
Ideally, then, since systemd is handling hibernation, there should be a
hook in the Ceph client to ensure it and the Ceph servers are at peace
with each other while it's down. Probably better reporting on the Ceph
admin side as to the hostnames of clients with connection issues, but
that's just icing.
Do not use a network file system with hibernating machines. It will
not work reliably.
Both NFS and especially CephFS can delegate responsibility to the
clients. In case of NFSv4 these are the delegates, in case of CephFS
it are cephfs capabilities. So the filesystem needs to have a back
channel to the clients in case capabilities / delegates have to be
revoked.
This won't work if the client is not reachable. Access to the
files/directory protected by a capability might not be possible
depending on the type of capability granted. E.g. in case of a write
capability read access might be stalled; in case of a read capability
any other client requesting write access will be blocked.
To make things worse, the cephfs capabilities also cover cached files.
So even files not actively used by a client might be blocked....
A hook might reduce the impact of the problem, but it won't resolve
it. At least the mountpoint itself will be as active capability on the
client. This is the cap list of a client without any active file
access and after dropping the caches:
# cat caps
total 8193
avail 8192
used 1
reserved 0
min 8192
ino mds issued implemented
--------------------------------------------------
0x1 0 pAsLsXsFs pAsLsXsFs
Waiters:
--------
tgid ino need want
-----------------------------------------------------
(see /sys/kernel/debug/ceph/<fs id>.<client id>/caps)
TL;DR: do not use cephfs with hibernating clients.
We had the same problem with our desktops a while ago, and decided to
switch to a NFS re-export of the CephFS filesystem. This has proven to
be much more reliable in case of hibernation. But as you already
mentioned NFS also has other problems...
Regards,
Burkhard Linke
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
