Dear list members,, after upgrading to reef (18.2.0) I spent some time with CephFS, NFS & HA(Ingress). I can confirm that Ingress (count either 1 or 2) works well IF only ONE backend server is configured. But this is, of course, no HA. ;-) Two or more backend servers won't work because there isn't ANY "*check*" directive used in HAproxy backend config. This means any client will stay/stick on the last NFS backend server used. IMHO in haproxy.cfg this should look more like the following: [..] backend backend mode tcp option tcp-check tcp-check connect port 2049 balance source hash-type consistent server nfs.nfstest-st.0 10.100.1.111:2049 check server nfs.nfstest-st.1 10.100.1.112:2049 check BTW: There are plenty of configuration directives to get a proper working NFS backend. Please refer to: https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#5.2-check https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#stick-table https://infohub.delltechnologies.com/l/ecs-with-haproxy-load-balancer-2/nfs-configuration-definitions/ Did (NFS) HA ever work in previous versions? Unfortunately, I had no reason to test this in advance. Currently, only a manual installation of HAproxy based ingress controller(s) should help. And, can we hope for a correction in future versions? Thanks, Christoph Am Mo., 4. Sept. 2023 um 07:43 Uhr schrieb Thorne Lawler <thorne@xxxxxxxxxxx >: > One more question, for John or anyone: > > It looks like Ganesha NFS wants to use dbus to talk to other services > (although I'm not sure how well this works across containers). > > I just realised that my build (on AlmaLinux 8 'minimal') did not include > dbus, and Ceph has not installed it. > > I have manually installed the default 'dbus' package and its deps, but > this doesn't seem to have had any effect. > > * Does Ganesha use dbus to talk to other services, even in a cephadm > containerised build? > * Is there a non-destructive way to make Ganesha retry gsh_dbus_pkginit ? > * Is the default dbus package sufficient, or does Ceph require > specific dbus plugins? > > Thank you. > > On 4/09/2023 9:55 am, Thorne Lawler wrote: > > John, > > > > Thanks for getting back to me. I am indeed using cephadm, and I will > > dig up those configurations. > > > > Even if Ceph Quincy is current completely incapable of configuring its > > own HA failover, I would really like to know what the /imagined/ > > process would be for detecting a node failure and failing over. > > > > Can you elaborate about those changes that need to happen, or point me > > to the forums (ideally the posts_ where this work is broken down in > > more detail? > > > > It's too late for me to change this; I am 100% committed to using Ceph > > for HA NFS, no matter what that involves. > > > > Thanks. > > > > On 31/08/2023 11:18 pm, John Mulligan wrote: > >> On Wednesday, August 30, 2023 8:38:21 PM EDT Thorne Lawler wrote: > >>> If there isn't any documentation for this yet, can anyone tell me: > >>> > >>> * How do I inspect/change my NFS/haproxy/keepalived configuration? > >>> * What is it supposed to look like? Does someone have a working > >>> example? > >> The configuration for haproxy, keepalive, and ganesha are generated. > >> I'm assuming you are using cephadm orchestration. If you want to see > >> what it > >> generated configs contain look in /var/lib/ceph/<FSID>/<svc-name>/ > >> under those dirs may be additional subdirectories like etc/ or > >> config/ (it > >> varies from service to service) > >> > >> It's not simple to customize those files directly. It's not > >> impossible but it's > >> probably not worth it (IMO). > >> > >> Also, speaking only for my personal opinion, the current NFS > >> situation is one > >> of OK scale-out but is HA in name only. Fail over is not mature. To > >> make it > >> so, changes need to happen throughout the stack including in > >> nfs-ganesha. I > >> know there are some conversations happening around this topic but I > >> don't know > >> the best place to get involved upstream. > >> > >> I know this probably isn't very satisfactory, but I hope the information > >> helps. > >> > >> _______________________________________________ > >> ceph-users mailing list --ceph-users@xxxxxxx > >> To unsubscribe send an email toceph-users-leave@xxxxxxx > -- > > Regards, > > Thorne Lawler - Senior System Administrator > *DDNS* | ABN 76 088 607 265 > First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 > P +61 499 449 170 > > _DDNS > > /_*Please note:* The information contained in this email message and any > attached files may be confidential information, and may also be the > subject of legal professional privilege. _If you are not the intended > recipient any use, disclosure or copying of this email is unauthorised. > _If you received this email in error, please notify Discount Domain Name > Services Pty Ltd on 03 9815 6868 to report this matter and delete all > copies of this transmission together with any attachments. / > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |