When we switched (Was using the compat balancer previously), I: 1. turned off the balancer 2. forced the client minimum (new centos7 clients are ok being forced to luminious even though they report as jewel. There's an email thread elsewhere describing it) 3. slowly reweighted the crush compat weight set back to completely matching. I manually ran the upmap balancer in between reweights to try and keep it not too unbalanced. 4. removed the crush compat weight set. didn't have any noticeable affect when all the weighs were back to matching. 5. ran some more manual balances to get things really well balanced 6. turned on the balancer We run the cluster near full and its quite busy so we did things much more slowly that way. The new balance using upmap is a significant improvement over the compat balancer. Kevin ________________________________________ From: Stolte, Felix <f.stolte@xxxxxxxxxxxxx> Sent: Wednesday, November 30, 2022 4:20 AM To: Dan van der Ster Cc: ceph-users@xxxxxxx Subject: Re: osd set-require-min-compat-client Check twice before you click! This email originated from outside PNNL. Hi Dan, thanks for your reply. I wasn’t worried about the setting itself, but about the balancer starting to use the pg-upmap feature (which currently fails, because of the jewel setting). I would assume though, that the balancer is using pg-upmap in a throttled way to avoid performance issues. I will execute the command on the weekend, just to be safe. Best regards Felix --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Am 30.11.2022 um 12:48 schrieb Dan van der Ster <dvanders@xxxxxxxxx>: Hi Felix, This change won't trigger any rebalancing. It will prevent older clients from connecting, but since this isn't a crush tunable it won't directly affect data placement. Best, Dan On Wed, Nov 30, 2022, 12:33 Stolte, Felix <f.stolte@xxxxxxxxxxxxx<mailto:f.stolte@xxxxxxxxxxxxx>> wrote: Hey guys, our ceph cluster is on pacific, but started on jewel years ago. While i was going through the logs of the mrg daemon i stumbled about the following entry: ========= [balancer ERROR root] execute error: r = -1, detail = min_compat_client jewel < luminous, which is required for pg-upmap. Try 'ceph osd set-require-min-compat-client luminous' before using the new interface ========= I could confirm that with `ceph osd get-require-min-compat-client` my value is still value. Reading the docs it looks to me, we really want to set this to luminous to benefit from a better pg distribution. My question for you is the following: Do I have to expect a major rebalancing after applying the 'ceph osd set-require-min-compat-client luminous‘ command, affecting my cluster IO? All my daemons are on pacific and all clients at least on nautilus. Thanks in advance and best regards Felix --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |