Thanks! Appreciate everyone who responded :) After reading up on stretch mode, it appears some of the exact things it was created to prevent happened, so this would be the solution! Cheers, D. -----Original Message----- From: Frank Schilder [mailto:frans@xxxxxx] Sent: Tuesday, November 29, 2022 1:49 AM To: Wolfpaw - Dale Corse <dale@xxxxxxxxxxx>; 'ceph-users' <ceph-users@xxxxxxx> Subject: Re: PGs stuck down Hi Dale, > we thought we had set it up to prevent.. and with size = 4 and > min_size set = 1 I'm afraid this is exactly what you didn't. Firstly, min_size=1 is always a bad idea. Secondly, if you have 2 data centres, the only way to get this to work is to use stretch mode. Even if you had min_size=2 (which, by the way you should have in any case), without stretch mode you would not be guaranteed that you have all PGs active+clean after one DC goes down (or cable gets cut). There is a quite long and very detailed explanation of why this is the case and with min_size=1 you are very certain to hit one of these cases or even loose data. What you could check in your situation are these two: mon_osd_min_up_ratio mon_osd_min_in_ratio My guess is that these prevented the mons from marking sufficiently many OSDs as out and therefore they got stuck peering (maybe even nothing was marked down?). The other thing is that you almost certainly had exactly the split brain situation that stretch mode is there to prevent. You probably ended up with 2 sub-clusters with 2 mons each and now what? If the third mon could still see the other 2 I don't think you get a meaningful quorum. Stretch mode will actually change the crush rule depending on a decision by the tie-breaking monitor to re-configure the pool to use only OSDs in one of the 2 DCs so that no cross-site peering happens. Maybe if you explicitly shut down one of the DC-mons you get stuff to work in one of the DCs? Without stretch mode you need 3 DCs and a geo-replicated 3(2) pool. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Wolfpaw - Dale Corse <dale@xxxxxxxxxxx> Sent: 29 November 2022 07:20:20 To: 'ceph-users' Subject: PGs stuck down Hi All, We had a fiber cut tonight between 2 data centers, and a ceph cluster didn't do very well :( We ended up with 98% of PGs as down. This setup has 2 data centers defined, with 4 copies across both, and a minimum of size of 1. We have 1 mon/mgr in each DC, with one in a 3rd data center connected to each of the other 2 by VPN. When I did a pg query on the PG's that were stuck it said they were blocked from coming up because they couldn't contact 2 of the OSDs (located in the other data center that it was unable to reach).. but the other 2 were fine. I'm at a loss because this was exactly the thing we thought we had set it up to prevent.. and with size = 4 and min_size set = 1 I understood that it would continue without a problem? :( Crush map is below .. if anyone has any ideas? I would sincerely appreciate it :) Thanks! Dale # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable straw_calc_version 1 # devices device 0 osd.0 class ssd device 1 osd.1 class ssd device 2 osd.2 class ssd device 3 osd.3 class ssd device 4 osd.4 class ssd device 5 osd.5 class ssd device 6 osd.6 class ssd device 7 osd.7 class ssd device 8 osd.8 class ssd device 9 osd.9 class ssd device 10 osd.10 class ssd device 11 osd.11 class ssd device 12 osd.12 class ssd device 13 osd.13 class ssd device 14 osd.14 class ssd device 15 osd.15 class ssd device 16 osd.16 class ssd device 17 osd.17 class ssd device 18 osd.18 class ssd device 19 osd.19 class ssd device 20 osd.20 class ssd device 21 osd.21 class ssd device 22 osd.22 class ssd device 23 osd.23 class ssd device 24 osd.24 class ssd device 25 osd.25 class ssd device 26 osd.26 class ssd device 27 osd.27 class ssd device 28 osd.28 class ssd device 29 osd.29 class ssd device 30 osd.30 class ssd device 31 osd.31 class ssd device 32 osd.32 class ssd device 33 osd.33 class ssd device 34 osd.34 class ssd device 35 osd.35 class ssd device 36 osd.36 class ssd device 37 osd.37 class ssd device 38 osd.38 class ssd device 39 osd.39 class ssd device 40 osd.40 class ssd device 41 osd.41 class ssd device 42 osd.42 class ssd device 43 osd.43 class ssd device 44 osd.44 class ssd device 45 osd.45 class ssd device 46 osd.46 class ssd device 47 osd.47 class ssd device 49 osd.49 class ssd # types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 region type 10 root # buckets host Pnode01 { id -8 # do not change unnecessarily id -9 class ssd # do not change unnecessarily # weight 0.000 alg straw2 hash 0 # rjenkins1 } host node01 { id -2 # do not change unnecessarily id -15 class ssd # do not change unnecessarily # weight 14.537 alg straw2 hash 0 # rjenkins1 item osd.4 weight 1.817 item osd.1 weight 1.817 item osd.3 weight 1.817 item osd.2 weight 1.817 item osd.6 weight 1.817 item osd.9 weight 1.817 item osd.5 weight 1.817 item osd.0 weight 1.818 } host node02 { id -3 # do not change unnecessarily id -16 class ssd # do not change unnecessarily # weight 14.536 alg straw2 hash 0 # rjenkins1 item osd.10 weight 1.817 item osd.11 weight 1.817 item osd.12 weight 1.817 item osd.13 weight 1.817 item osd.14 weight 1.817 item osd.15 weight 1.817 item osd.16 weight 1.817 item osd.19 weight 1.817 } host node03 { id -4 # do not change unnecessarily id -17 class ssd # do not change unnecessarily # weight 14.536 alg straw2 hash 0 # rjenkins1 item osd.20 weight 1.817 item osd.21 weight 1.817 item osd.22 weight 1.817 item osd.23 weight 1.817 item osd.25 weight 1.817 item osd.26 weight 1.817 item osd.29 weight 1.817 item osd.24 weight 1.817 } datacenter EDM1 { id -11 # do not change unnecessarily id -14 class ssd # do not change unnecessarily # weight 43.609 alg straw hash 0 # rjenkins1 item node01 weight 14.537 item node02 weight 14.536 item node03 weight 14.536 } host node04 { id -5 # do not change unnecessarily id -18 class ssd # do not change unnecessarily # weight 14.536 alg straw2 hash 0 # rjenkins1 item osd.30 weight 1.817 item osd.31 weight 1.817 item osd.32 weight 1.817 item osd.33 weight 1.817 item osd.34 weight 1.817 item osd.35 weight 1.817 item osd.36 weight 1.817 item osd.39 weight 1.817 } host node05 { id -6 # do not change unnecessarily id -19 class ssd # do not change unnecessarily # weight 14.536 alg straw2 hash 0 # rjenkins1 item osd.40 weight 1.817 item osd.41 weight 1.817 item osd.42 weight 1.817 item osd.43 weight 1.817 item osd.44 weight 1.817 item osd.45 weight 1.817 item osd.46 weight 1.817 item osd.49 weight 1.817 } host node06 { id -7 # do not change unnecessarily id -20 class ssd # do not change unnecessarily # weight 16.353 alg straw2 hash 0 # rjenkins1 item osd.47 weight 1.817 item osd.37 weight 1.817 item osd.27 weight 1.817 item osd.38 weight 1.817 item osd.7 weight 1.817 item osd.28 weight 1.817 item osd.8 weight 1.817 item osd.17 weight 1.817 item osd.18 weight 1.817 } datacenter EDM3 { id -12 # do not change unnecessarily id -13 class ssd # do not change unnecessarily # weight 45.425 alg straw hash 0 # rjenkins1 item node04 weight 14.536 item node05 weight 14.536 item node06 weight 16.353 } datacenter EDM2 { id -10 # do not change unnecessarily id -22 class ssd # do not change unnecessarily # weight 0.000 alg straw hash 0 # rjenkins1 } root default { id -1 # do not change unnecessarily id -21 class ssd # do not change unnecessarily # weight 89.034 alg straw2 hash 0 # rjenkins1 item Pnode01 weight 0.000 item EDM1 weight 43.609 item EDM3 weight 45.425 item EDM2 weight 0.000 } # rules rule replicated_ruleset { id 0 type replicated min_size 1 max_size 10 step take default step choose firstn 2 type datacenter step chooseleaf firstn 2 type host step emit } _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |