These settings are safe to change. I think you can leave set_choose_tries at 100 though. If you are somewhat uncertain, you can pull the OSD map from your production cluster and inject the new crush map into this osd map first. Osdmaptool then allows you to compute the actual mappings of your production system for a pool and you can check if the bad mapping goes away. This is a dry run and will show you exactly what will happen in your production cluster. If you search the user list for osdmaptool and my e-mail, you should find command examples. The ticket https://tracker.ceph.com/issues/57348 contains examples of how the output looks like. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Nicola Mori <mori@xxxxxxxxxx> Sent: 03 November 2022 10:57:03 To: ceph-users Subject: Re: Missing OSD in up set Hi Frank, I set choose_total_tries 250 and set_choose_tries 1000: I get no bad mappings and up to 239 tries. I guess I might try this rule in production, what do you suggest? On 03/11/22 10:46, Frank Schilder wrote: > Hi Nicola, > > you are hit hard by the problem, having so many mappings requiring 49 or more tries. The parameter you need to tune is not set_choose_tries inside the rule, but choose_total_tries at the beginning of the crush map file. You need to decompile, modify and compile again. The start of our file looks like this: > > # begin crush map > tunable choose_local_tries 0 > tunable choose_local_fallback_tries 0 > tunable choose_total_tries 250 > tunable chooseleaf_descend_once 1 > tunable chooseleaf_vary_r 1 > tunable chooseleaf_stable 1 > tunable straw_calc_version 1 > tunable allowed_bucket_algs 54 > > The default for choose_total_tries was 50 in my case and way too small. It will get better once you have more host buckets to choose OSDs from. > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Nicola Mori <mori@xxxxxxxxxx> > Sent: 03 November 2022 10:39:30 > To: ceph-users > Subject: Re: Missing OSD in up set > > Hi Frank, I checked the first hypothesis, and I found something strange. > This is the decompiled rule: > > rule wizard_data { > id 1 > type erasure > step set_chooseleaf_tries 5 > step set_choose_tries 100 > step take default > step chooseleaf indep 0 type host > step emit > } > > As you see it already contains step set_choose_tries 100. But when I > test it with crushtool I get: > > # crushtool -i crush.map --test --show-bad-mappings --rule 1 --num-rep 8 > --min-x 1 --max-x 1000 --show-choose-tries > bad mapping rule 1 x 319 num_rep 8 result [43,40,58,69,2147483647,21,11,31] > bad mapping rule 1 x 542 num_rep 8 result [50,75,53,55,66,43,61,2147483647] > bad mapping rule 1 x 721 num_rep 8 result [35,59,72,24,23,41,2147483647,15] > 0: 0 > 1: 7999 > 2: 12 > 3: 26 > 4: 36 > 5: 44 > 6: 52 > . . . > 48: 3 > 49: 4 > > As far as I understand the crushtool output, the maximum number of tries > is 49 < 100 so I should get no bad mapping. But maybe the bad mappings > are not considered in the tries count, so I tried to increase > set_choose_tries setting it to 1000. This seems to fix the bad mappings, > but the maximum number of tries does not change: > > # crushtool -i better-crush.map --test --show-bad-mappings --rule 1 > --num-rep 8 --min-x 1 --max-x 1000 --show-choose-tries > 0: 0 > 1: 8002 > 2: 12 > 3: 26 > 4: 36 > 5: 44 > 6: 52 > . . . > 48: 3 > 49: 4 > > I get 3 more PG placed with 1 try, which I guess might be the ones > having bad mappings when using set_choose_tries 100. If I'm correct then > I really don't understand why they need just one try with > set_choose_tries 1000 but fail with set_choose_tries 100... > > Anyway, do you think it would be worth trying set_choose_tries 1000 in > production? -- Nicola Mori, Ph.D. INFN sezione di Firenze Via Bruno Rossi 1, 50019 Sesto F.no (Italy) +390554572660 mori@xxxxxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |