Hi Nicola,
you are hit hard by the problem, having so many mappings requiring 49 or more tries. The parameter you need to tune is not set_choose_tries inside the rule, but choose_total_tries at the beginning of the crush map file. You need to decompile, modify and compile again. The start of our file looks like this:
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 250
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54
The default for choose_total_tries was 50 in my case and way too small. It will get better once you have more host buckets to choose OSDs from.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Nicola Mori <mori@xxxxxxxxxx>
Sent: 03 November 2022 10:39:30
To: ceph-users
Subject: Re: Missing OSD in up set
Hi Frank, I checked the first hypothesis, and I found something strange.
This is the decompiled rule:
rule wizard_data {
id 1
type erasure
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default
step chooseleaf indep 0 type host
step emit
}
As you see it already contains step set_choose_tries 100. But when I
test it with crushtool I get:
# crushtool -i crush.map --test --show-bad-mappings --rule 1 --num-rep 8
--min-x 1 --max-x 1000 --show-choose-tries
bad mapping rule 1 x 319 num_rep 8 result [43,40,58,69,2147483647,21,11,31]
bad mapping rule 1 x 542 num_rep 8 result [50,75,53,55,66,43,61,2147483647]
bad mapping rule 1 x 721 num_rep 8 result [35,59,72,24,23,41,2147483647,15]
0: 0
1: 7999
2: 12
3: 26
4: 36
5: 44
6: 52
. . .
48: 3
49: 4
As far as I understand the crushtool output, the maximum number of tries
is 49 < 100 so I should get no bad mapping. But maybe the bad mappings
are not considered in the tries count, so I tried to increase
set_choose_tries setting it to 1000. This seems to fix the bad mappings,
but the maximum number of tries does not change:
# crushtool -i better-crush.map --test --show-bad-mappings --rule 1
--num-rep 8 --min-x 1 --max-x 1000 --show-choose-tries
0: 0
1: 8002
2: 12
3: 26
4: 36
5: 44
6: 52
. . .
48: 3
49: 4
I get 3 more PG placed with 1 try, which I guess might be the ones
having bad mappings when using set_choose_tries 100. If I'm correct then
I really don't understand why they need just one try with
set_choose_tries 1000 but fail with set_choose_tries 100...
Anyway, do you think it would be worth trying set_choose_tries 1000 in
production?
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
