Hi all,
We got 4 PG active+remapped in our cluster. We set the pool?s ruleset to ruleset 0 and got HEALTH_OK. After we set the ruleset to ruleset 1, 4 pg is active+remapped. The testing result from crushtool also shows some bad mapping exists. Anyone happened to know the reason?
pool 0 'rbd' replicated size 3 min_size 2 crush_ruleset 1 object_hash rjenkins pg_num 256 pgp_num 256 last_change 421 flags hashpspool stripe_width 0
[root at Ceph-Stress-01 ~]# ceph pg ls remapped
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP
0.33 0 0 0 0 0 0 0 0 active+remapped 2017-03-07 22:53:02.453447 0'0 419:110 [3,22] 3 [3,22,4] 3 0'0 2017-03-07 13:29:32.110280 0'0 2017-03-07 13:29:32.110280
0.3b 0 0 0 0 0 0 0 0 active+remapped 2017-03-07 22:53:02.619526 0'0 419:110 [2,20] 2 [2,20,17] 2 0'0 2017-03-07 13:29:32.110287 0'0 2017-03-07 13:29:32.110287
0.49 0 0 0 0 0 0 0 0 active+remapped 2017-03-07 22:53:02.453239 0'0 419:104 [4,20] 4 [4,20,19] 4 0'0 2017-03-07 13:29:32.110257 0'0 2017-03-07 13:29:32.110257
0.54 0 0 0 0 0 0 0 0 active+remapped 2017-03-07 22:53:02.101725 0'0 419:207 [19,3] 19 [19,3,20] 19 0'0 2017-03-07 13:29:32.110262 0'0 2017-03-07 13:29:32.110262
rule replicated_ruleset_one_host {
ruleset 0
type replicated
min_size 1
max_size 10
step take default
step choose firstn 0 type osd
step emit
}
rule replicated_ruleset_multi_host {
ruleset 1
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 15.00000 root default
-2 7.00000 host Ceph-Stress-01
0 1.00000 osd.0 up 1.00000 1.00000
1 1.00000 osd.1 up 1.00000 1.00000
2 1.00000 osd.2 up 1.00000 1.00000
3 1.00000 osd.3 up 1.00000 1.00000
4 1.00000 osd.4 up 1.00000 1.00000
5 1.00000 osd.5 up 1.00000 1.00000
6 1.00000 osd.6 up 1.00000 1.00000
-4 7.00000 host Ceph-Stress-03
16 1.00000 osd.16 up 1.00000 1.00000
17 1.00000 osd.17 up 1.00000 1.00000
18 1.00000 osd.18 up 1.00000 1.00000
19 1.00000 osd.19 up 1.00000 1.00000
20 1.00000 osd.20 up 1.00000 1.00000
21 1.00000 osd.21 up 1.00000 1.00000
22 1.00000 osd.22 up 1.00000 1.00000
-3 1.00000 host Ceph-Stress-02
7 1.00000 osd.7 up 1.00000 1.00000
[root at Ceph-Stress-02 ~]# crushtool --test -i crushmap --rule 1 --min-x 1 --max-x 5 --num-rep 3 --show-utilization --show-mappings --show-bad-mappings
rule 1 (replicated_ruleset_multi_host), x = 1..5, numrep = 3..3
CRUSH rule 1 x 1 [5,7,21]
CRUSH rule 1 x 2 [17,6]
bad mapping rule 1 x 2 num_rep 3 result [17,6]
CRUSH rule 1 x 3 [19,7,1]
CRUSH rule 1 x 4 [5,22,7]
CRUSH rule 1 x 5 [21,0]
bad mapping rule 1 x 5 num_rep 3 result [21,0]
rule 1 (replicated_ruleset_multi_host) num_rep 3 result size == 2: 2/5
rule 1 (replicated_ruleset_multi_host) num_rep 3 result size == 3: 3/5
device 0: stored : 1 expected : 1
device 1: stored : 1 expected : 1
device 5: stored : 2 expected : 1
device 6: stored : 1 expected : 1
device 7: stored : 3 expected : 1
device 17: stored : 1 expected : 1
device 19: stored : 1 expected : 1
device 21: stored : 2 expected : 1
device 22: stored : 1 expected : 1
Thanks,
Tim
