Hi,
The autoscale settings of some pools seem to be disabled in some of my clusters.
This problem seems to be caused by overlapping root. My cluster is for RGW
and there are two shadow trees, one is for index (SSD), and the other
is for data (HDD).
This overlapping is caused by the existence of
`ceph-poc-object-store-ssd-index.rgw.otp` pool
which root is not one of the shadow roots, but the default root. In
addition, some other
clusters for RGW, which also have two shadow trees, don't have pools
which names are
postfixed by `rgw.opt`.
Could you tell me how to fix this problem and what is the `...rgw.opt` pool.
Here is the detail.
My cluster uses the following software.
- Ceph: v16.2.7
- Rook: v1.9.6
The result of `ceph osd pool ls`.
```
$ kubectl -n ceph-poc exec deploy/rook-ceph-tools -- ceph osd pool ls
ceph-poc-block-pool
ceph-poc-object-store-ssd-index.rgw.control
ceph-poc-object-store-ssd-index.rgw.meta
ceph-poc-object-store-ssd-index.rgw.log
ceph-poc-object-store-ssd-index.rgw.buckets.index
ceph-poc-object-store-ssd-index.rgw.buckets.non-ec
.rgw.root
ceph-poc-object-store-ssd-index.rgw.buckets.data
ceph-poc-object-store-hdd-index.rgw.control
ceph-poc-object-store-hdd-index.rgw.meta
ceph-poc-object-store-hdd-index.rgw.log
ceph-poc-object-store-hdd-index.rgw.buckets.index
ceph-poc-object-store-hdd-index.rgw.buckets.non-ec
ceph-poc-object-store-hdd-index.rgw.buckets.data
device_health_metrics
ceph-poc-object-store-ssd-index.rgw.otp
```
Some pools are missing in the result of `ceph osd pool autoscale-status`.
```
$ kubectl -n ceph-poc exec deploy/rook-ceph-tools -- ceph osd pool
autoscale-status
POOL SIZE TARGET
SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS
PG_NUM NEW PG_NUM AUTOSCALE BULK
ceph-poc-object-store-ssd-index.rgw.control 0
3.0 6144G 0.0000 1.0
8 on False
ceph-poc-object-store-ssd-index.rgw.meta 3910
3.0 6144G 0.0000 1.0
8 on False
ceph-poc-object-store-ssd-index.rgw.log 29328M
3.0 6144G 0.0140 1.0
8 on False
ceph-poc-object-store-ssd-index.rgw.buckets.index 4042
3.0 6144G 0.0000 1.0
128 8 off False
ceph-poc-object-store-ssd-index.rgw.buckets.non-ec 0
3.0 6144G 0.0000 1.0
8 on False
.rgw.root 9592
3.0 6144G 0.0000 1.0
8 on False
device_health_metrics 8890k
3.0 6144G 0.0000 1.0
32 on Fals
```
CRUSH rules are as follows.
```
$ kubectl -n ceph-poc exec deploy/rook-ceph-tools -- ceph osd crush
tree --show-shadow
ID CLASS WEIGHT TYPE NAME
-3 ssd 6.00000 root default~ssd
-9 ssd 2.00000 zone rack0~ssd
-8 ssd 1.00000 host 10-69-0-10~ssd
14 ssd 1.00000 osd.14
-51 ssd 0 host 10-69-0-22~ssd
...
...
-2 hdd 781.32037 root default~hdd
-7 hdd 130.99301 zone rack0~hdd
-6 hdd 0 host 10-69-0-10~hdd
-50 hdd 14.55478 host 10-69-0-22~hdd
8 hdd 7.27739 osd.8
...
-1 787.32037 root default
-5 132.99301 zone rack0
-4 1.00000 host 10-69-0-10
14 ssd 1.00000 osd.14
-49 14.55478 host 10-69-0-22
8 hdd 7.27739 osd.8
...
```
The rule of `...rgw.otp` pools is "replicated_rule".
```
$ kubectl -n ceph-poc exec deploy/rook-ceph-tools -- ceph osd pool get
ceph-poc-object-store-ssd-index.rgw.otp crush_rule
crush_rule: replicated_rule
```
The root of this rule is "default".
```
$ kubectl -n ceph-poc exec deploy/rook-ceph-tools -- ceph osd crush
rule dump replicated_rule
{
"rule_id": 0,
"rule_name": "replicated_rule",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
```
mgr daemon claims the overlapping root. Here is the log of mgr.
```
...
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.720+0000
7fe64d5d5700 0 [progress INFO root] Processing OSDMap change
175926..175926
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.445+0000
7fe64de96700 0 [pg_autoscaler WARNING root] pool 17 contains an
overlapping root -1... skipping scaling
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.444+0000
7fe64de96700 0 [pg_autoscaler INFO root] Pool 'device_health_metrics'
root_id -3 using 4.139842076256173e-06 of space, bias 1.0, pg target
0.000816928836381218 quantized to 32 (current 32)
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.444+0000
7fe64de96700 0 [pg_autoscaler INFO root] effective_target_ratio 0.0
0.0 0 6597069766656
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.442+0000
7fe64de96700 0 [pg_autoscaler WARNING root] pool 15 contains an
overlapping root -2... skipping scaling
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.441+0000
7fe64de96700 0 [pg_autoscaler WARNING root] pool 14 contains an
overlapping root -2... skipping scaling
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.440+0000
7fe64de96700 0 [pg_autoscaler WARNING root] pool 13 contains an
overlapping root -2... skipping scaling
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.438+0000
7fe64de96700 0 [pg_autoscaler WARNING root] pool 12 contains an
overlapping root -2... skipping scaling
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.437+0000
7fe64de96700 0 [pg_autoscaler WARNING root] pool 11 contains an
overlapping root -2... skipping scaling
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.436+0000
7fe64de96700 0 [pg_autoscaler WARNING root] pool 10 contains an
overlapping root -2... skipping scaling
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.434+0000
7fe64de96700 0 [pg_autoscaler WARNING root] pool 9 contains an
overlapping root -2... skipping scaling
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.433+0000
7fe64de96700 0 [pg_autoscaler INFO root] Pool '.rgw.root' root_id -3
using 4.361936589702964e-09 of space, bias 1.0, pg target
8.607554870347182e-07 quantized to 8 (current 8)
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.433+0000
7fe64de96700 0 [pg_autoscaler INFO root] effective_target_ratio 0.0
0.0 0 6597069766656
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.432+0000
7fe64de96700 0 [pg_autoscaler INFO root] Pool
'ceph-poc-object-store-ssd-index.rgw.buckets.non-ec' root_id -3 using
0.0 of space, bias 1.0, pg target 0.0 quantized to 8 (current 8)
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.432+0000
7fe64de96700 0 [pg_autoscaler INFO root] effective_target_ratio 0.0
0.0 0 6597069766656
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.430+0000
7fe64de96700 0 [pg_autoscaler INFO root] Pool
'ceph-poc-object-store-ssd-index.rgw.buckets.index' root_id -3 using
1.838088792283088e-09 of space, bias 1.0, pg target
3.627161883438627e-07 quantized to 8 (current 128)
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.430+0000
7fe64de96700 0 [pg_autoscaler INFO root] effective_target_ratio 0.0
0.0 0 6597069766656
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.429+0000
7fe64de96700 0 [pg_autoscaler INFO root] Pool
'ceph-poc-object-store-ssd-index.rgw.log' root_id -3 using
0.013985070909257047 of space, bias 1.0, pg target 2.7970141818514094
quantized to 8 (current 8)
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.429+0000
7fe64de96700 0 [pg_autoscaler INFO root] effective_target_ratio 0.0
0.0 0 6597069766656
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.427+0000
7fe64de96700 0 [pg_autoscaler INFO root] Pool
'ceph-poc-object-store-ssd-index.rgw.meta' root_id -3 using
1.7780621419660747e-09 of space, bias 1.0, pg target
3.5561242839321494e-07 quantized to 8 (current 8)
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.427+0000
7fe64de96700 0 [pg_autoscaler INFO root] effective_target_ratio 0.0
0.0 0 6597069766656
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.426+0000
7fe64de96700 0 [pg_autoscaler INFO root] Pool
'ceph-poc-object-store-ssd-index.rgw.control' root_id -3 using 0.0 of
space, bias 1.0, pg target 0.0 quantized to 8 (current 8)
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.426+0000
7fe64de96700 0 [pg_autoscaler INFO root] effective_target_ratio 0.0
0.0 0 6597069766656
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.425+0000
7fe64de96700 0 [pg_autoscaler WARNING root] pool 2 contains an
overlapping root -2... skipping scaling
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.421+0000
7fe64de96700 0 [pg_autoscaler ERROR root] pool 17 has overlapping
roots: {-2, -1}
2022-10-24T10:42:31+09:00 debug 2022-10-24T01:42:31.356+0000
7fe64de96700 0 [pg_autoscaler INFO root] _maybe_adjust
...
```
Regards,
Satoru
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
