Hi,
with EC you need to take into account the total number of chunks for
PG calculation. Given your 16348 target, devide that number by 12 (if
I'm reading your crush rule correctly) results in 1362, this would be
either 1024 or 2048 PGs for that pool.
By the way, I wouldn't recommend to use all 12 hosts for your EC
profile since that would mean a degraded cluster as soon as one host
fails. You should consider one or two spare nodes so the PGs can be
recovered in case one node fails.
Regards,
Eugen
Zitat von Gunnar Bandelow <gunnar.bandelow@xxxxxxxxxxxxxxxxx>:
Hello Everyone,
i've run into problems with placement groups.
We have a 12 Host Ceph-Cluster with 408 OSDs (hdd and ssd).
I create a replicated pool with a large pg_num (16384). No problems
everything works.
If I do this with an erasure pool, i get a warning which is fixable
by extending the max_pgs_per_osd and afterwards i get a health_warn
in the ceph status due too few PGs.
Checking the pg_num after pool creation, it is capped at 1024.
I'm stuck at this point. Maybe i did something fundamentally wrong?
To illustrate my steps I tried to summarize everything in a small example:
# ceph -v
ceph version 14.2.7 (fb8e34a687d76cd3bd45c2a0fb445432ab69b4ff)
nautilus (stable)
# ceph osd erasure-code-profile get myerasurehdd
crush-device-class=hdd
crush-failure-domain=host
crush-root=default
jerasure-per-chunk-alignment=false
k=7
m=5
plugin=jerasure
technique=reed_sol_van
w=8
# ceph osd crush rule dump sas_rule
{
"rule_id": 0,
"rule_name": "sas_rule",
"ruleset": 0,
"type": 3,
"min_size": 1,
"max_size": 12,
"steps": [
{
"op": "take",
"item": -2,
"item_name": "default~hdd"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "rack"
},
{
"op": "emit"
}
]
}
# ceph osd pool create sas-pool 16384 16384 erasure myerasurehdd sas_rule
Error ERANGE: pg_num 16384 size 12 would mean 196704 total pgs,
which exceeds max 102000 (mon_max_pg_per_osd 250 * num_in_osds 408)
# ceph tell mon.\* injectargs '--mon-max-pg-per-osd=500'
mon.ceph-fs01: injectargs:mon_max_pg_per_osd = '500' (not observed,
change may require restart)
mon.ceph-fs05: injectargs:mon_max_pg_per_osd = '500' (not observed,
change may require restart)
mon.ceph-fs09: injectargs:mon_max_pg_per_osd = '500' (not observed,
change may require restart)
ceph-fs01:/opt/ceph-setup# ceph osd pool create sas-pool 16384 16384
erasure myerasurehdd sas_rule
pool 'sas-pool' created
# ceph -s
cluster:
id: b9471b57-95a2-4e58-8f69-b5e6048bea7c
health: HEALTH_WARN
Reduced data availability: 1024 pgs incomplete
too few PGs per OSD (7 < min 30)
# ceph osd pool get sas-pool pg_num
pg_num: 1024
Best regards,
Gunnar
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
