Arguably the error should say something like
" TASK ERROR: error with 'osd pool create': mon_command failed - pg_num 512 size 8 would result in 251 cumulative PGs per OSD (22148 total PG replicas on 88 ‘in’ OSDs), which exceeds the mon_max_pg_per_osd value of 250.
IMHO this is sort of mixing up max_pgs_per_osd with mon_max_pool_pg_num.
Here’s the code
auto max_pgs = max_pgs_per_osd * num_osds;
if (projected > max_pgs) {
if (pool >= 0) {
*ss << "pool id " << pool;
}
*ss << " pg_num " << pg_num << " size " << size
<< " would mean " << projected
<< " total pgs, which exceeds max " << max_pgs
<< " (mon_max_pg_per_osd " << max_pgs_per_osd
<< " * num_in_osds " << num_osds << ")";
return -ERANGE;
}
IMHO this is sort of confusing max_pgs_per_osd with mon_max_pool_pg_num.
Here’s a PR with a suggested cleanup for clarity:
https://github.com/ceph/ceph/pull/49180
> On Dec 1, 2022, at 13:00, Eugen Block <eblock@xxxxxx> wrote:
>
> Hi,
>
> you need to take into account the number of replicas as well. With 88 OSDs and the default max PGs per OSD of 250 you get the mentioned 22000 PGs (including replica): 88 x 250 = 22000.
> With EC pools each chunk counts as one replica. So you should consider shrinking your pools or let autoscaler do that for you.
>
> Regards
> Eugen
>
> Zitat von Rainer Krienke <krienke@xxxxxxxxxxxxxx>:
>
>> Hello,
>>
>> I run a a hyperconverged pve cluster (V7.2) with 11 nodes. Each node has 8 4TB disks. pve and ceph are installed an running.
>>
>> I wanted to create some ceph-pools with each 512 pgs. Since I want to use erasure coding (5+3) when creating a pool one rbd pool for metadata and the data pool are created. I used pveceph pool this command:
>>
>> pveceph pool create px-e --erasure-coding k=5,m=3 --pg_autoscale_mode off --pg_num 512 --pg_num_min 128
>>
>> I was able to create two pools in this way but the third pveceph call threw this error:
>>
>> "got unexpected control message: TASK ERROR: error with 'osd pool create': mon_command failed - pg_num 512 size 8 would mean 22148 total pgs, which exceeds max 22000 (mon_max_pg_per_osd 250 * num_in_osds 88)"
>>
>> I also tried the direct way to create a new pool using:
>> ceph osd pool create <pool> 512 128 erasure <profile> but the error message below remains.
>>
>> What I do not understand now are the calculations behind the scenes for the calculated total pg number of 22148. How is this total number "22148" calculated?
>>
>> I already reduced the number of pgs for the metadata pool of each ec-pool and so I was able to create 4 pools in this way. But just for fun I now tried to create ec-pool number 5 and I see the message from above again.
>>
>> Here are the pools created by now (scraped from ceph osd pool autoscale-status):
>> Pool: Size: Bias: PG_NUM:
>> rbd 4599 1.0 32
>> px-a-data 528.2G 1.0 512
>> px-a-metadata 838.1k 1.0 128
>> px-b-data 0 1.0 512
>> px-b-metadata 19 1.0 128
>> px-c-data 0 1.0 512
>> px-c-metadata 19 1.0 128
>> px-d-data 0 1.0 512
>> px-d-metadata 0 1.0 128
>>
>> So the total number of pgs for all pools is currently 2592 which is far from 22148 pgs?
>>
>> Any ideas?
>> Thanks Rainer
>> --
>> Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
>> 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
>> PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@xxxxxxx
>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
