HI,
We currently segregate ceph pool PG allocation using the crush device class ruleset as described: https://ceph.com/community/new-luminous-crush-device-classes/
simply using the following command to define the rule : ceph osd crush rule create-replicated <RULE> default host <DEVICE CLASS>
However, we noticed that the rule is not strict in certain scenarios. By that, I mean that if there is no OSD of the specific device class ceph will allocate PG for this pool to any other OSD available ( creating an issue with the PG calculation when we want to add new pool)
Simple scenario :
1. create 1 Pool : <pool1> , replication 2 with 4 nodes , 1 OSD each . belonging to class <pool1>
2. remove all OSD ( delete them )
3. create 4 new OSD (using same disk but different ID) but this time tag them with class <pool2>
4. Try to create pool <pool2> -> this will fail with
the pool creation will fail with output : Error ERANGE: pg_num 256 size 2 would mean 1024 total pgs, which exceeds max 800 (mon_max_pg_per_osd 200 * num_in_osds 4)"
Pool1 simply started allocating PG to OSD that doesn't belong to the ruleset
Which leads me to the following question: is there a way to make the crush rule a hard requirement. E.g : if we do not have any osd matching the device class , it won't start trying to allocate pg to OSD that doesn't match it?
Is there any way to prevent pool 1 to use the OSD ?
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
