Hi, we have a ceph cluster with 3 cluster nodes and 20 OSD's, with 6-7-7 2 TB HDD/s per node. In long term we want to use 7-9 pools, and for 20 OSD and 8 pools I calculate that the ideal pg_num was 250 (20 * 100 / 8). In this case normally each OSD store 100 pg's, that is the recommanded. I have few problems: 1. I have 1736 pg's, and if I want to create a new pool with 270 pg's, I got the error: Error ERANGE: pg_num 270 size 2 would mean 4012 total pgs, which exceeds max 4000 (mon_max_pg_per_osd 200 * num_in_osds 20) 2. Now we have 8 pools, but only one of them store huge amount of data, and for this reason I got a warning: health: HEALTH_WARN 1 pools have many more objects per pg than average But in past I remember that I got a warning that the pg_num for a pool is less/more then the average pg_num in cluster. In this case how can I set the optimal pg_num for my pools? Some debug data: OSD number: 20 data: pools: 8 pools, 1736 pgs objects: 560k objects, 1141 GB usage: 2331 GB used, 30053 GB / 32384 GB avail pgs: 1736 active+clean POOLS: NAME ID USED %USED MAX AVAIL OBJECTS kvmpool 5 34094M 0.24 13833G 8573 rbd 6 155G 1.11 13833G 94056 lxdhv04 15 29589M 0.21 13833G 12805 lxdhv01 16 14480M 0.10 13833G 9732 lxdhv02 17 14840M 0.10 13833G 7931 lxdhv03 18 18735M 0.13 13833G 7567 cephfs-metadata 22 40433k 0 13833G 11336 cephfs-data 23 876G 5.96 13833G 422108 pool 5 'kvmpool' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 1909 lfor 0/1906 owner 18446744073709551615 flags hashpspool stripe_width 0 application rbd pool 6 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 8422 lfor 0/2375 owner 18446744073709551615 flags hashpspool stripe_width 0 application rbd pool 15 'lxdhv04' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 3053 flags hashpspool stripe_width 0 application rbd pool 16 'lxdhv01' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 3054 flags hashpspool stripe_width 0 application rbd pool 17 'lxdhv02' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 8409 flags hashpspool stripe_width 0 application rbd pool 18 'lxdhv03' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 3066 flags hashpspool stripe_width 0 application rbd pool 22 'cephfs-metadata' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 8405 flags hashpspool stripe_width 0 application cephfs pool 23 'cephfs-data' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 8405 flags hashpspool stripe_width 0 application cephfs -- Ákos _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |