When you increase your pg number, the new pgs will
have to peer first and during this time they will be unreachable.So you need to put the cluster in maintenance mode for this operation.
The way to upgrade the number of PG and the PGP of a running cluster is :
The way to upgrade the number of PG and the PGP of a running cluster is :
- First, it's very important to modify your Ceph settings in ceph.conf to keep your cluster responsive for client operations. Otherwise, all the IO and CPU will be used for the recovery operations and your cluster will be unreachable. Be sure that all these new parameters are in place and taken in account by all the running process before upgrading your cluster
osd_max_backfills = 1 (you can increase a little bit if the recovery is to slow but stay < 5)
osd_recovery_threads = 1
osd_recovery_threads = 1
osd_recovery_max_active = 1 (you can increase a little bit if the recovery is to slow but stay < 5)
osd_client_op_priority = 63
osd_recovery_op_priority = 1
osd_recovery_op_priority = 1
- stop scrub and deep-scrub operations (and wait till no scrub or deepscrub operation are running)
ceph osd set noscrub
ceph osd set nodeep-scrub
- set you cluster in maintenance mode with :
ceph osd set norecover
ceph osd set nobackfill
ceph osd set nodown
ceph osd set noout
- upgrade the pg number with a small increment like 256
- wait for the cluster to create and peer the new pgs (about 30 seconds)
- upgrade the pgp number with the same increment
- wait for the cluster to create and peer (about 30 seconds)
(Repeat the last 4 operations until you reach the number of pg and pgp you want
At this time, your cluster is still functionnal.
- Now you have to unset the maintenance mode
ceph osd unset noout
ceph osd unset nodown
ceph osd unset nobackfill
ceph osd unset norecover
ceph osd unset nodown
ceph osd unset nobackfill
ceph osd unset norecover
It
will take some time to rebalance all the data to the new pgs but at the end you will have
a cluster with all pgs active+clean.During all the operation,your
cluster will still be functionnal if you have respected the first settings
- When all the pgs are active+clean, you can re-enable the scrub and deep-scrub operations
ceph osd unset noscrub
ceph osd unset nodeep-scrub
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
