Hi, we maintain a cluster with 126 OSDs, replication 3 and appr. 148T raw used space. We store data objects basically on two pools, the one being appr. 300x larger in data stored and # of objects terms than the other. Based on the formula provided here http://ceph.com/docs/master/rados/operations/placement-groups/ we computed that we need to increase our per pool pg_num & pgp_num to appr 6300 PGs / pool (100 * 126 / 2). We started by increasing the pg & pgp number on the smaller pool from 1800 to 2048 PGs (first the pg_num, then the pgp_num) and we experienced a 10X increase in Ceph total operations and an appr 3X disk latency increase in some underlying OSD disks. At the same time, for appr 10 seconds we experienced very low values of client io and op/s Should we be worried that the pg/pgp num increase on the bigger pool will have a 300X larger impact? Can we throttle this impact by injecting any thresholds or applying an appropriate configuration on our ceph conf? Regards, Kostis
 |