Hi Zhang, On Tue, 22 Sep 2015, Z Zhang wrote: > Hi ceph-devel, > > We enhanced the crushtool to simulate real pg distribution for a > specific pool. Recently we had encountered pg uneven issue again after > expanding our cluster, although we had re-weighted the cluster to make > pg evenly-distribution at the time we built up the cluster. > > It could be painful to re-weighted osds after expanding cluster because > in order to achieve pg evenly-distribution, we may need to re-weight > couple of times and each time will trigger data movement. > > Now we could use this crushtool to simulate pg distribution against > crush map, re-weight osds and test the crush map again and again until > we satisfy with pg distribution. Then we could set back the final crush > map and trigger data movement for one time. > > Please check this PR: https://github.com/ceph/ceph/pull/6004 if your > guys are interested. This looks like it will work, but I'm not sure it's the right place to put it. The osdmaptool has a --test-map-pgs option that maps all PGs and gives you the osd distribution and pg min/max per OSD. The procedure is then slightly different: ceph osd getmap -o om osdmaptool om --export-crush cm repeat: adjust crush map cm... osdmaptool om --import-crush cm --test-map-pgs ceph osd setcrushmap -i cm ...but it will keep the details of ceph pools out of crushtool (where they probably don't belong). What do you think? sage
