Question about the ideal number of PGs. This is the advice I've read for a single pool: 50-100 PGs per OSD or total_PGs = (OSDs * 100) / Replicas What happens as the number of pools increases? Should each pool have that same number of PGs, or do I need to increase or decrease the number of PGs per pool as the number of pools grows? Also, if I am using replication 3 (which is original plus two replicas?) should I use a value of 2 for replicas in the formula shown above? Or use the replication value of 3? Thanks! -Joe >-----Original Message----- >From: ceph-users-bounces@xxxxxxxxxxxxxx [mailto:ceph-users- >bounces@xxxxxxxxxxxxxx] On Behalf Of Niklas Goerke >Sent: Friday, September 27, 2013 4:29 AM >To: ceph-users@xxxxxxxxxxxxxx >Subject: Re: PG distribution scattered > >Sorry for replying only now, I did not get to try it earlier… > >On Thu, 19 Sep 2013 08:43:11 -0500, Mark Nelson wrote: >> On 09/19/2013 08:36 AM, Niklas Goerke wrote: >>> […] >>> >>> My Setup: >>> * Two Hosts with 45 Disks each --> 90 OSDs >>> * Only one newly created pool with 4500 PGs and a Replica Size of 2 >>> --> >>> should be about 100 PGs per OSD >>> >>> What I found was that one OSD only had 72 PGs, while another had 123 >>> PGs [1]. That means that - if I did the math correctly - I can only >>> fill the cluster to about 81%, because thats when the first OSD is >>> completely full[2]. >> >> Does distribution improve if you make a pool with significantly more >> PGs? >> >Yes it does. I tried 45000 PGs and got a range of minimum 922 to a maximum >of 1066 PGs per OSD (average is 1000). This is better, I can now fill my cluster >up to 93,8% (theoretically) but I still don't get why I would want to limit myself >to that. Also 1000 PGs are was to many for one OSD (I think 100 is suggested). >What should I do about this? > >>> I did some experimenting and found, that if I add another pool with >>> 4500 >>> PGs, each OSD will have exacly doubled the amount of PGs as with one >>> pool. So this is not an accident (tried it multiple times). On >>> another test-cluster with 4 Hosts and 15 Disks each, the Distribution >>> was similarly worse. >> >> This is a bug that causes each pool to more or less be distributed the >> same way on the same hosts. We have a fix, but it impacts backwards >> compatibility so it's off by default. If you set: >> >> osd pool default flag hashpspool = true >> >> Theoretically that will cause different pools to be distributed more >> randomly. >> >I did not try this, becuase in my production scenario we will probably only >have one or two very large pools, so it does not matter all that much to me. > >>> […] > >_______________________________________________ >ceph-users mailing list >ceph-users@xxxxxxxxxxxxxx >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |