Re: Increasing number of PGs by not a factor of two?

[Bryan Banister <bbanister@xxxxxxxxxxxxxxx>]

+1

 

From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Kai Wagner
Sent: Thursday, May 17, 2018 4:20 PM
To: David Turner <drakonstein@xxxxxxxxx>
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re: [ceph-users] Increasing number of PGs by not a factor of two?

 

Great summary David. Wouldn't this be worth a blog post?

 

On 17.05.2018 20:36, David Turner wrote:

By sticking with PG numbers as a base 2 number (1024, 16384, etc) all of your PGs will be the same size and easier to balance and manage.  What happens when you have a non base 2 number is something like this.  Say you have 4 PGs that are all 2GB in size.  If you increase pg(p)_num to 6, then you will have 2 PGs that are 2GB and 4 PGs that are 1GB as you've split 2 of the PGs into 4 to get to the 6 total.  If you increase the pg(p)_num to 8, then all 8 PGs will be 1GB.  Depending on how you manage your cluster, that doesn't really matter, but for some methods of balancing your cluster, that will greatly imbalance things.

 

This would be a good time to go to a base 2 number.  I think you're thinking about Gluster where if you have 4 bricks and you want to increase your capacity, going to anything other than a multiple of 4 (8, 12, 16) kills performance (worse than increasing storage already does) and takes longer as it has to weirdly divide the data instead of splitting a single brick up to multiple bricks.

 

As you increase your PGs, do this slowly and in a loop.  I like to increase my PGs by 256, wait for all PGs to create, activate, and peer, rinse/repate until I get to my target.  [1] This is an example of a script that should accomplish this with no interference.  Notice the use of flags while increasing the PGs.  It will make things take much longer if you have an OSD OOM itself or die for any reason by adding to the peering needing to happen.  It will also be wasted IO to start backfilling while you're still making changes; it's best to wait until you finish increasing your PGs and everything peers before you let data start moving.

 

Another thing to keep in mind is how long your cluster will be moving data around.  Increasing your PG count on a pool full of data is one of the most intensive operations you can tell a cluster to do.  The last time I had to do this, I increased pg(p)_num by 4k PGs from 16k to 32k, let it backfill, rinse/repeat until the desired PG count was achieved.  For me, that 4k PGs would take 3-5 days depending on other cluster load and how full the cluster was.  If you do decide to increase your PGs by 4k instead of the full increase, change the 16384 to the number you decide to go to, backfill, continue. 

 

 

[1]

# Make sure to set pool variable as well as the number ranges to the appropriate values.

flags="nodown nobackfill norecover"

for flag in $flags; do

  ceph osd set $flag

done

pool=rbd

echo "$pool currently has $(ceph osd pool get $pool pg_num) PGs"

# The first number is your current PG count for the pool, the second number is the target PG count, and the third number is how many to increase it by each time through the loop.

for num in {7700..16384..256}; do

  ceph osd pool set $pool pg_num $num

  while sleep 10; do

    ceph osd health | grep -q 'peering\|stale\|activating\|creating\|inactive' || break

  done

  ceph osd pool set $pool pgp_num $num

  while sleep 10; do

    ceph osd health | grep -q 'peering\|stale\|activating\|creating\|inactive' || break

  done

done

for flag in $flags; do

  ceph osd unset $flag

done

 

On Thu, May 17, 2018 at 9:27 AM Kai Wagner <kwagner@xxxxxxxx> wrote:

Hi Oliver,

a good value is 100-150 PGs per OSD. So in your case between 20k and 30k.

You can increase your PGs, but keep in mind that this will keep the
cluster quite busy for some while. That said I would rather increase in
smaller steps than in one large move.

Kai


On 17.05.2018 01:29, Oliver Schulz wrote:
> Dear all,
>
> we have a Ceph cluster that has slowly evolved over several
> years and Ceph versions (started with 18 OSDs and 54 TB
> in 2013, now about 200 OSDs and 1.5 PB, still the same
> cluster, with data continuity). So there are some
> "early sins" in the cluster configuration, left over from
> the early days.
>
> One of these sins is the number of PGs in our CephFS "data"
> pool, which is 7200 and therefore not (as recommended)
> a power of two. Pretty much all of our data is in the
> "data" pool, the only other pools are "rbd" and "metadata",
> both contain little data (and they have way too many PGs
> already, another early sin).
>
> Is it possible - and safe - to change the number of "data"
> pool PGs from 7200 to 8192 or 16384? As we recently added
> more OSDs, I guess it would be time to increase the number
> of PGs anyhow. Or would we have to go to 14400 instead of
> 16384?
>
>
> Thanks for any advice,
>
> Oliver
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

--
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)

---

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com