Just checking, are you aware of this ? http://ceph.com/pgcalc/ FYI, the warning is given based on the following logic. int per = sum_pg_up / num_in; if (per > g_conf->mon_pg_warn_max_per_osd) { //raise warning.. } This is not considering any resources..It is solely depends on number of in OSDs and total number of PGs in the cluster. Default mon_pg_warn_max_per_osd = 300, so, in your cluster per OSD is serving > 300 PGs it seems. It will be good if you assign PGs in your pool keeping the above calculation in mind i.e no more than 300 PGs/ OSD.. But, if you feel you OSD is in fast disk and box has lot of compute power, you may want to try out with more number of PGs/OSD. In this case, raise the mon_pg_warn_max_per_osd to something big and warning should go away. Hope this helps, Thanks & Regards Somnath -----Original Message----- From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Stuart Longland Sent: Wednesday, May 06, 2015 3:48 PM To: ceph-users@xxxxxxxxxxxxxx Subject: Re: "too many PGs per OSD" in Hammer On 07/05/15 07:53, Chris Armstrong wrote: > Thanks for the feedback. That language is confusing to me, then, since > the first paragraph seems to suggest using a pg_num of 128 in cases > where we have less than 5 OSDs, as we do here. > > The warning below that is: "As the number of OSDs increases, chosing > the right value for pg_num becomes more important because it has a > significant influence on the behavior of the cluster as well as the > durability of the data when something goes wrong (i.e. the probability > that a catastrophic event leads to data loss).", which suggests that > this could be an issue with more OSDs, which doesn't apply here. > > Do we know if this warning is calculated based on the resources of the > host? If I try with larger machines, will this warning change? I'd be interested in an answer here too. I just did an update from Giant to Hammer and struck the same dreaded error message. When I initially deployed Ceph (with Emperor), I worked out according to the formula given on the site: > # We have: 3 OSD nodes with 2 OSDs each > # giving us 6 OSDs total. > # There are 3 replicas, so the recommended number of > # placement groups is: > # 6 * 100 / 3 > # which gives: 200 placement groups. > # Rounding this up to the nearest power of two gives: > osd pool default pg num = 256 > osd pool default pgp num = 256 It seems this was a bad value to use. I now have a problem of a biggish lump of data sitting in a pool with an inappropriate number of placement groups. It seems I needed to divide this number by the number of pools. For now I've shut it up with the following: > [mon] > mon warn on legacy crush tunables = false > # New warning on move to Hammer > mon pg warn max per osd = 2048 Question is, how does one go about fixing this? I'd rather not blow away production pools just at this point although right now we only have one major production load, so if we're going to do it at any time, now is the time to do it. Worst bit is this will probably change: so I can see me hitting this problem time and time again as a new pool is added some time later. Is there a way of tuning the number of placement groups without destroying data? Regards, -- _ ___ Stuart Longland - Systems Engineer \ /|_) | T: +61 7 3535 9619 \/ | \ | 38b Douglas Street F: +61 7 3535 9699 SYSTEMS Milton QLD 4064 http://www.vrt.com.au _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |