On Fri, Jul 29, 2016 at 11:47:59AM +0900, Christian Balzer wrote: > On Fri, 29 Jul 2016 09:59:38 +0800 Chengwei Yang wrote: > > > Hi list, > > > > I just followed the placement group guide to set pg_num for the rbd pool. > > > How many other pools do you have, or is that the only pool? Yes, this is the only one. > > The numbers mentioned are for all pools, not per pool, something that > isn't abundantly clear from the documentation either. Exactly, especially for newbie like me. :-) > > > " > > Less than 5 OSDs set pg_num to 128 > > Between 5 and 10 OSDs set pg_num to 512 > > Between 10 and 50 OSDs set pg_num to 4096 > > If you have more than 50 OSDs, you need to understand the tradeoffs and how to > > calculate the pg_num value by yourself > > For calculating pg_num value by yourself please take help of pgcalc tool > > " > > > You should have headed the hint about pgcalc, which is by far the best > thing to do. > The above numbers are an (imprecise) attempt to give a quick answer to a > complex question. > > > Since I have 40 OSDs, so I set pg_num to 4096 according to the above > > recommendation. > > > > However, after configured pg_num and pgp_num both to 4096, I found that my > > ceph cluster in **HEALTH_WARN** status, which does surprised me and still > > confusing me. > > > PGcalc would recommend 2048 PGs at most (for a single pool) with 40 OSDs. BTW, I read PGcal and found that it may also has some flaw as it says: " If the value of the above calculation is less than the value of (OSD#) / (Size), then the value is updated to the value of ((OSD#) / (Size)). This is to ensure even load / data distribution by allocating at least one Primary or Secondary PG to every OSD for every Pool. " However, in the above **OpenStack w RGW** use case, there are a lot of small pool with 32 PG that apparently smaller than OSD / Size(100/3 ～= 33.33). I do mean it though it's not smaller a lot. :-) > > I assume the above high number of 4096 stems from the wisdom that with > small clusters more PGs than normally recommended (100 per OSD) can be > helpful. > It was also probably written before those WARN calculations were added to > Ceph. > > The above would better read: > --- > Use PGcalc! > [...] > Between 10 and 20 OSDs set pg_num to 1024 > Between 20 and 40 OSDs set pg_num to 2048 > > Over 40 definitely use and understand PGcalc. > --- > > > > cluster bf6fa9e4-56db-481e-8585-29f0c8317773 > > health HEALTH_WARN > > too many PGs per OSD (307 > max 300) > > > > I see the cluster also says "4096 active+clean" so it's safe, but I do not like > > the HEALTH_WARN in anyway. > > > You can ignore it, but yes, it is annoying. > > > As I know(from ceph -s output), the recommended pg_num per OSD is [30, 300], any > > other pg_num out of this range with bring cluster to HEALTH_WARN. > > > > So what I would like to say: is the document misleading? Should we fix it? > > > Definitely. OK, I'd like to submit a PR. > > Christian > -- > Christian Balzer Network/Systems Engineer > chibi@xxxxxxx Global OnLine Japan/Rakuten Communications > http://www.gol.com/ -- Thanks, Chengwei
Attachment:
signature.asc
Description: Digital signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
