Hi all, during a lot of discussions in the past the comment that having "many PGs per OSD can lead to issues" came up without ever explaining what these issues will (not might!) be or how one would notice. It comes up as kind of a rumor without any factual or even anecdotal backing. As far as I can tell from experience, any increase of resource utilization due to an increase of the PG count per OSD is more than offset by the performance impact of the reduced size of the PGs. Everything seems to benefit from smaller PGs, recovery, user IO, scrubbing. Yet, I'm holding back on an increase of PG count due to these rumors. My situation: I would like to split PGs on large HDDs. Currently, we have on average 135PGs per OSD and I would like to go for something like 450. I heard in related rumors that some users have 1000+ PGs per OSD without problems. I would be very much interested in a non-rumor answer, that is, not an answer of the form "it might use more RAM", "it might stress xyz". I don't care what a rumor says it might do. I would like to know what it will do. I'm looking for answers of the form "a PG per OSD requires X amount of RAM fixed plus Y amount per object" or "searching/indexing stuff of kind A in N PGs per OSD requires N log N/N²/... operations", "peering of N PGs per OSD requires N/N log N/N²/N*#peers/... operations". In other words, what are the *actual* resources required to host N PGs with M objects on an OSD (note that N*M is a constant per OSD). With that info one could make an informed decision, informed by facts not rumors. An additional question of interest is: Has anyone ever observed any detrimental effects of increasing the PG count per OSD to large values>500? Thanks a lot for any clarifications in this matter! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |