Hi Anthony. > ... Bump up pg_num on pools and see how the average / P90 ceph-osd process size changes? > Grafana FTW. osd_map_cache_size I think defaults to 50 now; I want to say it used to be much higher. That's not an option. What would help is a-priori information based on the implementation. I'm looking at a pool with 5PB of data and 8192 PGs. If I increase that by, say a factor 4, its in one step, not gradual to avoid excessive redundant data movement. I don't want to spend hardware life for nothing and also don't want to wait for months or more for this to complete or get stuck along the way due to something catastrophic. What I would like to know is is there a fundamental scaling limit in the PG implementation that someone who was staring at the code for a long time knows about. This is usually something that grows much worse than N log N in time- or memory complexity. The answer to this is in the code and boils down to "why the recommendation of 100 PGs per OSD" and not 200 or 1000 or 100 per TB - the latter would make a looooot more sense). There ought to be a reason other than "we didn't know what else to write". I would like to know the scaling in (worst-case) complexity as a function of the number of PGs. Making a fixed recommendation of a specific number independent of anything else is something really weird. It indicates that there is something catastrophic in the code that will blow up once an (unknown/undocumented!!) threshold is crossed. For example, a tiny but important function that is exponential in the number of PGs. If there is nothing catastrophic in the code, then why is the recommendation not floating, specifying what increase in resource consumption one should expect. None of the discussions I have seen so far address this extreme weirdness of the recommendation. If there is an unsolved scaling problem, please anyone state what it is, why its there and what the critical threshold is. What part of the code will explode? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Anthony D'Atri <anthony.datri@xxxxxxxxx> Sent: Wednesday, October 9, 2024 3:52 PM To: Frank Schilder Cc: ceph-users@xxxxxxx Subject: Re: What is the problem with many PGs per OSD > Unfortunately, it doesn't really help answering my questions either. Sometimes the best we can do is grunt and shrug :-/. Before Nautilus we couldn’t merge PGs, so we could raise pg_num for a pool but not decrease it, so a certain fear of overshooting was established. Mark is the go-to here. > That's why deploying multiple OSDs per SSD is such a great way to improve performance on devices where 4K random IO throughput scales with iodepth. Mark’s testing have shown this to not be so much the case with recent releases — do you still see this? Until recently I was expecting 30TB TLC SSDs for RBD, and in the next year perhaps as large as 122T for object so I was thinking of splitting just because of the size - and the systems in question were overequipped with CPU. > Memory: I have never used file store, so can't relate to that. XFS - I experienced a lot of ballooning, to the point of OOMkilling. In mixed clusters under duress the BlueStore OSDs consistently behaved better. > 9000 PGs/OSD was too much for what kind of system? What CPU? How much RAM? How many OSDs per host? Those were Cisco UCS… C240m3. Dual 16c Sandy Bridge IIRC, 10x SATA HDD OSDs @ 3TB, 64GB I think. > Did it even work with 200PGs with the same data (recovery after power loss)? I didn’t have remote power control, and being a shared lab it was difficult to take a cluster down for such testing. We did have a larger integration cluster (450 OSDs) with a PG ratio of ~~ 200 where we tested a rack power drop. Ceph was fine (this was …. Firefly I think) but the LSI RoC HBAs lost data like crazy due to hardware, firmware, and utility bugs. > Was it maybe the death spiral (https://ceph-users.ceph.narkive.com/KAzvjjPc/explanation-for-ceph-osd-set-nodown-and-ceph-osd-cluster-snap) that prevented the cluster from coming up and not so much the PG count? Not in this case, though I’ve seen a similar cascading issue in another context. > Rumors: Yes, 1000 PGs/OSD on spinners without issues. I guess we are not talking about barely working home systems with lack of all sorts of resources here. I’d be curious how such systems behave under duress. I’ve seen a cluster that had grown - the mons ended up with enough RAM to run but not to boot, so I did urgent RAM upgrades on the mons. That was the mixed Filestore / BlueStore cluster (Luminous 12.2.2) where the Filestore OSDs were much more affected by a cascading event than the [mostly larger] BlueStore OSDs. I suspect that had the whole cluster been BlueStore it might not have cascaded. > > The goal: Let's say I want to go 500-1000PGs/OSD on 16T spinners to trim PGs to about 10-20G each. What are the resources that count will require compared with, say, 200 PGs/OSD? That's the interesting question and if I can make the resources available I would consider doing that. The proof is in the proverbial pudding. Bump up pg_num on pools and see how the average / P90 ceph-osd process size changes? Grafana FTW. osd_map_cache_size I think defaults to 50 now; I want to say it used to be much higher. > > Thanks and best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Anthony D'Atri <aad@xxxxxxxxxxxxxx> > Sent: Wednesday, October 9, 2024 2:40 AM > To: Frank Schilder > Cc: ceph-users@xxxxxxx > Subject: Re: What is the problem with many PGs per OSD > > I’ve sprinkled minimizers below. Free advice and worth every penny. ymmv. Do not taunt Happy Fun Ball. > > >> during a lot of discussions in the past the comment that having "many PGs per OSD can lead to issues" came up without ever explaining what these issues will (not might!) be or how one would notice. It comes up as kind of a rumor without any factual or even anecdotal backing. > > A handful of years ago Sage IIRC retconned PG ratio guidance from 200 to 100 to help avoid OOMing, the idea being that more PGs = more RAM usage on each daemon that stores the maps. With BlueStore’s osd_memory_target, my sense is that the ballooning seen with Filestore is much less of an issue. > >> As far as I can tell from experience, any increase of resource utilization due to an increase of the PG count per OSD is more than offset by the performance impact of the reduced size of the PGs. Everything seems to benefit from smaller PGs, recovery, user IO, scrubbing. > > My understanding is that there is serialization in the PG code, and thus the PG ratio can be thought of as the degree of parallelism the OSD device can handle. SAS/SATA SSDs don’t seek so they can handle more than HDDS, and NVMe devices can handle more than SAS/SATA. > >> Yet, I'm holding back on an increase of PG count due to these rumors. > > My personal sense: > > HDD OSD: PG ratio 100-200 > SATA/SAS SSD OSD: 200-300 > NVMe SSD OSD: 300-400 > > These are not empirical figures. ymmv. > > >> My situation: I would like to split PGs on large HDDs. Currently, we have on average 135PGs per OSD and I would like to go for something like 450. > > The good Mr. Nelson may have more precise advice, but my personal sense is that I wouldn’t go higher than 200 on an HDD. If you were at like 20 (I’ve seen it!) that would be a different story, my sense is that there are diminishing returns over say 150. Seek thrashing fu, elevator scheduling fu, op re-ordering fu, etc. Assuming you’re on Nautilus or later, it doesn’t hurt to experiment with your actual workload since you can scale pg_num back down. Without Filestore colocated journals, the seek thrashing may be less of an issue than it used to be. > >> I heard in related rumors that some users have 1000+ PGs per OSD without problems. > > On spinners? Or NVMe? On a 60-120 TB NVMe OSD I’d be sorely tempted to try 500-1000. > >> I would be very much interested in a non-rumor answer, that is, not an answer of the form "it might use more RAM", "it might stress xyz". I don't care what a rumor says it might do. I would like to know what it will do. > > It WILL use more RAM. > >> I'm looking for answers of the form "a PG per OSD requires X amount of RAM fixed plus Y amount per object” > > Derive the size of your map and multiple by the number of OSDs per system. My sense is that it’s on the order of MBs per OSD. After a certain point the RAM delta might have more impact by raising osd_memory_target instead. > >> or "searching/indexing stuff of kind A in N PGs per OSD requires N log N/N²/... operations", "peering of N PGs per OSD requires N/N log N/N²/N*#peers/... operations". In other words, what are the *actual* resources required to host N PGs with M objects on an OSD (note that N*M is a constant per OSD). With that info one could make an informed decision, informed by facts not rumors. >> >> An additional question of interest is: Has anyone ever observed any detrimental effects of increasing the PG count per OSD to large values>500? > > Consider this scenario: > > An unmanaged lab setup used for successive OpenStack deployments, each of which created two RBD pools and the panoply of RGW pools. Which nobody cleaned up before redeploys, so they accreted like plaque in the arteries of an omnivore. Such that the PG ratio hits 9000. Yes, 9000. Then the building loses power. The systems don’t have nearly enough RAM to boot, peer, and activate, so the entire cluster has to be wiped and redeployed from scratch. An extreme example, but remember that I don’t make stuff up. > >> >> Thanks a lot for any clarifications in this matter! >> ================= >> Frank Schilder >> AIT Risø Campus >> Bygning 109, rum S14 >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |