Hello Josh, > > Was there PG movement (backfill) happening in this cluster recently? > Do the OSDs have stray PGs (e.g. 'ceph daemon osd.NN perf dump | grep > numpg_stray' - run this against an affected OSD from the housing > node)? yes, some nodes have stray pgs (1..5) shell I do something about those? > > I'm wondering if you're running into > https://tracker.ceph.com/issues/45765, where cleaning of PGs from OSDs hmm, yes, this seems very familiar, problems started with using balancer, forgot to mention that! > leads to a high read rate from disk due to a combination of rocksdb > behaviour and caching issues. Turning on bluefs_buffered_io (on by > default in 14.2.22) is a mitigation for this problem, but has some > side effects to watch out for (write IOPS amplification, for one). > Fixes for that linked issue went into 14.2.17, 14.2.22, and then > Pacific; we found the 14.2.17 changes to be quite effective by > themselves. > > Even if you don't have stray PGs, trying bluefs_buffered_io might be > an interesting experiment. An alternative would be to compact rocksdb > for each of your OSDs and see if that helps; compacting eliminates the > tombstoned data that can cause problems during iteration, but if you > have a workload that generates a lot of rocksdb tombstones (like PG > cleaning does), then the problem will return a while after compaction. > hmm, I'll try enabling bluefs_buffered_io (it was indeed false) and do the compaction as well anyways.. I'll report back, thanks for the hints! BR nik > Josh > -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis@xxxxxxxxxxx ------------------------------------- _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |