Hi Boris, OK, what I'm wondering is whether https://tracker.ceph.com/issues/58530 is involved. There are two aspects to that ticket: * A measurable increase in the number of bytes written to disk in Pacific as compared to Nautilus * The same, but for IOPS Per the current theory, both are due to the loss of rocksdb log recycling when using default recovery options in rocksdb 6.8; Octopus uses version 6.1.2, Pacific uses 6.8.1. 16.2.11 largely addressed the bytes-written amplification, but the IOPS amplification remains. In practice, whether this results in a write performance degradation depends on the speed of the underlying media and the workload, and thus the things I mention in the next paragraph may or may not be applicable to you. There's no known workaround or solution for this at this time. In some cases I've seen that disabling bluefs_buffered_io (which itself can cause IOPS amplification in some cases) can help; I think most folks do this by setting it in local conf and then restarting OSDs in order to gain the config change. Something else to consider is https://docs.ceph.com/en/quincy/start/hardware-recommendations/#write-caches, as sometimes disabling these write caches can improve the IOPS performance of SSDs. Josh On Tue, Feb 28, 2023 at 7:19 AM Boris Behrens <bb@xxxxxxxxx> wrote: > > Hi Josh, > we upgraded 15.2.17 -> 16.2.11 and we only use rbd workload. > > > > Am Di., 28. Feb. 2023 um 15:00 Uhr schrieb Josh Baergen <jbaergen@xxxxxxxxxxxxxxxx>: >> >> Hi Boris, >> >> Which version did you upgrade from and to, specifically? And what >> workload are you running (RBD, etc.)? >> >> Josh >> >> On Tue, Feb 28, 2023 at 6:51 AM Boris Behrens <bb@xxxxxxxxx> wrote: >> > >> > Hi, >> > today I did the first update from octopus to pacific, and it looks like the >> > avg apply latency went up from 1ms to 2ms. >> > >> > All 36 OSDs are 4TB SSDs and nothing else changed. >> > Someone knows if this is an issue, or am I just missing a config value? >> > >> > Cheers >> > Boris >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users@xxxxxxx >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |