Hi Josh, thanks a lot for the breakdown and the links. I disabled the write cache but it didn't change anything. Tomorrow I will try to disable bluefs_buffered_io. It doesn't sound that I can mitigate the problem with more SSDs. Am Di., 28. Feb. 2023 um 15:42 Uhr schrieb Josh Baergen < jbaergen@xxxxxxxxxxxxxxxx>: > Hi Boris, > > OK, what I'm wondering is whether > https://tracker.ceph.com/issues/58530 is involved. There are two > aspects to that ticket: > * A measurable increase in the number of bytes written to disk in > Pacific as compared to Nautilus > * The same, but for IOPS > > Per the current theory, both are due to the loss of rocksdb log > recycling when using default recovery options in rocksdb 6.8; Octopus > uses version 6.1.2, Pacific uses 6.8.1. > > 16.2.11 largely addressed the bytes-written amplification, but the > IOPS amplification remains. In practice, whether this results in a > write performance degradation depends on the speed of the underlying > media and the workload, and thus the things I mention in the next > paragraph may or may not be applicable to you. > > There's no known workaround or solution for this at this time. In some > cases I've seen that disabling bluefs_buffered_io (which itself can > cause IOPS amplification in some cases) can help; I think most folks > do this by setting it in local conf and then restarting OSDs in order > to gain the config change. Something else to consider is > > https://docs.ceph.com/en/quincy/start/hardware-recommendations/#write-caches > , > as sometimes disabling these write caches can improve the IOPS > performance of SSDs. > > Josh > > On Tue, Feb 28, 2023 at 7:19 AM Boris Behrens <bb@xxxxxxxxx> wrote: > > > > Hi Josh, > > we upgraded 15.2.17 -> 16.2.11 and we only use rbd workload. > > > > > > > > Am Di., 28. Feb. 2023 um 15:00 Uhr schrieb Josh Baergen < > jbaergen@xxxxxxxxxxxxxxxx>: > >> > >> Hi Boris, > >> > >> Which version did you upgrade from and to, specifically? And what > >> workload are you running (RBD, etc.)? > >> > >> Josh > >> > >> On Tue, Feb 28, 2023 at 6:51 AM Boris Behrens <bb@xxxxxxxxx> wrote: > >> > > >> > Hi, > >> > today I did the first update from octopus to pacific, and it looks > like the > >> > avg apply latency went up from 1ms to 2ms. > >> > > >> > All 36 OSDs are 4TB SSDs and nothing else changed. > >> > Someone knows if this is an issue, or am I just missing a config > value? > >> > > >> > Cheers > >> > Boris > >> > _______________________________________________ > >> > ceph-users mailing list -- ceph-users@xxxxxxx > >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > > > > -- > > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groüen Saal. > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |