Hello ceph-users,
after an upgrade from Ceph Nautilus to Octopus we ran into extreme
performance issues leading to an unusable cluster
when doing a larger snapshot delete and the cluster doing snaptrims, see
i.e. https://tracker.ceph.com/issues/50511#note-13.
Since this was not an issue prior to the upgrade, maybe the conversion
of the OSD to OMAP caused this degradation of the RocksDB data
structures, maybe not. (We were running bluefs_buffered_io=true, so that
was NOT the issue here).
But I've noticed there are a few reports of such issues which boil down
to RocksDB being in a somewhat degraded state and running a simple
compact fixed those issues, see:
* https://tracker.ceph.com/issues/50511
*
https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/XSEBOIT43TGIBVIGKC5WAHMB7NSD7D2B/
*
https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/BTWAQIEXBBEGTSTSJ4SK25PEWDEHIAUR/
*
https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/Z4ADQFTGC5HMMTCJZW3WHOTNLMU5Q4JR/
* Maybe also:
https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/N74C4U2POOSHZGE6ZVKFLVKN3LZ2XEEC/
I know improvements in this regard are actively worked on for pg
removal, i.e.
* https://tracker.ceph.com/issues/47174
** https://github.com/ceph/ceph/pull/37314
** https://github.com/ceph/ceph/pull/37496
but am wondering if this will help with snaptrims as well?
In any case I was just wondering if any of you also experienced this
condition with RocksDB and am wondering what you do to monitor or to
actively mitigate this prior to having flapping OSDs and queuing up
(snaptrim) operations?
With Ceph Pacific it's possible to enable offline compaction on every
start of an OSD (osd_compact_on_start), but is this really sufficient then?
Regards
Christian
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
