Thanks for the suggestion, Josh!
That part is relatively simple: the container gets ceph.conf from the
host's filesystem, for example:
"HostConfig": {
"Binds": [
"/dev:/dev",
"/run/udev:/run/udev",
"/var/run/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/run/ceph:z",
"/var/log/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/log/ceph:z",
"/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z",
"/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph05:/var/lib/ceph/mon/ceph-ceph05:z",
"/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph05/config:/etc/ceph/ceph.conf:z"
],
When I stop the monitor, edit the file directly and restart the monitor,
mon_rocksdb_options seem to be applied correctly!
Unfortunately, I specify global mon_rocksdb_options and redeploy the
monitor, the new ceph.conf doesn't have mon_rocksdb_options at all. I am
not sure that this is a reliable way to enable compression, but it works -
so it's better than other ways which don't work :-)
/Z
On Mon, 16 Oct 2023 at 16:16, Josh Baergen <jbaergen@xxxxxxxxxxxxxxxx>
wrote:
> > the resulting ceph.conf inside the monitor container doesn't have
> mon_rocksdb_options
>
> I don't know where this particular ceph.conf copy comes from, but I
> still suspect that this is where this particular option needs to be
> set. The reason I think this is that rocksdb mount options are needed
> _before_ the mon is able to access any of the centralized conf data,
> which I believe is itself stored in rocksdb.
>
> Josh
>
> On Sun, Oct 15, 2023 at 10:29 PM Zakhar Kirpichenko <zakhar@xxxxxxxxx>
> wrote:
> >
> > Out of curiosity, I tried setting mon_rocksdb_options via ceph.conf.
> This didn't work either: ceph.conf gets overridden at monitor start, the
> resulting ceph.conf inside the monitor container doesn't have
> mon_rocksdb_options, the monitor starts with no RocksDB compression.
> >
> > I would appreciate it if someone from the Ceph team could please chip in
> and suggest a working way to enable RocksDB compression in Ceph monitors.
> >
> > /Z
> >
> > On Sat, 14 Oct 2023 at 19:16, Zakhar Kirpichenko <zakhar@xxxxxxxxx>
> wrote:
> >>
> >> Thanks for your response, Josh. Our ceph.conf doesn't have anything but
> the mon addresses, modern Ceph versions store their configuration in the
> monitor configuration database.
> >>
> >> This works rather well for various Ceph components, including the
> monitors. RocksDB options are also applied to monitors correctly, but for
> some reason are being ignored.
> >>
> >> /Z
> >>
> >> On Sat, 14 Oct 2023, 17:40 Josh Baergen, <jbaergen@xxxxxxxxxxxxxxxx>
> wrote:
> >>>
> >>> Apologies if you tried this already and I missed it - have you tried
> >>> configuring that setting in /etc/ceph/ceph.conf (or wherever your conf
> >>> file is) instead of via 'ceph config'? I wonder if mon settings like
> >>> this one won't actually apply the way you want because they're needed
> >>> before the mon has the ability to obtain configuration from,
> >>> effectively, itself.
> >>>
> >>> Josh
> >>>
> >>> On Sat, Oct 14, 2023 at 1:32 AM Zakhar Kirpichenko <zakhar@xxxxxxxxx>
> wrote:
> >>> >
> >>> > I also tried setting RocksDB compression options and deploying a new
> >>> > monitor. The monitor started with no RocksDB compression again.
> >>> >
> >>> > Ceph monitors seem to ignore mon_rocksdb_options set at runtime, at
> mon
> >>> > start and at mon deploy. How can I enable RocksDB compression in Ceph
> >>> > monitors?
> >>> >
> >>> > Any input from anyone, please?
> >>> >
> >>> > /Z
> >>> >
> >>> > On Fri, 13 Oct 2023 at 23:01, Zakhar Kirpichenko <zakhar@xxxxxxxxx>
> wrote:
> >>> >
> >>> > > Hi,
> >>> > >
> >>> > > I'm still trying to fight large Ceph monitor writes. One option I
> >>> > > considered is enabling RocksDB compression, as our nodes have more
> than
> >>> > > sufficient RAM and CPU. Unfortunately, monitors seem to completely
> ignore
> >>> > > the compression setting:
> >>> > >
> >>> > > I tried:
> >>> > >
> >>> > > - setting ceph config set mon.ceph05 mon_rocksdb_options
> >>> > >
> "write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true",
> >>> > > restarting the test monitor. The monitor started with no RocksDB
> >>> > > compression:
> >>> > >
> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
> Compression
> >>> > > algorithms supported:
> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
> >>> > > kZSTDNotFinalCompression supported: 0
> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
> >>> > > kXpressCompression supported: 0
> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
> >>> > > kLZ4HCCompression supported: 1
> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
> >>> > > kLZ4Compression supported: 1
> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
> >>> > > kBZip2Compression supported: 0
> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
> >>> > > kZlibCompression supported: 1
> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
> >>> > > kSnappyCompression supported: 1
> >>> > > ...
> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
> >>> > > Options.compression: NoCompression
> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb:
> >>> > > Options.bottommost_compression: Disabled
> >>> > >
> >>> > > - setting ceph config set mon mon_rocksdb_options
> >>> > >
> "write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true",
> >>> > > restarting the test monitor. The monitor started with no RocksDB
> >>> > > compression, the same way as above.
> >>> > >
> >>> > > In each case config options were correctly set and readable with
> config
> >>> > > get. I also found a suggestion in ceph-users (
> >>> > >
> https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/message/KJM232IHN7FKYI5LODUREN7SVO45BL42/
> )
> >>> > > to set compression in a similar manner. Unfortunately, these
> options appear
> >>> > > to be ignored.
> >>> > >
> >>> > > How can I enable RocksDB compression in Ceph monitors?
> >>> > >
> >>> > > I would very much appreciate your advices and comments.
> >>> > >
> >>> > > Best regards,
> >>> > > Zakhar
> >>> > >
> >>> > >
> >>> > >
> >>> > _______________________________________________
> >>> > ceph-users mailing list -- ceph-users@xxxxxxx
> >>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
