Thanks for this, Eugen. I think I'll stick to adding the option to the config file, it seems like a safer way to do it. /Z On Tue, 17 Oct 2023, 15:21 Eugen Block, <eblock@xxxxxx> wrote: > Hi, > > I managed to get the compression setting into the MONs by using the > extra-entrypoint-arguments [1]: > > ceph01:~ # cat mon-specs.yaml > service_type: mon > placement: > hosts: > - ceph01 > - ceph02 > - ceph03 > extra_entrypoint_args: > - > > '--mon-rocksdb-options=write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true,bottommost_compression=kLZ4HCCompression,max_background_jobs=4,max_subcompactions=2' > > Just note that if you make a mistake and run 'ceph orch apply -i > mon-specs.yaml' with incorrect options your MON containers will all > fail. So test that in a non-critical environment first. In case the > daemons fail to start you can remove those options from the unit.run > file and get them up again. > But for me that worked and the daemons have the compression setting > enabled now. What remains unclear is which config options can be > changed as usual with the config database and which require this > extra-entrypoint-argument. > > Thanks again, Mykola! > Eugen > > [1] > > https://docs.ceph.com/en/quincy/cephadm/services/#extra-entrypoint-arguments > > Zitat von Zakhar Kirpichenko <zakhar@xxxxxxxxx>: > > > Thanks for the suggestion, Josh! > > > > That part is relatively simple: the container gets ceph.conf from the > > host's filesystem, for example: > > > > "HostConfig": { > > "Binds": [ > > "/dev:/dev", > > "/run/udev:/run/udev", > > > > "/var/run/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/run/ceph:z", > > > > "/var/log/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/log/ceph:z", > > > > > "/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z", > > > > > "/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph05:/var/lib/ceph/mon/ceph-ceph05:z", > > > > > "/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph05/config:/etc/ceph/ceph.conf:z" > > ], > > > > When I stop the monitor, edit the file directly and restart the monitor, > > mon_rocksdb_options seem to be applied correctly! > > > > Unfortunately, I specify global mon_rocksdb_options and redeploy the > > monitor, the new ceph.conf doesn't have mon_rocksdb_options at all. I am > > not sure that this is a reliable way to enable compression, but it works > - > > so it's better than other ways which don't work :-) > > > > /Z > > > > On Mon, 16 Oct 2023 at 16:16, Josh Baergen <jbaergen@xxxxxxxxxxxxxxxx> > > wrote: > > > >> > the resulting ceph.conf inside the monitor container doesn't have > >> mon_rocksdb_options > >> > >> I don't know where this particular ceph.conf copy comes from, but I > >> still suspect that this is where this particular option needs to be > >> set. The reason I think this is that rocksdb mount options are needed > >> _before_ the mon is able to access any of the centralized conf data, > >> which I believe is itself stored in rocksdb. > >> > >> Josh > >> > >> On Sun, Oct 15, 2023 at 10:29 PM Zakhar Kirpichenko <zakhar@xxxxxxxxx> > >> wrote: > >> > > >> > Out of curiosity, I tried setting mon_rocksdb_options via ceph.conf. > >> This didn't work either: ceph.conf gets overridden at monitor start, the > >> resulting ceph.conf inside the monitor container doesn't have > >> mon_rocksdb_options, the monitor starts with no RocksDB compression. > >> > > >> > I would appreciate it if someone from the Ceph team could please chip > in > >> and suggest a working way to enable RocksDB compression in Ceph > monitors. > >> > > >> > /Z > >> > > >> > On Sat, 14 Oct 2023 at 19:16, Zakhar Kirpichenko <zakhar@xxxxxxxxx> > >> wrote: > >> >> > >> >> Thanks for your response, Josh. Our ceph.conf doesn't have anything > but > >> the mon addresses, modern Ceph versions store their configuration in the > >> monitor configuration database. > >> >> > >> >> This works rather well for various Ceph components, including the > >> monitors. RocksDB options are also applied to monitors correctly, but > for > >> some reason are being ignored. > >> >> > >> >> /Z > >> >> > >> >> On Sat, 14 Oct 2023, 17:40 Josh Baergen, <jbaergen@xxxxxxxxxxxxxxxx> > >> wrote: > >> >>> > >> >>> Apologies if you tried this already and I missed it - have you tried > >> >>> configuring that setting in /etc/ceph/ceph.conf (or wherever your > conf > >> >>> file is) instead of via 'ceph config'? I wonder if mon settings like > >> >>> this one won't actually apply the way you want because they're > needed > >> >>> before the mon has the ability to obtain configuration from, > >> >>> effectively, itself. > >> >>> > >> >>> Josh > >> >>> > >> >>> On Sat, Oct 14, 2023 at 1:32 AM Zakhar Kirpichenko < > zakhar@xxxxxxxxx> > >> wrote: > >> >>> > > >> >>> > I also tried setting RocksDB compression options and deploying a > new > >> >>> > monitor. The monitor started with no RocksDB compression again. > >> >>> > > >> >>> > Ceph monitors seem to ignore mon_rocksdb_options set at runtime, > at > >> mon > >> >>> > start and at mon deploy. How can I enable RocksDB compression in > Ceph > >> >>> > monitors? > >> >>> > > >> >>> > Any input from anyone, please? > >> >>> > > >> >>> > /Z > >> >>> > > >> >>> > On Fri, 13 Oct 2023 at 23:01, Zakhar Kirpichenko < > zakhar@xxxxxxxxx> > >> wrote: > >> >>> > > >> >>> > > Hi, > >> >>> > > > >> >>> > > I'm still trying to fight large Ceph monitor writes. One option > I > >> >>> > > considered is enabling RocksDB compression, as our nodes have > more > >> than > >> >>> > > sufficient RAM and CPU. Unfortunately, monitors seem to > completely > >> ignore > >> >>> > > the compression setting: > >> >>> > > > >> >>> > > I tried: > >> >>> > > > >> >>> > > - setting ceph config set mon.ceph05 mon_rocksdb_options > >> >>> > > > >> > "write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true", > >> >>> > > restarting the test monitor. The monitor started with no RocksDB > >> >>> > > compression: > >> >>> > > > >> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > >> Compression > >> >>> > > algorithms supported: > >> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > >> >>> > > kZSTDNotFinalCompression supported: 0 > >> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > >> >>> > > kXpressCompression supported: 0 > >> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > >> >>> > > kLZ4HCCompression supported: 1 > >> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > >> >>> > > kLZ4Compression supported: 1 > >> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > >> >>> > > kBZip2Compression supported: 0 > >> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > >> >>> > > kZlibCompression supported: 1 > >> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > >> >>> > > kSnappyCompression supported: 1 > >> >>> > > ... > >> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > >> >>> > > Options.compression: NoCompression > >> >>> > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > >> >>> > > Options.bottommost_compression: Disabled > >> >>> > > > >> >>> > > - setting ceph config set mon mon_rocksdb_options > >> >>> > > > >> > "write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true", > >> >>> > > restarting the test monitor. The monitor started with no RocksDB > >> >>> > > compression, the same way as above. > >> >>> > > > >> >>> > > In each case config options were correctly set and readable with > >> config > >> >>> > > get. I also found a suggestion in ceph-users ( > >> >>> > > > >> > https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/message/KJM232IHN7FKYI5LODUREN7SVO45BL42/ > >> ) > >> >>> > > to set compression in a similar manner. Unfortunately, these > >> options appear > >> >>> > > to be ignored. > >> >>> > > > >> >>> > > How can I enable RocksDB compression in Ceph monitors? > >> >>> > > > >> >>> > > I would very much appreciate your advices and comments. > >> >>> > > > >> >>> > > Best regards, > >> >>> > > Zakhar > >> >>> > > > >> >>> > > > >> >>> > > > >> >>> > _______________________________________________ > >> >>> > ceph-users mailing list -- ceph-users@xxxxxxx > >> >>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |