Hi everyone,
while evaluating different config options at our Ceph cluster, I discovered that there are multiple ways to apply (ephemeral) config changes to specific running daemons. But even after researching docs and manpages, and doing some experiments, I fail to understand when to use which of the commands. Even worse, the behaviour of these commands appears to depend on the actual config option that is changed.
DISCLAIMER: I did my experiments on a Ceph Luminous cluster. While this clearly is a deprecated Ceph release, I am still interested in the subtleties of the individual commands even when you can only answer this for current Ceph releases.
To my understanding, there are at least 4 commands to change ceph daemon config at runtime:
- `ceph tell <type.id> injectargs --option_name=value`
- goes via monitor, mon intructs the daemon to do the injectargs
- `ceph daemon <type.id> config set "config option" value`
- local daemons only, via adminsocket
- also allows a `config get`
- `ceph tell <type.id> config set "option name" value`
- `ceph config set "config option" value`
- via some central monitor config store
- `config get` not yet implemented in Luminous
Is this overview correct, or do I have some misconceptions?
My experiments have shown that the behaviour of the commands differs depending on the config option changed:
Experiment 1: changing "osd max backfill" to 5 on daemon osd.6 and looking whether the value returned by `ceph daemon config get "osd max backfill" has changed.
the value has changed after: `ceph tell injectargs`, `ceph tell config set`, `ceph daemon config set`
the value has not changed after: `ceph config set`
Experiment 2: changing "mon pg warn max object skew" to 5 on daemon mon.cartman09 and looking whether the value returned by `ceph daemon config get "mon pg warn max object skew" has changed.
the value has changed after: `ceph tell injectargs`, `ceph daemon config set`
the value has not changed after: `ceph tell config set`, `ceph config set`
Even more confusing, in a cluster with a MANY_OBJECTS_PER_PG warning, only `ceph config set "mon pg warn max object skew" 20` resolved that warning, while a `ceph tell mon.\* injectargs --mon-pg-warn-max-object-skew=20` did not resolve the warning.
Can someone explain the subtle differences of these 4 commands to me? How does the central monitor config store relate to individual daemon configurations?
The full experiment logs can be found at the end of the mail.
Big thanks in advance
--
Oliver Schmidt · os@xxxxxxxxxxxxxxx · Systems Engineer
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick
========
Experiment 1:
```
~ # ceph daemon osd.6 config get "osd max backfills"
{
"osd max backfills": "2"
}
~ # ceph tell osd.6 injectargs --osd-max-backfills=5
osd_max_backfills = '5' rocksdb_separate_wal_dir = 'true' (not observed, change may require restart)
~ # ceph daemon osd.6 config get "osd max backfills"
{
"osd max backfills": "5"
}
~ # <daemon restart>
~ # ceph daemon osd.6 config get "osd max backfills"
{
"osd max backfills": "2"
}
~ # ceph tell osd.6 config set "osd max backfills" 5
Set osd_max_backfills to 5
~ # ceph daemon osd.6 config get "osd max backfills"
{
"osd max backfills": "5"
}
~ # <daemon restart>
~ # ceph daemon osd.6 config get "osd max backfills"
{
"osd max backfills": "2"
}
~ # ceph daemon osd.6 config set "osd max backfills" 5
{
"success": "osd_max_backfills = '5' rocksdb_separate_wal_dir = 'true' (not observed, change may require restart) "
}
~ # ceph daemon osd.6 config get "osd max backfills"
{
"osd max backfills": "5"
}
~ # <daemon restart>
~ # ceph daemon osd.6 config get "osd max backfills"
{
"osd max backfills": "2"
}
~ # ceph config set "osd max backfills" 5
Set osd_max_backfills to 5
~ # ceph daemon osd.6 config get "osd max backfills"
{
"osd max backfills": "2"
}
```
=======
Experiment 2:
```
root@cartman09 ~ # ceph daemon mon.cartman09 config get "mon pg warn max object skew"
{
"mon pg warn max object skew": "10.000000"
}
root@cartman09 ~ # ceph tell mon.cartman09 injectargs --mon_pg_warn_max_object_skew=20
injectargs:mon_pg_warn_max_object_skew = '20.000000' (not observed, change may require restart)
root@cartman09 ~ # ceph daemon mon.cartman09 config get "mon pg warn max object skew"
{
"mon pg warn max object skew": "20.000000"
}
root@cartman09 ~ # <daemon restart>
root@cartman09 ~ # ceph daemon mon.cartman09 config get "mon pg warn max object skew"
{
"mon pg warn max object skew": "10.000000"
}
root@cartman09 ~ # ceph tell mon.cartman09 config set "mon pg warn max object skew" 20
Set mon_pg_warn_max_object_skew to 20
root@cartman09 ~ # ceph daemon mon.cartman09 config get "mon pg warn max object skew"
{
"mon pg warn max object skew": "10.000000"
}
root@cartman09 ~ # <daemon restart>
root@cartman09 ~ # ceph daemon mon.cartman09 config set "mon pg warn max object skew" 20
{
"success": "mon_pg_warn_max_object_skew = '20.000000' (not observed, change may require restart) "
}
root@cartman09 ~ # ceph daemon mon.cartman09 config get "mon pg warn max object skew"
{
"mon pg warn max object skew": "20.000000"
}
root@cartman09 ~ # <daemon restart>
root@cartman09 ~ # ceph daemon mon.cartman09 config get "mon pg warn max object skew"
{
"mon pg warn max object skew": "10.000000"
}
root@cartman09 ~ # ceph config set "mon pg warn max object skew" 20
Set mon_pg_warn_max_object_skew to 20
root@cartman09 ~ # ceph daemon mon.cartman09 config get "mon pg warn max object skew"
{
"mon pg warn max object skew": "10.000000"
}
```
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
