cephadm sysctl-dir parameter does not affect location of /usr/lib/sysctl.d/90-ceph-${fsid}-osd.conf

["Gosch, Torsten" <Torsten.Gosch@xxxxxxxxxxxxxx>]

Hello,

I'm attempting to run the latest Ceph release (v16.2.5) on Fedora CoreOS (stable release 34.20210808.3.0). I was yet able to manage bootstrapping the cluster without any significant problems. But unfortunately one thing prevents further progress. The OSD containers won't start because they can't write the osd.conf tuning file because /usr/lib/sysctl.d is not writable (/usr is immutbale on CoreOS in general).

Output:

Deploy daemon osd.0 ...
Traceback (most recent call last):
  File "/var/lib/ceph/7768c8f2-0cbf-11ec-a9d8-0050569cdc1f/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in <module>
    main()
  File "/var/lib/ceph/7768c8f2-0cbf-11ec-a9d8-0050569cdc1f/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main
    r = ctx.func(ctx)
  File "/var/lib/ceph/7768c8f2-0cbf-11ec-a9d8-0050569cdc1f/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image
    return func(ctx)
  File "/var/lib/ceph/7768c8f2-0cbf-11ec-a9d8-0050569cdc1f/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4322, in command_deploy
    deploy_daemon(ctx, ctx.fsid, daemon_type, daemon_id, c, uid, gid,
  File "/var/lib/ceph/7768c8f2-0cbf-11ec-a9d8-0050569cdc1f/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2631, in deploy_daemon
    deploy_daemon_units(ctx, fsid, uid, gid, daemon_type, daemon_id,
  File "/var/lib/ceph/7768c8f2-0cbf-11ec-a9d8-0050569cdc1f/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2801, in deploy_daemon_units
    install_sysctl(ctx, fsid, daemon_type)
  File "/var/lib/ceph/7768c8f2-0cbf-11ec-a9d8-0050569cdc1f/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2963, in install_sysctl
    _write(conf, lines)
  File "/var/lib/ceph/7768c8f2-0cbf-11ec-a9d8-0050569cdc1f/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2948, in _write
    with open(conf, 'w') as f:
OSError: [Errno 30] Read-only file system: '/usr/lib/sysctl.d/90-ceph-7768c8f2-0cbf-11ec-a9d8-0050569cdc1f-osd.conf'
```

An explanation why this is also done system-widely in a containerized setup can be read at https://tracker.ceph.com/issues/47873. After some investigation i figured out that cephadm provides an optional parameter `sysctl-dir` (the description of cephadm argument parser is "location of sysctl configuration files"). I tried to apply this setting for cephadm bootstrap (--sysctl-dir /etc/sysctl.d) but without any effect on the deployment location (even changing SYSCTL_DIR variable ;-)

Don't really know if there is already another config option available to change the location or it's a cephadm bug after all.

Any hints welcome.

Thanks, Torsten
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx