Ah I thought they were fixed. Still weird, the IDs inside the container definitely were not the same as on the host.
I think adding a check would be quite nice. Not sure if I have the skills for that but I don't mind giving it a try.
Should it check if the IDs on the host and container match?
On Thu, 26 Aug 2021 at 14:00, Sebastian Wagner <sewagner@xxxxxxxxxx> wrote:
Hi Massim,
cephadm looks for the IDs in the container:
https://github.com/ceph/ceph/blob/master/src/cephadm/cephadm#L4499
so as long as the host's (in this case the VM) IDs stay the same thorough the lifecycle, everything works. This seems to be an edge case for me. Would you be interested in adding a check to cephadm? Maybe to
https://github.com/ceph/ceph/blob/b7b73d5daa020643d0d4b1bba0d6fb6b091b005b/src/cephadm/cephadm#L5798
?
Am 26.08.21 um 13:33 schrieb Massim Knaapen:
Hi,
I got this cluster setup on 3VMs for testing purposes. Every time one of the boxes rebooted the OSDs daemons failed.They worked fine, as long as I didn't reboot....
Then I noticed that the UID and GID for the Ceph user were set incorrectly.If I understand correctly: the `ceph) user needs to have a UID and GID set to 167.In the Docker container this was correct, but on the host it was a different story.
The fix was quite easy of course:$ usermod -u 167 ceph$ groupmod -u 167 ceph
But the errors are not that clear (only found some reference about this in the OSD service log, see below).
If Ceph needs to have this user set to this specific ID. Maybe it would be nice to add this to the Troubleshooting section of the docs?
This is the output of the OSD service log (in case someone tries to search for this error):`Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.133+0000 7f025ef4a080 0 set uid:gid to 167:167 (ceph:ceph)
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.133+0000 7f025ef4a080 0 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable), process ceph-osd, pid 8
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.133+0000 7f025ef4a080 0 pidfile_write: ignore empty --pid-file
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bdev(0x5646bc068800 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bdev(0x5646bc068800 /var/lib/ceph/osd/ceph-0/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size>
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bdev(0x5646bc068c00 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bdev(0x5646bc068c00 /var/lib/ceph/osd/ceph-0/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size>
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 200 GiB
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bdev(0x5646bc068c00 /var/lib/ceph/osd/ceph-0/block) close
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.161+0000 7f025ef4a080 1 bdev(0x5646bc068800 /var/lib/ceph/osd/ceph-0/block) close
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.425+0000 7f025ef4a080 0 starting osd.0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.425+0000 7f025ef4a080 -1 Falling back to public interface
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.429+0000 7f025ef4a080 0 load: jerasure load: lrc load: isa
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.437+0000 7f025ef4a080 1 bdev(0x5646bc03c400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.437+0000 7f025ef4a080 -1 bdev(0x5646bc03c400 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied `
_______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
_______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
