Hi,
I got this cluster setup on 3VMs for testing purposes. Every time one of the boxes rebooted the OSDs daemons failed.
They worked fine, as long as I didn't reboot....
Then I noticed that the UID and GID for the Ceph user were set incorrectly.
If I understand correctly: the `ceph) user needs to have a UID and GID set to 167.
In the Docker container this was correct, but on the host it was a different story.
The fix was quite easy of course:
$ usermod -u 167 ceph
$ groupmod -u 167 ceph
But the errors are not that clear (only found some reference about this in the OSD service log, see below).
If
Ceph needs to have this user set to this specific ID. Maybe it would be
nice to add this to the Troubleshooting section of the docs?
This is the output of the OSD service log (in case someone tries to search for this error):
`Aug
24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.133+0000
7f025ef4a080 0 set uid:gid to 167:167 (ceph:ceph)
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.133+0000 7f025ef4a080 0 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable), process ceph-osd, pid 8
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.133+0000 7f025ef4a080 0 pidfile_write: ignore empty --pid-file
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bdev(0x5646bc068800 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bdev(0x5646bc068800 /var/lib/ceph/osd/ceph-0/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size>
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bdev(0x5646bc068c00 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bdev(0x5646bc068c00 /var/lib/ceph/osd/ceph-0/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size>
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 200 GiB
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bdev(0x5646bc068c00 /var/lib/ceph/osd/ceph-0/block) close
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.161+0000 7f025ef4a080 1 bdev(0x5646bc068800 /var/lib/ceph/osd/ceph-0/block) close
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.425+0000 7f025ef4a080 0 starting osd.0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.425+0000 7f025ef4a080 -1 Falling back to public interface
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.429+0000 7f025ef4a080 0 load: jerasure load: lrc load: isa
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.437+0000 7f025ef4a080 1 bdev(0x5646bc03c400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.437+0000 7f025ef4a080 -1 bdev(0x5646bc03c400 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied `
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.133+0000 7f025ef4a080 0 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable), process ceph-osd, pid 8
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.133+0000 7f025ef4a080 0 pidfile_write: ignore empty --pid-file
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bdev(0x5646bc068800 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bdev(0x5646bc068800 /var/lib/ceph/osd/ceph-0/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size>
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bdev(0x5646bc068c00 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bdev(0x5646bc068c00 /var/lib/ceph/osd/ceph-0/block) open size 214744170496 (0x31ffc00000, 200 GiB) block_size>
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 200 GiB
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.137+0000 7f025ef4a080 1 bdev(0x5646bc068c00 /var/lib/ceph/osd/ceph-0/block) close
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.161+0000 7f025ef4a080 1 bdev(0x5646bc068800 /var/lib/ceph/osd/ceph-0/block) close
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.425+0000 7f025ef4a080 0 starting osd.0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.425+0000 7f025ef4a080 -1 Falling back to public interface
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.429+0000 7f025ef4a080 0 load: jerasure load: lrc load: isa
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.437+0000 7f025ef4a080 1 bdev(0x5646bc03c400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
Aug 24 17:00:09 alpha0001 bash[7301]: debug 2021-08-24T17:00:09.437+0000 7f025ef4a080 -1 bdev(0x5646bc03c400 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied `
_______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
