UID and guid should both be 167 I believe. Make a note of the current values and change them to 167 using usermod and groupmod. I had just this issue. It’s partly to do with how perms are used within the containers I think. I changed the values to 167 in passwd everything worked again. Symptoms for me were OSDs not starting and permissions/file not found errors. Sent from my iPhone On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer@xxxxxxxxx> wrote: Hello, Permissions are correct. guid/uid is 64045/64045 ls -alh total 32K drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 . drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 .. lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/... -rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid -rw------- 1 ceph ceph 37 Apr 4 14:11 fsid -rw------- 1 ceph ceph 56 Apr 4 14:11 keyring -rw------- 1 ceph ceph 6 Apr 4 14:11 ready -rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release -rw------- 1 ceph ceph 10 Apr 4 14:11 type -rw------- 1 ceph ceph 3 Apr 4 14:11 whoami On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <andrew_jbrown@xxxxxxxxxxx<mailto:andrew_jbrown@xxxxxxxxxxx>> wrote: Are the file permissions correct and UID/guid in passwd both 167? Sent from my iPhone On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer@xxxxxxxxx<mailto:lomlaizer@xxxxxxxxx>> wrote: Hello, +1 Am facing the same problem in ubuntu after upgrade to pacific 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/ ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block: (1) Operation not permitted 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or directoryESC[0m On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <khoshbakhtib@xxxxxxxxx<mailto:khoshbakhtib@xxxxxxxxx>> wrote: > It worth mentioning as I issue the following command, the Ceph OSD starts > and joins the cluster: > /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph > > > > On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <khoshbakhtib@xxxxxxxxx<mailto:khoshbakhtib@xxxxxxxxx>> > wrote: > >> Hi all, >> >> As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the >> manual upgrade using the precompiled packages, the OSDs was down with the >> following messages: >> >> root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all >> --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 >> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph > prime-osd-dir >> --dev >> > /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 >> --path /var/lib/ceph/osd/ceph-2 --no-mon-config >> Running command: /usr/bin/ln -snf >> > /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 >> /var/lib/ceph/osd/ceph-2/block >> Running command: /usr/bin/chown -h ceph:ceph > /var/lib/ceph/osd/ceph-2/block >> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 >> Running command: /usr/bin/systemctl enable >> ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 >> Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 >> Running command: /usr/bin/systemctl start ceph-osd@2 >> --> ceph-volume lvm activate successful for osd ID: 2 >> >> Content of /var/log/ceph/ceph-osd.2.log >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 >> (ceph:ceph) >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 >> (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process >> ceph-osd, pid 5484 >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty >> --pid-file >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* >> bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open >> /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open >> OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or > directory* >> >> >> root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 >> â— ceph-osd@2.service - Ceph object storage daemon osd.2 >> Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; >> vendor preset: enabled) >> Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 >> +0430; 50s ago >> Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh >> --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) >> Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} > --id >> 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) >> Main PID: 5484 (code=exited, status=1/FAILURE) >> >> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart >> job, restart counter is at 3. >> Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon > osd.2. >> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request >> repeated too quickly. >> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result >> 'exit-code'. >> Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage >> daemon osd.2. >> root@osd03:/var/lib/ceph/osd/ceph-2# >> >> root@osd03:~# lsblk >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT >> fd0 2:0 1 4K 0 disk >> loop0 7:0 0 55.5M 1 loop >> /snap/core18/1988 >> loop1 7:1 0 69.9M 1 loop >> /snap/lxd/19188 >> loop2 7:2 0 55.5M 1 loop >> /snap/core18/1997 >> loop3 7:3 0 70.4M 1 loop >> /snap/lxd/19647 >> loop4 7:4 0 32.3M 1 loop >> /snap/snapd/11402 >> loop5 7:5 0 32.3M 1 loop >> /snap/snapd/11107 >> sda 8:0 0 80G 0 disk >> ├─sda1 8:1 0 1M 0 part >> ├─sda2 8:2 0 1G 0 part /boot >> └─sda3 8:3 0 79G 0 part >> └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / >> sdb 8:16 0 16G 0 disk >> └─sdb1 8:17 0 16G 0 part >> >> > └─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4 >> b89--bcd4--105b2df26352 >> 253:1 0 16G 0 lvm >> root@osd03:~# >> >> root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph >> tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) >> root@osd03:/var/lib/ceph/osd/ceph-2# >> >> any help is much appreciated >> -- >> >> Regards >> Behzad Khoshbakhti >> Computer Network Engineer (CCIE #58887) >> >> > > -- > > Regards > Behzad Khoshbakhti > Computer Network Engineer (CCIE #58887) > +989128610474 > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> > To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |