Re: Upgrade and lost osds Operation not permitted

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Thanks for reporting this issue. It has been fixed by
https://github.com/ceph/ceph/pull/40845 and will be released in the
next pacific point release.

Neha

On Mon, Apr 19, 2021 at 8:19 AM Behzad Khoshbakhti
<khoshbakhtib@xxxxxxxxx> wrote:
>
> thanks by commenting the ProtectClock directive, the issue is resolved.
> Thanks for the support.
>
> On Sun, Apr 18, 2021 at 9:28 AM Lomayani S. Laizer <lomlaizer@xxxxxxxxx>
> wrote:
>
> > Hello,
> >
> > Uncomment ProtectClock=true in /lib/systemd/system/ceph-osd@.service
> > should fix the issue
> >
> >
> >
> > On Thu, Apr 8, 2021 at 9:49 AM Behzad Khoshbakhti <khoshbakhtib@xxxxxxxxx>
> > wrote:
> >
> >> I believe there is some of problem in the systemd as the ceph starts
> >> successfully by running manually using the ceph-osd command.
> >>
> >> On Thu, Apr 8, 2021, 10:32 AM Enrico Kern <enrico.kern@xxxxxxxxxxxxxxx>
> >> wrote:
> >>
> >> > I agree. But why does the process start manual without systemd which
> >> > obviously has nothing to do with uid/gid 167 ? It is also not really a
> >> fix
> >> > to let all users change uid/gids...
> >> >
> >> > On Wed, Apr 7, 2021 at 7:39 PM Wladimir Mutel <mwg@xxxxxxxxx> wrote:
> >> >
> >> > > Could there be more smooth migration? On my Ubuntu I have the same
> >> > > behavior and my ceph uid/gud are also 64045.
> >> > > I started with Luminous in 2018 when it was not containerized, and
> >> still
> >> > > continue updating it with apt.
> >> > > Since when we have got this hardcoded value of 167 ???
> >> > >
> >> > > Andrew Walker-Brown wrote:
> >> > > > UID and guid should both be 167 I believe.
> >> > > >
> >> > > > Make a note of the current values and change them to 167 using
> >> usermod
> >> > > and groupmod.
> >> > > >
> >> > > > I had just this issue. It’s partly to do with how perms are used
> >> within
> >> > > the containers I think.
> >> > > >
> >> > > > I changed the values to 167 in passwd everything worked again.
> >> Symptoms
> >> > > for me were OSDs not starting and permissions/file not found errors.
> >> > > >
> >> > > > Sent from my iPhone
> >> > > >
> >> > > > On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer@xxxxxxxxx>
> >> > wrote:
> >> > > >
> >> > > > 
> >> > > > Hello,
> >> > > > Permissions are correct. guid/uid is 64045/64045
> >> > > >
> >> > > > ls -alh
> >> > > > total 32K
> >> > > > drwxrwxrwt 2 ceph ceph  200 Apr  4 14:11 .
> >> > > > drwxr-xr-x 8 ceph ceph 4.0K Sep 18  2018 ..
> >> > > > lrwxrwxrwx 1 ceph ceph   93 Apr  4 14:11 block -> /dev/...
> >> > > > -rw------- 1 ceph ceph   37 Apr  4 14:11 ceph_fsid
> >> > > > -rw------- 1 ceph ceph   37 Apr  4 14:11 fsid
> >> > > > -rw------- 1 ceph ceph   56 Apr  4 14:11 keyring
> >> > > > -rw------- 1 ceph ceph    6 Apr  4 14:11 ready
> >> > > > -rw------- 1 ceph ceph    3 Apr  4 14:11 require_osd_release
> >> > > > -rw------- 1 ceph ceph   10 Apr  4 14:11 type
> >> > > > -rw------- 1 ceph ceph    3 Apr  4 14:11 whoami
> >> > > >
> >> > > > On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <
> >> > > andrew_jbrown@xxxxxxxxxxx<mailto:andrew_jbrown@xxxxxxxxxxx>> wrote:
> >> > > > Are the file permissions correct and UID/guid in passwd  both 167?
> >> > > >
> >> > > > Sent from my iPhone
> >> > > >
> >> > > > On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer@xxxxxxxxx
> >> > <mailto:
> >> > > lomlaizer@xxxxxxxxx>> wrote:
> >> > > >
> >> > > > Hello,
> >> > > >
> >> > > > +1 Am facing the same problem in ubuntu after upgrade to pacific
> >> > > >
> >> > > > 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1
> >> > bluestore(/var/lib/ceph/osd/
> >> > > > ceph-29/block) _read_bdev_label failed to open
> >> > > /var/lib/ceph/osd/ceph-29/block:
> >> > > > (1) Operation not permitted
> >> > > > 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR:
> >> unable
> >> > > to
> >> > > > open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file
> >> or
> >> > > > directoryESC[0m
> >> > > >
> >> > > > On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <
> >> > > khoshbakhtib@xxxxxxxxx<mailto:khoshbakhtib@xxxxxxxxx>>
> >> > > > wrote:
> >> > > >
> >> > > >> It worth mentioning as I issue the following command, the Ceph OSD
> >> > > starts
> >> > > >> and joins the cluster:
> >> > > >> /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph
> >> --setgroup
> >> > > ceph
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <
> >> > > khoshbakhtib@xxxxxxxxx<mailto:khoshbakhtib@xxxxxxxxx>>
> >> > > >> wrote:
> >> > > >>
> >> > > >>> Hi all,
> >> > > >>>
> >> > > >>> As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during
> >> the
> >> > > >>> manual upgrade using the precompiled packages, the OSDs was down
> >> with
> >> > > the
> >> > > >>> following messages:
> >> > > >>>
> >> > > >>> root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate
> >> --all
> >> > > >>> --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
> >> > > >>> Running command: /usr/bin/chown -R ceph:ceph
> >> /var/lib/ceph/osd/ceph-2
> >> > > >>> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
> >> > > >> prime-osd-dir
> >> > > >>> --dev
> >> > > >>>
> >> > > >>
> >> > >
> >> >
> >> /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
> >> > > >>> --path /var/lib/ceph/osd/ceph-2 --no-mon-config
> >> > > >>> Running command: /usr/bin/ln -snf
> >> > > >>>
> >> > > >>
> >> > >
> >> >
> >> /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
> >> > > >>> /var/lib/ceph/osd/ceph-2/block
> >> > > >>> Running command: /usr/bin/chown -h ceph:ceph
> >> > > >> /var/lib/ceph/osd/ceph-2/block
> >> > > >>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
> >> > > >>> Running command: /usr/bin/chown -R ceph:ceph
> >> /var/lib/ceph/osd/ceph-2
> >> > > >>> Running command: /usr/bin/systemctl enable
> >> > > >>> ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
> >> > > >>> Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
> >> > > >>> Running command: /usr/bin/systemctl start ceph-osd@2
> >> > > >>> --> ceph-volume lvm activate successful for osd ID: 2
> >> > > >>>
> >> > > >>> Content of /var/log/ceph/ceph-osd.2.log
> >> > > >>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00  0 set uid:gid to
> >> > 64045:64045
> >> > > >>> (ceph:ceph)
> >> > > >>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00  0 ceph version 16.2.0
> >> > > >>> (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable),
> >> process
> >> > > >>> ceph-osd, pid 5484
> >> > > >>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00  0 pidfile_write: ignore
> >> > > empty
> >> > > >>> --pid-file
> >> > > >>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
> >> > > >>> bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed
> >> to
> >> > > open
> >> > > >>> /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
> >> > > >>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1  *** ERROR: unable to
> >> > open
> >> > > >>> OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
> >> > > >> directory*
> >> > > >>>
> >> > > >>>
> >> > > >>> root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
> >> > > >>> â— ceph-osd@2.service - Ceph object storage daemon osd.2
> >> > > >>>      Loaded: loaded (/lib/systemd/system/ceph-osd@.service;
> >> enabled;
> >> > > >>> vendor preset: enabled)
> >> > > >>>      Active: failed (Result: exit-code) since Sun 2021-04-04
> >> 14:55:06
> >> > > >>> +0430; 50s ago
> >> > > >>>     Process: 5471
> >> ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
> >> > > >>> --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS)
> >> > > >>>     Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster
> >> ${CLUSTER}
> >> > > >> --id
> >> > > >>> 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
> >> > > >>>    Main PID: 5484 (code=exited, status=1/FAILURE)
> >> > > >>>
> >> > > >>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled
> >> > > restart
> >> > > >>> job, restart counter is at 3.
> >> > > >>> Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage
> >> daemon
> >> > > >> osd.2.
> >> > > >>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start
> >> request
> >> > > >>> repeated too quickly.
> >> > > >>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with
> >> > > result
> >> > > >>> 'exit-code'.
> >> > > >>> Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object
> >> storage
> >> > > >>> daemon osd.2.
> >> > > >>> root@osd03:/var/lib/ceph/osd/ceph-2#
> >> > > >>>
> >> > > >>> root@osd03:~# lsblk
> >> > > >>> NAME                                  MAJ:MIN RM  SIZE RO TYPE
> >> > > MOUNTPOINT
> >> > > >>> fd0                                     2:0    1    4K  0 disk
> >> > > >>> loop0                                   7:0    0 55.5M  1 loop
> >> > > >>> /snap/core18/1988
> >> > > >>> loop1                                   7:1    0 69.9M  1 loop
> >> > > >>> /snap/lxd/19188
> >> > > >>> loop2                                   7:2    0 55.5M  1 loop
> >> > > >>> /snap/core18/1997
> >> > > >>> loop3                                   7:3    0 70.4M  1 loop
> >> > > >>> /snap/lxd/19647
> >> > > >>> loop4                                   7:4    0 32.3M  1 loop
> >> > > >>> /snap/snapd/11402
> >> > > >>> loop5                                   7:5    0 32.3M  1 loop
> >> > > >>> /snap/snapd/11107
> >> > > >>> sda                                     8:0    0   80G  0 disk
> >> > > >>> ├─sda1                                  8:1    0    1M  0 part
> >> > > >>> ├─sda2                                  8:2    0    1G  0 part
> >> /boot
> >> > > >>> └─sda3                                  8:3    0   79G  0 part
> >> > > >>>   └─ubuntu--vg-ubuntu--lv             253:0    0 69.5G  0 lvm  /
> >> > > >>> sdb                                     8:16   0   16G  0 disk
> >> > > >>> └─sdb1                                  8:17   0   16G  0 part
> >> > > >>>
> >> > > >>>
> >> > > >>
> >> > >
> >> >
> >> └─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
> >> > > >>>      b89--bcd4--105b2df26352
> >> > > >>>                                       253:1    0   16G  0 lvm
> >> > > >>> root@osd03:~#
> >> > > >>>
> >> > > >>> root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph
> >> > > >>> tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime)
> >> > > >>> root@osd03:/var/lib/ceph/osd/ceph-2#
> >> > > >>>
> >> > > >>> any help is much appreciated
> >> > > >>> --
> >> > > >>>
> >> > > >>> Regards
> >> > > >>> Behzad Khoshbakhti
> >> > > >>> Computer Network Engineer (CCIE #58887)
> >> > > >>>
> >> > > >>>
> >> > > >>
> >> > > >> --
> >> > > >>
> >> > > >> Regards
> >> > > >> Behzad Khoshbakhti
> >> > > >> Computer Network Engineer (CCIE #58887)
> >> > > >> +989128610474
> >> > > >> _______________________________________________
> >> > > >> ceph-users mailing list -- ceph-users@xxxxxxx<mailto:
> >> > ceph-users@xxxxxxx
> >> > > >
> >> > > >> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:
> >> > > ceph-users-leave@xxxxxxx>
> >> > > >>
> >> > > > _______________________________________________
> >> > > > ceph-users mailing list -- ceph-users@xxxxxxx<mailto:
> >> > ceph-users@xxxxxxx>
> >> > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:
> >> > > ceph-users-leave@xxxxxxx>
> >> > > > _______________________________________________
> >> > > > ceph-users mailing list -- ceph-users@xxxxxxx
> >> > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >> > > >
> >> > >
> >> > > _______________________________________________
> >> > > ceph-users mailing list -- ceph-users@xxxxxxx
> >> > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >> > >
> >> >
> >> >
> >> > --
> >> > UG (haftungsbeschränkt)
> >> > *   Enrico Kern, Geschäftsführer*
> >> >    Südstraße 26
> >> >    01877 Demitz-Thumitz
> >> >
> >> >    https://www.stackxperts.com
> >> >    enrico.kern@xxxxxxxxxxxxxxx
> >> >     +49 (0) 152 26814501
> >> >     skype: flyersa
> >> >
> >> >     Geschäftsführer Enrico Kern
> >> >     Amtsgericht Dresden | HRB 38717
> >> > _______________________________________________
> >> > ceph-users mailing list -- ceph-users@xxxxxxx
> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >> >
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users@xxxxxxx
> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>
> >
>
> --
>
> Regards
>  Behzad Khoshbakhti
>  Computer Network Engineer (CCIE #58887)
>  +989128610474
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux