Re: - permission denied on journal after reboot

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Piotr,
is your partition GUID right?

Look with sgdisk:
# sgdisk --info=2 /dev/sdd
Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
Partition unique GUID: 396A0C50-738C-449E-9FC6-B2D3A4469E51
First sector: 2048 (at 1024.0 KiB)
Last sector: 10485760 (at 5.0 GiB)
Partition size: 10483713 sectors (5.0 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph journal'

# sgdisk --info=2 /dev/sdc
Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
Partition unique GUID: 31E9A040-A2C2-4F8F-906E-19D8A24DBDAB
First sector: 2048 (at 1024.0 KiB)
Last sector: 10485760 (at 5.0 GiB)
Partition size: 10483713 sectors (5.0 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph journal'



Udo

Am 2017-02-13 16:13, schrieb Piotr Dzionek:
I run it on CentOS Linux release 7.3.1611. After running "udevadm test
/sys/block/sda/sda1" I don't see that this rule apply to this disk.

Hmm I remember that it used to work properly, but some time ago I
retested journal disk recreation. I followed the same tutorial like
the one pasted here by Wido den Hollander :

"The udev rules of Ceph should chown the journal to ceph:ceph if it's
set to the right partition UUID.
This blog shows it
partially:http://ceph.com/planet/ceph-recover-osds-after-ssd-journal-failure/";

I think that my journals were not recreated in a proper way, but I
don't know what is missing.

My SSD journal disk looks like this:
/Disk /dev/sda: 120GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name Flags
 1      1049kB  27.3GB  27.3GB               ceph journal
 2      27.3GB  54.6GB  27.3GB               ceph journal
 3      54.6GB  81.9GB  27.3GB               ceph journal
 4      81.9GB  109GB   27.3GB               ceph journal/

and blkid:
blkid | grep sda
/dev/sda1: PARTLABEL="ceph journal"
PARTUUID="a5ea6883-b2b2-4d53-b8ba-9ff8bcddead5"
/dev/sda2: PARTLABEL="ceph journal"
PARTUUID="adae4442-380c-418c-bdc0-05890fcf633e"
/dev/sda3: PARTLABEL="ceph journal"
PARTUUID="a8637452-fd9c-4d68-924f-69a43c75442c"
/dev/sda4: PARTLABEL="ceph journal"
PARTUUID="615a208a-19e0-4e02-8ef3-19d618a71103"

Do you have any idea what may be wrong?

W dniu 13.02.2017 o 12:45, Craig Chi pisze:
Hi,
What is your OS? The permission of journal partition should be changed by udev rules: /lib/udev/rules.d/95-ceph-osd.rules
In this file, it is described as:
# JOURNAL_UUID
ACTION=="add", SUBSYSTEM=="block", \
  ENV{DEVTYPE}=="partition", \
ENV{ID_PART_ENTRY_TYPE}=="45b0969e-9b03-4f30-b4c6-b4b80ceff106", \
  OWNER:="ceph", GROUP:="ceph", MODE:="660", \
  RUN+="/usr/sbin/ceph-disk --log-stdout -v trigger /dev/$name"
You can also use udevadm command to test whether the partition has been processed by the correct udev rule. Like following:
#> udevadm test /sys/block/sdb/sdb2
...
starting 'probe-bcache -o udev /dev/sdb2'
Process 'probe-bcache -o udev /dev/sdb2' succeeded.
OWNER 64045 /lib/udev/rules.d/95-ceph-osd.rules:16
GROUP 64045 /lib/udev/rules.d/95-ceph-osd.rules:16
MODE 0660 /lib/udev/rules.d/95-ceph-osd.rules:16
RUN '/usr/sbin/ceph-disk --log-stdout -v trigger /dev/$name' /lib/udev/rules.d/95-ceph-osd.rules:16
...
Then /dev/sdb2 will have ceph:ceph permission automatically.
#> ls -l /dev/sdb2
brw-rw---- 1 ceph ceph 8, 18 Feb 13 19:43 /dev/sdb2
Sincerely,
Craig Chi
On 2017-02-13 19:06, Piotr Dzionek <piotr.dzionek@xxxxxxxx> wrote:

    Hi,

    I am running ceph Jewel 10.2.5 with separate journals - ssd disks.
    It runs pretty smooth, however I stumble upon an issue after
    system reboot. Journal disks become owned by root and ceph failed
    to start.

    /starting osd.4 at :/0 osd_data /var/lib/ceph/osd/ceph-4
    /var/lib/ceph/osd/ceph-4/journal//
    / /2017-02-10 16:24:29.924126 7fd07ab40800 -1
    filestore(/var/lib/ceph/osd/ceph-4) mount failed to open journal
    /var/lib/ceph/osd/ceph-4/journal: (13) Permission denied//
    / /2017-02-10 16:24:29.924210 7fd07ab40800 -1 osd.4 0 OSD:init:
    unable to mount object store//
    / /2017-02-10 16:24:29.924217 7fd07ab40800 -1 #033[0;31m ** ERROR:
    osd init failed: (13) Permission denied#033[0m/

    I fixed this issue by finding journal disks in /dev dir and chown
    to ceph:ceph. I remember that I had a similar issue after I
    installed it for a first time. Is it a bug ? or do I have to set
    some kind of udev rules for this disks?

    FYI, I have this issue after every restart now.

    Kind regards,
    Piotr Dzionek

_______________________________________________ ceph-users mailing
    list ceph-users@xxxxxxxxxxxxxx
    http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux