Re: mkjournal error creating journal ... : (13) Permission denied

Peter Maloney <peter.maloney@xxxxxxxxxxxxxxxxxxxx> · Wed, 15 Mar 2017 10:46:03 +0100



    On 03/15/17 08:43, Gunwoo Gim wrote:

    
           After a reboot, all the partitions of LVM don't show up
            in /dev/mapper -nor in the /dev/dm-<dm-num> or
            /proc/partitions- though the whole disks show up; I have to
            make the hosts run one 'partprobe' every time they boot so
            as to have the partitions all show up.
        
      
    Maybe you need this after partprobe:

    
        udevadm trigger

    
         I've found out that the udev rules have never triggered even
        when I removed the DEVTYPE checking part; checked with a udev
        line: RUN+="/bin/echo 'add /dev/$name' >> /root/log.txt" 
        
           I've also tried chowning all the /dev/dm-<num> to
            ceph:disk in vain. Do I have to use the udev rules even if
            the /dev/dm-<num> s are already owned by ceph:ceph?
          

    No, I think you just need them owned by ceph:ceph. Test that with
    something like:

    
        sudo -u ceph hexdump -C /dev/dm-${number} | head

    
    (which reads, not writes...so not a full test, but close enough)

    
    And also make sure the files in /var/lib/ceph/{osd,mon,...} are
    owned by ceph:ceph too. Maybe you have a mix of root and ceph, which
    is easy to cause by running it as root when ceph owns some files.

    
    And FYI, I don't like udev, and did not use ceph-deploy or
    ceph-disk. I did it with a very simple init script instead:

    
    #!/bin/bash

      mkdir -p /var/run/ceph

      chown ceph:ceph /var/run/ceph

      chgrp -R ceph /var/log/ceph

      for d in /var/lib/ceph/osd/*/journal; do

          d=$(readlink -f "$d")

          chown ceph:ceph "$d"

      done

    
    This works on ubuntu 14.04 as is, a badly written init script, but I
    think centos will not accept it without the lsb tags.

    
    A side effect of doing it this way is you have to manually run the
    script again when replacing or adding disks, since it is not run
    after hot swap like udev is.

    
           Thank you very much for reading.
          

          Best Regards,
          Nicholas.
        
      
        On Wed, Mar 15, 2017 at 1:06 AM Gunwoo Gim <wind8702@xxxxxxxxx>
          wrote:

        
             Thank you very much, Peter.
            

             I'm sorry for not clarifying the
              version number; it's kraken and 11.2.0-1xenial.

            
             I guess the udev rules in this file
              are supposed to change them :
              /lib/udev/rules.d/95-ceph-osd.rules
             ...but the rules' filters don't seem
              to match the DEVTYPE part of the prepared partitions on
              the LVs I've got on the host.
            

             Would it have been the cause of
              trouble? I'd love to be informed of a good way to make it
              work with the logical volumes; should I fix the udev rule?
            

              ~ # cat
                /lib/udev/rules.d/95-ceph-osd.rules | head -n 19
              # OSD_UUID
              ACTION="" SUBSYSTEM=="block",
                \
                ENV{DEVTYPE}=="partition", \
               
                ENV{ID_PART_ENTRY_TYPE}=="4fbd7e29-9d25-41b8-afd0-062c0ceff05d",
                \
                OWNER:="ceph", GROUP:="ceph",
                MODE:="660", \
                RUN+="/usr/sbin/ceph-disk
                --log-stdout -v trigger /dev/$name"
              ACTION=""
                SUBSYSTEM=="block", \
               
                ENV{ID_PART_ENTRY_TYPE}=="4fbd7e29-9d25-41b8-afd0-062c0ceff05d",
                \
                OWNER="ceph", GROUP="ceph",
                MODE="660"
              

              # JOURNAL_UUID
              ACTION="" SUBSYSTEM=="block",
                \
                ENV{DEVTYPE}=="partition", \
               
                ENV{ID_PART_ENTRY_TYPE}=="45b0969e-9b03-4f30-b4c6-b4b80ceff106",
                \
                OWNER:="ceph", GROUP:="ceph",
                MODE:="660", \
                RUN+="/usr/sbin/ceph-disk
                --log-stdout -v trigger /dev/$name"
              ACTION=""
                SUBSYSTEM=="block", \
               
                ENV{ID_PART_ENTRY_TYPE}=="45b0969e-9b03-4f30-b4c6-b4b80ceff106",
                \
                OWNER="ceph", GROUP="ceph",
                MODE="660"
            
            
              ~ # udevadm info
                /dev/mapper/vg--ssd1-lv--ssd1p1 | grep
                ID_PART_ENTRY_TYPE
              E:
                ID_PART_ENTRY_TYPE=45b0969e-9b03-4f30-b4c6-b4b80ceff106
              

              ~ # udevadm info
                /dev/mapper/vg--ssd1-lv--ssd1p1 | grep DEVTYPE
              E: DEVTYPE=disk
            
            
            Best Regards,
            Nicholas.
          
          
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com