Re: Can't activate osd in infernalis

David Riedl <david.riedl@xxxxxxxxxxx> · Thu, 19 Nov 2015 14:13:12 +0100

    Thanks for the fix!

    Two questions though:

    Is that the right place for the udev rule? I have CentOS 7. The
    folder exists, but all the other udev rules are in
    /usr/lib/udev/rules.d/.

    Can I just create a new file named "89-ceph-journal.rules"  in the
    /usr/lib/udev/rules.d/ folder?

    Regards

    David

    On 19.11.2015 14:02, Mykola Dvornik
      wrote:

        cat /etc/udev/rules.d/89-ceph-journal.rules

        KERNEL=="sdd?" SUBSYSTEM=="block" OWNER="ceph" GROUP="disk"
          MODE="0660"

        On 19 November 2015 at 13:54, Mykola <mykola.dvornik@xxxxxxxxx>
          wrote:

                I am afraid one would need an udev
                  rule to make it persistent.

                Sent from Outlook Mail for Windows 10
                  phone

                    From: David Riedl

                    Sent: Thursday, November 19, 2015 1:42 PM

                    To: ceph-users@xxxxxxxx

                    Subject: Re:  Can't activate osd
                    in infernalis

                    I fixed the issue and opened a
                      ticket on the ceph-deploy bug tracker
                    http://tracker.ceph.com/issues/13833

                    tl;dr:
                    change permission of the ssd
                      journal partition with
                    chown ceph:ceph /dev/sdd1

                    On 19.11.2015 11:38, David
                      Riedl wrote:
                    > Hi everyone.
                    > I updated one of my hammer
                      osd nodes to infernalis today.
                    > After many problems with
                      the upgrading process of the running OSDs, I 
                    > decided to wipe them and
                      start anew.
                    > I reinstalled all packages
                      and deleted all partitions on the OSDs and 
                    > the SSD journal drive.
                    > I zapped the disks with
                      ceph-deploy and also prepared them with 
                    > ceph-deploy.
                    > Selinux state is enabled
                      (disabling it didn't help though).
                    > 
                    > After executing
                      "ceph-deploy osd activate
                      ceph01:/dev/sda1:/dev/sdd1" 
                    > I get the following error
                      message from ceph-deploy:
                    > 
                    > 
                    > [ceph01][WARNIN]
                      INFO:ceph-disk:Running command: /usr/bin/ceph 
                    > --cluster ceph --name
                      client.bootstrap-osd --keyring 
                    >
                      /var/lib/ceph/bootstrap-osd/ceph.keyring mon
                      getmap -o 
                    >
                      /var/lib/ceph/tmp/mnt.pmHRuu/activate.monmap
                    > [ceph01][WARNIN]
                      2015-11-19 11:22:53.974765 7f1a06852700  0 -- 
                    > :/3225863658 >> 10.20.60.10:6789/0
                      pipe(0x7f19f8062590 sd=4 :0 s=1 
                    > pgs=0 cs=0 l=1
                      c=0x7f19f805c1b0).fault
                    > [ceph01][WARNIN] got
                      monmap epoch 16
                    > [ceph01][WARNIN]
                      INFO:ceph-disk:Running command: /usr/bin/ceph-osd

                    > --cluster ceph --mkfs
                      --mkkey -i 0 --monmap 
                    >
                      /var/lib/ceph/tmp/mnt.pmHRuu/activate.monmap
                      --osd-data 
                    >
                      /var/lib/ceph/tmp/mnt.pmHRuu --osd-journal 
                    >
                      /var/lib/ceph/tmp/mnt.pmHRuu/journal --osd-uuid 
                    >
                      de162e24-16b6-4796-b6b9-774fdb8ec234 --keyring 
                    >
                      /var/lib/ceph/tmp/mnt.pmHRuu/keyring --setuser
                      ceph --setgroup ceph
                    > [ceph01][WARNIN]
                      2015-11-19 11:22:57.237096 7fb458bb7900 -1 
                    >
                      filestore(/var/lib/ceph/tmp/mnt.pmHRuu) mkjournal
                      error creating 
                    > journal on
                      /var/lib/ceph/tmp/mnt.pmHRuu/journal: (13)
                      Permission denied
                    > [ceph01][WARNIN]
                      2015-11-19 11:22:57.237118 7fb458bb7900 -1
                      OSD::mkfs: 
                    > ObjectStore::mkfs failed
                      with error -13
                    > [ceph01][WARNIN]
                      2015-11-19 11:22:57.237157 7fb458bb7900 -1  **
                      ERROR: 
                    > error creating empty
                      object store in /var/lib/ceph/tmp/mnt.pmHRuu: 
                    > (13) Permission denied
                    > [ceph01][WARNIN]
                      ERROR:ceph-disk:Failed to activate
                    > [ceph01][WARNIN]
                      DEBUG:ceph-disk:Unmounting
                      /var/lib/ceph/tmp/mnt.pmHRuu
                    > [ceph01][WARNIN]
                      INFO:ceph-disk:Running command: /bin/umount -- 
                    >
                      /var/lib/ceph/tmp/mnt.pmHRuu
                    > [ceph01][WARNIN] Traceback
                      (most recent call last):
                    > [ceph01][WARNIN]   File
                      "/usr/sbin/ceph-disk", line 3576, in
                      <module>
                    > [ceph01][WARNIN]    
                      main(sys.argv[1:])
                    > [ceph01][WARNIN]   File
                      "/usr/sbin/ceph-disk", line 3530, in main
                    > [ceph01][WARNIN]   
                       args.func(args)
                    > [ceph01][WARNIN]   File
                      "/usr/sbin/ceph-disk", line 2424, in 
                    > main_activate
                    > [ceph01][WARNIN]    
                      dmcrypt_key_dir=args.dmcrypt_key_dir,
                    > [ceph01][WARNIN]   File
                      "/usr/sbin/ceph-disk", line 2197, in 
                    > mount_activate
                    > [ceph01][WARNIN]    
                      (osd_id, cluster) = activate(path, 
                    > activate_key_template,
                      init)
                    > [ceph01][WARNIN]   File
                      "/usr/sbin/ceph-disk", line 2360, in activate
                    > [ceph01][WARNIN]    
                      keyring=keyring,
                    > [ceph01][WARNIN]   File
                      "/usr/sbin/ceph-disk", line 1950, in mkfs
                    > [ceph01][WARNIN]    
                      '--setgroup', get_ceph_user(),
                    > [ceph01][WARNIN]   File
                      "/usr/sbin/ceph-disk", line 349, in 
                    > command_check_call
                    > [ceph01][WARNIN]    
                      return subprocess.check_call(arguments)
                    > [ceph01][WARNIN]   File
                      "/usr/lib64/python2.7/subprocess.py", line 
                    > 542, in check_call
                    > [ceph01][WARNIN]     raise
                      CalledProcessError(retcode, cmd)
                    > [ceph01][WARNIN]
                      subprocess.CalledProcessError: Command 
                    > '['/usr/bin/ceph-osd',
                      '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', 
                    > '0', '--monmap',
                      '/var/lib/ceph/tmp/mnt.pmHRuu/activate.monmap', 
                    > '--osd-data',
                      '/var/lib/ceph/tmp/mnt.pmHRuu', '--osd-journal', 
                    >
                      '/var/lib/ceph/tmp/mnt.pmHRuu/journal',
                      '--osd-uuid', 
                    >
                      'de162e24-16b6-4796-b6b9-774fdb8ec234',
                      '--keyring', 
                    >
                      '/var/lib/ceph/tmp/mnt.pmHRuu/keyring',
                      '--setuser', 'ceph', 
                    > '--setgroup', 'ceph']'
                      returned non-zero exit status 1
                    > [ceph01][ERROR ]
                      RuntimeError: command returned non-zero exit
                      status: 1
                    > [ceph_deploy][ERROR ]
                      RuntimeError: Failed to execute command: 
                    > ceph-disk -v activate
                      --mark-init systemd --mount /dev/sda1
                    > 
                    > The output of ls -lahn in
                      /var/lib/ceph/ is
                    > 
                    > drwxr-x---.  9 167 167
                      4,0K 19. Nov 10:32 .
                    > drwxr-xr-x. 28   0   0
                      4,0K 19. Nov 11:14 ..
                    > drwxr-x---.  2 167 167   
                      6 10. Nov 13:06 bootstrap-mds
                    > drwxr-x---.  2 167 167  
                      25 19. Nov 10:48 bootstrap-osd
                    > drwxr-x---.  2 167 167   
                      6 10. Nov 13:06 bootstrap-rgw
                    > drwxr-x---.  2 167 167   
                      6 10. Nov 13:06 mds
                    > drwxr-x---.  2 167 167   
                      6 10. Nov 13:06 mon
                    > drwxr-x---.  2 167 167   
                      6 10. Nov 13:06 osd
                    > drwxr-x---.  2 167 167  
                      65 19. Nov 11:22 tmp
                    > 
                    > 
                    > I hope someone can help
                      me, I am really lost right now.
                    > 

                    -- 
                    Mit freundlichen Grüßen

                    David Riedl

                    WINGcon GmbH Wireless New
                      Generation - Consulting & Solutions

                    Phone: +49 (0)
                        7543 9661 - 26
                    E-Mail: david.riedl@xxxxxxxxxxx
                    Web: http://www.wingcon.com

                    Sitz der Gesellschaft:
                      Langenargen
                    Registergericht: ULM, HRB
                      632019
                    USt-Id.: DE232931635, WEEE-Id.:
                      DE74015979
                    Geschäftsführer: Norbert
                      Schäfer, Fritz R. Paul

                    _______________________________________________
                    ceph-users mailing list
                    ceph-users@xxxxxxxxxxxxxx
                    http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

        -- 

           Mykola 

    -- 
Mit freundlichen Grüßen

David Riedl

WINGcon GmbH Wireless New Generation - Consulting & Solutions

Phone: +49 (0) 7543 9661 - 26
E-Mail: david.riedl@xxxxxxxxxxx
Web: http://www.wingcon.com

Sitz der Gesellschaft: Langenargen
Registergericht: ULM, HRB 632019
USt-Id.: DE232931635, WEEE-Id.: DE74015979
Geschäftsführer: Norbert Schäfer, Fritz R. Paul 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com