Re: Ceph client failed to mount RBD device after reboot

JC Lopez <jelopez@xxxxxxxxxx> · Fri, 6 Sep 2019 13:42:12 -0700

Hi
ceph auth caps client.n01 mon ‘profile rbd’ osd ‘profile rbd’

should do the trick.

JC

On Sep 6, 2019, at 05:23, Vang Le-Quy <vle@xxxxxxxxxx> wrote:

Hi Jason
My client key content looks like the following:

#####[
[client.n01]
                key = AQA301KcfFVtKxAAyJxY9uTvNh3gn7eH3iHhKw==
#####

So Paul and you may be right, the key lacks suitable caps. Can I just manually add caps to the key file or it has to be regenerated from the CEPH cluster?

From: Jason Dillaman <jdillama@xxxxxxxxxx>
Reply-To: "dillaman@xxxxxxxxxx" <dillaman@xxxxxxxxxx>
Date: Friday, 6 September 2019 at 14.06
To: Vang Le-Quy <vle@xxxxxxxxxx>
Cc: Paul Emmerich <paul.emmerich@xxxxxxxx>, "ceph-users@xxxxxxx" <ceph-users@xxxxxxx>
Subject: Re:  Re: Ceph client failed to mount RBD device after reboot

On Fri, Sep 6, 2019 at 7:33 AM Vang Le-Quy <vle@xxxxxxxxxx> wrote:
Hi Paul
Thank you for suggestion.
Firstly, a correction “I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1” is incorrect. I meant that version 12.2.8-0ubuntu0.18.04.1 on node N01 works, . 12.2.12-0ubuntu0.18.04.2 is having problem.

After upgrading to version 12.2.12-0ubuntu0.18.04.2, it doesn’t work anymore. No changes on client keys file or on the Ceph cluster. Therefore I deduce that it is a problem with client (ceph-common) software.

Looks like a CephX caps issue for your client. Please refer to the documentation and verify you are using the correct caps [1].

~ Vang

From: Paul Emmerich <paul.emmerich@xxxxxxxx>
Date: Friday, 6 September 2019 at 13.13
To: Vang Le-Quy <vle@xxxxxxxxxx>
Cc: "ceph-users@xxxxxxx" <ceph-users@xxxxxxx>
Subject: Re:  Ceph client failed to mount RBD device after reboot

failure to break locks is usually due to incorrect permissions on the key, check if the client key has "profile rbd" mon permissions

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

On Fri, Sep 6, 2019 at 1:06 PM Vang Le-Quy <vle@xxxxxxxxxx> wrote:
I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1. We have too ceph client n01 and n03 with following software versions.

# n01 ceph-common:
  Installed: 12.2.8-0ubuntu0.18.04.1
  Candidate: 12.2.12-0ubuntu0.18.04.2

# n03 ceph-common:
  Installed: 12.2.11-0ubuntu0.18.04.2
  Candidate: 12.2.12-0ubuntu0.18.04.2

# n01 uname -a
Linux n01 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

# n03 uname -a
Linux n03 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Problem:
N01 can mount its RBD after reboot whereas N03 can’t. To verify that it is something on client side, I upgrade n01 to have the same package version to n03 including ceph-common. And n01 does indeed fail to mount its CEPH RBD partition. Syslog shows messages like the following:
Sep  6 12:52:33 n01 kernel: [  875.160275] rbd: rbd0: no lock owners detected
Sep  6 12:52:33 n01 kernel: [  875.161069] rbd: rbd0: client2508785 seems dead, breaking lock
Sep  6 12:52:33 n01 kernel: [  875.161723] rbd: rbd0: blacklist of client2508785 failed: -13
Sep  6 12:52:33 n01 kernel: [  875.161724] rbd: rbd0: failed to acquire lock: -13
Sep  6 12:52:33 n01 kernel: [  875.161956] rbd: rbd0: no lock owners detected
Sep  6 12:52:33 n01 kernel: [  875.162755] rbd: rbd0: client2508785 seems dead, breaking lock
Sep  6 12:52:33 n01 kernel: [  875.163415] rbd: rbd0: blacklist of client2508785 failed: -13
Sep  6 12:52:33 n01 kernel: [  875.163415] rbd: rbd0: failed to acquire lock: -13
Sep  6 12:52:33 n01 kernel: [  875.163645] rbd: rbd0: no lock owners detected
Sep  6 12:52:33 n01 kernel: [  875.164431] rbd: rbd0: client2508785 seems dead, breaking lock
Sep  6 12:52:33 n01 kernel: [  875.165082] rbd: rbd0: blacklist of client2508785 failed: -13
Sep  6 12:52:33 n01 kernel: [  875.165082] rbd: rbd0: failed to acquire lock: -13
Sep  6 12:52:33 n01 kernel: [  875.165355] rbd: rbd0: no lock owners detected

<image001.png>
Vang Quy Le
Special Consultant in Data Science and Infrastructure 

T: (+45) 9940 7710 | Email: vle@xxxxxxxxxx
Kontor 0-1-91 | Selma Lagerløfs Vej 300 | DK-9220 Aalborg Ø |

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

[1] https://docs.ceph.com/docs/luminous/rbd/rados-rbd-cmds/#create-a-block-device-user

-- 
Jason
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx