Hi Paul
Thank you for suggestion.
Firstly, a correction “I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1” is incorrect. I meant that version 12.2.8-0ubuntu0.18.04.1 on node N01 works, . 12.2.12-0ubuntu0.18.04.2 is having problem.
After upgrading to version 12.2.12-0ubuntu0.18.04.2, it doesn’t work anymore. No changes on client keys file or on the Ceph cluster. Therefore I deduce that it is a problem with client (ceph-common) software.
_______________________________________________
~ Vang
From: Paul Emmerich <paul.emmerich@xxxxxxxx>
Date: Friday, 6 September 2019 at 13.13
To: Vang Le-Quy <vle@xxxxxxxxxx>
Cc: "ceph-users@xxxxxxx" <ceph-users@xxxxxxx>
Subject: Re: Ceph client failed to mount RBD device after reboot
failure to break locks is usually due to incorrect permissions on the key, check if the client key has "profile rbd" mon permissions
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Fri, Sep 6, 2019 at 1:06 PM Vang Le-Quy <vle@xxxxxxxxxx> wrote:
I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1. We have too ceph client n01 and n03 with following software versions.
# n01 ceph-common:
Installed: 12.2.8-0ubuntu0.18.04.1
Candidate: 12.2.12-0ubuntu0.18.04.2
# n03 ceph-common:
Installed: 12.2.11-0ubuntu0.18.04.2
Candidate: 12.2.12-0ubuntu0.18.04.2
# n01 uname -a
Linux n01 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
# n03 uname -a
Linux n03 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Problem:
N01 can mount its RBD after reboot whereas N03 can’t. To verify that it is something on client side, I upgrade n01 to have the same package version to n03 including ceph-common. And n01 does indeed fail to mount its CEPH RBD partition. Syslog shows messages like the following:
Sep 6 12:52:33 n01 kernel: [ 875.160275] rbd: rbd0: no lock owners detected
Sep 6 12:52:33 n01 kernel: [ 875.161069] rbd: rbd0: client2508785 seems dead, breaking lock
Sep 6 12:52:33 n01 kernel: [ 875.161723] rbd: rbd0: blacklist of client2508785 failed: -13
Sep 6 12:52:33 n01 kernel: [ 875.161724] rbd: rbd0: failed to acquire lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.161956] rbd: rbd0: no lock owners detected
Sep 6 12:52:33 n01 kernel: [ 875.162755] rbd: rbd0: client2508785 seems dead, breaking lock
Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: blacklist of client2508785 failed: -13
Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: failed to acquire lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.163645] rbd: rbd0: no lock owners detected
Sep 6 12:52:33 n01 kernel: [ 875.164431] rbd: rbd0: client2508785 seems dead, breaking lock
Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: blacklist of client2508785 failed: -13
Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: failed to acquire lock: -13
Sep 6 12:52:33 n01 kernel: [ 875.165355] rbd: rbd0: no lock owners detected
Vang Quy Le
Special Consultant in Data Science and Infrastructure
T: (+45) 9940 7710 | Email: vle@xxxxxxxxxx
Kontor 0-1-91 | Selma Lagerløfs Vej 300 | DK-9220 Aalborg Ø |
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
