RBD map issue

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi all,

This is my first post to this user group, I’m not a ceph expert, sorry if I say/ask anything trivial.

On a Kubernetes cluster I have an issue in creating volumes from a (csi) ceph EC pool.

I can reproduce the problem from rbd cli like this from one of the k8s worker nodes:

“””
root@fulen-w006:~# ceph -v
ceph version 15.2.14 (cd3bb7e87a2f62c1b862ff3fd8b1eec13391a5be) octopus (stable)

root@fulen-w006:~# rbd -m 148.187.20.141:6789 --id fulen --keyfile key create test-loreg-3 --data-pool fulen-hdd-data --pool fulen-nvme-meta --size 1G

root@fulen-w006:~# rbd -m 148.187.20.141:6789 --id fulen --keyfile key map fulen-nvme-meta/test-loreg-3
...
rbd: sysfs write failed
...
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (1) Operation not permitted
“””

The same sequence of operations works on a different node (not part of the k8s cluster, completely different setup):

“””
root@storagesmw: # ceph -v
ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)

root@storagesmw: # rbd -m 148.187.20.141:6789 --id fulen --keyfile client.fulen.key create test-loreg-4 --data-pool fulen-hdd-data --pool fulen-nvme-meta --size 1G

root@storagesmw: # rbd -m 148.187.20.141:6789 --id fulen --keyfile client.fulen.key info fulen-nvme-meta/test-loreg-4
rbd image 'test-loreg-4':
        size 1 GiB in 256 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: cafc436ff3573
        data_pool: fulen-hdd-data
        block_name_prefix: rbd_data.36.cafc436ff3573
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool
        op_features:
        flags:
        create_timestamp: Thu Feb 10 18:23:26 2022
        access_timestamp: Thu Feb 10 18:23:26 2022
        modify_timestamp: Thu Feb 10 18:23:26 2022

root@storagesmw: # rbd -m 148.187.20.141:6789 --id fulen --keyfile client.fulen.key map fulen-nvme-meta/test-loreg-4
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable fulen-nvme-meta/test-loreg-4 object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address

root@storagesmw: # rbd -m 148.187.20.141:6789 --id fulen --keyfile client.fulen.key feature disable fulen-nvme-meta/test-loreg-4 object-map fast-diff deep-flatten

root@storagesmw: # rbd -m 148.187.20.141:6789 --id fulen --keyfile client.fulen.key map fulen-nvme-meta/test-loreg-4
/dev/rbd0
“””

The 2 nodes OS release and kernel are below.

Does anyone have any advice on how to debug this?

Thanks in advance,

Giuseppe

============================================== Fulen-w006:

root@fulen-w006:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/";
SUPPORT_URL="https://help.ubuntu.com/";
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/";
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy";
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
root@fulen-w006:~# uname -a
Linux fulen-w006.cscs.ch 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Storagesmw:

root@storagesmw:~/loreg/ceph_conf # cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.8 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.8"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.8:GA:server"
HOME_URL="https://www.redhat.com/";
BUG_REPORT_URL="https://bugzilla.redhat.com/";

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.8
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.8"
root@storagesmw:~/loreg/ceph_conf # uname -a
Linux storagesmw.cscs.ch 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 11 19:12:04 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux





_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux