Hi all,
This is my first post to this user group, I’m not a ceph expert, sorry if I say/ask anything trivial.
On a Kubernetes cluster I have an issue in creating volumes from a (csi) ceph EC pool.
I can reproduce the problem from rbd cli like this from one of the k8s worker nodes:
“””
root@fulen-w006:~# ceph -v
ceph version 15.2.14 (cd3bb7e87a2f62c1b862ff3fd8b1eec13391a5be) octopus (stable)
root@fulen-w006:~# rbd -m 148.187.20.141:6789 --id fulen --keyfile key create test-loreg-3 --data-pool fulen-hdd-data --pool fulen-nvme-meta --size 1G
root@fulen-w006:~# rbd -m 148.187.20.141:6789 --id fulen --keyfile key map fulen-nvme-meta/test-loreg-3
...
rbd: sysfs write failed
...
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (1) Operation not permitted
“””
The same sequence of operations works on a different node (not part of the k8s cluster, completely different setup):
“””
root@storagesmw: # ceph -v
ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)
root@storagesmw: # rbd -m 148.187.20.141:6789 --id fulen --keyfile client.fulen.key create test-loreg-4 --data-pool fulen-hdd-data --pool fulen-nvme-meta --size 1G
root@storagesmw: # rbd -m 148.187.20.141:6789 --id fulen --keyfile client.fulen.key info fulen-nvme-meta/test-loreg-4
rbd image 'test-loreg-4':
size 1 GiB in 256 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: cafc436ff3573
data_pool: fulen-hdd-data
block_name_prefix: rbd_data.36.cafc436ff3573
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool
op_features:
flags:
create_timestamp: Thu Feb 10 18:23:26 2022
access_timestamp: Thu Feb 10 18:23:26 2022
modify_timestamp: Thu Feb 10 18:23:26 2022
root@storagesmw: # rbd -m 148.187.20.141:6789 --id fulen --keyfile client.fulen.key map fulen-nvme-meta/test-loreg-4
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable fulen-nvme-meta/test-loreg-4 object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
root@storagesmw: # rbd -m 148.187.20.141:6789 --id fulen --keyfile client.fulen.key feature disable fulen-nvme-meta/test-loreg-4 object-map fast-diff deep-flatten
root@storagesmw: # rbd -m 148.187.20.141:6789 --id fulen --keyfile client.fulen.key map fulen-nvme-meta/test-loreg-4
/dev/rbd0
“””
The 2 nodes OS release and kernel are below.
Does anyone have any advice on how to debug this?
Thanks in advance,
Giuseppe
============================================== Fulen-w006:
root@fulen-w006:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/";
SUPPORT_URL="https://help.ubuntu.com/";
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/";
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy";
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
root@fulen-w006:~# uname -a
Linux fulen-w006.cscs.ch 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Storagesmw:
root@storagesmw:~/loreg/ceph_conf # cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.8 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.8"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.8:GA:server"
HOME_URL="https://www.redhat.com/";
BUG_REPORT_URL="https://bugzilla.redhat.com/";
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.8
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.8"
root@storagesmw:~/loreg/ceph_conf # uname -a
Linux storagesmw.cscs.ch 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 11 19:12:04 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
