Re: Ceph Openstack deployment

[Iban Cabrillo <cabrillo@xxxxxxxxxxxxxx>]

Hi,

  One more step debugging this issue (hypervisor/nova-compute node is XEN 4.4.2):

  I think the problem is that libvirt is not getting the correct user or credentials tu access pool, on instance qemu log i see:

  

xen be: qdisk-51760: error: Could not open 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or directory

xen be: qdisk-51760: initialise() failed

xen be: qdisk-51760: error: Could not open 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or directory

xen be: qdisk-51760: initialise() failed

xen be: qdisk-51760: error: Could not open 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or directory

But using the user cinder on pool volumes :

rbd ls -p volumes --id cinder

test

volume-4d26bb31-91e8-4646-8010-82127b775c8e

volume-5e2ab5c2-4710-4c28-9755-b5bc4ff6a52a

volume-7da08f12-fb0f-4269-931a-d528c1507fee

Using:

qemu-img info -f rbd rbd:volumes/test

Does not work, but using directly the user cinder and the ceph.conf file works fine:

qemu-img info -f rbd rbd:volumes/test:id=cinder:conf=/etc/ceph/ceph.conf

I think nova.conf is set correctly (section libvirt):

images_rbd_pool = volumes

images_rbd_ceph_conf = /etc/ceph/ceph.conf

hw_disk_discard=unmap

rbd_user = cinder

rbd_secret_uuid = 67a6d4a1-e53a-42c7-9bc9-XXXXXXXXXXXX

And looking at libvirt:

# virsh secret-list

setlocale: No such file or directory

 UUID                                  Usage

--------------------------------------------------------------------------------

 67a6d4a1-e53a-42c7-9bc9-XXXXXXXXXXXX  ceph client.cinder secret

virsh secret-get-value 67a6d4a1-e53a-42c7-9bc9-XXXXXXXXXXXX

setlocale: No such file or directory

AQAonAdWS3iMJxxxxxxj9iErv001a0k+vyFdUg==

cat /etc/ceph/ceph.client.cinder.keyring 

[client.cinder]

key = AQAonAdWS3iMJxxxxxxj9iErv001a0k+vyFdUg==

Any idea will be welcomed.

regards, I

2015-11-04 10:51 GMT+01:00 Iban Cabrillo <cabrillo@xxxxxxxxxxxxxx>:

Dear Cephers,

   I still can attach volume to my cloud machines, ceph version is  0.94.5 (9764da52395923e0b32908d83a9f7304401fee43) and Openstack Juno

   Nova+cinder are able to create volumes on Ceph

cephvolume:~ # rados ls --pool volumes

rbd_header.1f7784a9e1c2e

rbd_id.volume-5e2ab5c2-4710-4c28-9755-b5bc4ff6a52a

rbd_directory

rbd_id.volume-7da08f12-fb0f-4269-931a-d528c1507fee

rbd_header.23d5e33b4c15c

rbd_id.volume-4d26bb31-91e8-4646-8010-82127b775c8e

rbd_header.20407190ce77f

cloud:~ # cinder list

+--------------------------------------+--------+--------------+------+-------------+----------+------------------------------------------------------------------------------------------+

|                  ID                                               | Status  | Display Name | Size | Volume Type | Bootable |                                    Attached to              |

+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------|---------------------------------------------------------+

| 4d26bb31-91e8-4646-8010-82127b775c8e | in-use |     None     |         2   |             rbd     |  false       | 59aa021e-bb4c-4154-9b18-9d09f5fd3aeb |

+--------------------------------------+--------+--------------+------+-------------+----------+------------------------------------------------------------------------------------------+

   nova:~ # nova volume-attach 59aa021e-bb4c-4154-9b18-9d09f5fd3aeb 4d26bb31-91e8-4646-8010-82127b775c8e auto

+----------+------------------------------------------------------------+

| Property |                              Value                                |

+----------+------------------------------------------------------------+

| device      | /dev/xvdd                                                     |

| id             | 4d26bb31-91e8-4646-8010-82127b775c8e |

| serverId   | 59aa021e-bb4c-4154-9b18-9d09f5fd3aeb  |

| volumeId | 4d26bb31-91e8-4646-8010-82127b775c8e |

+----------+--------------------------------------+

From nova-compute (Ubuntu 14.04 LTS \n \l) node I see the attaching/detaching:

cloud01:~ # dpkg -l | grep ceph

ii  ceph-common                         0.94.5-1trusty                      amd64        common utilities to mount and interact with a ceph storage cluster

ii  libcephfs1                               0.94.5-1trusty                      amd64        Ceph distributed file system client library

ii  python-cephfs                         0.94.5-1trusty                      amd64        Python libraries for the Ceph libcephfs library

ii  librbd1                                    0.94.5-1trusty                      amd64        RADOS block device client library

ii  python-rbd                              0.94.5-1trusty                      amd64        Python libraries for the Ceph librbd library

at cinder.conf

 rbd_user = cinder

rbd_secret_uuid = 67a6d4a1-e53a-42c7-9bc9-xxxxxxxxxxx

[rbd-cephvolume]

volume_backend_name=rbd

volume_driver = cinder.volume.drivers.rbd.RBDDriver

rbd_pool = volumes

rbd_ceph_conf = /etc/ceph/ceph.conf

rbd_flatten_volume_from_snapshot = false

rbd_max_clone_depth = 5

rbd_store_chunk_size = 4

rados_connect_timeout = -1

glance_api_version = 2

in nova.conf

rbd_user=cinder

# The libvirt UUID of the secret for the rbd_uservolumes

# (string value)

rbd_secret_uuid=67a6d4a1-e53a-42c7-9bc9-xxxxxxxxxxx

images_rbd_pool=volumes

# Path to the ceph configuration file to use (string value)

images_rbd_ceph_conf=/etc/ceph/ceph.conf

ls -la /etc/libvirt/secrets

total 16

drwx------ 2 root root 4096 Nov  4 10:28 .

drwxr-xr-x 7 root root 4096 Oct 22 13:15 ..

-rw------- 1 root root   40 Nov  4 10:28 67a6d4a1-e53a-42c7-9bc9-xxxxxxxxx.base64

-rw------- 1 root root  170 Nov  4 10:25 67a6d4a1-e53a-42c7-9bc9-xxxxxxxxxx.xml

2015-11-04 10:39:42.573 11653 INFO nova.compute.manager [req-8b2a9793-4b39-4cb0-b291-e492c350387e b7aadbb4a85745feb498b74e437129cc ce2dd2951bd24c1ea3b43c3b3716f604 - - -] [instance: 59aa021e-bb4c-4154-9b18-9d09f5fd3aeb] Detach volume 4d26bb31-91e8-4646-8010-82127b775c8e from mountpoint /dev/xvdd

2015-11-04 10:40:43.266 11653 INFO nova.compute.manager [req-35218de0-3f26-496b-aad9-5c839143da17 b7aadbb4a85745feb498b74e437129cc ce2dd2951bd24c1ea3b43c3b3716f604 - - -] [instance: 59aa021e-bb4c-4154-9b18-9d09f5fd3aeb] Attaching volume 4d26bb31-91e8-4646-8010-82127b775c8e to /dev/xvdd

but one on cloud machine (SL6) the volume y never showed (xvdd).

[root@cloud5 ~]# cat /proc/partitions 

major minor  #blocks  name

 202        0   20971520 xvda

 202       16  209715200 xvdb

 202       32   10485760 xvdc

Thanks in advance, I

2015-11-03 11:18 GMT+01:00 Iban Cabrillo <cabrillo@xxxxxxxxxxxxxx>:

Hi all,

    During last week I been trying to deploy the pre-existing ceph cluster with out openstack intance.

    The ceph-cinder integration was easy (or at least I think so!!)

    There is only one volume to attach block storage to out cloud machines.

    The client.cinder has permission on this volume (following the guides)

    ...............

    client.cinder

key: AQAonXXXXXXXRAAPIAj9iErv001a0k+vyFdUg==

caps: [mon] allow r

caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=volumes

   ceph.conf file seems to be OK:

   

[global]

fsid = 6f5a65a7-316c-4825-afcb-428608941dd1

mon_initial_members = cephadm, cephmon02, cephmon03

mon_host = 10.10.3.1,10.10.3.2,10.10.3.3

auth_cluster_required = cephx

auth_service_required = cephx

auth_client_required = cephx

filestore_xattr_use_omap = true

osd_pool_default_size = 2

public_network = 10.10.0.0/16

cluster_network = 192.168.254.0/27

[osd]

osd_journal_size = 20000

[client.cinder]

keyring = /etc/ceph/ceph.client.cinder.keyring

[client]

rbd cache = true

rbd cache writethrough until flush = true

admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok

The trouble seems that blocks are created using the client.admin instead of client.cinder

From cinder machine:

cinder:~ # rados ls --pool volumes

rbd_id.volume-5e2ab5c2-4710-4c28-9755-b5bc4ff6a52a

rbd_directory

rbd_id.volume-7da08f12-fb0f-4269-931a-d528c1507fee

rbd_header.23d5e33b4c15c

rbd_header.20407190ce77f

But if I try to look for using cinder client:

  cinder:~ #rados ls --pool volumes --secret client.cinder

  "empty answer"

cinder:~ # ls -la /etc/ceph

total 24

drwxr-xr-x   2 root   root   4096 nov  3 10:17 .

drwxr-xr-x 108 root   root   4096 oct 29 09:52 ..

-rw-------   1 root   root     63 nov  3 10:17 ceph.client.admin.keyring

-rw-r--r--   1 cinder cinder   67 oct 28 13:44 ceph.client.cinder.keyring

-rw-r--r--   1 root   root    454 oct  1 13:56 ceph.conf

-rw-r--r--   1 root   root     73 sep 27 09:36 ceph.mon.keyring

from a client (I have supposed that this machine only need the cinder key...)

cloud28:~ # ls -la /etc/ceph/

total 28

drwx------   2 root root  4096 nov  3 11:01 .

drwxr-xr-x 116 root root 12288 oct 30 14:37 ..

-rw-r--r--   1 nova nova    67 oct 28 11:43 ceph.client.cinder.keyring

-rw-r--r--   1 root root   588 nov  3 10:59 ceph.conf

-rw-r--r--   1 root root    92 oct 26 16:59 rbdmap

cloud28:~ # rbd -p volumes ls

2015-11-03 11:01:58.782795 7fc6c714b840 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication

2015-11-03 11:01:58.782800 7fc6c714b840  0 librados: client.admin initialization error (2) No such file or directory

rbd: couldn't connect to the cluster!

Any help will be welcome.

--

############################################################################
Iban Cabrillo Bartolome
Instituto de Fisica de Cantabria (IFCA)
Santander, Spain
Tel: +34942200969

PGP PUBLIC KEY: http://pgp.mit.edu/pks/lookup?op=get&search=0xD9DF0B3D6C8C08AC
############################################################################
Bertrand Russell:
"El problema con el mundo es que los estúpidos están seguros de todo y los inteligentes están llenos de dudas"

--

############################################################################
Iban Cabrillo Bartolome
Instituto de Fisica de Cantabria (IFCA)
Santander, Spain
Tel: +34942200969

PGP PUBLIC KEY: http://pgp.mit.edu/pks/lookup?op=get&search=0xD9DF0B3D6C8C08AC
############################################################################
Bertrand Russell:
"El problema con el mundo es que los estúpidos están seguros de todo y los inteligentes están llenos de dudas"

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com