There must be something in /var/log/cinder/volume.log or /var/log/nova/nova-compute.log that points to the problem. Can you post it here? 2015-11-06 20:14 GMT+08:00 Iban Cabrillo <cabrillo@xxxxxxxxxxxxxx>: > Hi Vasilly, > Thanks, but I still see the same error: > > cinder.conf (of course I just restart the cinder-volume service) > > # default volume type to use (string value) > > [rbd-cephvolume] > rbd_user = cinder > rbd_secret_uuid = 67a6d4a1-e53a-42c7-9bc9-xxxxxxxxxxx > volume_backend_name=rbd > volume_driver = cinder.volume.drivers.rbd.RBDDriver > rbd_pool = volumes > rbd_ceph_conf = /etc/ceph/ceph.conf > rbd_flatten_volume_from_snapshot = false > rbd_max_clone_depth = 5 > rbd_store_chunk_size = 4 > rados_connect_timeout = -1 > glance_api_version = 2 > > > xen be: qdisk-51760: error: Could not open > 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or > directory > xen be: qdisk-51760: initialise() failed > xen be: qdisk-51760: error: Could not open > 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or > directory > xen be: qdisk-51760: initialise() failed > xen be: qdisk-51760: error: Could not open > 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or > directory > xen be: qdisk-51760: initialise() failed > xen be: qdisk-51760: error: Could not open > 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or > directory > xen be: qdisk-51760: initialise() failed > xen be: qdisk-51760: error: Could not open > 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or > directory > xen be: qdisk-51760: initialise() failed > xen be: qdisk-51760: error: Could not open > 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or > directory > xen be: qdisk-51760: initialise() failed > > Regards, I > > 2015-11-06 13:00 GMT+01:00 Vasiliy Angapov <angapov@xxxxxxxxx>: >> >> At cinder.conf you should place this options: >> >> rbd_user = cinder >> rbd_secret_uuid = 67a6d4a1-e53a-42c7-9bc9-xxxxxxxxxxx >> >> to [rbd-cephvolume] section instead of DEFAULT. >> >> 2015-11-06 19:45 GMT+08:00 Iban Cabrillo <cabrillo@xxxxxxxxxxxxxx>: >> > Hi, >> > One more step debugging this issue (hypervisor/nova-compute node is >> > XEN >> > 4.4.2): >> > >> > I think the problem is that libvirt is not getting the correct user or >> > credentials tu access pool, on instance qemu log i see: >> > >> > xen be: qdisk-51760: error: Could not open >> > 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or >> > directory >> > xen be: qdisk-51760: initialise() failed >> > xen be: qdisk-51760: error: Could not open >> > 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or >> > directory >> > xen be: qdisk-51760: initialise() failed >> > xen be: qdisk-51760: error: Could not open >> > 'volumes/volume-4d26bb31-91e8-4646-8010-82127b775c8e': No such file or >> > directory >> > >> > But using the user cinder on pool volumes : >> > >> > rbd ls -p volumes --id cinder >> > test >> > volume-4d26bb31-91e8-4646-8010-82127b775c8e >> > volume-5e2ab5c2-4710-4c28-9755-b5bc4ff6a52a >> > volume-7da08f12-fb0f-4269-931a-d528c1507fee >> > >> > Using: >> > qemu-img info -f rbd rbd:volumes/test >> > Does not work, but using directly the user cinder and the ceph.conf file >> > works fine: >> > >> > qemu-img info -f rbd rbd:volumes/test:id=cinder:conf=/etc/ceph/ceph.conf >> > >> > I think nova.conf is set correctly (section libvirt): >> > images_rbd_pool = volumes >> > images_rbd_ceph_conf = /etc/ceph/ceph.conf >> > hw_disk_discard=unmap >> > rbd_user = cinder >> > rbd_secret_uuid = 67a6d4a1-e53a-42c7-9bc9-XXXXXXXXXXXX >> > >> > And looking at libvirt: >> > >> > # virsh secret-list >> > setlocale: No such file or directory >> > UUID Usage >> > >> > -------------------------------------------------------------------------------- >> > 67a6d4a1-e53a-42c7-9bc9-XXXXXXXXXXXX ceph client.cinder secret >> > >> > >> > virsh secret-get-value 67a6d4a1-e53a-42c7-9bc9-XXXXXXXXXXXX >> > setlocale: No such file or directory >> > AQAonAdWS3iMJxxxxxxj9iErv001a0k+vyFdUg== >> > cat /etc/ceph/ceph.client.cinder.keyring >> > [client.cinder] >> > key = AQAonAdWS3iMJxxxxxxj9iErv001a0k+vyFdUg== >> > >> > >> > Any idea will be welcomed. >> > regards, I >> > >> > 2015-11-04 10:51 GMT+01:00 Iban Cabrillo <cabrillo@xxxxxxxxxxxxxx>: >> >> >> >> Dear Cephers, >> >> >> >> I still can attach volume to my cloud machines, ceph version is >> >> 0.94.5 >> >> (9764da52395923e0b32908d83a9f7304401fee43) and Openstack Juno >> >> >> >> Nova+cinder are able to create volumes on Ceph >> >> cephvolume:~ # rados ls --pool volumes >> >> rbd_header.1f7784a9e1c2e >> >> rbd_id.volume-5e2ab5c2-4710-4c28-9755-b5bc4ff6a52a >> >> rbd_directory >> >> rbd_id.volume-7da08f12-fb0f-4269-931a-d528c1507fee >> >> rbd_header.23d5e33b4c15c >> >> rbd_id.volume-4d26bb31-91e8-4646-8010-82127b775c8e >> >> rbd_header.20407190ce77f >> >> >> >> cloud:~ # cinder list >> >> >> >> >> >> +--------------------------------------+--------+--------------+------+-------------+----------+------------------------------------------------------------------------------------------+ >> >> | ID | >> >> Status | Display Name | Size | Volume Type | Bootable | >> >> Attached to | >> >> >> >> >> >> +--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------|---------------------------------------------------------+ >> >> | 4d26bb31-91e8-4646-8010-82127b775c8e | in-use | None | >> >> 2 >> >> | rbd | false | >> >> 59aa021e-bb4c-4154-9b18-9d09f5fd3aeb >> >> | >> >> >> >> >> >> +--------------------------------------+--------+--------------+------+-------------+----------+------------------------------------------------------------------------------------------+ >> >> >> >> >> >> nova:~ # nova volume-attach 59aa021e-bb4c-4154-9b18-9d09f5fd3aeb >> >> 4d26bb31-91e8-4646-8010-82127b775c8e auto >> >> >> >> +----------+------------------------------------------------------------+ >> >> | Property | Value >> >> | >> >> >> >> +----------+------------------------------------------------------------+ >> >> | device | /dev/xvdd >> >> | >> >> | id | 4d26bb31-91e8-4646-8010-82127b775c8e | >> >> | serverId | 59aa021e-bb4c-4154-9b18-9d09f5fd3aeb | >> >> | volumeId | 4d26bb31-91e8-4646-8010-82127b775c8e | >> >> +----------+--------------------------------------+ >> >> >> >> From nova-compute (Ubuntu 14.04 LTS \n \l) node I see the >> >> attaching/detaching: >> >> cloud01:~ # dpkg -l | grep ceph >> >> ii ceph-common 0.94.5-1trusty >> >> amd64 common utilities to mount and interact with a ceph storage >> >> cluster >> >> ii libcephfs1 0.94.5-1trusty >> >> amd64 Ceph distributed file system client library >> >> ii python-cephfs 0.94.5-1trusty >> >> amd64 Python libraries for the Ceph libcephfs library >> >> ii librbd1 0.94.5-1trusty >> >> amd64 RADOS block device client library >> >> ii python-rbd 0.94.5-1trusty >> >> amd64 Python libraries for the Ceph librbd library >> >> >> >> at cinder.conf >> >> >> >> rbd_user = cinder >> >> rbd_secret_uuid = 67a6d4a1-e53a-42c7-9bc9-xxxxxxxxxxx >> >> >> >> [rbd-cephvolume] >> >> volume_backend_name=rbd >> >> volume_driver = cinder.volume.drivers.rbd.RBDDriver >> >> rbd_pool = volumes >> >> rbd_ceph_conf = /etc/ceph/ceph.conf >> >> rbd_flatten_volume_from_snapshot = false >> >> rbd_max_clone_depth = 5 >> >> rbd_store_chunk_size = 4 >> >> rados_connect_timeout = -1 >> >> glance_api_version = 2 >> >> >> >> in nova.conf >> >> rbd_user=cinder >> >> >> >> # The libvirt UUID of the secret for the rbd_uservolumes >> >> # (string value) >> >> rbd_secret_uuid=67a6d4a1-e53a-42c7-9bc9-xxxxxxxxxxx >> >> >> >> images_rbd_pool=volumes >> >> >> >> # Path to the ceph configuration file to use (string value) >> >> images_rbd_ceph_conf=/etc/ceph/ceph.conf >> >> >> >> ls -la /etc/libvirt/secrets >> >> total 16 >> >> drwx------ 2 root root 4096 Nov 4 10:28 . >> >> drwxr-xr-x 7 root root 4096 Oct 22 13:15 .. >> >> -rw------- 1 root root 40 Nov 4 10:28 >> >> 67a6d4a1-e53a-42c7-9bc9-xxxxxxxxx.base64 >> >> -rw------- 1 root root 170 Nov 4 10:25 >> >> 67a6d4a1-e53a-42c7-9bc9-xxxxxxxxxx.xml >> >> >> >> >> >> >> >> 2015-11-04 10:39:42.573 11653 INFO nova.compute.manager >> >> [req-8b2a9793-4b39-4cb0-b291-e492c350387e >> >> b7aadbb4a85745feb498b74e437129cc >> >> ce2dd2951bd24c1ea3b43c3b3716f604 - - -] [instance: >> >> 59aa021e-bb4c-4154-9b18-9d09f5fd3aeb] Detach volume >> >> 4d26bb31-91e8-4646-8010-82127b775c8e from mountpoint /dev/xvdd >> >> 2015-11-04 10:40:43.266 11653 INFO nova.compute.manager >> >> [req-35218de0-3f26-496b-aad9-5c839143da17 >> >> b7aadbb4a85745feb498b74e437129cc >> >> ce2dd2951bd24c1ea3b43c3b3716f604 - - -] [instance: >> >> 59aa021e-bb4c-4154-9b18-9d09f5fd3aeb] Attaching volume >> >> 4d26bb31-91e8-4646-8010-82127b775c8e to /dev/xvdd >> >> >> >> but one on cloud machine (SL6) the volume y never showed (xvdd). >> >> [root@cloud5 ~]# cat /proc/partitions >> >> major minor #blocks name >> >> >> >> 202 0 20971520 xvda >> >> 202 16 209715200 xvdb >> >> 202 32 10485760 xvdc >> >> >> >> Thanks in advance, I >> >> >> >> 2015-11-03 11:18 GMT+01:00 Iban Cabrillo <cabrillo@xxxxxxxxxxxxxx>: >> >>> >> >>> Hi all, >> >>> During last week I been trying to deploy the pre-existing ceph >> >>> cluster with out openstack intance. >> >>> The ceph-cinder integration was easy (or at least I think so!!) >> >>> There is only one volume to attach block storage to out cloud >> >>> machines. >> >>> >> >>> The client.cinder has permission on this volume (following the >> >>> guides) >> >>> ............... >> >>> client.cinder >> >>> key: AQAonXXXXXXXRAAPIAj9iErv001a0k+vyFdUg== >> >>> caps: [mon] allow r >> >>> caps: [osd] allow class-read object_prefix rbd_children, allow rwx >> >>> pool=volumes >> >>> >> >>> ceph.conf file seems to be OK: >> >>> >> >>> [global] >> >>> fsid = 6f5a65a7-316c-4825-afcb-428608941dd1 >> >>> mon_initial_members = cephadm, cephmon02, cephmon03 >> >>> mon_host = 10.10.3.1,10.10.3.2,10.10.3.3 >> >>> auth_cluster_required = cephx >> >>> auth_service_required = cephx >> >>> auth_client_required = cephx >> >>> filestore_xattr_use_omap = true >> >>> osd_pool_default_size = 2 >> >>> public_network = 10.10.0.0/16 >> >>> cluster_network = 192.168.254.0/27 >> >>> >> >>> [osd] >> >>> osd_journal_size = 20000 >> >>> >> >>> [client.cinder] >> >>> keyring = /etc/ceph/ceph.client.cinder.keyring >> >>> >> >>> [client] >> >>> rbd cache = true >> >>> rbd cache writethrough until flush = true >> >>> admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok >> >>> >> >>> >> >>> The trouble seems that blocks are created using the client.admin >> >>> instead >> >>> of client.cinder >> >>> >> >>> From cinder machine: >> >>> >> >>> cinder:~ # rados ls --pool volumes >> >>> rbd_id.volume-5e2ab5c2-4710-4c28-9755-b5bc4ff6a52a >> >>> rbd_directory >> >>> rbd_id.volume-7da08f12-fb0f-4269-931a-d528c1507fee >> >>> rbd_header.23d5e33b4c15c >> >>> rbd_header.20407190ce77f >> >>> >> >>> But if I try to look for using cinder client: >> >>> >> >>> >> >>> cinder:~ #rados ls --pool volumes --secret client.cinder >> >>> "empty answer" >> >>> >> >>> cinder:~ # ls -la /etc/ceph >> >>> total 24 >> >>> drwxr-xr-x 2 root root 4096 nov 3 10:17 . >> >>> drwxr-xr-x 108 root root 4096 oct 29 09:52 .. >> >>> -rw------- 1 root root 63 nov 3 10:17 >> >>> ceph.client.admin.keyring >> >>> -rw-r--r-- 1 cinder cinder 67 oct 28 13:44 >> >>> ceph.client.cinder.keyring >> >>> -rw-r--r-- 1 root root 454 oct 1 13:56 ceph.conf >> >>> -rw-r--r-- 1 root root 73 sep 27 09:36 ceph.mon.keyring >> >>> >> >>> >> >>> from a client (I have supposed that this machine only need the cinder >> >>> key...) >> >>> >> >>> cloud28:~ # ls -la /etc/ceph/ >> >>> total 28 >> >>> drwx------ 2 root root 4096 nov 3 11:01 . >> >>> drwxr-xr-x 116 root root 12288 oct 30 14:37 .. >> >>> -rw-r--r-- 1 nova nova 67 oct 28 11:43 ceph.client.cinder.keyring >> >>> -rw-r--r-- 1 root root 588 nov 3 10:59 ceph.conf >> >>> -rw-r--r-- 1 root root 92 oct 26 16:59 rbdmap >> >>> >> >>> cloud28:~ # rbd -p volumes ls >> >>> 2015-11-03 11:01:58.782795 7fc6c714b840 -1 monclient(hunting): ERROR: >> >>> missing keyring, cannot use cephx for authentication >> >>> 2015-11-03 11:01:58.782800 7fc6c714b840 0 librados: client.admin >> >>> initialization error (2) No such file or directory >> >>> rbd: couldn't connect to the cluster! >> >>> >> >>> Any help will be welcome. >> >>> >> >> >> >> >> >> >> >> -- >> >> >> >> >> >> ############################################################################ >> >> Iban Cabrillo Bartolome >> >> Instituto de Fisica de Cantabria (IFCA) >> >> Santander, Spain >> >> Tel: +34942200969 >> >> PGP PUBLIC KEY: >> >> http://pgp.mit.edu/pks/lookup?op=get&search=0xD9DF0B3D6C8C08AC >> >> >> >> >> >> ############################################################################ >> >> Bertrand Russell: >> >> "El problema con el mundo es que los estúpidos están seguros de todo y >> >> los >> >> inteligentes están llenos de dudas" >> > >> > >> > >> > >> > -- >> > >> > ############################################################################ >> > Iban Cabrillo Bartolome >> > Instituto de Fisica de Cantabria (IFCA) >> > Santander, Spain >> > Tel: +34942200969 >> > PGP PUBLIC KEY: >> > http://pgp.mit.edu/pks/lookup?op=get&search=0xD9DF0B3D6C8C08AC >> > >> > ############################################################################ >> > Bertrand Russell: >> > "El problema con el mundo es que los estúpidos están seguros de todo y >> > los >> > inteligentes están llenos de dudas" >> > >> > _______________________________________________ >> > ceph-users mailing list >> > ceph-users@xxxxxxxxxxxxxx >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > > > > -- > ############################################################################ > Iban Cabrillo Bartolome > Instituto de Fisica de Cantabria (IFCA) > Santander, Spain > Tel: +34942200969 > PGP PUBLIC KEY: > http://pgp.mit.edu/pks/lookup?op=get&search=0xD9DF0B3D6C8C08AC > ############################################################################ > Bertrand Russell: > "El problema con el mundo es que los estúpidos están seguros de todo y los > inteligentes están llenos de dudas" _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |