Hi All,
tl;dr - does glance/rbd and cinder/rbd play together nicely in grizzly?
I'm currently testing a ceph/rados back end with an openstack installation. I have the following things working OK:
1. cinder configured to create volumes in RBD
2. nova configured to boot from RBD backed cinder volumes (libvirt UUID secret set etc)
3. glance configured to use RBD as a back end store for images
With this setup, when I create a bootable volume in cinder, passing an id of an image in glance, the image gets downloaded, converted to raw, and then created as an RBD object and made available to cinder. The correct metadata field for the cinder volume is populated (volume_image_metadata) and so the cinder client marks the volume as bootable. This is all fine.
If I want to take advantage of the fact that both glance images and cinder volumes are stored in RBD, I can add the following flag to the glance-api.conf:
show_image_direct_url = True
This enables cinder to see that the glance image is stored in RBD, and the cinder rbd driver then, instead of downloading the image and creating an RBD image from it, just issues an 'rbd clone' command (seen in the cinder-volume.log):
rbd clone --pool images --image dcb2f16d-a09d-4064-9198-1965274e214d --snap snap --dest-pool volumes --dest volume-20987f9d-b4fb-463d-8b8f-fa667bd47c6d
This is all very nice, and the cinder volume is available immediately as you'd expect. The problem is that the metadata field is not populated so it's not seen as bootable. Even manually populating this field leaves the volume unbootable. The volume can not even be attached to another instance for inspection.
libvirt doesn't seem to be able to access the rbd device. From nova-compute.log:
qemu-system-x86_64: -drive file=rbd:volumes/volume-20987f9d-b4fb-463d-8b8f-fa667bd47c6d:id=volumes:key=AQAnAy9ScPB4IRAAtxD/V1rDciqFiT9AMPPr+A==:auth_supported=cephx\;none,if=none,id=drive-virtio-disk0,format=raw,serial=20987f9d-b4fb-463d-8b8f-fa667bd47c6d,cache=none: error reading header from volume-20987f9d-b4fb-463d-8b8f-fa667bd47c6d
qemu-system-x86_64: -drive file=rbd:volumes/volume-20987f9d-b4fb-463d-8b8f-fa667bd47c6d:id=volumes:key=AQAnAy9ScPB4IRAAtxD/V1rDciqFiT9AMPPr+A==:auth_supported=cephx\;none,if=none,id=drive-virtio-disk0,format=raw,serial=20987f9d-b4fb-463d-8b8f-fa667bd47c6d,cache=none: could not open disk image rbd:volumes/volume-20987f9d-b4fb-463d-8b8f-fa667bd47c6d:id=volumes:key=AQAnAy9ScPB4IRAAtxD/V1rDciqFiT9AMPPr+A==:auth_supported=cephx\;none: Operation not permitted
It's almost like a permission issue, but my ceph/rbd knowledge is still fledgeling.
I know that the cinder rbd driver has been rewritten to use librbd in havana, and I'm wondering if this will change any of this behaviour? I'm also wondering if anyone has actually got this working with grizzly, and how?
Many thanks
Darren
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
