Re: Boot from volume with Dumpling on RDO/CentOS 6 (using backported QEMU 0.12)

Josh Durgin <josh.durgin@xxxxxxxxxxx> · Mon, 21 Oct 2013 10:38:19 -0700

On 10/21/2013 10:35 AM, Andrew Richards wrote:
Thanks for the response Josh!

If the Ceph CLI tool still needs to be there for Cinder in Havana, then
am I correct in assuming that I still also need to export
"CEPH_ARGS='--id volumes'" in my cinder init script for the sake of
cephx like I had to do in Grizzly?

No, that's no longer necessary.

Josh

Thanks,
Andy

On Oct 21, 2013, at 12:26 PM, Josh Durgin <josh.durgin@xxxxxxxxxxx
<mailto:josh.durgin@xxxxxxxxxxx>> wrote:

On 10/21/2013 09:03 AM, Andrew Richards wrote:
Hi Everybody,

I'm attempting to get Ceph working for CentOS 6.4 running RDO Havana for
Cinder volume storage and boot-from-volume, and I keep bumping into a
very unhelpful errors on my nova-compute test node and my cinder
controller node.

Here is what I see on my cinder-volume controller (Node #1) when I try
to attach a RBD-backed Cinder volume to a Nova VM using either the GUI
or nova volume-attach (/var/log/cinder/volume.log):

2013-10-20 18:21:05.880 13668 ERROR cinder.openstack.common.rpc.amqp
[req-bd62cb07-42e7-414a-86dc-f26f7a569de6
9bfee22cd15b4dc0a2e203d7c151edbc 8431635821f84285afdd0f5faf1ce1aa]
Exception during message handling
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
Traceback (most recent call last):
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
File
"/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py",
line 441, in _process_data
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
**args)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
File
"/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py",
line 148, in dispatch
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
return getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in
wrapper
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
return func(self, *args, **kwargs)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line
624, in initialize_connection
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
conn_info = self.driver.initialize_connection(volume, connector)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py",
line 665, in initialize_connection
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
hosts, ports = self._get_mon_addrs()
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py",
line 312, in _get_mon_addrs
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
out, _ = self._execute(*args)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in
execute
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
return processutils.execute(*cmd, **kwargs)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
File
"/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py",
line 158, in execute
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
shell=shell)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
File "/usr/lib/python2.6/site-packages/eventlet/green/subprocess.py",
line 25, in __init__
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
errread, errwrite)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
raise child_exception
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
OSError: [Errno 2] No such file or directory
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp
2013-10-20 18:21:05.883 13668 ERROR cinder.openstack.common.rpc.common
[req-bd62cb07-42e7-414a-86dc-f26f7a569de6
9bfee22cd15b4dc0a2e203d7c151edbc 8431635821f84285afdd0f5faf1ce1aa]
Returning exception [Errno 2] No such file or directory to caller

Here is what I see on my nova-compute node (Node #2) when I try to boot
from volume (/var/log/nova/compute.log):

ERROR nova.compute.manager [req-ced59268-4766-4f57-9cdb-4ba451b0faaa
9bfee22cd15b4dc0a2e203d7c151edbc 8431635821f84285afdd0f5faf1ce1aa]
[instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] Error: The server has
either erred or is incapable of performing the requested operation.
(HTTP 500) (Request-ID: req-44557bfa-6777-41a6-8183-e08dedf0611b)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] Traceback (most recent call last):
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]   File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1028,
in _build_instance
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]     context, instance, bdms)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]   File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1393,
in _prep_block_device
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]     instance=instance)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]   File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1371,
in _prep_block_device
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]
self._await_block_device_map_created) +
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]   File
"/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 283,
in attach_block_devices
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]     block_device_mapping)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]   File
"/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 170,
in attach
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]     connector)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]   File
"/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 176, in
wrapper
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]     res = method(self, ctx,
volume_id, *args, **kwargs)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]   File
"/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 274, in
initialize_connection
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]     connector)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]   File
"/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 306,
in initialize_connection
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]     {'connector':
connector})[1]['connection_info']
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]   File
"/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 237,
in _action
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]     return
self.api.client.post(url, body=body)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]   File
"/usr/lib/python2.6/site-packages/cinderclient/client.py", line 210,
in post
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]     return self._cs_request(url,
'POST', **kwargs)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]   File
"/usr/lib/python2.6/site-packages/cinderclient/client.py", line 174, in
_cs_request
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]     **kwargs)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]   File
"/usr/lib/python2.6/site-packages/cinderclient/client.py", line 157, in
request
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]     raise
exceptions.from_response(resp, body)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0] ClientException: The server has
either erred or is incapable of performing the requested operation.
(HTTP 500) (Request-ID: req-44557bfa-6777-41a6-8183-e08dedf0611b)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance:
c80a053f-b84c-401c-8e29-022d4c6f56a0]

More info on my setup:

 * I'm running the most recent release of Dumpling libraries (0.67.4)
   and python-ceph, but I do not have ceph itself installed as it
   should not be needed on Havana

This is where the error is coming from - although most of the rbd
driver was converted to use librbd, there are a couple things that
still rely on the cli tools, since they had some functionality that
wasn't present in librados in cuttlefish or bobtail.

Specifically, the attach is failing because cinder is trying to use
the 'ceph' command to get a list of all monitors to pass to nova.
Installing
ceph-common on the node running cinder-volume should fix it.

 * cephx is in use, and I set up pools and keyrings per the guide at
http://ceph.com/docs/next/rbd/rbd-openstack/

These aren't updated for havana yet, but hopefully your email helps in
the interim. This is the main new requirement - librbd python bindings
and cli tools are needed for cinder-volume.

Josh

 * I'm using the ceph-extras repo to install the backported QEMU
   packages as described at
http://openstack.redhat.com/Using_Ceph_for_Block_Storage_with_RDO
 * I'm also using Neutron+OVS and thus edited my qemu.conf according to
   this libvirt wiki page
http://wiki.libvirt.org/page/Guest_won't_start_-_warning:_could_not_open_/dev/net/tun_('generic_ethernet'_interface)
<http://wiki.libvirt.org/page/Guest_won%27t_start_-_warning:_could_not_open_/dev/net/tun_%28%27generic_ethernet%27_interface%29>
   <http://wiki.libvirt.org/page/Guest_won%27t_start_-_warning:_could_not_open_/dev/net/tun_%28%27generic_ethernet%27_interface%29>
 * I am presently not configuring Nova to put its ephemeral disk image
   on RBD (new option in Havana)

Things I've been able to do so far:

 * I stored an image to Glance backed by RBD
 * I used that image to create a Cinder volume backed by RBD
 * I instantiated a working ephemeral VM with Nova based on the image
   from Glance backed by RBD
 * I created a libvirt VM manually with virsh on the same compute node
   that attached a volume from the Cinder RBD pool
 * I created a VM with Nova (ephemeral local boot) that I was then able
   to use virsh to successfully attach a Cinder volume backed by RBD
 * I can use qemu-img to create volumes in the cinder RBD pool, but
   only if I have the client.admin keyring installed on the compute node

The error traced above is what happens every time I try to boot from
volume using that RBD-backed Cinder volume spawned from the RBD-backed
Glance image.  What did work led me to believe that QEMU was the
problem, so I tried the following:

 * I changed the user and group qemu runs as from root to nova and from
   nova to qemu to see if their permissions had any affect; no change
 * I tried the above tests with matching permissions on the contents of
   /etc/ceph/ (ceph.conf and the keyrings for admin, cinder, and
   glance); no change

It seems like Nova is somehow failing to get from Cinder that Cinder is
using RBD for its backend, but I can't understand why.  All my configs
align with every instance of documentation I've been able to find for
making OpenStack work with Ceph.  Has anyone done what I'm trying to do
on CentOS 6 or even on some version of Fedora?  I am cross-posting this
on the OpenStack listserv as well.

Thanks for your time,

Andy

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com