On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
I have some problems. Can you help me ??
ceph cluster:
ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
kvm hypervisor:
kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
oneiric x86_64.
I create image from qemu-img on machine with kvm VM's, and it works very well:
qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240 cluster_size=0
in ceph cluster machines:
rados -p kvmtest ls
kvmtest1.rbd
rbd_directory
rbd_info
But when i try to attach new device to kvm VM like this:
virsh attach-device one-10 /tmp/kvm.xml
with kvm.xml like this:
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='writeback'/>
<source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
<host name='10.177.64.4' port='6789'/>
</source>
<target dev='vdc' bus='virtio'/>
</disk>
output of virsh:
error: Failed to attach device from /tmp/kvm.xml
error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or directory
From the monitor output below, I'm assuming this is just a bad error
message from libvirt.
and mon.0 log in ceph cluster.
2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
new session MonSession: client.? 10.177.32.66:0/25020608 is open for
client.? 10.177.32.66:0/25020608
2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
preprocess_query auth(proto 0 21 bytes) v1 from client.?
10.177.32.66:0/25020608
2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
supported) v1 -- ?+0 0x2f30600 con 0x297adc0
Here you can see qemu isn't authenticating correctly, probably because
the rados user isn't being set by libvirt. With your libvirt version,
you'll need to change:
<source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
to
<source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>
to authenticate as client.foo. Any ceph config option can be set like
that, separated by colons.
If this still doesn't work, I'd suggest setting
debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more
info in the libvirt instance log (usually
/var/log/libvirt/qemu/instance_name.log). There may be apparmour
restrictions preventing qemu from reading the keyring or ceph config files.
ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
in same directory:
; global
[global]
; enable secure authentication
auth supported = cephx
keyring = /etc/ceph/keyring.bin
debug rgw = 1
rgw print continue = false
rgw socket path = /var/run/radosgw.sock
; monitors
; You need at least one. You need at least three if you want to
; tolerate any node failures. Always create an odd number.
[mon]
mon data = /vol0/data/mon.$id
; some minimal logging (just message traffic) to aid debugging
debug ms = 1 ; see message traffic
debug mon = 0 ; monitor
debug paxos = 0 ; monitor replication
debug auth = 0 ;
mon allowed clock drift = 2
[mon.0]
host = 10-177-64-4
mon addr = 10.177.64.4:6789
; radosgw client list
[client.radosgw.10-177-64-4]
host = 10-177-64-4
log file = /var/log/ceph/$name.log
debug rgw = 1
debug ms = 1
; osd
; You need at least one. Two if you want data to be replicated.
; Define as many as you like.
[osd]
; This is where the btrfs volume will be mounted.
osd data = /vol0/data/osd.$id
; Ideally, make this a separate disk or partition. A few GB
; is usually enough; more if you have fast disks. You can use
; a file under the osd data dir if need be
; (e.g. /data/osd$id/journal), but it will be slower than a
; separate disk or partition.
osd journal = /vol0/data/osd.$id/journal
; If the OSD journal is a file, you need to specify the size.
This is specified in MB.
osd journal size = 512
filestore journal writeahead = 1
osd heartbeat grace = 5
debug ms = 0 ; message traffic
debug osd = 0
debug filestore = 0 ; local object storage
debug journal = 0 ; local journaling
debug monc = 0
debug rados = 1
[osd.0]
host = 10-177-64-4
osd data = /vol0/data/osd.0
keyring = /vol0/data/osd.0/keyring
[osd.1]
host = 10-177-64-4
osd data = /vol0/data/osd.1
keyring = /vol0/data/osd.1/keyring
[osd.2]
host = 10-177-64-4
osd data = /vol0/data/osd.2
keyring = /vol0/data/osd.2/keyring
[osd.3]
host = 10-177-64-4
osd data = /vol0/data/osd.3
keyring = /vol0/data/osd.3/keyring
[osd.4]
host = 10-177-64-4
osd data = /vol0/data/osd.4
keyring = /vol0/data/osd.4/keyring
[osd.5]
host = 10-177-64-4
osd data = /vol0/data/osd.5
keyring = /vol0/data/osd.5/keyring
[osd.6]
host = 10-177-64-4
osd data = /vol0/data/osd.6
keyring = /vol0/data/osd.6/keyring
[osd.7]
host = 10-177-64-4
osd data = /vol0/data/osd.7
keyring = /vol0/data/osd.7/keyring
[osd.8]
host = 10-177-64-4
osd data = /vol0/data/osd.8
keyring = /vol0/data/osd.8/keyring
[osd.9]
host = 10-177-64-4
osd data = /vol0/data/osd.9
keyring = /vol0/data/osd.9/keyring
[osd.10]
host = 10-177-64-4
osd data = /vol0/data/osd.10
keyring = /vol0/data/osd.10/keyring
[osd.11]
host = 10-177-64-4
osd data = /vol0/data/osd.11
keyring = /vol0/data/osd.11/keyring
[osd.12]
host = 10-177-64-4
osd data = /vol0/data/osd.12
keyring = /vol0/data/osd.12/keyring
[osd.13]
host = 10-177-64-4
osd data = /vol0/data/osd.13
keyring = /vol0/data/osd.13/keyring
[osd.14]
host = 10-177-64-4
osd data = /vol0/data/osd.14
keyring = /vol0/data/osd.14/keyring
[osd.15]
host = 10-177-64-4
osd data = /vol0/data/osd.15
keyring = /vol0/data/osd.15/keyring
[osd.16]
host = 10-177-64-4
osd data = /vol0/data/osd.16
keyring = /vol0/data/osd.16/keyring
[osd.17]
host = 10-177-64-4
osd data = /vol0/data/osd.17
keyring = /vol0/data/osd.17/keyring
[osd.18]
host = 10-177-64-4
osd data = /vol0/data/osd.18
keyring = /vol0/data/osd.18/keyring
[osd.19]
host = 10-177-64-4
osd data = /vol0/data/osd.19
keyring = /vol0/data/osd.19/keyring
[osd.20]
host = 10-177-64-4
osd data = /vol0/data/osd.20
keyring = /vol0/data/osd.20/keyring
[osd.21]
host = 10-177-64-4
osd data = /vol0/data/osd.21
keyring = /vol0/data/osd.21/keyring
[osd.22]
host = 10-177-64-4
osd data = /vol0/data/osd.22
keyring = /vol0/data/osd.22/keyring
[osd.23]
host = 10-177-64-4
osd data = /vol0/data/osd.23
keyring = /vol0/data/osd.23/keyring
[osd.24]
host = 10-177-64-4
osd data = /vol0/data/osd.24
keyring = /vol0/data/osd.24/keyring
[osd.25]
host = 10-177-64-4
osd data = /vol0/data/osd.25
keyring = /vol0/data/osd.25/keyring
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
