On 12/13/2011 04:56 AM, Sławomir Skowron wrote:
Finally, i manage the problem with rbd with kvm 1.0, and libvirt
0.9.8, or i think i manage :), but i get stuck with one thing after.
2011-12-13 12:13:31.173+0000: 21512: error :
qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU
command 'device_add': Bus 'pci.0' does not support hotplugging
2011-12-13 12:13:31.173+0000: 21512: warning :
qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on
file=rbd:rbd/testdysk:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx
none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk2,format=raw
(virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2)
2011-12-13 12:13:31.173+0000: 21512: error :
virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink
rbd/testdysk: No such file or directory2011-12-13 12:13:31.173+0000:
21512: warning : qemuDomainAttachPciDiskDevice:287 : Unable to restore
security label on rbd/testdysk
and i can't find any info about that.
my kvm.xml inject file:
<disk type="network" device="disk">
<driver name="qemu" type="raw"/>
<auth username="admin">
<secret type="ceph"
uuid="0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f"/>
</auth>
<source protocol="rbd" name="rbd/testdysk">
<host name="10.177.64.4" port="6789"/>
</source>
<target dev="sdc" bus="virtio"/>
</disk>
secret:
cat /tmp/secret.xml
<secret ephemeral='no' private='no'>
<uuid>0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f</uuid>
<usage type='ceph'>
<domain>rbd/admin</domain>
<name>admin</name>
</usage>
</secret>
virsh secret-define /tmp/secret.xml
Secret 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f created
virsh secret-set-value 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f
AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==
Secret value set
Your xml and configuration are all fine, you're hitting the libvirt bug
that I mentioned before. The fix isn't in any stable release yet, and
0.9.8 was just released yesterday, so I guess you'll have to compile it
yourself or disable the libvirt security driver.
2011/12/12 Josh Durgin<josh.durgin@xxxxxxxxxxxxx>:
On 12/09/2011 04:48 AM, Sławomir Skowron wrote:
Sorry, for my lag, but i was sick.
I handled problem with apparmor before and now it's not a problem,
even when i send mail before, it was solved.
Ok when i create image from qemu-img it's looks like that, and works
perfect, (attachment with log of this opperation.)
But when i use virsh, it's still a problem. Just like he don't know
what user is going to be auth.
I try, as client.admin (default i hope :)) only with id=admin in
kvm.xml, or client=admin, and other combinations - like client.rbd
etc., Still nothing.
admin is the default, and the correct option for client.admin is id=admin,
or id=foo for client.foo.
I try, improve a keyring file, adding client rbd into keyring on
machine with kvm hypervisor in /etc/ceph/keyring.bin
[client.rbd]
key = AAAAAAAAAAAAAAAA
auid = 18446744073709551615
caps mds = "allow"
caps mon = "allow r"
caps osd = "allow rw pool=kvmtest"
and on cluster site like above. I was added, a section [client.rbd]
with keyring path, and it's still a problem.
Did you run 'ceph auth add -i /etc/ceph/keyring.bin'?
Make sure 'ceph auth list' shows client.rbd with the right key.
Overall i want to use admin id, and i think now it's correctly used in xml.
2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
root@s3-10-177-64-4:~# tail -n10000 /var/log/ceph/mon.0.log | grep
13:29:11.
2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
2011-12-09 13:29:11.519624 7fe61b090700 -- 10.177.64.4:6789/0<==
client.? 10.177.32.66:0/58020608 1 ==== auth(proto 0 21 bytes) v1 ====
47+0+0 (3310669357 0 0) 0x17f2e00 con 0x19d2c80
2011-12-09 13:29:11.519645 7fe61b090700 mon.0@0(leader) e1 have connection
2011-12-09 13:29:11.519655 7fe61b090700 mon.0@0(leader) e1 do not have
session, making new one
2011-12-09 13:29:11.519668 7fe61b090700 mon.0@0(leader) e1 ms_dispatch
new session MonSession: client.? 10.177.32.66:0/58020608 is open for
client.? 10.177.32.66:0/58020608
2011-12-09 13:29:11.519674 7fe61b090700 mon.0@0(leader) e1 setting
timeout on session
2011-12-09 13:29:11.519680 7fe61b090700 mon.0@0(leader) e1 caps
2011-12-09 13:29:11.519688 7fe61b090700 mon.0@0(leader).auth v353
AuthMonitor::update_from_paxos()
2011-12-09 13:29:11.519697 7fe61b090700 mon.0@0(leader).auth v353
preprocess_query auth(proto 0 21 bytes) v1 from client.?
10.177.32.66:0/58020608
2011-12-09 13:29:11.519703 7fe61b090700 mon.0@0(leader).auth v353
prep_auth() blob_size=21
2011-12-09 13:29:11.519735 7fe61b090700 -- 10.177.64.4:6789/0 -->
10.177.32.66:0/58020608 -- auth_reply(proto 0 -95 Operation not
supported) v1 -- ?+0 0x1246200 con 0x19d2c80
from libvirt.log
23:20:56.689: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
: cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
or directory
23:20:56.882: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
23:21:09.172: 1309: error : qemuMonitorTextAddDrive:2418 : operation
failed: open disk image file failed
23:21:09.172: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
: cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
or directory
23:21:09.364: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
23:21:49.645: 1309: error : qemuMonitorTextAddDrive:2418 : operation
failed: open disk image file failed
23:21:49.645: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
: cannot resolve symlink rbd:kvmtest/kvmtest1:id=admin: No such file
or directory
23:21:49.853: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
Unable to restore security label on rbd:kvmtest/kvmtest1:id=admin
13:08:08.924: 1309: error : qemuMonitorTextAddDrive:2418 : operation
failed: open disk image file failed
13:08:08.924: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
directory
13:08:09.117: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
Unable to restore security label on rbd:kvmtest/kvmtest1
13:09:45.402: 1309: error : qemuMonitorTextAddDrive:2418 : operation
failed: open disk image file failed
13:29:05.554: 1310: error : qemuMonitorTextAddDrive:2418 : operation
failed: open disk image file failed
13:29:05.554: 1310: error : virSecurityDACRestoreSecurityFileLabel:143
: cannot resolve symlink kvmtest/kvmtest1: No such file or directory
13:29:05.751: 1310: warning : qemuDomainAttachPciDiskDevice:250 :
Unable to restore security label on kvmtest/kvmtest1
13:29:05.751: 1310: warning : virEventPollUpdateHandle:147 : Ignoring
invalid update watch -1
This is libvirt mistakenly trying to treat the rbd disk as a file - fixed in
20e1233c31e3150d259073c523077acdccb5d419 to libvirt. If you don't want to
upgrade libvirt, someone on IRC last week reported that installing
libapparmor1 fixed this problem for them, since libvirt no longer fell back
to it's own DAC implementation.
hmm is this commited to any stable version now ?? i have installed
libapparmor1,
ri libapparmor1 2.7.0~beta1+bzr1774-1ubuntu2
changehat AppArmor library
and i use newest stable libvirt 0.9.8.
2011/12/1 Josh Durgin<josh.durgin@xxxxxxxxxxxxx>:
On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
I have some problems. Can you help me ??
ceph cluster:
ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
kvm hypervisor:
kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
oneiric x86_64.
I create image from qemu-img on machine with kvm VM's, and it works very
well:
qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240
cluster_size=0
in ceph cluster machines:
rados -p kvmtest ls
kvmtest1.rbd
rbd_directory
rbd_info
But when i try to attach new device to kvm VM like this:
virsh attach-device one-10 /tmp/kvm.xml
with kvm.xml like this:
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='writeback'/>
<source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
<host name='10.177.64.4' port='6789'/>
</source>
<target dev='vdc' bus='virtio'/>
</disk>
output of virsh:
error: Failed to attach device from /tmp/kvm.xml
error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
directory
From the monitor output below, I'm assuming this is just a bad error
message
from libvirt.
and mon.0 log in ceph cluster.
2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
new session MonSession: client.? 10.177.32.66:0/25020608 is open for
client.? 10.177.32.66:0/25020608
2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
preprocess_query auth(proto 0 21 bytes) v1 from client.?
10.177.32.66:0/25020608
2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
supported) v1 -- ?+0 0x2f30600 con 0x297adc0
Here you can see qemu isn't authenticating correctly, probably because
the
rados user isn't being set by libvirt. With your libvirt version, you'll
need to change:
<source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
to
<source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>
to authenticate as client.foo. Any ceph config option can be set like
that,
separated by colons.
If this still doesn't work, I'd suggest setting
debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more info
in
the libvirt instance log (usually
/var/log/libvirt/qemu/instance_name.log).
There may be apparmour restrictions preventing qemu from reading the
keyring
or ceph config files.
ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
in same directory:
; global
[global]
; enable secure authentication
auth supported = cephx
keyring = /etc/ceph/keyring.bin
debug rgw = 1
rgw print continue = false
rgw socket path = /var/run/radosgw.sock
; monitors
; You need at least one. You need at least three if you want to
; tolerate any node failures. Always create an odd number.
[mon]
mon data = /vol0/data/mon.$id
; some minimal logging (just message traffic) to aid debugging
debug ms = 1 ; see message traffic
debug mon = 0 ; monitor
debug paxos = 0 ; monitor replication
debug auth = 0 ;
mon allowed clock drift = 2
[mon.0]
host = 10-177-64-4
mon addr = 10.177.64.4:6789
; radosgw client list
[client.radosgw.10-177-64-4]
host = 10-177-64-4
log file = /var/log/ceph/$name.log
debug rgw = 1
debug ms = 1
; osd
; You need at least one. Two if you want data to be replicated.
; Define as many as you like.
[osd]
; This is where the btrfs volume will be mounted.
osd data = /vol0/data/osd.$id
; Ideally, make this a separate disk or partition. A few GB
; is usually enough; more if you have fast disks. You can use
; a file under the osd data dir if need be
; (e.g. /data/osd$id/journal), but it will be slower than a
; separate disk or partition.
osd journal = /vol0/data/osd.$id/journal
; If the OSD journal is a file, you need to specify the size.
This is specified in MB.
osd journal size = 512
filestore journal writeahead = 1
osd heartbeat grace = 5
debug ms = 0 ; message traffic
debug osd = 0
debug filestore = 0 ; local object storage
debug journal = 0 ; local journaling
debug monc = 0
debug rados = 1
[osd.0]
host = 10-177-64-4
osd data = /vol0/data/osd.0
keyring = /vol0/data/osd.0/keyring
[osd.1]
host = 10-177-64-4
osd data = /vol0/data/osd.1
keyring = /vol0/data/osd.1/keyring
[osd.2]
host = 10-177-64-4
osd data = /vol0/data/osd.2
keyring = /vol0/data/osd.2/keyring
[osd.3]
host = 10-177-64-4
osd data = /vol0/data/osd.3
keyring = /vol0/data/osd.3/keyring
[osd.4]
host = 10-177-64-4
osd data = /vol0/data/osd.4
keyring = /vol0/data/osd.4/keyring
[osd.5]
host = 10-177-64-4
osd data = /vol0/data/osd.5
keyring = /vol0/data/osd.5/keyring
[osd.6]
host = 10-177-64-4
osd data = /vol0/data/osd.6
keyring = /vol0/data/osd.6/keyring
[osd.7]
host = 10-177-64-4
osd data = /vol0/data/osd.7
keyring = /vol0/data/osd.7/keyring
[osd.8]
host = 10-177-64-4
osd data = /vol0/data/osd.8
keyring = /vol0/data/osd.8/keyring
[osd.9]
host = 10-177-64-4
osd data = /vol0/data/osd.9
keyring = /vol0/data/osd.9/keyring
[osd.10]
host = 10-177-64-4
osd data = /vol0/data/osd.10
keyring = /vol0/data/osd.10/keyring
[osd.11]
host = 10-177-64-4
osd data = /vol0/data/osd.11
keyring = /vol0/data/osd.11/keyring
[osd.12]
host = 10-177-64-4
osd data = /vol0/data/osd.12
keyring = /vol0/data/osd.12/keyring
[osd.13]
host = 10-177-64-4
osd data = /vol0/data/osd.13
keyring = /vol0/data/osd.13/keyring
[osd.14]
host = 10-177-64-4
osd data = /vol0/data/osd.14
keyring = /vol0/data/osd.14/keyring
[osd.15]
host = 10-177-64-4
osd data = /vol0/data/osd.15
keyring = /vol0/data/osd.15/keyring
[osd.16]
host = 10-177-64-4
osd data = /vol0/data/osd.16
keyring = /vol0/data/osd.16/keyring
[osd.17]
host = 10-177-64-4
osd data = /vol0/data/osd.17
keyring = /vol0/data/osd.17/keyring
[osd.18]
host = 10-177-64-4
osd data = /vol0/data/osd.18
keyring = /vol0/data/osd.18/keyring
[osd.19]
host = 10-177-64-4
osd data = /vol0/data/osd.19
keyring = /vol0/data/osd.19/keyring
[osd.20]
host = 10-177-64-4
osd data = /vol0/data/osd.20
keyring = /vol0/data/osd.20/keyring
[osd.21]
host = 10-177-64-4
osd data = /vol0/data/osd.21
keyring = /vol0/data/osd.21/keyring
[osd.22]
host = 10-177-64-4
osd data = /vol0/data/osd.22
keyring = /vol0/data/osd.22/keyring
[osd.23]
host = 10-177-64-4
osd data = /vol0/data/osd.23
keyring = /vol0/data/osd.23/keyring
[osd.24]
host = 10-177-64-4
osd data = /vol0/data/osd.24
keyring = /vol0/data/osd.24/keyring
[osd.25]
host = 10-177-64-4
osd data = /vol0/data/osd.25
keyring = /vol0/data/osd.25/keyring
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
