Finally, i manage the problem with rbd with kvm 1.0, and libvirt
0.9.8, or i think i manage :), but i get stuck with one thing after.
2011-12-13 12:13:31.173+0000: 21512: error :
qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU
command 'device_add': Bus 'pci.0' does not support hotplugging
2011-12-13 12:13:31.173+0000: 21512: warning :
qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on
file=rbd:rbd/testdysk:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx
none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk2,format=raw
(virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2)
2011-12-13 12:13:31.173+0000: 21512: error :
virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink
rbd/testdysk: No such file or directory2011-12-13 12:13:31.173+0000:
21512: warning : qemuDomainAttachPciDiskDevice:287 : Unable to restore
security label on rbd/testdysk
and i can't find any info about that.
my kvm.xml inject file:
<disk type="network" device="disk">
<driver name="qemu" type="raw"/>
<auth username="admin">
<secret type="ceph"
uuid="0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f"/>
</auth>
<source protocol="rbd" name="rbd/testdysk">
<host name="10.177.64.4" port="6789"/>
</source>
<target dev="sdc" bus="virtio"/>
</disk>
secret:
cat /tmp/secret.xml
<secret ephemeral='no' private='no'>
<uuid>0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f</uuid>
<usage type='ceph'>
<domain>rbd/admin</domain>
<name>admin</name>
</usage>
</secret>
virsh secret-define /tmp/secret.xml
Secret 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f created
virsh secret-set-value 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f
AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==
Secret value set
2011/12/12 Josh Durgin <josh.durgin@xxxxxxxxxxxxx>:
> On 12/09/2011 04:48 AM, Sławomir Skowron wrote:
>>
>> Sorry, for my lag, but i was sick.
>>
>> I handled problem with apparmor before and now it's not a problem,
>> even when i send mail before, it was solved.
>>
>> Ok when i create image from qemu-img it's looks like that, and works
>> perfect, (attachment with log of this opperation.)
>>
>> But when i use virsh, it's still a problem. Just like he don't know
>> what user is going to be auth.
>>
>> I try, as client.admin (default i hope :)) only with id=admin in
>> kvm.xml, or client=admin, and other combinations - like client.rbd
>> etc., Still nothing.
>
>
> admin is the default, and the correct option for client.admin is id=admin,
> or id=foo for client.foo.
>
>
>>
>> I try, improve a keyring file, adding client rbd into keyring on
>> machine with kvm hypervisor in /etc/ceph/keyring.bin
>>
>> [client.rbd]
>> key = AAAAAAAAAAAAAAAA
>> auid = 18446744073709551615
>> caps mds = "allow"
>> caps mon = "allow r"
>> caps osd = "allow rw pool=kvmtest"
>>
>> and on cluster site like above. I was added, a section [client.rbd]
>> with keyring path, and it's still a problem.
>>
>
> Did you run 'ceph auth add -i /etc/ceph/keyring.bin'?
> Make sure 'ceph auth list' shows client.rbd with the right key.
Overall i want to use admin id, and i think now it's correctly used in xml.
>
>
>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>> root@s3-10-177-64-4:~# tail -n10000 /var/log/ceph/mon.0.log | grep
>> 13:29:11.
>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>> 2011-12-09 13:29:11.519624 7fe61b090700 -- 10.177.64.4:6789/0<==
>> client.? 10.177.32.66:0/58020608 1 ==== auth(proto 0 21 bytes) v1 ====
>> 47+0+0 (3310669357 0 0) 0x17f2e00 con 0x19d2c80
>> 2011-12-09 13:29:11.519645 7fe61b090700 mon.0@0(leader) e1 have connection
>> 2011-12-09 13:29:11.519655 7fe61b090700 mon.0@0(leader) e1 do not have
>> session, making new one
>> 2011-12-09 13:29:11.519668 7fe61b090700 mon.0@0(leader) e1 ms_dispatch
>> new session MonSession: client.? 10.177.32.66:0/58020608 is open for
>> client.? 10.177.32.66:0/58020608
>> 2011-12-09 13:29:11.519674 7fe61b090700 mon.0@0(leader) e1 setting
>> timeout on session
>> 2011-12-09 13:29:11.519680 7fe61b090700 mon.0@0(leader) e1 caps
>> 2011-12-09 13:29:11.519688 7fe61b090700 mon.0@0(leader).auth v353
>> AuthMonitor::update_from_paxos()
>> 2011-12-09 13:29:11.519697 7fe61b090700 mon.0@0(leader).auth v353
>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>> 10.177.32.66:0/58020608
>> 2011-12-09 13:29:11.519703 7fe61b090700 mon.0@0(leader).auth v353
>> prep_auth() blob_size=21
>> 2011-12-09 13:29:11.519735 7fe61b090700 -- 10.177.64.4:6789/0 -->
>> 10.177.32.66:0/58020608 -- auth_reply(proto 0 -95 Operation not
>> supported) v1 -- ?+0 0x1246200 con 0x19d2c80
>>
>> from libvirt.log
>>
>> 23:20:56.689: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
>> or directory
>> 23:20:56.882: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>> 23:21:09.172: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>> failed: open disk image file failed
>> 23:21:09.172: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
>> or directory
>> 23:21:09.364: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>> 23:21:49.645: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>> failed: open disk image file failed
>> 23:21:49.645: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>> : cannot resolve symlink rbd:kvmtest/kvmtest1:id=admin: No such file
>> or directory
>> 23:21:49.853: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>> Unable to restore security label on rbd:kvmtest/kvmtest1:id=admin
>> 13:08:08.924: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>> failed: open disk image file failed
>> 13:08:08.924: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>> : cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>> directory
>> 13:08:09.117: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>> Unable to restore security label on rbd:kvmtest/kvmtest1
>> 13:09:45.402: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>> failed: open disk image file failed
>> 13:29:05.554: 1310: error : qemuMonitorTextAddDrive:2418 : operation
>> failed: open disk image file failed
>> 13:29:05.554: 1310: error : virSecurityDACRestoreSecurityFileLabel:143
>> : cannot resolve symlink kvmtest/kvmtest1: No such file or directory
>> 13:29:05.751: 1310: warning : qemuDomainAttachPciDiskDevice:250 :
>> Unable to restore security label on kvmtest/kvmtest1
>> 13:29:05.751: 1310: warning : virEventPollUpdateHandle:147 : Ignoring
>> invalid update watch -1
>
>
> This is libvirt mistakenly trying to treat the rbd disk as a file - fixed in
> 20e1233c31e3150d259073c523077acdccb5d419 to libvirt. If you don't want to
> upgrade libvirt, someone on IRC last week reported that installing
> libapparmor1 fixed this problem for them, since libvirt no longer fell back
> to it's own DAC implementation.
hmm is this commited to any stable version now ?? i have installed
libapparmor1,
ri libapparmor1 2.7.0~beta1+bzr1774-1ubuntu2
changehat AppArmor library
and i use newest stable libvirt 0.9.8.
>
>
>> 2011/12/1 Josh Durgin<josh.durgin@xxxxxxxxxxxxx>:
>>>
>>> On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
>>>>
>>>>
>>>> I have some problems. Can you help me ??
>>>>
>>>> ceph cluster:
>>>> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
>>>>
>>>> kvm hypervisor:
>>>> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
>>>> oneiric x86_64.
>>>>
>>>> I create image from qemu-img on machine with kvm VM's, and it works very
>>>> well:
>>>>
>>>> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
>>>> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240
>>>> cluster_size=0
>>>>
>>>> in ceph cluster machines:
>>>>
>>>> rados -p kvmtest ls
>>>> kvmtest1.rbd
>>>> rbd_directory
>>>> rbd_info
>>>>
>>>> But when i try to attach new device to kvm VM like this:
>>>>
>>>> virsh attach-device one-10 /tmp/kvm.xml
>>>>
>>>> with kvm.xml like this:
>>>>
>>>> <disk type='network' device='disk'>
>>>> <driver name='qemu' type='raw' cache='writeback'/>
>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>> <host name='10.177.64.4' port='6789'/>
>>>> </source>
>>>> <target dev='vdc' bus='virtio'/>
>>>> </disk>
>>>>
>>>> output of virsh:
>>>>
>>>> error: Failed to attach device from /tmp/kvm.xml
>>>> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>>> directory
>>>
>>>
>>>
>>> From the monitor output below, I'm assuming this is just a bad error
>>> message
>>> from libvirt.
>>>
>>>
>>>> and mon.0 log in ceph cluster.
>>>>
>>>> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
>>>> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
>>>> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
>>>> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
>>>> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
>>>> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
>>>> new session MonSession: client.? 10.177.32.66:0/25020608 is open for
>>>> client.? 10.177.32.66:0/25020608
>>>> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>>> 10.177.32.66:0/25020608
>>>> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
>>>> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
>>>> supported) v1 -- ?+0 0x2f30600 con 0x297adc0
>>>
>>>
>>>
>>> Here you can see qemu isn't authenticating correctly, probably because
>>> the
>>> rados user isn't being set by libvirt. With your libvirt version, you'll
>>> need to change:
>>>
>>>
>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>
>>> to
>>>
>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>
>>>
>>> to authenticate as client.foo. Any ceph config option can be set like
>>> that,
>>> separated by colons.
>>>
>>> If this still doesn't work, I'd suggest setting
>>> debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more info
>>> in
>>> the libvirt instance log (usually
>>> /var/log/libvirt/qemu/instance_name.log).
>>> There may be apparmour restrictions preventing qemu from reading the
>>> keyring
>>> or ceph config files.
>>>
>>>
>>>> ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
>>>> in same directory:
>>>> ; global
>>>> [global]
>>>> ; enable secure authentication
>>>>
>>>> auth supported = cephx
>>>> keyring = /etc/ceph/keyring.bin
>>>>
>>>> debug rgw = 1
>>>>
>>>> rgw print continue = false
>>>> rgw socket path = /var/run/radosgw.sock
>>>>
>>>> ; monitors
>>>> ; You need at least one. You need at least three if you want to
>>>> ; tolerate any node failures. Always create an odd number.
>>>> [mon]
>>>> mon data = /vol0/data/mon.$id
>>>>
>>>> ; some minimal logging (just message traffic) to aid debugging
>>>>
>>>> debug ms = 1 ; see message traffic
>>>> debug mon = 0 ; monitor
>>>> debug paxos = 0 ; monitor replication
>>>> debug auth = 0 ;
>>>>
>>>> mon allowed clock drift = 2
>>>>
>>>> [mon.0]
>>>> host = 10-177-64-4
>>>> mon addr = 10.177.64.4:6789
>>>>
>>>> ; radosgw client list
>>>> [client.radosgw.10-177-64-4]
>>>>
>>>> host = 10-177-64-4
>>>> log file = /var/log/ceph/$name.log
>>>> debug rgw = 1
>>>> debug ms = 1
>>>>
>>>> ; osd
>>>> ; You need at least one. Two if you want data to be replicated.
>>>> ; Define as many as you like.
>>>> [osd]
>>>> ; This is where the btrfs volume will be mounted.
>>>>
>>>> osd data = /vol0/data/osd.$id
>>>>
>>>> ; Ideally, make this a separate disk or partition. A few GB
>>>> ; is usually enough; more if you have fast disks. You can use
>>>> ; a file under the osd data dir if need be
>>>> ; (e.g. /data/osd$id/journal), but it will be slower than a
>>>> ; separate disk or partition.
>>>>
>>>> osd journal = /vol0/data/osd.$id/journal
>>>>
>>>> ; If the OSD journal is a file, you need to specify the size.
>>>> This is specified in MB.
>>>>
>>>> osd journal size = 512
>>>>
>>>> filestore journal writeahead = 1
>>>> osd heartbeat grace = 5
>>>>
>>>> debug ms = 0 ; message traffic
>>>> debug osd = 0
>>>> debug filestore = 0 ; local object storage
>>>> debug journal = 0 ; local journaling
>>>> debug monc = 0
>>>> debug rados = 1
>>>>
>>>> [osd.0]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.0
>>>> keyring = /vol0/data/osd.0/keyring
>>>>
>>>> [osd.1]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.1
>>>> keyring = /vol0/data/osd.1/keyring
>>>>
>>>> [osd.2]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.2
>>>> keyring = /vol0/data/osd.2/keyring
>>>>
>>>> [osd.3]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.3
>>>> keyring = /vol0/data/osd.3/keyring
>>>>
>>>> [osd.4]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.4
>>>> keyring = /vol0/data/osd.4/keyring
>>>>
>>>> [osd.5]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.5
>>>> keyring = /vol0/data/osd.5/keyring
>>>>
>>>> [osd.6]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.6
>>>> keyring = /vol0/data/osd.6/keyring
>>>>
>>>> [osd.7]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.7
>>>> keyring = /vol0/data/osd.7/keyring
>>>>
>>>> [osd.8]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.8
>>>> keyring = /vol0/data/osd.8/keyring
>>>>
>>>> [osd.9]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.9
>>>> keyring = /vol0/data/osd.9/keyring
>>>>
>>>> [osd.10]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.10
>>>> keyring = /vol0/data/osd.10/keyring
>>>>
>>>> [osd.11]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.11
>>>> keyring = /vol0/data/osd.11/keyring
>>>>
>>>> [osd.12]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.12
>>>> keyring = /vol0/data/osd.12/keyring
>>>>
>>>> [osd.13]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.13
>>>> keyring = /vol0/data/osd.13/keyring
>>>>
>>>> [osd.14]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.14
>>>> keyring = /vol0/data/osd.14/keyring
>>>>
>>>> [osd.15]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.15
>>>> keyring = /vol0/data/osd.15/keyring
>>>>
>>>> [osd.16]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.16
>>>> keyring = /vol0/data/osd.16/keyring
>>>>
>>>> [osd.17]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.17
>>>> keyring = /vol0/data/osd.17/keyring
>>>>
>>>> [osd.18]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.18
>>>> keyring = /vol0/data/osd.18/keyring
>>>>
>>>> [osd.19]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.19
>>>> keyring = /vol0/data/osd.19/keyring
>>>>
>>>> [osd.20]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.20
>>>> keyring = /vol0/data/osd.20/keyring
>>>>
>>>> [osd.21]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.21
>>>> keyring = /vol0/data/osd.21/keyring
>>>>
>>>> [osd.22]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.22
>>>> keyring = /vol0/data/osd.22/keyring
>>>>
>>>> [osd.23]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.23
>>>> keyring = /vol0/data/osd.23/keyring
>>>>
>>>> [osd.24]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.24
>>>> keyring = /vol0/data/osd.24/keyring
>>>>
>>>> [osd.25]
>>>> host = 10-177-64-4
>>>> osd data = /vol0/data/osd.25
>>>> keyring = /vol0/data/osd.25/keyring
>>>>
>>>>
>>>
>>
>>
>>
>
--
-----
Pozdrawiam
Sławek "sZiBis" Skowron
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
