2011/12/13 Josh Durgin <josh.durgin@xxxxxxxxxxxxx>: > On 12/13/2011 04:56 AM, Sławomir Skowron wrote: >> >> Finally, i manage the problem with rbd with kvm 1.0, and libvirt >> 0.9.8, or i think i manage :), but i get stuck with one thing after. >> >> 2011-12-13 12:13:31.173+0000: 21512: error : >> qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU >> command 'device_add': Bus 'pci.0' does not support hotplugging >> 2011-12-13 12:13:31.173+0000: 21512: warning : >> qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on >> >> file=rbd:rbd/testdysk:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx >> none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk2,format=raw >> >> (virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2) >> 2011-12-13 12:13:31.173+0000: 21512: error : >> virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink >> rbd/testdysk: No such file or directory2011-12-13 12:13:31.173+0000: >> 21512: warning : qemuDomainAttachPciDiskDevice:287 : Unable to restore >> security label on rbd/testdysk >> and i can't find any info about that. >> >> my kvm.xml inject file: >> >> <disk type="network" device="disk"> >> <driver name="qemu" type="raw"/> >> <auth username="admin"> >> <secret type="ceph" >> uuid="0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f"/> >> </auth> >> <source protocol="rbd" name="rbd/testdysk"> >> <host name="10.177.64.4" port="6789"/> >> </source> >> <target dev="sdc" bus="virtio"/> >> </disk> >> >> secret: >> >> cat /tmp/secret.xml >> <secret ephemeral='no' private='no'> >> <uuid>0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f</uuid> >> <usage type='ceph'> >> <domain>rbd/admin</domain> >> <name>admin</name> >> </usage> >> </secret> >> >> virsh secret-define /tmp/secret.xml >> Secret 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f created >> >> virsh secret-set-value 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f >> AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog== >> Secret value set >> > > Your xml and configuration are all fine, you're hitting the libvirt bug that > I mentioned before. The fix isn't in any stable release yet, and 0.9.8 was > just released yesterday, so I guess you'll have to compile it yourself or > disable the libvirt security driver. Yes libvirt 0.9.8 with this patch solved problem with rbd, now it's clean. Thanks for that. But, a second problem appear. error: Failed to attach device from /root/kvm-ceph.xml error: internal error unable to execute QEMU command 'device_add': Property 'virtio-blk-pci.drive' can't find value 'drive-virtio-disk3' /var/log/libvirt/libvirtd.log 2011-12-16 12:03:04.950+0000: 1070: error : qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU command 'device_add': Property 'virtio-blk-pci.drive' can't find value 'drive-virtio-disk3' 2011-12-16 12:03:04.950+0000: 1070: warning : qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on file=rbd:rbd/testdysk:rbd_writeback_window=8000000:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk3,format=raw (virtio-blk-pci,bus=pci.0,addr=0xd,drive=drive-virtio-disk3,id=virtio-disk3) if i attach a scsi device from img file everything is OK. Anyone have, any concept with this ?? > > >> 2011/12/12 Josh Durgin<josh.durgin@xxxxxxxxxxxxx>: >>> >>> On 12/09/2011 04:48 AM, Sławomir Skowron wrote: >>>> >>>> >>>> Sorry, for my lag, but i was sick. >>>> >>>> I handled problem with apparmor before and now it's not a problem, >>>> even when i send mail before, it was solved. >>>> >>>> Ok when i create image from qemu-img it's looks like that, and works >>>> perfect, (attachment with log of this opperation.) >>>> >>>> But when i use virsh, it's still a problem. Just like he don't know >>>> what user is going to be auth. >>>> >>>> I try, as client.admin (default i hope :)) only with id=admin in >>>> kvm.xml, or client=admin, and other combinations - like client.rbd >>>> etc., Still nothing. >>> >>> >>> >>> admin is the default, and the correct option for client.admin is >>> id=admin, >>> or id=foo for client.foo. >>> >>> >>>> >>>> I try, improve a keyring file, adding client rbd into keyring on >>>> machine with kvm hypervisor in /etc/ceph/keyring.bin >>>> >>>> [client.rbd] >>>> key = AAAAAAAAAAAAAAAA >>>> auid = 18446744073709551615 >>>> caps mds = "allow" >>>> caps mon = "allow r" >>>> caps osd = "allow rw pool=kvmtest" >>>> >>>> and on cluster site like above. I was added, a section [client.rbd] >>>> with keyring path, and it's still a problem. >>>> >>> >>> Did you run 'ceph auth add -i /etc/ceph/keyring.bin'? >>> Make sure 'ceph auth list' shows client.rbd with the right key. >> >> >> Overall i want to use admin id, and i think now it's correctly used in >> xml. >> >>> >>> >>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1 >>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0 >>>> root@s3-10-177-64-4:~# tail -n10000 /var/log/ceph/mon.0.log | grep >>>> 13:29:11. >>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1 >>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0 >>>> 2011-12-09 13:29:11.519624 7fe61b090700 -- 10.177.64.4:6789/0<== >>>> client.? 10.177.32.66:0/58020608 1 ==== auth(proto 0 21 bytes) v1 ==== >>>> 47+0+0 (3310669357 0 0) 0x17f2e00 con 0x19d2c80 >>>> 2011-12-09 13:29:11.519645 7fe61b090700 mon.0@0(leader) e1 have >>>> connection >>>> 2011-12-09 13:29:11.519655 7fe61b090700 mon.0@0(leader) e1 do not have >>>> session, making new one >>>> 2011-12-09 13:29:11.519668 7fe61b090700 mon.0@0(leader) e1 ms_dispatch >>>> new session MonSession: client.? 10.177.32.66:0/58020608 is open for >>>> client.? 10.177.32.66:0/58020608 >>>> 2011-12-09 13:29:11.519674 7fe61b090700 mon.0@0(leader) e1 setting >>>> timeout on session >>>> 2011-12-09 13:29:11.519680 7fe61b090700 mon.0@0(leader) e1 caps >>>> 2011-12-09 13:29:11.519688 7fe61b090700 mon.0@0(leader).auth v353 >>>> AuthMonitor::update_from_paxos() >>>> 2011-12-09 13:29:11.519697 7fe61b090700 mon.0@0(leader).auth v353 >>>> preprocess_query auth(proto 0 21 bytes) v1 from client.? >>>> 10.177.32.66:0/58020608 >>>> 2011-12-09 13:29:11.519703 7fe61b090700 mon.0@0(leader).auth v353 >>>> prep_auth() blob_size=21 >>>> 2011-12-09 13:29:11.519735 7fe61b090700 -- 10.177.64.4:6789/0 --> >>>> 10.177.32.66:0/58020608 -- auth_reply(proto 0 -95 Operation not >>>> supported) v1 -- ?+0 0x1246200 con 0x19d2c80 >>>> >>>> from libvirt.log >>>> >>>> 23:20:56.689: 1309: error : virSecurityDACRestoreSecurityFileLabel:143 >>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file >>>> or directory >>>> 23:20:56.882: 1309: warning : qemuDomainAttachPciDiskDevice:250 : >>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin >>>> 23:21:09.172: 1309: error : qemuMonitorTextAddDrive:2418 : operation >>>> failed: open disk image file failed >>>> 23:21:09.172: 1309: error : virSecurityDACRestoreSecurityFileLabel:143 >>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file >>>> or directory >>>> 23:21:09.364: 1309: warning : qemuDomainAttachPciDiskDevice:250 : >>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin >>>> 23:21:49.645: 1309: error : qemuMonitorTextAddDrive:2418 : operation >>>> failed: open disk image file failed >>>> 23:21:49.645: 1309: error : virSecurityDACRestoreSecurityFileLabel:143 >>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:id=admin: No such file >>>> or directory >>>> 23:21:49.853: 1309: warning : qemuDomainAttachPciDiskDevice:250 : >>>> Unable to restore security label on rbd:kvmtest/kvmtest1:id=admin >>>> 13:08:08.924: 1309: error : qemuMonitorTextAddDrive:2418 : operation >>>> failed: open disk image file failed >>>> 13:08:08.924: 1309: error : virSecurityDACRestoreSecurityFileLabel:143 >>>> : cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or >>>> directory >>>> 13:08:09.117: 1309: warning : qemuDomainAttachPciDiskDevice:250 : >>>> Unable to restore security label on rbd:kvmtest/kvmtest1 >>>> 13:09:45.402: 1309: error : qemuMonitorTextAddDrive:2418 : operation >>>> failed: open disk image file failed >>>> 13:29:05.554: 1310: error : qemuMonitorTextAddDrive:2418 : operation >>>> failed: open disk image file failed >>>> 13:29:05.554: 1310: error : virSecurityDACRestoreSecurityFileLabel:143 >>>> : cannot resolve symlink kvmtest/kvmtest1: No such file or directory >>>> 13:29:05.751: 1310: warning : qemuDomainAttachPciDiskDevice:250 : >>>> Unable to restore security label on kvmtest/kvmtest1 >>>> 13:29:05.751: 1310: warning : virEventPollUpdateHandle:147 : Ignoring >>>> invalid update watch -1 >>> >>> >>> >>> This is libvirt mistakenly trying to treat the rbd disk as a file - fixed >>> in >>> 20e1233c31e3150d259073c523077acdccb5d419 to libvirt. If you don't want to >>> upgrade libvirt, someone on IRC last week reported that installing >>> libapparmor1 fixed this problem for them, since libvirt no longer fell >>> back >>> to it's own DAC implementation. >> >> >> hmm is this commited to any stable version now ?? i have installed >> libapparmor1, >> >> ri libapparmor1 2.7.0~beta1+bzr1774-1ubuntu2 >> changehat AppArmor library >> >> and i use newest stable libvirt 0.9.8. >> >>> >>> >>>> 2011/12/1 Josh Durgin<josh.durgin@xxxxxxxxxxxxx>: >>>>> >>>>> >>>>> On 12/01/2011 11:37 AM, Sławomir Skowron wrote: >>>>>> >>>>>> >>>>>> >>>>>> I have some problems. Can you help me ?? >>>>>> >>>>>> ceph cluster: >>>>>> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine. >>>>>> >>>>>> kvm hypervisor: >>>>>> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu >>>>>> oneiric x86_64. >>>>>> >>>>>> I create image from qemu-img on machine with kvm VM's, and it works >>>>>> very >>>>>> well: >>>>>> >>>>>> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G >>>>>> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240 >>>>>> cluster_size=0 >>>>>> >>>>>> in ceph cluster machines: >>>>>> >>>>>> rados -p kvmtest ls >>>>>> kvmtest1.rbd >>>>>> rbd_directory >>>>>> rbd_info >>>>>> >>>>>> But when i try to attach new device to kvm VM like this: >>>>>> >>>>>> virsh attach-device one-10 /tmp/kvm.xml >>>>>> >>>>>> with kvm.xml like this: >>>>>> >>>>>> <disk type='network' device='disk'> >>>>>> <driver name='qemu' type='raw' cache='writeback'/> >>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'> >>>>>> <host name='10.177.64.4' port='6789'/> >>>>>> </source> >>>>>> <target dev='vdc' bus='virtio'/> >>>>>> </disk> >>>>>> >>>>>> output of virsh: >>>>>> >>>>>> error: Failed to attach device from /tmp/kvm.xml >>>>>> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or >>>>>> directory >>>>> >>>>> >>>>> >>>>> >>>>> From the monitor output below, I'm assuming this is just a bad error >>>>> message >>>>> from libvirt. >>>>> >>>>> >>>>>> and mon.0 log in ceph cluster. >>>>>> >>>>>> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1 >>>>>> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0 >>>>>> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<== >>>>>> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ==== >>>>>> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0 >>>>>> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch >>>>>> new session MonSession: client.? 10.177.32.66:0/25020608 is open for >>>>>> client.? 10.177.32.66:0/25020608 >>>>>> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146 >>>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.? >>>>>> 10.177.32.66:0/25020608 >>>>>> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 --> >>>>>> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not >>>>>> supported) v1 -- ?+0 0x2f30600 con 0x297adc0 >>>>> >>>>> >>>>> >>>>> >>>>> Here you can see qemu isn't authenticating correctly, probably because >>>>> the >>>>> rados user isn't being set by libvirt. With your libvirt version, >>>>> you'll >>>>> need to change: >>>>> >>>>> >>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'> >>>>> >>>>> to >>>>> >>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'> >>>>> >>>>> to authenticate as client.foo. Any ceph config option can be set like >>>>> that, >>>>> separated by colons. >>>>> >>>>> If this still doesn't work, I'd suggest setting >>>>> debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more >>>>> info >>>>> in >>>>> the libvirt instance log (usually >>>>> /var/log/libvirt/qemu/instance_name.log). >>>>> There may be apparmour restrictions preventing qemu from reading the >>>>> keyring >>>>> or ceph config files. >>>>> >>>>> >>>>>> ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm >>>>>> in same directory: >>>>>> ; global >>>>>> [global] >>>>>> ; enable secure authentication >>>>>> >>>>>> auth supported = cephx >>>>>> keyring = /etc/ceph/keyring.bin >>>>>> >>>>>> debug rgw = 1 >>>>>> >>>>>> rgw print continue = false >>>>>> rgw socket path = /var/run/radosgw.sock >>>>>> >>>>>> ; monitors >>>>>> ; You need at least one. You need at least three if you want to >>>>>> ; tolerate any node failures. Always create an odd number. >>>>>> [mon] >>>>>> mon data = /vol0/data/mon.$id >>>>>> >>>>>> ; some minimal logging (just message traffic) to aid debugging >>>>>> >>>>>> debug ms = 1 ; see message traffic >>>>>> debug mon = 0 ; monitor >>>>>> debug paxos = 0 ; monitor replication >>>>>> debug auth = 0 ; >>>>>> >>>>>> mon allowed clock drift = 2 >>>>>> >>>>>> [mon.0] >>>>>> host = 10-177-64-4 >>>>>> mon addr = 10.177.64.4:6789 >>>>>> >>>>>> ; radosgw client list >>>>>> [client.radosgw.10-177-64-4] >>>>>> >>>>>> host = 10-177-64-4 >>>>>> log file = /var/log/ceph/$name.log >>>>>> debug rgw = 1 >>>>>> debug ms = 1 >>>>>> >>>>>> ; osd >>>>>> ; You need at least one. Two if you want data to be replicated. >>>>>> ; Define as many as you like. >>>>>> [osd] >>>>>> ; This is where the btrfs volume will be mounted. >>>>>> >>>>>> osd data = /vol0/data/osd.$id >>>>>> >>>>>> ; Ideally, make this a separate disk or partition. A few GB >>>>>> ; is usually enough; more if you have fast disks. You can use >>>>>> ; a file under the osd data dir if need be >>>>>> ; (e.g. /data/osd$id/journal), but it will be slower than a >>>>>> ; separate disk or partition. >>>>>> >>>>>> osd journal = /vol0/data/osd.$id/journal >>>>>> >>>>>> ; If the OSD journal is a file, you need to specify the size. >>>>>> This is specified in MB. >>>>>> >>>>>> osd journal size = 512 >>>>>> >>>>>> filestore journal writeahead = 1 >>>>>> osd heartbeat grace = 5 >>>>>> >>>>>> debug ms = 0 ; message traffic >>>>>> debug osd = 0 >>>>>> debug filestore = 0 ; local object storage >>>>>> debug journal = 0 ; local journaling >>>>>> debug monc = 0 >>>>>> debug rados = 1 >>>>>> >>>>>> [osd.0] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.0 >>>>>> keyring = /vol0/data/osd.0/keyring >>>>>> >>>>>> [osd.1] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.1 >>>>>> keyring = /vol0/data/osd.1/keyring >>>>>> >>>>>> [osd.2] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.2 >>>>>> keyring = /vol0/data/osd.2/keyring >>>>>> >>>>>> [osd.3] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.3 >>>>>> keyring = /vol0/data/osd.3/keyring >>>>>> >>>>>> [osd.4] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.4 >>>>>> keyring = /vol0/data/osd.4/keyring >>>>>> >>>>>> [osd.5] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.5 >>>>>> keyring = /vol0/data/osd.5/keyring >>>>>> >>>>>> [osd.6] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.6 >>>>>> keyring = /vol0/data/osd.6/keyring >>>>>> >>>>>> [osd.7] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.7 >>>>>> keyring = /vol0/data/osd.7/keyring >>>>>> >>>>>> [osd.8] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.8 >>>>>> keyring = /vol0/data/osd.8/keyring >>>>>> >>>>>> [osd.9] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.9 >>>>>> keyring = /vol0/data/osd.9/keyring >>>>>> >>>>>> [osd.10] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.10 >>>>>> keyring = /vol0/data/osd.10/keyring >>>>>> >>>>>> [osd.11] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.11 >>>>>> keyring = /vol0/data/osd.11/keyring >>>>>> >>>>>> [osd.12] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.12 >>>>>> keyring = /vol0/data/osd.12/keyring >>>>>> >>>>>> [osd.13] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.13 >>>>>> keyring = /vol0/data/osd.13/keyring >>>>>> >>>>>> [osd.14] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.14 >>>>>> keyring = /vol0/data/osd.14/keyring >>>>>> >>>>>> [osd.15] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.15 >>>>>> keyring = /vol0/data/osd.15/keyring >>>>>> >>>>>> [osd.16] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.16 >>>>>> keyring = /vol0/data/osd.16/keyring >>>>>> >>>>>> [osd.17] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.17 >>>>>> keyring = /vol0/data/osd.17/keyring >>>>>> >>>>>> [osd.18] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.18 >>>>>> keyring = /vol0/data/osd.18/keyring >>>>>> >>>>>> [osd.19] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.19 >>>>>> keyring = /vol0/data/osd.19/keyring >>>>>> >>>>>> [osd.20] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.20 >>>>>> keyring = /vol0/data/osd.20/keyring >>>>>> >>>>>> [osd.21] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.21 >>>>>> keyring = /vol0/data/osd.21/keyring >>>>>> >>>>>> [osd.22] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.22 >>>>>> keyring = /vol0/data/osd.22/keyring >>>>>> >>>>>> [osd.23] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.23 >>>>>> keyring = /vol0/data/osd.23/keyring >>>>>> >>>>>> [osd.24] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.24 >>>>>> keyring = /vol0/data/osd.24/keyring >>>>>> >>>>>> [osd.25] >>>>>> host = 10-177-64-4 >>>>>> osd data = /vol0/data/osd.25 >>>>>> keyring = /vol0/data/osd.25/keyring >>>>>> >>>>>> >>>>> >>>> >>>> >>>> >>> >> >> >> > -- ----- Pozdrawiam Sławek "sZiBis" Skowron -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
