Sorry, for my lag, but i was sick.
I handled problem with apparmor before and now it's not a problem,
even when i send mail before, it was solved.
Ok when i create image from qemu-img it's looks like that, and works
perfect, (attachment with log of this opperation.)
But when i use virsh, it's still a problem. Just like he don't know
what user is going to be auth.
I try, as client.admin (default i hope :)) only with id=admin in
kvm.xml, or client=admin, and other combinations - like client.rbd
etc., Still nothing.
I try, improve a keyring file, adding client rbd into keyring on
machine with kvm hypervisor in /etc/ceph/keyring.bin
[client.rbd]
key = AAAAAAAAAAAAAAAA
auid = 18446744073709551615
caps mds = "allow"
caps mon = "allow r"
caps osd = "allow rw pool=kvmtest"
and on cluster site like above. I was added, a section [client.rbd]
with keyring path, and it's still a problem.
2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
root@s3-10-177-64-4:~# tail -n10000 /var/log/ceph/mon.0.log | grep 13:29:11.
2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
2011-12-09 13:29:11.519624 7fe61b090700 -- 10.177.64.4:6789/0 <==
client.? 10.177.32.66:0/58020608 1 ==== auth(proto 0 21 bytes) v1 ====
47+0+0 (3310669357 0 0) 0x17f2e00 con 0x19d2c80
2011-12-09 13:29:11.519645 7fe61b090700 mon.0@0(leader) e1 have connection
2011-12-09 13:29:11.519655 7fe61b090700 mon.0@0(leader) e1 do not have
session, making new one
2011-12-09 13:29:11.519668 7fe61b090700 mon.0@0(leader) e1 ms_dispatch
new session MonSession: client.? 10.177.32.66:0/58020608 is open for
client.? 10.177.32.66:0/58020608
2011-12-09 13:29:11.519674 7fe61b090700 mon.0@0(leader) e1 setting
timeout on session
2011-12-09 13:29:11.519680 7fe61b090700 mon.0@0(leader) e1 caps
2011-12-09 13:29:11.519688 7fe61b090700 mon.0@0(leader).auth v353
AuthMonitor::update_from_paxos()
2011-12-09 13:29:11.519697 7fe61b090700 mon.0@0(leader).auth v353
preprocess_query auth(proto 0 21 bytes) v1 from client.?
10.177.32.66:0/58020608
2011-12-09 13:29:11.519703 7fe61b090700 mon.0@0(leader).auth v353
prep_auth() blob_size=21
2011-12-09 13:29:11.519735 7fe61b090700 -- 10.177.64.4:6789/0 -->
10.177.32.66:0/58020608 -- auth_reply(proto 0 -95 Operation not
supported) v1 -- ?+0 0x1246200 con 0x19d2c80
from libvirt.log
23:20:56.689: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
: cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
or directory
23:20:56.882: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
23:21:09.172: 1309: error : qemuMonitorTextAddDrive:2418 : operation
failed: open disk image file failed
23:21:09.172: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
: cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
or directory
23:21:09.364: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
23:21:49.645: 1309: error : qemuMonitorTextAddDrive:2418 : operation
failed: open disk image file failed
23:21:49.645: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
: cannot resolve symlink rbd:kvmtest/kvmtest1:id=admin: No such file
or directory
23:21:49.853: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
Unable to restore security label on rbd:kvmtest/kvmtest1:id=admin
13:08:08.924: 1309: error : qemuMonitorTextAddDrive:2418 : operation
failed: open disk image file failed
13:08:08.924: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
directory
13:08:09.117: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
Unable to restore security label on rbd:kvmtest/kvmtest1
13:09:45.402: 1309: error : qemuMonitorTextAddDrive:2418 : operation
failed: open disk image file failed
13:29:05.554: 1310: error : qemuMonitorTextAddDrive:2418 : operation
failed: open disk image file failed
13:29:05.554: 1310: error : virSecurityDACRestoreSecurityFileLabel:143
: cannot resolve symlink kvmtest/kvmtest1: No such file or directory
13:29:05.751: 1310: warning : qemuDomainAttachPciDiskDevice:250 :
Unable to restore security label on kvmtest/kvmtest1
13:29:05.751: 1310: warning : virEventPollUpdateHandle:147 : Ignoring
invalid update watch -1
2011/12/1 Josh Durgin <josh.durgin@xxxxxxxxxxxxx>:
> On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
>>
>> I have some problems. Can you help me ??
>>
>> ceph cluster:
>> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
>>
>> kvm hypervisor:
>> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
>> oneiric x86_64.
>>
>> I create image from qemu-img on machine with kvm VM's, and it works very
>> well:
>>
>> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
>> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240 cluster_size=0
>>
>> in ceph cluster machines:
>>
>> rados -p kvmtest ls
>> kvmtest1.rbd
>> rbd_directory
>> rbd_info
>>
>> But when i try to attach new device to kvm VM like this:
>>
>> virsh attach-device one-10 /tmp/kvm.xml
>>
>> with kvm.xml like this:
>>
>> <disk type='network' device='disk'>
>> <driver name='qemu' type='raw' cache='writeback'/>
>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>> <host name='10.177.64.4' port='6789'/>
>> </source>
>> <target dev='vdc' bus='virtio'/>
>> </disk>
>>
>> output of virsh:
>>
>> error: Failed to attach device from /tmp/kvm.xml
>> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>> directory
>
>
> From the monitor output below, I'm assuming this is just a bad error message
> from libvirt.
>
>
>> and mon.0 log in ceph cluster.
>>
>> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
>> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
>> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
>> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
>> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
>> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
>> new session MonSession: client.? 10.177.32.66:0/25020608 is open for
>> client.? 10.177.32.66:0/25020608
>> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>> 10.177.32.66:0/25020608
>> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
>> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
>> supported) v1 -- ?+0 0x2f30600 con 0x297adc0
>
>
> Here you can see qemu isn't authenticating correctly, probably because the
> rados user isn't being set by libvirt. With your libvirt version, you'll
> need to change:
>
>
> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>
> to
>
> <source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>
>
> to authenticate as client.foo. Any ceph config option can be set like that,
> separated by colons.
>
> If this still doesn't work, I'd suggest setting
> debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more info in
> the libvirt instance log (usually /var/log/libvirt/qemu/instance_name.log).
> There may be apparmour restrictions preventing qemu from reading the keyring
> or ceph config files.
>
>
>> ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
>> in same directory:
>> ; global
>> [global]
>> ; enable secure authentication
>>
>> auth supported = cephx
>> keyring = /etc/ceph/keyring.bin
>>
>> debug rgw = 1
>>
>> rgw print continue = false
>> rgw socket path = /var/run/radosgw.sock
>>
>> ; monitors
>> ; You need at least one. You need at least three if you want to
>> ; tolerate any node failures. Always create an odd number.
>> [mon]
>> mon data = /vol0/data/mon.$id
>>
>> ; some minimal logging (just message traffic) to aid debugging
>>
>> debug ms = 1 ; see message traffic
>> debug mon = 0 ; monitor
>> debug paxos = 0 ; monitor replication
>> debug auth = 0 ;
>>
>> mon allowed clock drift = 2
>>
>> [mon.0]
>> host = 10-177-64-4
>> mon addr = 10.177.64.4:6789
>>
>> ; radosgw client list
>> [client.radosgw.10-177-64-4]
>>
>> host = 10-177-64-4
>> log file = /var/log/ceph/$name.log
>> debug rgw = 1
>> debug ms = 1
>>
>> ; osd
>> ; You need at least one. Two if you want data to be replicated.
>> ; Define as many as you like.
>> [osd]
>> ; This is where the btrfs volume will be mounted.
>>
>> osd data = /vol0/data/osd.$id
>>
>> ; Ideally, make this a separate disk or partition. A few GB
>> ; is usually enough; more if you have fast disks. You can use
>> ; a file under the osd data dir if need be
>> ; (e.g. /data/osd$id/journal), but it will be slower than a
>> ; separate disk or partition.
>>
>> osd journal = /vol0/data/osd.$id/journal
>>
>> ; If the OSD journal is a file, you need to specify the size.
>> This is specified in MB.
>>
>> osd journal size = 512
>>
>> filestore journal writeahead = 1
>> osd heartbeat grace = 5
>>
>> debug ms = 0 ; message traffic
>> debug osd = 0
>> debug filestore = 0 ; local object storage
>> debug journal = 0 ; local journaling
>> debug monc = 0
>> debug rados = 1
>>
>> [osd.0]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.0
>> keyring = /vol0/data/osd.0/keyring
>>
>> [osd.1]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.1
>> keyring = /vol0/data/osd.1/keyring
>>
>> [osd.2]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.2
>> keyring = /vol0/data/osd.2/keyring
>>
>> [osd.3]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.3
>> keyring = /vol0/data/osd.3/keyring
>>
>> [osd.4]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.4
>> keyring = /vol0/data/osd.4/keyring
>>
>> [osd.5]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.5
>> keyring = /vol0/data/osd.5/keyring
>>
>> [osd.6]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.6
>> keyring = /vol0/data/osd.6/keyring
>>
>> [osd.7]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.7
>> keyring = /vol0/data/osd.7/keyring
>>
>> [osd.8]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.8
>> keyring = /vol0/data/osd.8/keyring
>>
>> [osd.9]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.9
>> keyring = /vol0/data/osd.9/keyring
>>
>> [osd.10]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.10
>> keyring = /vol0/data/osd.10/keyring
>>
>> [osd.11]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.11
>> keyring = /vol0/data/osd.11/keyring
>>
>> [osd.12]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.12
>> keyring = /vol0/data/osd.12/keyring
>>
>> [osd.13]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.13
>> keyring = /vol0/data/osd.13/keyring
>>
>> [osd.14]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.14
>> keyring = /vol0/data/osd.14/keyring
>>
>> [osd.15]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.15
>> keyring = /vol0/data/osd.15/keyring
>>
>> [osd.16]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.16
>> keyring = /vol0/data/osd.16/keyring
>>
>> [osd.17]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.17
>> keyring = /vol0/data/osd.17/keyring
>>
>> [osd.18]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.18
>> keyring = /vol0/data/osd.18/keyring
>>
>> [osd.19]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.19
>> keyring = /vol0/data/osd.19/keyring
>>
>> [osd.20]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.20
>> keyring = /vol0/data/osd.20/keyring
>>
>> [osd.21]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.21
>> keyring = /vol0/data/osd.21/keyring
>>
>> [osd.22]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.22
>> keyring = /vol0/data/osd.22/keyring
>>
>> [osd.23]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.23
>> keyring = /vol0/data/osd.23/keyring
>>
>> [osd.24]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.24
>> keyring = /vol0/data/osd.24/keyring
>>
>> [osd.25]
>> host = 10-177-64-4
>> osd data = /vol0/data/osd.25
>> keyring = /vol0/data/osd.25/keyring
>>
>>
>
--
-----
Pozdrawiam
Sławek "sZiBis" Skowron
Attachment:
mon.0.log
Description: Binary data
