Hi,
Actual setup:
ii libvirt-bin 0.9.2-4ubuntu15.1
the programs for the libvirt library
ii libvirt0 0.9.2-4ubuntu15.1
library for interfacing with different virtualization systems
ii qemu-kvm 1.0.0+dfsg+rc2-1~oneiric1
Full virtualization on i386 and amd64 hardware
I change many thing in system, and successfully attached rbd drives via libvirt:
virsh attach-device one-888 kvm-ceph.xml
Device attached successfully
A new set of disks, appears in libvirt vm xml, but not in vm machine.
After reboot, all new rbd drives appear in VM.
Is there any chance to hotadd rbd device to working VM without reboot ??
2011/12/16 Sławomir Skowron <szibis@xxxxxxxxx>:
> 2011/12/13 Josh Durgin <josh.durgin@xxxxxxxxxxxxx>:
>> On 12/13/2011 04:56 AM, Sławomir Skowron wrote:
>>>
>>> Finally, i manage the problem with rbd with kvm 1.0, and libvirt
>>> 0.9.8, or i think i manage :), but i get stuck with one thing after.
>>>
>>> 2011-12-13 12:13:31.173+0000: 21512: error :
>>> qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU
>>> command 'device_add': Bus 'pci.0' does not support hotplugging
>>> 2011-12-13 12:13:31.173+0000: 21512: warning :
>>> qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on
>>>
>>> file=rbd:rbd/testdysk:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx
>>> none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk2,format=raw
>>>
>>> (virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2)
>>> 2011-12-13 12:13:31.173+0000: 21512: error :
>>> virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink
>>> rbd/testdysk: No such file or directory2011-12-13 12:13:31.173+0000:
>>> 21512: warning : qemuDomainAttachPciDiskDevice:287 : Unable to restore
>>> security label on rbd/testdysk
>>> and i can't find any info about that.
>>>
>>> my kvm.xml inject file:
>>>
>>> <disk type="network" device="disk">
>>> <driver name="qemu" type="raw"/>
>>> <auth username="admin">
>>> <secret type="ceph"
>>> uuid="0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f"/>
>>> </auth>
>>> <source protocol="rbd" name="rbd/testdysk">
>>> <host name="10.177.64.4" port="6789"/>
>>> </source>
>>> <target dev="sdc" bus="virtio"/>
>>> </disk>
>>>
>>> secret:
>>>
>>> cat /tmp/secret.xml
>>> <secret ephemeral='no' private='no'>
>>> <uuid>0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f</uuid>
>>> <usage type='ceph'>
>>> <domain>rbd/admin</domain>
>>> <name>admin</name>
>>> </usage>
>>> </secret>
>>>
>>> virsh secret-define /tmp/secret.xml
>>> Secret 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f created
>>>
>>> virsh secret-set-value 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f
>>> AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==
>>> Secret value set
>>>
>>
>> Your xml and configuration are all fine, you're hitting the libvirt bug that
>> I mentioned before. The fix isn't in any stable release yet, and 0.9.8 was
>> just released yesterday, so I guess you'll have to compile it yourself or
>> disable the libvirt security driver.
>
> Yes libvirt 0.9.8 with this patch solved problem with rbd, now it's
> clean. Thanks for that.
>
> But, a second problem appear.
>
> error: Failed to attach device from /root/kvm-ceph.xml
> error: internal error unable to execute QEMU command 'device_add':
> Property 'virtio-blk-pci.drive' can't find value 'drive-virtio-disk3'
>
> /var/log/libvirt/libvirtd.log
> 2011-12-16 12:03:04.950+0000: 1070: error :
> qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU
> command 'device_add': Property 'virtio-blk-pci.drive' can't find value
> 'drive-virtio-disk3'
> 2011-12-16 12:03:04.950+0000: 1070: warning :
> qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on
> file=rbd:rbd/testdysk:rbd_writeback_window=8000000:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx
> none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk3,format=raw
> (virtio-blk-pci,bus=pci.0,addr=0xd,drive=drive-virtio-disk3,id=virtio-disk3)
>
> if i attach a scsi device from img file everything is OK.
>
> Anyone have, any concept with this ??
>
>>
>>
>>> 2011/12/12 Josh Durgin<josh.durgin@xxxxxxxxxxxxx>:
>>>>
>>>> On 12/09/2011 04:48 AM, Sławomir Skowron wrote:
>>>>>
>>>>>
>>>>> Sorry, for my lag, but i was sick.
>>>>>
>>>>> I handled problem with apparmor before and now it's not a problem,
>>>>> even when i send mail before, it was solved.
>>>>>
>>>>> Ok when i create image from qemu-img it's looks like that, and works
>>>>> perfect, (attachment with log of this opperation.)
>>>>>
>>>>> But when i use virsh, it's still a problem. Just like he don't know
>>>>> what user is going to be auth.
>>>>>
>>>>> I try, as client.admin (default i hope :)) only with id=admin in
>>>>> kvm.xml, or client=admin, and other combinations - like client.rbd
>>>>> etc., Still nothing.
>>>>
>>>>
>>>>
>>>> admin is the default, and the correct option for client.admin is
>>>> id=admin,
>>>> or id=foo for client.foo.
>>>>
>>>>
>>>>>
>>>>> I try, improve a keyring file, adding client rbd into keyring on
>>>>> machine with kvm hypervisor in /etc/ceph/keyring.bin
>>>>>
>>>>> [client.rbd]
>>>>> key = AAAAAAAAAAAAAAAA
>>>>> auid = 18446744073709551615
>>>>> caps mds = "allow"
>>>>> caps mon = "allow r"
>>>>> caps osd = "allow rw pool=kvmtest"
>>>>>
>>>>> and on cluster site like above. I was added, a section [client.rbd]
>>>>> with keyring path, and it's still a problem.
>>>>>
>>>>
>>>> Did you run 'ceph auth add -i /etc/ceph/keyring.bin'?
>>>> Make sure 'ceph auth list' shows client.rbd with the right key.
>>>
>>>
>>> Overall i want to use admin id, and i think now it's correctly used in
>>> xml.
>>>
>>>>
>>>>
>>>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>>>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>>>>> root@s3-10-177-64-4:~# tail -n10000 /var/log/ceph/mon.0.log | grep
>>>>> 13:29:11.
>>>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1
>>>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0
>>>>> 2011-12-09 13:29:11.519624 7fe61b090700 -- 10.177.64.4:6789/0<==
>>>>> client.? 10.177.32.66:0/58020608 1 ==== auth(proto 0 21 bytes) v1 ====
>>>>> 47+0+0 (3310669357 0 0) 0x17f2e00 con 0x19d2c80
>>>>> 2011-12-09 13:29:11.519645 7fe61b090700 mon.0@0(leader) e1 have
>>>>> connection
>>>>> 2011-12-09 13:29:11.519655 7fe61b090700 mon.0@0(leader) e1 do not have
>>>>> session, making new one
>>>>> 2011-12-09 13:29:11.519668 7fe61b090700 mon.0@0(leader) e1 ms_dispatch
>>>>> new session MonSession: client.? 10.177.32.66:0/58020608 is open for
>>>>> client.? 10.177.32.66:0/58020608
>>>>> 2011-12-09 13:29:11.519674 7fe61b090700 mon.0@0(leader) e1 setting
>>>>> timeout on session
>>>>> 2011-12-09 13:29:11.519680 7fe61b090700 mon.0@0(leader) e1 caps
>>>>> 2011-12-09 13:29:11.519688 7fe61b090700 mon.0@0(leader).auth v353
>>>>> AuthMonitor::update_from_paxos()
>>>>> 2011-12-09 13:29:11.519697 7fe61b090700 mon.0@0(leader).auth v353
>>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>>>> 10.177.32.66:0/58020608
>>>>> 2011-12-09 13:29:11.519703 7fe61b090700 mon.0@0(leader).auth v353
>>>>> prep_auth() blob_size=21
>>>>> 2011-12-09 13:29:11.519735 7fe61b090700 -- 10.177.64.4:6789/0 -->
>>>>> 10.177.32.66:0/58020608 -- auth_reply(proto 0 -95 Operation not
>>>>> supported) v1 -- ?+0 0x1246200 con 0x19d2c80
>>>>>
>>>>> from libvirt.log
>>>>>
>>>>> 23:20:56.689: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
>>>>> or directory
>>>>> 23:20:56.882: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>>>>> 23:21:09.172: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>>> failed: open disk image file failed
>>>>> 23:21:09.172: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such file
>>>>> or directory
>>>>> 23:21:09.364: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin
>>>>> 23:21:49.645: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>>> failed: open disk image file failed
>>>>> 23:21:49.645: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:id=admin: No such file
>>>>> or directory
>>>>> 23:21:49.853: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:id=admin
>>>>> 13:08:08.924: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>>> failed: open disk image file failed
>>>>> 13:08:08.924: 1309: error : virSecurityDACRestoreSecurityFileLabel:143
>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>>>> directory
>>>>> 13:08:09.117: 1309: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1
>>>>> 13:09:45.402: 1309: error : qemuMonitorTextAddDrive:2418 : operation
>>>>> failed: open disk image file failed
>>>>> 13:29:05.554: 1310: error : qemuMonitorTextAddDrive:2418 : operation
>>>>> failed: open disk image file failed
>>>>> 13:29:05.554: 1310: error : virSecurityDACRestoreSecurityFileLabel:143
>>>>> : cannot resolve symlink kvmtest/kvmtest1: No such file or directory
>>>>> 13:29:05.751: 1310: warning : qemuDomainAttachPciDiskDevice:250 :
>>>>> Unable to restore security label on kvmtest/kvmtest1
>>>>> 13:29:05.751: 1310: warning : virEventPollUpdateHandle:147 : Ignoring
>>>>> invalid update watch -1
>>>>
>>>>
>>>>
>>>> This is libvirt mistakenly trying to treat the rbd disk as a file - fixed
>>>> in
>>>> 20e1233c31e3150d259073c523077acdccb5d419 to libvirt. If you don't want to
>>>> upgrade libvirt, someone on IRC last week reported that installing
>>>> libapparmor1 fixed this problem for them, since libvirt no longer fell
>>>> back
>>>> to it's own DAC implementation.
>>>
>>>
>>> hmm is this commited to any stable version now ?? i have installed
>>> libapparmor1,
>>>
>>> ri libapparmor1 2.7.0~beta1+bzr1774-1ubuntu2
>>> changehat AppArmor library
>>>
>>> and i use newest stable libvirt 0.9.8.
>>>
>>>>
>>>>
>>>>> 2011/12/1 Josh Durgin<josh.durgin@xxxxxxxxxxxxx>:
>>>>>>
>>>>>>
>>>>>> On 12/01/2011 11:37 AM, Sławomir Skowron wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have some problems. Can you help me ??
>>>>>>>
>>>>>>> ceph cluster:
>>>>>>> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine.
>>>>>>>
>>>>>>> kvm hypervisor:
>>>>>>> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu
>>>>>>> oneiric x86_64.
>>>>>>>
>>>>>>> I create image from qemu-img on machine with kvm VM's, and it works
>>>>>>> very
>>>>>>> well:
>>>>>>>
>>>>>>> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G
>>>>>>> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240
>>>>>>> cluster_size=0
>>>>>>>
>>>>>>> in ceph cluster machines:
>>>>>>>
>>>>>>> rados -p kvmtest ls
>>>>>>> kvmtest1.rbd
>>>>>>> rbd_directory
>>>>>>> rbd_info
>>>>>>>
>>>>>>> But when i try to attach new device to kvm VM like this:
>>>>>>>
>>>>>>> virsh attach-device one-10 /tmp/kvm.xml
>>>>>>>
>>>>>>> with kvm.xml like this:
>>>>>>>
>>>>>>> <disk type='network' device='disk'>
>>>>>>> <driver name='qemu' type='raw' cache='writeback'/>
>>>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>>>>> <host name='10.177.64.4' port='6789'/>
>>>>>>> </source>
>>>>>>> <target dev='vdc' bus='virtio'/>
>>>>>>> </disk>
>>>>>>>
>>>>>>> output of virsh:
>>>>>>>
>>>>>>> error: Failed to attach device from /tmp/kvm.xml
>>>>>>> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or
>>>>>>> directory
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> From the monitor output below, I'm assuming this is just a bad error
>>>>>> message
>>>>>> from libvirt.
>>>>>>
>>>>>>
>>>>>>> and mon.0 log in ceph cluster.
>>>>>>>
>>>>>>> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1
>>>>>>> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0
>>>>>>> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<==
>>>>>>> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 ====
>>>>>>> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0
>>>>>>> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 ms_dispatch
>>>>>>> new session MonSession: client.? 10.177.32.66:0/25020608 is open for
>>>>>>> client.? 10.177.32.66:0/25020608
>>>>>>> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146
>>>>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.?
>>>>>>> 10.177.32.66:0/25020608
>>>>>>> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 -->
>>>>>>> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not
>>>>>>> supported) v1 -- ?+0 0x2f30600 con 0x297adc0
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Here you can see qemu isn't authenticating correctly, probably because
>>>>>> the
>>>>>> rados user isn't being set by libvirt. With your libvirt version,
>>>>>> you'll
>>>>>> need to change:
>>>>>>
>>>>>>
>>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'>
>>>>>>
>>>>>> to
>>>>>>
>>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'>
>>>>>>
>>>>>> to authenticate as client.foo. Any ceph config option can be set like
>>>>>> that,
>>>>>> separated by colons.
>>>>>>
>>>>>> If this still doesn't work, I'd suggest setting
>>>>>> debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more
>>>>>> info
>>>>>> in
>>>>>> the libvirt instance log (usually
>>>>>> /var/log/libvirt/qemu/instance_name.log).
>>>>>> There may be apparmour restrictions preventing qemu from reading the
>>>>>> keyring
>>>>>> or ceph config files.
>>>>>>
>>>>>>
>>>>>>> ceph.conf, and keyring.bin exists on server, and machine with qemu-kvm
>>>>>>> in same directory:
>>>>>>> ; global
>>>>>>> [global]
>>>>>>> ; enable secure authentication
>>>>>>>
>>>>>>> auth supported = cephx
>>>>>>> keyring = /etc/ceph/keyring.bin
>>>>>>>
>>>>>>> debug rgw = 1
>>>>>>>
>>>>>>> rgw print continue = false
>>>>>>> rgw socket path = /var/run/radosgw.sock
>>>>>>>
>>>>>>> ; monitors
>>>>>>> ; You need at least one. You need at least three if you want to
>>>>>>> ; tolerate any node failures. Always create an odd number.
>>>>>>> [mon]
>>>>>>> mon data = /vol0/data/mon.$id
>>>>>>>
>>>>>>> ; some minimal logging (just message traffic) to aid debugging
>>>>>>>
>>>>>>> debug ms = 1 ; see message traffic
>>>>>>> debug mon = 0 ; monitor
>>>>>>> debug paxos = 0 ; monitor replication
>>>>>>> debug auth = 0 ;
>>>>>>>
>>>>>>> mon allowed clock drift = 2
>>>>>>>
>>>>>>> [mon.0]
>>>>>>> host = 10-177-64-4
>>>>>>> mon addr = 10.177.64.4:6789
>>>>>>>
>>>>>>> ; radosgw client list
>>>>>>> [client.radosgw.10-177-64-4]
>>>>>>>
>>>>>>> host = 10-177-64-4
>>>>>>> log file = /var/log/ceph/$name.log
>>>>>>> debug rgw = 1
>>>>>>> debug ms = 1
>>>>>>>
>>>>>>> ; osd
>>>>>>> ; You need at least one. Two if you want data to be replicated.
>>>>>>> ; Define as many as you like.
>>>>>>> [osd]
>>>>>>> ; This is where the btrfs volume will be mounted.
>>>>>>>
>>>>>>> osd data = /vol0/data/osd.$id
>>>>>>>
>>>>>>> ; Ideally, make this a separate disk or partition. A few GB
>>>>>>> ; is usually enough; more if you have fast disks. You can use
>>>>>>> ; a file under the osd data dir if need be
>>>>>>> ; (e.g. /data/osd$id/journal), but it will be slower than a
>>>>>>> ; separate disk or partition.
>>>>>>>
>>>>>>> osd journal = /vol0/data/osd.$id/journal
>>>>>>>
>>>>>>> ; If the OSD journal is a file, you need to specify the size.
>>>>>>> This is specified in MB.
>>>>>>>
>>>>>>> osd journal size = 512
>>>>>>>
>>>>>>> filestore journal writeahead = 1
>>>>>>> osd heartbeat grace = 5
>>>>>>>
>>>>>>> debug ms = 0 ; message traffic
>>>>>>> debug osd = 0
>>>>>>> debug filestore = 0 ; local object storage
>>>>>>> debug journal = 0 ; local journaling
>>>>>>> debug monc = 0
>>>>>>> debug rados = 1
>>>>>>>
>>>>>>> [osd.0]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.0
>>>>>>> keyring = /vol0/data/osd.0/keyring
>>>>>>>
>>>>>>> [osd.1]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.1
>>>>>>> keyring = /vol0/data/osd.1/keyring
>>>>>>>
>>>>>>> [osd.2]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.2
>>>>>>> keyring = /vol0/data/osd.2/keyring
>>>>>>>
>>>>>>> [osd.3]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.3
>>>>>>> keyring = /vol0/data/osd.3/keyring
>>>>>>>
>>>>>>> [osd.4]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.4
>>>>>>> keyring = /vol0/data/osd.4/keyring
>>>>>>>
>>>>>>> [osd.5]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.5
>>>>>>> keyring = /vol0/data/osd.5/keyring
>>>>>>>
>>>>>>> [osd.6]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.6
>>>>>>> keyring = /vol0/data/osd.6/keyring
>>>>>>>
>>>>>>> [osd.7]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.7
>>>>>>> keyring = /vol0/data/osd.7/keyring
>>>>>>>
>>>>>>> [osd.8]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.8
>>>>>>> keyring = /vol0/data/osd.8/keyring
>>>>>>>
>>>>>>> [osd.9]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.9
>>>>>>> keyring = /vol0/data/osd.9/keyring
>>>>>>>
>>>>>>> [osd.10]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.10
>>>>>>> keyring = /vol0/data/osd.10/keyring
>>>>>>>
>>>>>>> [osd.11]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.11
>>>>>>> keyring = /vol0/data/osd.11/keyring
>>>>>>>
>>>>>>> [osd.12]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.12
>>>>>>> keyring = /vol0/data/osd.12/keyring
>>>>>>>
>>>>>>> [osd.13]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.13
>>>>>>> keyring = /vol0/data/osd.13/keyring
>>>>>>>
>>>>>>> [osd.14]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.14
>>>>>>> keyring = /vol0/data/osd.14/keyring
>>>>>>>
>>>>>>> [osd.15]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.15
>>>>>>> keyring = /vol0/data/osd.15/keyring
>>>>>>>
>>>>>>> [osd.16]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.16
>>>>>>> keyring = /vol0/data/osd.16/keyring
>>>>>>>
>>>>>>> [osd.17]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.17
>>>>>>> keyring = /vol0/data/osd.17/keyring
>>>>>>>
>>>>>>> [osd.18]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.18
>>>>>>> keyring = /vol0/data/osd.18/keyring
>>>>>>>
>>>>>>> [osd.19]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.19
>>>>>>> keyring = /vol0/data/osd.19/keyring
>>>>>>>
>>>>>>> [osd.20]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.20
>>>>>>> keyring = /vol0/data/osd.20/keyring
>>>>>>>
>>>>>>> [osd.21]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.21
>>>>>>> keyring = /vol0/data/osd.21/keyring
>>>>>>>
>>>>>>> [osd.22]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.22
>>>>>>> keyring = /vol0/data/osd.22/keyring
>>>>>>>
>>>>>>> [osd.23]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.23
>>>>>>> keyring = /vol0/data/osd.23/keyring
>>>>>>>
>>>>>>> [osd.24]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.24
>>>>>>> keyring = /vol0/data/osd.24/keyring
>>>>>>>
>>>>>>> [osd.25]
>>>>>>> host = 10-177-64-4
>>>>>>> osd data = /vol0/data/osd.25
>>>>>>> keyring = /vol0/data/osd.25/keyring
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>
>
>
> --
> -----
> Pozdrawiam
>
> Sławek "sZiBis" Skowron
--
-----
Pozdrawiam
Sławek "sZiBis" Skowron
2011-12-19 16:16:13.434408 7f904f32b700 mon.0@0(leader) e1 ms_verify_authorizer 10.177.32.66:0/13014391 client protocol 0
2011-12-19 16:16:13.434797 7f905426c700 -- 10.177.64.4:6789/0 <== client.? 10.177.32.66:0/13014391 1 ==== auth(proto 0 30 bytes) v1 ==== 56+0+0 (625540289 0 0) 0x11fc000 con 0xcd7280
2011-12-19 16:16:13.434814 7f905426c700 mon.0@0(leader) e1 have connection
2011-12-19 16:16:13.434821 7f905426c700 mon.0@0(leader) e1 do not have session, making new one
2011-12-19 16:16:13.434833 7f905426c700 mon.0@0(leader) e1 ms_dispatch new session MonSession: client.? 10.177.32.66:0/13014391 is open for client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.434838 7f905426c700 mon.0@0(leader) e1 setting timeout on session
2011-12-19 16:16:13.434844 7f905426c700 mon.0@0(leader) e1 caps
2011-12-19 16:16:13.434852 7f905426c700 mon.0@0(leader).auth v626 update_from_paxos()
2011-12-19 16:16:13.434861 7f905426c700 mon.0@0(leader).auth v626 preprocess_query auth(proto 0 30 bytes) v1 from client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.434867 7f905426c700 mon.0@0(leader).auth v626 prep_auth() blob_size=30
2011-12-19 16:16:13.434890 7f905426c700 mon.0@0(leader).auth v626 AuthMonitor::assign_global_id m=auth(proto 0 30 bytes) v1 mon=0/1 last_allocated=6252 max_global_id=6296
2011-12-19 16:16:13.434896 7f905426c700 mon.0@0(leader).auth v626 next_global_id should be 6253
2011-12-19 16:16:13.434936 7f905426c700 cephx server client.admin: start_session server_challenge 543a42451923b4cf
2011-12-19 16:16:13.434963 7f905426c700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/13014391 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x19b4a00 con 0xcd7280
2011-12-19 16:16:13.435651 7f905426c700 -- 10.177.64.4:6789/0 <== client.? 10.177.32.66:0/13014391 2 ==== auth(proto 2 32 bytes) v1 ==== 58+0+0 (2091681013 0 0) 0xd9c200 con 0xcd7280
2011-12-19 16:16:13.435667 7f905426c700 mon.0@0(leader) e1 have connection
2011-12-19 16:16:13.435678 7f905426c700 mon.0@0(leader) e1 ms_dispatch existing session MonSession: client.? 10.177.32.66:0/13014391 is open for client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.435684 7f905426c700 mon.0@0(leader) e1 caps
2011-12-19 16:16:13.435691 7f905426c700 mon.0@0(leader).auth v626 update_from_paxos()
2011-12-19 16:16:13.435700 7f905426c700 mon.0@0(leader).auth v626 preprocess_query auth(proto 2 32 bytes) v1 from client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.435706 7f905426c700 mon.0@0(leader).auth v626 prep_auth() blob_size=32
2011-12-19 16:16:13.435712 7f905426c700 cephx server client.admin: handle_request get_auth_session_key for client.admin
2011-12-19 16:16:13.435757 7f905426c700 cephx server client.admin: checking key: req.key=e84566446375700e expected_key=e84566446375700e
2011-12-19 16:16:13.435796 7f905426c700 cephx keyserverdata: get_service_secret service auth id 46 AQBeee5OgBTXBxAAOW5Tzjrs3mIzVrZ8N+lfsg== expires 2011-12-20 00:38:06.131500
2011-12-19 16:16:13.435811 7f905426c700 cephx: build_service_ticket_reply encoding 1 tickets with secret AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==
2011-12-19 16:16:13.435831 7f905426c700 cephx: build_service_ticket service auth secret_id 46 ticket_info.ticket.name=client.admin
2011-12-19 16:16:13.435854 7f905426c700 cephx: service_ticket_blob is 0000 : 01 2e 00 00 00 00 00 00 00 60 00 00 00 19 86 a9 : .........`......
2011-12-19 16:16:13.435863 010 : dc d6 55 ef 5b 8d d5 54 17 68 0c 82 c2 de 57 e4 : ..U.[..T.h....W.
2011-12-19 16:16:13.435871 020 : 78 21 f3 d0 71 ef a3 17 c6 f5 88 24 e1 0b f9 ea : x!..q......$....
2011-12-19 16:16:13.435878 030 : 79 c5 b2 be 36 d3 67 09 76 e1 88 47 75 da ec 4f : y...6.g.v..Gu..O
2011-12-19 16:16:13.435886 040 : ae 75 f1 b4 24 33 a2 fc f7 f9 fd 20 a2 d5 f4 5a : .u..$3..... ...Z
2011-12-19 16:16:13.435894 050 : 1c 56 03 43 9b 1b 36 af d7 1e 99 76 22 63 83 99 : .V.C..6....v"c..
2011-12-19 16:16:13.435901 060 : ac af a4 9c 09 67 09 a4 82 7e 06 c4 6b : .....g...~..k
2011-12-19 16:16:13.435905 2011-12-19 16:16:13.435912 7f905426c700 cephx keyserverdata: get_caps: name=client.admin
2011-12-19 16:16:13.435918 7f905426c700 cephx keyserverdata: get_secret: num of caps=3
2011-12-19 16:16:13.435942 7f905426c700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/13014391 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x11fc000 con 0xcd7280
2011-12-19 16:16:13.436699 7f905426c700 -- 10.177.64.4:6789/0 <== client.? 10.177.32.66:0/13014391 3 ==== auth(proto 2 165 bytes) v1 ==== 191+0+0 (1146995391 0 0) 0x11f8800 con 0xcd7280
2011-12-19 16:16:13.436715 7f905426c700 mon.0@0(leader) e1 have connection
2011-12-19 16:16:13.436727 7f905426c700 mon.0@0(leader) e1 ms_dispatch existing session MonSession: client.? 10.177.32.66:0/13014391 is openallow * for client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.436733 7f905426c700 mon.0@0(leader) e1 caps allow *
2011-12-19 16:16:13.436739 7f905426c700 mon.0@0(leader).auth v626 update_from_paxos()
2011-12-19 16:16:13.436748 7f905426c700 mon.0@0(leader).auth v626 preprocess_query auth(proto 2 165 bytes) v1 from client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.436754 7f905426c700 mon.0@0(leader).auth v626 prep_auth() blob_size=165
2011-12-19 16:16:13.436760 7f905426c700 cephx server client.admin: handle_request get_principal_session_key
2011-12-19 16:16:13.436767 7f905426c700 cephx: verify_authorizer decrypted service auth secret_id=46
2011-12-19 16:16:13.436802 7f905426c700 cephx: verify_authorizer global_id=6253
2011-12-19 16:16:13.436828 7f905426c700 cephx: verify_authorizer ok nonce 467dc1ff70cfb3a1 reply_bl.length()=36
2011-12-19 16:16:13.436836 7f905426c700 cephx server client.admin: ticket_req.keys = 5
2011-12-19 16:16:13.436841 7f905426c700 cephx server client.admin: adding key for service mon
2011-12-19 16:16:13.436855 7f905426c700 cephx keyserverdata: get_service_secret service mon id 544 AQAhP+9O0BaxKxAABySkS5uAWPm6f/lFOhKQhA== expires 2011-12-19 16:41:53.732991
2011-12-19 16:16:13.436879 7f905426c700 cephx keyserverdata: get_caps: name=client.admin
2011-12-19 16:16:13.436886 7f905426c700 cephx keyserverdata: get_secret: num of caps=3
2011-12-19 16:16:13.436893 7f905426c700 cephx server client.admin: adding key for service osd
2011-12-19 16:16:13.436905 7f905426c700 cephx keyserverdata: get_service_secret service osd id 544 AQAhP+9O4LqxKxAAwCUAcOmsSvqjkdBNjfO4sA== expires 2011-12-19 16:41:53.733047
2011-12-19 16:16:13.436922 7f905426c700 cephx keyserverdata: get_caps: name=client.admin
2011-12-19 16:16:13.436928 7f905426c700 cephx keyserverdata: get_secret: num of caps=3
2011-12-19 16:16:13.436941 7f905426c700 cephx: build_service_ticket_reply encoding 2 tickets with secret AQA9Ve9OQIn5GRAAkwGuF4Gf2EOOXCVqR1PqnA==
2011-12-19 16:16:13.436960 7f905426c700 cephx: build_service_ticket service mon secret_id 544 ticket_info.ticket.name=client.admin
2011-12-19 16:16:13.436982 7f905426c700 cephx: service_ticket_blob is 0000 : 01 20 02 00 00 00 00 00 00 70 00 00 00 1e 63 28 : . .......p....c(
2011-12-19 16:16:13.436992 010 : eb b5 cd 8e 7d 4a 33 33 d6 b0 69 31 dc 60 a8 61 : ....}J33..i1.`.a
2011-12-19 16:16:13.437000 020 : 54 23 55 17 6f 0a 86 54 f1 17 64 c9 99 5d 81 80 : T#U.o..T..d..]..
2011-12-19 16:16:13.437008 030 : 2c 85 83 dc d1 fe 44 1d 0f f0 aa c2 9e b6 fb 0b : ,.....D.........
2011-12-19 16:16:13.437015 040 : 02 52 69 3e cc b8 29 24 93 6a a1 8c 7d b8 98 28 : .Ri>..)$.j..}..(
2011-12-19 16:16:13.437023 050 : dc 20 e3 24 3a 06 c5 b4 8b a3 84 9c 54 8b 6c ff : . .$:.......T.l.
2011-12-19 16:16:13.437030 060 : 90 e2 a1 73 4f 7b 7f 30 ba 2b ca 20 89 96 57 22 : ...sO{.0.+. ..W"
2011-12-19 16:16:13.437037 070 : 87 ab ce a1 df 67 b9 60 99 97 b2 e4 05 : .....g.`.....
2011-12-19 16:16:13.437041 2011-12-19 16:16:13.437057 7f905426c700 cephx: build_service_ticket service osd secret_id 544 ticket_info.ticket.name=client.admin
2011-12-19 16:16:13.437078 7f905426c700 cephx: service_ticket_blob is 0000 : 01 20 02 00 00 00 00 00 00 70 00 00 00 1e 7d 8f : . .......p....}.
2011-12-19 16:16:13.437087 010 : 25 75 55 8a 08 da 1e a1 2c 16 62 ab ea 19 30 40 : %uU.....,.b...0@
2011-12-19 16:16:13.437095 020 : 2e a9 2d 1b 83 14 c4 21 e5 f8 5a 5d 5c f3 2c 2d : ..-....!..Z]\.,-
2011-12-19 16:16:13.437102 030 : a7 6a ea 37 c9 d1 53 db a0 02 b4 df 43 f5 e0 dd : .j.7..S.....C...
2011-12-19 16:16:13.437110 040 : 29 d6 01 cf 78 fe 5e 2f a7 f9 3d 07 ae fb 08 1a : )...x.^/..=.....
2011-12-19 16:16:13.437125 050 : 4b 38 7e da 98 81 14 07 05 d3 f3 01 c9 ca 15 6f : K8~............o
2011-12-19 16:16:13.437133 060 : bb ae bd a1 7e 76 70 0f b7 e6 42 5d 64 03 0f 6b : ....~vp...B]d..k
2011-12-19 16:16:13.437140 070 : 0b 28 7a f3 62 c3 e7 91 7f e5 33 15 5f : .(z.b.....3._
2011-12-19 16:16:13.437144 2011-12-19 16:16:13.437163 7f905426c700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/13014391 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0xd9c200 con 0xcd7280
2011-12-19 16:16:13.437750 7f905426c700 -- 10.177.64.4:6789/0 <== client.? 10.177.32.66:0/13014391 4 ==== mon_subscribe({monmap=0+}) v2 ==== 23+0+0 (1620593354 0 0) 0x128ea80 con 0xcd7280
2011-12-19 16:16:13.437768 7f905426c700 mon.0@0(leader) e1 have connection
2011-12-19 16:16:13.437786 7f905426c700 mon.0@0(leader) e1 ms_dispatch existing session MonSession: client.? 10.177.32.66:0/13014391 is openallow * for client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.437796 7f905426c700 mon.0@0(leader) e1 caps allow *
2011-12-19 16:16:13.437837 7f905426c700 mon.0@0(leader) e1 handle_subscribe mon_subscribe({monmap=0+}) v2
2011-12-19 16:16:13.437852 7f905426c700 mon.0@0(leader) e1 check_sub monmap next 0 have 1
2011-12-19 16:16:13.437875 7f905426c700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/13014391 -- mon_map v1 -- ?+0 0xea5a80 con 0xcd7280
2011-12-19 16:16:13.437899 7f905426c700 -- 10.177.64.4:6789/0 --> client.? 10.177.32.66:0/13014391 -- mon_subscribe_ack(300s) v1 -- ?+0 0xd48600
2011-12-19 16:16:13.437919 7f905426c700 -- 10.177.64.4:6789/0 <== client.6253 10.177.32.66:0/13014391 5 ==== mon_subscribe({monmap=0+,osdmap=0}) v2 ==== 42+0+0 (982583713 0 0) 0x14fd700 con 0xcd7280
2011-12-19 16:16:13.437926 7f905426c700 mon.0@0(leader) e1 have connection
2011-12-19 16:16:13.437937 7f905426c700 mon.0@0(leader) e1 ms_dispatch existing session MonSession: client.? 10.177.32.66:0/13014391 is openallow * for client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.437942 7f905426c700 mon.0@0(leader) e1 caps allow *
2011-12-19 16:16:13.437949 7f905426c700 mon.0@0(leader) e1 handle_subscribe mon_subscribe({monmap=0+,osdmap=0}) v2
2011-12-19 16:16:13.437957 7f905426c700 mon.0@0(leader) e1 check_sub monmap next 0 have 1
2011-12-19 16:16:13.437969 7f905426c700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/13014391 -- mon_map v1 -- ?+0 0x128ea80 con 0xcd7280
2011-12-19 16:16:13.438052 7f905426c700 -- 10.177.64.4:6789/0 --> client.? 10.177.32.66:0/13014391 -- osd_map(461..461 src has 1..461) v1 -- ?+0 0x11f8800
2011-12-19 16:16:13.438067 7f905426c700 -- 10.177.64.4:6789/0 --> client.6253 10.177.32.66:0/13014391 -- mon_subscribe_ack(300s) v1 -- ?+0 0x1851000
2011-12-19 16:16:13.438083 7f905426c700 -- 10.177.64.4:6789/0 <== client.6253 10.177.32.66:0/13014391 6 ==== mon_subscribe({monmap=0+,osdmap=0}) v2 ==== 42+0+0 (982583713 0 0) 0x14fd8c0 con 0xcd7280
2011-12-19 16:16:13.438091 7f905426c700 mon.0@0(leader) e1 have connection
2011-12-19 16:16:13.438100 7f905426c700 mon.0@0(leader) e1 ms_dispatch existing session MonSession: client.? 10.177.32.66:0/13014391 is openallow * for client.? 10.177.32.66:0/13014391
2011-12-19 16:16:13.438106 7f905426c700 mon.0@0(leader) e1 caps allow *
2011-12-19 16:16:13.438113 7f905426c700 mon.0@0(leader) e1 handle_subscribe mon_subscribe({monmap=0+,osdmap=0}) v2
2011-12-19 16:16:13.438120 7f905426c700 mon.0@0(leader) e1 check_sub monmap next 0 have 1
2011-12-19 16:16:13.438132 7f905426c700 -- 10.177.64.4:6789/0 --> 10.177.32.66:0/13014391 -- mon_map v1 -- ?+0 0x14fd700 con 0xcd7280
2011-12-19 16:16:13.438234 7f905426c700 -- 10.177.64.4:6789/0 --> client.? 10.177.32.66:0/13014391 -- osd_map(461..461 src has 1..461) v1 -- ?+0 0x19b4200
2011-12-19 16:16:13.438258 7f905426c700 -- 10.177.64.4:6789/0 --> client.6253 10.177.32.66:0/13014391 -- mon_subscribe_ack(300s) v1 -- ?+0 0xda7780
2011-12-19 16:16:13.649247 7f905426c700 -- 10.177.64.4:6789/0 <== osd.25 10.177.64.4:6851/5170 162 ==== pg_stats(0 pgs tid 72 v 461) v1 ==== 275+0+0 (1103829560 0 0) 0xd89d80 con 0xcd4dc0
