Ehhh too long with this :) I forget to load acpiphp. Thanks for everything, now its working beautifully. After 10h of iozone on rbd devices, and no hang, or problem. 2011/12/20 Josh Durgin <josh.durgin@xxxxxxxxxxxxx>: > On 12/19/2011 07:37 AM, Sławomir Skowron wrote: >> >> Hi, >> >> Actual setup: >> >> ii libvirt-bin 0.9.2-4ubuntu15.1 >> the programs for the libvirt library >> ii libvirt0 0.9.2-4ubuntu15.1 >> library for interfacing with different virtualization systems >> ii qemu-kvm 1.0.0+dfsg+rc2-1~oneiric1 >> Full virtualization on i386 and amd64 hardware >> >> I change many thing in system, and successfully attached rbd drives via >> libvirt: >> >> virsh attach-device one-888 kvm-ceph.xml >> Device attached successfully >> >> A new set of disks, appears in libvirt vm xml, but not in vm machine. >> After reboot, all new rbd drives appear in VM. >> >> Is there any chance to hotadd rbd device to working VM without reboot ?? > > > Hotplugging should work, but it does require guest support. Does your guest > kernel have hotplugging enabled? Does hotplugging work with a raw image file > using the same configuration? > > >> >> 2011/12/16 Sławomir Skowron<szibis@xxxxxxxxx>: >>> >>> 2011/12/13 Josh Durgin<josh.durgin@xxxxxxxxxxxxx>: >>>> >>>> On 12/13/2011 04:56 AM, Sławomir Skowron wrote: >>>>> >>>>> >>>>> Finally, i manage the problem with rbd with kvm 1.0, and libvirt >>>>> 0.9.8, or i think i manage :), but i get stuck with one thing after. >>>>> >>>>> 2011-12-13 12:13:31.173+0000: 21512: error : >>>>> qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU >>>>> command 'device_add': Bus 'pci.0' does not support hotplugging >>>>> 2011-12-13 12:13:31.173+0000: 21512: warning : >>>>> qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on >>>>> >>>>> >>>>> file=rbd:rbd/testdysk:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx >>>>> >>>>> none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk2,format=raw >>>>> >>>>> >>>>> (virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2) >>>>> 2011-12-13 12:13:31.173+0000: 21512: error : >>>>> virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink >>>>> rbd/testdysk: No such file or directory2011-12-13 12:13:31.173+0000: >>>>> 21512: warning : qemuDomainAttachPciDiskDevice:287 : Unable to restore >>>>> security label on rbd/testdysk >>>>> and i can't find any info about that. >>>>> >>>>> my kvm.xml inject file: >>>>> >>>>> <disk type="network" device="disk"> >>>>> <driver name="qemu" type="raw"/> >>>>> <auth username="admin"> >>>>> <secret type="ceph" >>>>> uuid="0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f"/> >>>>> </auth> >>>>> <source protocol="rbd" name="rbd/testdysk"> >>>>> <host name="10.177.64.4" port="6789"/> >>>>> </source> >>>>> <target dev="sdc" bus="virtio"/> >>>>> </disk> >>>>> >>>>> secret: >>>>> >>>>> cat /tmp/secret.xml >>>>> <secret ephemeral='no' private='no'> >>>>> <uuid>0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f</uuid> >>>>> <usage type='ceph'> >>>>> <domain>rbd/admin</domain> >>>>> <name>admin</name> >>>>> </usage> >>>>> </secret> >>>>> >>>>> virsh secret-define /tmp/secret.xml >>>>> Secret 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f created >>>>> >>>>> virsh secret-set-value 0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f >>>>> AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog== >>>>> Secret value set >>>>> >>>> >>>> Your xml and configuration are all fine, you're hitting the libvirt bug >>>> that >>>> I mentioned before. The fix isn't in any stable release yet, and 0.9.8 >>>> was >>>> just released yesterday, so I guess you'll have to compile it yourself >>>> or >>>> disable the libvirt security driver. >>> >>> >>> Yes libvirt 0.9.8 with this patch solved problem with rbd, now it's >>> clean. Thanks for that. >>> >>> But, a second problem appear. >>> >>> error: Failed to attach device from /root/kvm-ceph.xml >>> error: internal error unable to execute QEMU command 'device_add': >>> Property 'virtio-blk-pci.drive' can't find value 'drive-virtio-disk3' >>> >>> /var/log/libvirt/libvirtd.log >>> 2011-12-16 12:03:04.950+0000: 1070: error : >>> qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU >>> command 'device_add': Property 'virtio-blk-pci.drive' can't find value >>> 'drive-virtio-disk3' >>> 2011-12-16 12:03:04.950+0000: 1070: warning : >>> qemuDomainAttachPciDiskDevice:244 : qemuMonitorAddDevice failed on >>> >>> file=rbd:rbd/testdysk:rbd_writeback_window=8000000:id=admin:key=AQANeNFOiIx9DhAAv76MRXZjWNKn2sSSTQeJog==:auth_supported=cephx >>> none:mon_host=10.177.64.4\:6789,if=none,id=drive-virtio-disk3,format=raw >>> >>> (virtio-blk-pci,bus=pci.0,addr=0xd,drive=drive-virtio-disk3,id=virtio-disk3) >>> >>> if i attach a scsi device from img file everything is OK. >>> >>> Anyone have, any concept with this ?? >>> >>>> >>>> >>>>> 2011/12/12 Josh Durgin<josh.durgin@xxxxxxxxxxxxx>: >>>>>> >>>>>> >>>>>> On 12/09/2011 04:48 AM, Sławomir Skowron wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> Sorry, for my lag, but i was sick. >>>>>>> >>>>>>> I handled problem with apparmor before and now it's not a problem, >>>>>>> even when i send mail before, it was solved. >>>>>>> >>>>>>> Ok when i create image from qemu-img it's looks like that, and works >>>>>>> perfect, (attachment with log of this opperation.) >>>>>>> >>>>>>> But when i use virsh, it's still a problem. Just like he don't know >>>>>>> what user is going to be auth. >>>>>>> >>>>>>> I try, as client.admin (default i hope :)) only with id=admin in >>>>>>> kvm.xml, or client=admin, and other combinations - like client.rbd >>>>>>> etc., Still nothing. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> admin is the default, and the correct option for client.admin is >>>>>> id=admin, >>>>>> or id=foo for client.foo. >>>>>> >>>>>> >>>>>>> >>>>>>> I try, improve a keyring file, adding client rbd into keyring on >>>>>>> machine with kvm hypervisor in /etc/ceph/keyring.bin >>>>>>> >>>>>>> [client.rbd] >>>>>>> key = AAAAAAAAAAAAAAAA >>>>>>> auid = 18446744073709551615 >>>>>>> caps mds = "allow" >>>>>>> caps mon = "allow r" >>>>>>> caps osd = "allow rw pool=kvmtest" >>>>>>> >>>>>>> and on cluster site like above. I was added, a section [client.rbd] >>>>>>> with keyring path, and it's still a problem. >>>>>>> >>>>>> >>>>>> Did you run 'ceph auth add -i /etc/ceph/keyring.bin'? >>>>>> Make sure 'ceph auth list' shows client.rbd with the right key. >>>>> >>>>> >>>>> >>>>> Overall i want to use admin id, and i think now it's correctly used in >>>>> xml. >>>>> >>>>>> >>>>>> >>>>>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1 >>>>>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0 >>>>>>> root@s3-10-177-64-4:~# tail -n10000 /var/log/ceph/mon.0.log | grep >>>>>>> 13:29:11. >>>>>>> 2011-12-09 13:29:11.519096 7fe616b59700 mon.0@0(leader) e1 >>>>>>> ms_verify_authorizer 10.177.32.66:0/58020608 client protocol 0 >>>>>>> 2011-12-09 13:29:11.519624 7fe61b090700 -- 10.177.64.4:6789/0<== >>>>>>> client.? 10.177.32.66:0/58020608 1 ==== auth(proto 0 21 bytes) v1 >>>>>>> ==== >>>>>>> 47+0+0 (3310669357 0 0) 0x17f2e00 con 0x19d2c80 >>>>>>> 2011-12-09 13:29:11.519645 7fe61b090700 mon.0@0(leader) e1 have >>>>>>> connection >>>>>>> 2011-12-09 13:29:11.519655 7fe61b090700 mon.0@0(leader) e1 do not >>>>>>> have >>>>>>> session, making new one >>>>>>> 2011-12-09 13:29:11.519668 7fe61b090700 mon.0@0(leader) e1 >>>>>>> ms_dispatch >>>>>>> new session MonSession: client.? 10.177.32.66:0/58020608 is open for >>>>>>> client.? 10.177.32.66:0/58020608 >>>>>>> 2011-12-09 13:29:11.519674 7fe61b090700 mon.0@0(leader) e1 setting >>>>>>> timeout on session >>>>>>> 2011-12-09 13:29:11.519680 7fe61b090700 mon.0@0(leader) e1 caps >>>>>>> 2011-12-09 13:29:11.519688 7fe61b090700 mon.0@0(leader).auth v353 >>>>>>> AuthMonitor::update_from_paxos() >>>>>>> 2011-12-09 13:29:11.519697 7fe61b090700 mon.0@0(leader).auth v353 >>>>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.? >>>>>>> 10.177.32.66:0/58020608 >>>>>>> 2011-12-09 13:29:11.519703 7fe61b090700 mon.0@0(leader).auth v353 >>>>>>> prep_auth() blob_size=21 >>>>>>> 2011-12-09 13:29:11.519735 7fe61b090700 -- 10.177.64.4:6789/0 --> >>>>>>> 10.177.32.66:0/58020608 -- auth_reply(proto 0 -95 Operation not >>>>>>> supported) v1 -- ?+0 0x1246200 con 0x19d2c80 >>>>>>> >>>>>>> from libvirt.log >>>>>>> >>>>>>> 23:20:56.689: 1309: error : >>>>>>> virSecurityDACRestoreSecurityFileLabel:143 >>>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such >>>>>>> file >>>>>>> or directory >>>>>>> 23:20:56.882: 1309: warning : qemuDomainAttachPciDiskDevice:250 : >>>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin >>>>>>> 23:21:09.172: 1309: error : qemuMonitorTextAddDrive:2418 : operation >>>>>>> failed: open disk image file failed >>>>>>> 23:21:09.172: 1309: error : >>>>>>> virSecurityDACRestoreSecurityFileLabel:143 >>>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:name=admin: No such >>>>>>> file >>>>>>> or directory >>>>>>> 23:21:09.364: 1309: warning : qemuDomainAttachPciDiskDevice:250 : >>>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:name=admin >>>>>>> 23:21:49.645: 1309: error : qemuMonitorTextAddDrive:2418 : operation >>>>>>> failed: open disk image file failed >>>>>>> 23:21:49.645: 1309: error : >>>>>>> virSecurityDACRestoreSecurityFileLabel:143 >>>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1:id=admin: No such file >>>>>>> or directory >>>>>>> 23:21:49.853: 1309: warning : qemuDomainAttachPciDiskDevice:250 : >>>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1:id=admin >>>>>>> 13:08:08.924: 1309: error : qemuMonitorTextAddDrive:2418 : operation >>>>>>> failed: open disk image file failed >>>>>>> 13:08:08.924: 1309: error : >>>>>>> virSecurityDACRestoreSecurityFileLabel:143 >>>>>>> : cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or >>>>>>> directory >>>>>>> 13:08:09.117: 1309: warning : qemuDomainAttachPciDiskDevice:250 : >>>>>>> Unable to restore security label on rbd:kvmtest/kvmtest1 >>>>>>> 13:09:45.402: 1309: error : qemuMonitorTextAddDrive:2418 : operation >>>>>>> failed: open disk image file failed >>>>>>> 13:29:05.554: 1310: error : qemuMonitorTextAddDrive:2418 : operation >>>>>>> failed: open disk image file failed >>>>>>> 13:29:05.554: 1310: error : >>>>>>> virSecurityDACRestoreSecurityFileLabel:143 >>>>>>> : cannot resolve symlink kvmtest/kvmtest1: No such file or directory >>>>>>> 13:29:05.751: 1310: warning : qemuDomainAttachPciDiskDevice:250 : >>>>>>> Unable to restore security label on kvmtest/kvmtest1 >>>>>>> 13:29:05.751: 1310: warning : virEventPollUpdateHandle:147 : Ignoring >>>>>>> invalid update watch -1 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> This is libvirt mistakenly trying to treat the rbd disk as a file - >>>>>> fixed >>>>>> in >>>>>> 20e1233c31e3150d259073c523077acdccb5d419 to libvirt. If you don't want >>>>>> to >>>>>> upgrade libvirt, someone on IRC last week reported that installing >>>>>> libapparmor1 fixed this problem for them, since libvirt no longer fell >>>>>> back >>>>>> to it's own DAC implementation. >>>>> >>>>> >>>>> >>>>> hmm is this commited to any stable version now ?? i have installed >>>>> libapparmor1, >>>>> >>>>> ri libapparmor1 2.7.0~beta1+bzr1774-1ubuntu2 >>>>> changehat AppArmor library >>>>> >>>>> and i use newest stable libvirt 0.9.8. >>>>> >>>>>> >>>>>> >>>>>>> 2011/12/1 Josh Durgin<josh.durgin@xxxxxxxxxxxxx>: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 12/01/2011 11:37 AM, Sławomir Skowron wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I have some problems. Can you help me ?? >>>>>>>>> >>>>>>>>> ceph cluster: >>>>>>>>> ceph 0.38, oneiric, kernel 3.0.0 x86_64 - now only one machine. >>>>>>>>> >>>>>>>>> kvm hypervisor: >>>>>>>>> kvm version 1.0-rc4 (0.15.92), libvirt 0.9.2, kernel 3.0.0, Ubuntu >>>>>>>>> oneiric x86_64. >>>>>>>>> >>>>>>>>> I create image from qemu-img on machine with kvm VM's, and it works >>>>>>>>> very >>>>>>>>> well: >>>>>>>>> >>>>>>>>> qemu-img create -f rbd rbd:kvmtest/kvmtest1 10G >>>>>>>>> Formatting 'rbd:kvmtest/kvmtest1', fmt=rbd size=10737418240 >>>>>>>>> cluster_size=0 >>>>>>>>> >>>>>>>>> in ceph cluster machines: >>>>>>>>> >>>>>>>>> rados -p kvmtest ls >>>>>>>>> kvmtest1.rbd >>>>>>>>> rbd_directory >>>>>>>>> rbd_info >>>>>>>>> >>>>>>>>> But when i try to attach new device to kvm VM like this: >>>>>>>>> >>>>>>>>> virsh attach-device one-10 /tmp/kvm.xml >>>>>>>>> >>>>>>>>> with kvm.xml like this: >>>>>>>>> >>>>>>>>> <disk type='network' device='disk'> >>>>>>>>> <driver name='qemu' type='raw' cache='writeback'/> >>>>>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'> >>>>>>>>> <host name='10.177.64.4' port='6789'/> >>>>>>>>> </source> >>>>>>>>> <target dev='vdc' bus='virtio'/> >>>>>>>>> </disk> >>>>>>>>> >>>>>>>>> output of virsh: >>>>>>>>> >>>>>>>>> error: Failed to attach device from /tmp/kvm.xml >>>>>>>>> error: cannot resolve symlink rbd:kvmtest/kvmtest1: No such file or >>>>>>>>> directory >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> From the monitor output below, I'm assuming this is just a bad >>>>>>>> error >>>>>>>> message >>>>>>>> from libvirt. >>>>>>>> >>>>>>>> >>>>>>>>> and mon.0 log in ceph cluster. >>>>>>>>> >>>>>>>>> 2011-12-01 20:18:17.142595 7fe02dd04700 mon.0@0(leader) e1 >>>>>>>>> ms_verify_authorizer 10.177.32.66:0/25020608 client protocol 0 >>>>>>>>> 2011-12-01 20:18:17.143041 7fe03223b700 -- 10.177.64.4:6789/0<== >>>>>>>>> client.? 10.177.32.66:0/25020608 1 ==== auth(proto 0 21 bytes) v1 >>>>>>>>> ==== >>>>>>>>> 47+0+0 (3310669357 0 0) 0x29f6a00 con 0x297adc0 >>>>>>>>> 2011-12-01 20:18:17.143095 7fe03223b700 mon.0@0(leader) e1 >>>>>>>>> ms_dispatch >>>>>>>>> new session MonSession: client.? 10.177.32.66:0/25020608 is open >>>>>>>>> for >>>>>>>>> client.? 10.177.32.66:0/25020608 >>>>>>>>> 2011-12-01 20:18:17.143125 7fe03223b700 mon.0@0(leader).auth v146 >>>>>>>>> preprocess_query auth(proto 0 21 bytes) v1 from client.? >>>>>>>>> 10.177.32.66:0/25020608 >>>>>>>>> 2011-12-01 20:18:17.143168 7fe03223b700 -- 10.177.64.4:6789/0 --> >>>>>>>>> 10.177.32.66:0/25020608 -- auth_reply(proto 0 -95 Operation not >>>>>>>>> supported) v1 -- ?+0 0x2f30600 con 0x297adc0 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Here you can see qemu isn't authenticating correctly, probably >>>>>>>> because >>>>>>>> the >>>>>>>> rados user isn't being set by libvirt. With your libvirt version, >>>>>>>> you'll >>>>>>>> need to change: >>>>>>>> >>>>>>>> >>>>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1'> >>>>>>>> >>>>>>>> to >>>>>>>> >>>>>>>> <source protocol='rbd' name='rbd:kvmtest/kvmtest1:id=foo'> >>>>>>>> >>>>>>>> to authenticate as client.foo. Any ceph config option can be set >>>>>>>> like >>>>>>>> that, >>>>>>>> separated by colons. >>>>>>>> >>>>>>>> If this still doesn't work, I'd suggest setting >>>>>>>> debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2 to get more >>>>>>>> info >>>>>>>> in >>>>>>>> the libvirt instance log (usually >>>>>>>> /var/log/libvirt/qemu/instance_name.log). >>>>>>>> There may be apparmour restrictions preventing qemu from reading the >>>>>>>> keyring >>>>>>>> or ceph config files. >>>>>>>> >>>>>>>> >>>>>>>>> ceph.conf, and keyring.bin exists on server, and machine with >>>>>>>>> qemu-kvm >>>>>>>>> in same directory: >>>>>>>>> ; global >>>>>>>>> [global] >>>>>>>>> ; enable secure authentication >>>>>>>>> >>>>>>>>> auth supported = cephx >>>>>>>>> keyring = /etc/ceph/keyring.bin >>>>>>>>> >>>>>>>>> debug rgw = 1 >>>>>>>>> >>>>>>>>> rgw print continue = false >>>>>>>>> rgw socket path = /var/run/radosgw.sock >>>>>>>>> >>>>>>>>> ; monitors >>>>>>>>> ; You need at least one. You need at least three if you want to >>>>>>>>> ; tolerate any node failures. Always create an odd number. >>>>>>>>> [mon] >>>>>>>>> mon data = /vol0/data/mon.$id >>>>>>>>> >>>>>>>>> ; some minimal logging (just message traffic) to aid >>>>>>>>> debugging >>>>>>>>> >>>>>>>>> debug ms = 1 ; see message traffic >>>>>>>>> debug mon = 0 ; monitor >>>>>>>>> debug paxos = 0 ; monitor replication >>>>>>>>> debug auth = 0 ; >>>>>>>>> >>>>>>>>> mon allowed clock drift = 2 >>>>>>>>> >>>>>>>>> [mon.0] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> mon addr = 10.177.64.4:6789 >>>>>>>>> >>>>>>>>> ; radosgw client list >>>>>>>>> [client.radosgw.10-177-64-4] >>>>>>>>> >>>>>>>>> host = 10-177-64-4 >>>>>>>>> log file = /var/log/ceph/$name.log >>>>>>>>> debug rgw = 1 >>>>>>>>> debug ms = 1 >>>>>>>>> >>>>>>>>> ; osd >>>>>>>>> ; You need at least one. Two if you want data to be replicated. >>>>>>>>> ; Define as many as you like. >>>>>>>>> [osd] >>>>>>>>> ; This is where the btrfs volume will be mounted. >>>>>>>>> >>>>>>>>> osd data = /vol0/data/osd.$id >>>>>>>>> >>>>>>>>> ; Ideally, make this a separate disk or partition. A few >>>>>>>>> GB >>>>>>>>> ; is usually enough; more if you have fast disks. You can >>>>>>>>> use >>>>>>>>> ; a file under the osd data dir if need be >>>>>>>>> ; (e.g. /data/osd$id/journal), but it will be slower than a >>>>>>>>> ; separate disk or partition. >>>>>>>>> >>>>>>>>> osd journal = /vol0/data/osd.$id/journal >>>>>>>>> >>>>>>>>> ; If the OSD journal is a file, you need to specify the >>>>>>>>> size. >>>>>>>>> This is specified in MB. >>>>>>>>> >>>>>>>>> osd journal size = 512 >>>>>>>>> >>>>>>>>> filestore journal writeahead = 1 >>>>>>>>> osd heartbeat grace = 5 >>>>>>>>> >>>>>>>>> debug ms = 0 ; message traffic >>>>>>>>> debug osd = 0 >>>>>>>>> debug filestore = 0 ; local object storage >>>>>>>>> debug journal = 0 ; local journaling >>>>>>>>> debug monc = 0 >>>>>>>>> debug rados = 1 >>>>>>>>> >>>>>>>>> [osd.0] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.0 >>>>>>>>> keyring = /vol0/data/osd.0/keyring >>>>>>>>> >>>>>>>>> [osd.1] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.1 >>>>>>>>> keyring = /vol0/data/osd.1/keyring >>>>>>>>> >>>>>>>>> [osd.2] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.2 >>>>>>>>> keyring = /vol0/data/osd.2/keyring >>>>>>>>> >>>>>>>>> [osd.3] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.3 >>>>>>>>> keyring = /vol0/data/osd.3/keyring >>>>>>>>> >>>>>>>>> [osd.4] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.4 >>>>>>>>> keyring = /vol0/data/osd.4/keyring >>>>>>>>> >>>>>>>>> [osd.5] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.5 >>>>>>>>> keyring = /vol0/data/osd.5/keyring >>>>>>>>> >>>>>>>>> [osd.6] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.6 >>>>>>>>> keyring = /vol0/data/osd.6/keyring >>>>>>>>> >>>>>>>>> [osd.7] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.7 >>>>>>>>> keyring = /vol0/data/osd.7/keyring >>>>>>>>> >>>>>>>>> [osd.8] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.8 >>>>>>>>> keyring = /vol0/data/osd.8/keyring >>>>>>>>> >>>>>>>>> [osd.9] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.9 >>>>>>>>> keyring = /vol0/data/osd.9/keyring >>>>>>>>> >>>>>>>>> [osd.10] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.10 >>>>>>>>> keyring = /vol0/data/osd.10/keyring >>>>>>>>> >>>>>>>>> [osd.11] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.11 >>>>>>>>> keyring = /vol0/data/osd.11/keyring >>>>>>>>> >>>>>>>>> [osd.12] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.12 >>>>>>>>> keyring = /vol0/data/osd.12/keyring >>>>>>>>> >>>>>>>>> [osd.13] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.13 >>>>>>>>> keyring = /vol0/data/osd.13/keyring >>>>>>>>> >>>>>>>>> [osd.14] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.14 >>>>>>>>> keyring = /vol0/data/osd.14/keyring >>>>>>>>> >>>>>>>>> [osd.15] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.15 >>>>>>>>> keyring = /vol0/data/osd.15/keyring >>>>>>>>> >>>>>>>>> [osd.16] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.16 >>>>>>>>> keyring = /vol0/data/osd.16/keyring >>>>>>>>> >>>>>>>>> [osd.17] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.17 >>>>>>>>> keyring = /vol0/data/osd.17/keyring >>>>>>>>> >>>>>>>>> [osd.18] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.18 >>>>>>>>> keyring = /vol0/data/osd.18/keyring >>>>>>>>> >>>>>>>>> [osd.19] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.19 >>>>>>>>> keyring = /vol0/data/osd.19/keyring >>>>>>>>> >>>>>>>>> [osd.20] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.20 >>>>>>>>> keyring = /vol0/data/osd.20/keyring >>>>>>>>> >>>>>>>>> [osd.21] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.21 >>>>>>>>> keyring = /vol0/data/osd.21/keyring >>>>>>>>> >>>>>>>>> [osd.22] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.22 >>>>>>>>> keyring = /vol0/data/osd.22/keyring >>>>>>>>> >>>>>>>>> [osd.23] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.23 >>>>>>>>> keyring = /vol0/data/osd.23/keyring >>>>>>>>> >>>>>>>>> [osd.24] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.24 >>>>>>>>> keyring = /vol0/data/osd.24/keyring >>>>>>>>> >>>>>>>>> [osd.25] >>>>>>>>> host = 10-177-64-4 >>>>>>>>> osd data = /vol0/data/osd.25 >>>>>>>>> keyring = /vol0/data/osd.25/keyring >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>> >>> >>> >>> -- >>> ----- >>> Pozdrawiam >>> >>> Sławek "sZiBis" Skowron >> >> >> >> > -- ----- Pozdrawiam Sławek "sZiBis" Skowron -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
