thanks Luke, I will try that. Steve On Wed, 17 Sep 2014, Luke Jing Yuan wrote: > Hi, > > From the ones we managed to configure in our lab here. I noticed that using image format "raw" instead of "qcow2" worked for us. > > Regards, > Luke > > -----Original Message----- > From: ceph-users [mailto:ceph-users-bounces at lists.ceph.com] On Behalf Of Steven Timm > Sent: Thursday, 18 September, 2014 5:01 AM > To: ceph-users at lists.ceph.com > Subject: ceph issue: rbd vs. qemu-kvm > > > I am trying to use Ceph as a data store with OpenNebula 4.6 and have followed the instructions in OpenNebula's documentation at http://docs.opennebula.org/4.8/administration/storage/ceph_ds.html > > and compared them against the "using libvirt with ceph" > > http://ceph.com/docs/master/rbd/libvirt/ > > We are using the ceph-recompiled qemu-kvm and qemu-img as found at > > http://ceph.com/packages/qemu-kvm/ > > under Scientific Linux 6.5 which is a Redhat clone. Also a kernel-lt-3.10 kernel. > > [root at fgtest15 qemu]# kvm -version > QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2), Copyright (c) > 2003-2008 Fabrice Bellard > > > From qemu-img > > Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat qcow2 > qed parallels nbd blkdebug host_cdrom host_floppy host_device file rbd > > > ------------------ > Libvirt is trying to execute the following KVM command: > > 2014-09-17 19:50:12.774+0000: starting up > LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none > /usr/libexec/qemu-kvm -name one-60 -S -M rhel6.3.0 -enable-kvm -m 4096 > -smp 2,sockets=2,cores=1,threads=1 -uuid > 572499bf-07f3-3014-8d6a-dfa1ebb99aa4 -nodefconfig -nodefaults -chardev > socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-60.monitor,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc > -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive > file=rbd:one/one-19-60-0:id=libvirt2:key=AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none > -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 > -drive > file=/var/lib/one//datastores/102/60/disk.1,if=none,id=drive-virtio-disk1,format=raw,cache=none > -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 > -drive > file=/var/lib/one//datastores/102/60/disk.2,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw > -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 > -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device > virtio-net-pci,netdev=hostnet0,id=net0,mac=54:52:00:02:0b:04,bus=pci.0,addr=0x3 > -chardev pty,id=charserial0 -device > isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:60 -k en-us -vga > cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 > char device redirected to /dev/pts/3 > qemu-kvm: -drive > file=rbd:one/one-19-60-0:id=libvirt2:key=AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: > could not open disk image > rbd:one/one-19-60-0:id=libvirt2:key=AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789: > Invalid argument > 2014-09-17 19:50:12.980+0000: shutting down > > ----------- > > just to show that from the command line I can see the rbd pool fine > > [root at fgtest15 qemu]# rbd list one > foo > one-19 > one-19-58-0 > one-19-60-0 > [root at fgtest15 qemu]# rbd info one/one-19-60-0 > rbd image 'one-19-60-0': > size 40960 MB in 10240 objects > order 22 (4096 kB objects) > block_name_prefix: rb.0.3c39.238e1f29 > format: 1 > > > and even mount stuff with rbd map, etc. > > It's only inside libvirt that we had the problem. > > At first we were getting "permission denied" but then I upped the > permissions allowed to the libvirt user (client.libvirt2) and then > we are just getting "invalid argument" > > > client.libvirt2 > key: AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA== > caps: [mon] allow r > caps: [osd] allow *, allow rwx pool=one > > ------ > > Any idea why kvm doesn't like the argument I am delivering in the file= > argument? Better--does anyone have a working kvm command out > of either opennebula or openstack against which I can compare? > > Thanks > > Steve Timm > > > > > > ------------------------------------------------------------------ > Steven C. Timm, Ph.D (630) 840-8525 > timm at fnal.gov http://home.fnal.gov/~timm/ > Fermilab Scientific Computing Division, Scientific Computing Services Quad. > Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing > _______________________________________________ > ceph-users mailing list > ceph-users at lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ________________________________ > DISCLAIMER: > > > This e-mail (including any attachments) is for the addressee(s) only and may be confidential, especially as regards personal data. If you are not the intended recipient, please note that any dealing, review, distribution, printing, copying or use of this e-mail is strictly prohibited. If you have received this email in error, please notify the sender immediately and delete the original message (including any attachments). > > > MIMOS Berhad is a research and development institution under the purview of the Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions and other information in this e-mail that do not relate to the official business of MIMOS Berhad and/or its subsidiaries shall be understood as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts responsibility for the same. All liability arising from or in connection with computer viruses and/or corrupted e-mails is excluded to the fullest extent permitted by law. > ------------------------------------------------------------------ Steven C. Timm, Ph.D (630) 840-8525 timm at fnal.gov http://home.fnal.gov/~timm/ Fermilab Scientific Computing Division, Scientific Computing Services Quad. Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing
 |