Hi, >From the ones we managed to configure in our lab here. I noticed that using image format "raw" instead of "qcow2" worked for us. Regards, Luke -----Original Message----- From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Steven Timm Sent: Thursday, 18 September, 2014 5:01 AM To: ceph-users at lists.ceph.com Subject: ceph issue: rbd vs. qemu-kvm I am trying to use Ceph as a data store with OpenNebula 4.6 and have followed the instructions in OpenNebula's documentation at http://docs.opennebula.org/4.8/administration/storage/ceph_ds.html and compared them against the "using libvirt with ceph" http://ceph.com/docs/master/rbd/libvirt/ We are using the ceph-recompiled qemu-kvm and qemu-img as found at http://ceph.com/packages/qemu-kvm/ under Scientific Linux 6.5 which is a Redhat clone. Also a kernel-lt-3.10 kernel. [root at fgtest15 qemu]# kvm -version QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2), Copyright (c) 2003-2008 Fabrice Bellard >From qemu-img Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat qcow2 qed parallels nbd blkdebug host_cdrom host_floppy host_device file rbd ------------------ Libvirt is trying to execute the following KVM command: 2014-09-17 19:50:12.774+0000: starting up LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name one-60 -S -M rhel6.3.0 -enable-kvm -m 4096 -smp 2,sockets=2,cores=1,threads=1 -uuid 572499bf-07f3-3014-8d6a-dfa1ebb99aa4 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-60.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=rbd:one/one-19-60-0:id=libvirt2:key=AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/one//datastores/102/60/disk.1,if=none,id=drive-virtio-disk1,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/var/lib/one//datastores/102/60/disk.2,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=54:52:00:02:0b:04,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:60 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 char device redirected to /dev/pts/3 qemu-kvm: -drive file=rbd:one/one-19-60-0:id=libvirt2:key=AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: could not open disk image rbd:one/one-19-60-0:id=libvirt2:key=AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789: Invalid argument 2014-09-17 19:50:12.980+0000: shutting down ----------- just to show that from the command line I can see the rbd pool fine [root at fgtest15 qemu]# rbd list one foo one-19 one-19-58-0 one-19-60-0 [root at fgtest15 qemu]# rbd info one/one-19-60-0 rbd image 'one-19-60-0': size 40960 MB in 10240 objects order 22 (4096 kB objects) block_name_prefix: rb.0.3c39.238e1f29 format: 1 and even mount stuff with rbd map, etc. It's only inside libvirt that we had the problem. At first we were getting "permission denied" but then I upped the permissions allowed to the libvirt user (client.libvirt2) and then we are just getting "invalid argument" client.libvirt2 key: AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA== caps: [mon] allow r caps: [osd] allow *, allow rwx pool=one ------ Any idea why kvm doesn't like the argument I am delivering in the file= argument? Better--does anyone have a working kvm command out of either opennebula or openstack against which I can compare? Thanks Steve Timm ------------------------------------------------------------------ Steven C. Timm, Ph.D (630) 840-8525 timm at fnal.gov http://home.fnal.gov/~timm/ Fermilab Scientific Computing Division, Scientific Computing Services Quad. Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing _______________________________________________ ceph-users mailing list ceph-users at lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ________________________________ DISCLAIMER: This e-mail (including any attachments) is for the addressee(s) only and may be confidential, especially as regards personal data. If you are not the intended recipient, please note that any dealing, review, distribution, printing, copying or use of this e-mail is strictly prohibited. If you have received this email in error, please notify the sender immediately and delete the original message (including any attachments). MIMOS Berhad is a research and development institution under the purview of the Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions and other information in this e-mail that do not relate to the official business of MIMOS Berhad and/or its subsidiaries shall be understood as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts responsibility for the same. All liability arising from or in connection with computer viruses and/or corrupted e-mails is excluded to the fullest extent permitted by law.
 |