Yes--the image was converted back to raw. Since the image is mapped via rbd I can run fdisk on it and see both the partition tables and a normal set of files inside of it. My system datastore is local to each node. Have been in that mode for quite some time. Steve Timm ________________________________________ From: Luke Jing Yuan [jyluke@xxxxxxxx] Sent: Thursday, September 18, 2014 9:44 PM To: Steven C Timm Cc: ceph-users at lists.ceph.com Subject: RE: ceph issue: rbd vs. qemu-kvm Hi Steven, Assuming the original image was in qcow2 format, did you convert it back to raw before registering it? Another tweak I did was enabling and NFS shared the system datastore (id: 0) from the frontend to the other hosts: nebula at z4-hn01:~$ onedatastore list ID NAME SIZE AVAIL CLUSTER IMAGES TYPE DS TM 0 system 457.7G 92% z4-cluster-w 0 sys - shared 1 default 457.7G 92% - 0 img fs shared 2 files 457.7G 92% - 0 fil fs ssh 100 mi-cloud-ceph 7.8T 81% z4-cluster-w 3 img ceph ceph Regards, Luke -----Original Message----- From: Steven Timm [mailto:timm@xxxxxxxx] Sent: Friday, 19 September, 2014 5:18 AM To: Luke Jing Yuan Cc: ceph-users at lists.ceph.com Subject: Re: ceph issue: rbd vs. qemu-kvm Using image type raw actually got kvm to create the VM but then the virt-viewer console shows Booting from Hard Disk Geom Error ----------- We do not even get as far as GRUB. Below is the network stanza from XML. <disk type='network' device='disk'> <source protocol='rbd' name='one/one-21-61-0'> <host name='stkendca01a' port='6789'/> <host name='stkendca04a' port='6789'/> <host name='stkendca02a' port='6789'/> </source> <auth username='libvirt2'> <secret type='ceph' uuid='3bd5a6a5-6b2a-44b2-a07 5-ec9da47ae3f4'/> </auth> <target dev='vda'/> <driver name='qemu' type='raw' cache='none'/> </disk> Any other tweak I might be missing? Thanks Steve Timm By the way--the following is what the raw file in question looked like before I loaded it into CEPH [root at one4dev timm]# file gcso_sl6_giwms.raw gcso_sl6_giwms.raw: x86 boot sector; GRand Unified Bootloader, stage1 version 0x3, boot drive 0x80, 1st sector stage2 0x1307f70, GRUB version 0.94; partition 1: ID=0x83, active, starthead 32, startsector 2048, 6291456 sectors, code offset 0x4 On Thu, 18 Sep 2014, Steven Timm wrote: > thanks Luke, I will try that. > > Steve > > > On Wed, 17 Sep 2014, Luke Jing Yuan wrote: > >> Hi, >> >> From the ones we managed to configure in our lab here. I noticed >> that using image format "raw" instead of "qcow2" worked for us. >> >> Regards, >> Luke >> >> -----Original Message----- >> From: ceph-users [mailto:ceph-users-bounces at lists.ceph.com] On >> Behalf Of Steven Timm >> Sent: Thursday, 18 September, 2014 5:01 AM >> To: ceph-users at lists.ceph.com >> Subject: ceph issue: rbd vs. qemu-kvm >> >> >> I am trying to use Ceph as a data store with OpenNebula 4.6 and have >> followed the instructions in OpenNebula's documentation at >> http://docs.opennebula.org/4.8/administration/storage/ceph_ds.html >> >> and compared them against the "using libvirt with ceph" >> >> http://ceph.com/docs/master/rbd/libvirt/ >> >> We are using the ceph-recompiled qemu-kvm and qemu-img as found at >> >> http://ceph.com/packages/qemu-kvm/ >> >> under Scientific Linux 6.5 which is a Redhat clone. Also a >> kernel-lt-3.10 kernel. >> >> [root at fgtest15 qemu]# kvm -version >> QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2), Copyright (c) >> 2003-2008 Fabrice Bellard >> >> >> From qemu-img >> >> Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat >> qcow2 qed parallels nbd blkdebug host_cdrom host_floppy host_device >> file rbd >> >> >> ------------------ >> Libvirt is trying to execute the following KVM command: >> >> 2014-09-17 19:50:12.774+0000: starting up LC_ALL=C >> PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none >> /usr/libexec/qemu-kvm -name one-60 -S -M rhel6.3.0 -enable-kvm -m >> 4096 -smp 2,sockets=2,cores=1,threads=1 -uuid >> 572499bf-07f3-3014-8d6a-dfa1ebb99aa4 -nodefconfig -nodefaults >> -chardev >> socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-60.monitor,serve >> r,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc >> base=utc -no-shutdown -device >> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive >> file=rbd:one/one-19-60-0:id=libvirt2:key=AQAV5BlU2OV7NBAApurqxG0K8UkZ >> lQVy6hKmkA==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;s >> tkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,for >> mat=qcow2,cache=none >> -device >> >> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,i >> d=virtio-disk0,bootindex=1 >> -drive >> >> file=/var/lib/one//datastores/102/60/disk.1,if=none,id=drive-virtio-d >> isk1,format=raw,cache=none >> -device >> >> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,i >> d=virtio-disk1 >> -drive >> >> file=/var/lib/one//datastores/102/60/disk.2,if=none,media=cdrom,id=dr >> ive-ide0-1-0,readonly=on,format=raw >> -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 >> -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device >> >> virtio-net-pci,netdev=hostnet0,id=net0,mac=54:52:00:02:0b:04,bus=pci. >> 0,addr=0x3 >> -chardev pty,id=charserial0 -device >> isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:60 -k en-us >> -vga cirrus -device >> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 >> char device redirected to /dev/pts/3 >> qemu-kvm: -drive >> file=rbd:one/one-19-60-0:id=libvirt2:key=AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: >> could not open disk image >> rbd:one/one-19-60-0:id=libvirt2:key=AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789: >> Invalid argument >> 2014-09-17 19:50:12.980+0000: shutting down >> >> ----------- >> >> just to show that from the command line I can see the rbd pool fine >> >> [root at fgtest15 qemu]# rbd list one >> foo >> one-19 >> one-19-58-0 >> one-19-60-0 >> [root at fgtest15 qemu]# rbd info one/one-19-60-0 rbd image >> 'one-19-60-0': >> size 40960 MB in 10240 objects >> order 22 (4096 kB objects) >> block_name_prefix: rb.0.3c39.238e1f29 >> format: 1 >> >> >> and even mount stuff with rbd map, etc. >> >> It's only inside libvirt that we had the problem. >> >> At first we were getting "permission denied" but then I upped the >> permissions allowed to the libvirt user (client.libvirt2) and then >> we are just getting "invalid argument" >> >> >> client.libvirt2 >> key: AQAV5BlU2OV7NBAApurqxG0K8UkZlQVy6hKmkA== >> caps: [mon] allow r >> caps: [osd] allow *, allow rwx pool=one >> >> ------ >> >> Any idea why kvm doesn't like the argument I am delivering in the >> file= argument? Better--does anyone have a working kvm command out >> of either opennebula or openstack against which I can compare? >> >> Thanks >> >> Steve Timm >> >> >> >> >> >> ------------------------------------------------------------------ >> Steven C. Timm, Ph.D (630) 840-8525 timm at fnal.gov >> http://home.fnal.gov/~timm/ Fermilab Scientific Computing Division, >> Scientific Computing Services Quad. >> Grid and Cloud Services Dept., Associate Dept. Head for Cloud >> Computing _______________________________________________ >> ceph-users mailing list >> ceph-users at lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> ________________________________ >> DISCLAIMER: >> >> >> This e-mail (including any attachments) is for the addressee(s) only >> and may be confidential, especially as regards personal data. If you >> are not the intended recipient, please note that any dealing, >> review, distribution, printing, copying or use of this e-mail is >> strictly prohibited. If you have received this email in error, >> please notify the sender immediately and delete the original message >> (including any attachments). >> >> >> MIMOS Berhad is a research and development institution under the >> purview of the Malaysian Ministry of Science, Technology and >> Innovation. Opinions, conclusions and other information in this >> e-mail that do not relate to the official business of MIMOS Berhad >> and/or its subsidiaries shall be understood as neither given nor >> endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS >> Berhad nor its subsidiaries accepts responsibility for the same. All >> liability arising from or in connection with computer viruses and/or >> corrupted e-mails is excluded to the fullest extent permitted by law. >> > > ------------------------------------------------------------------ > Steven C. Timm, Ph.D (630) 840-8525 > timm at fnal.gov http://home.fnal.gov/~timm/ Fermilab Scientific > Computing Division, Scientific Computing Services Quad. > Grid and Cloud Services Dept., Associate Dept. Head for Cloud > Computing _______________________________________________ > ceph-users mailing list > ceph-users at lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ------------------------------------------------------------------ Steven C. Timm, Ph.D (630) 840-8525 timm at fnal.gov http://home.fnal.gov/~timm/ Fermilab Scientific Computing Division, Scientific Computing Services Quad. Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing ________________________________ DISCLAIMER: This e-mail (including any attachments) is for the addressee(s) only and may be confidential, especially as regards personal data. If you are not the intended recipient, please note that any dealing, review, distribution, printing, copying or use of this e-mail is strictly prohibited. If you have received this email in error, please notify the sender immediately and delete the original message (including any attachments). MIMOS Berhad is a research and development institution under the purview of the Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions and other information in this e-mail that do not relate to the official business of MIMOS Berhad and/or its subsidiaries shall be understood as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts responsibility for the same. All liability arising from or in connection with computer viruses and/or corrupted e-mails is excluded to the fullest extent permitted by law. ------------------------------------------------------------------ - - DISCLAIMER: This e-mail (including any attachments) is for the addressee(s) only and may contain confidential information. If you are not the intended recipient, please note that any dealing, review, distribution, printing, copying or use of this e-mail is strictly prohibited. If you have received this email in error, please notify the sender immediately and delete the original message. MIMOS Berhad is a research and development institution under the purview of the Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions and other information in this e- mail that do not relate to the official business of MIMOS Berhad and/or its subsidiaries shall be understood as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts responsibility for the same. All liability arising from or in connection with computer viruses and/or corrupted e-mails is excluded to the fullest extent permitted by law.
 |