On Mon, Jun 8, 2015 at 6:36 PM, Arnaud Virlet <avirlet@xxxxxxxxxxxxxxx> wrote: > > > On 06/08/2015 03:17 PM, Andrey Korolyov wrote: >> >> On Mon, Jun 8, 2015 at 3:44 PM, Arnaud Virlet <avirlet@xxxxxxxxxxxxxxx> >> wrote: >>> >>> >>> >>> On 06/08/2015 01:59 PM, Andrey Korolyov wrote: >>>> >>>> >>>> >>>> Am I understand you right that you are using certain template engine >>>> for both OCFS- and RBD-backed volumes within a single VM` config and >>>> it does not allow per-disk cache mode separation in a suggested way? >>>> >>> My VM has 3 disks on RBD backend. disks 1 and 2 have cache=writeback, >>> disk 3 >>> ( for ocfs2 ) has cache=none in my VM XML file. When I start the VM, >>> libvirt produce a launch string with cache=wtriteback for disk 1/2, and >>> with >>> cache=none for disk 3. >>> Even if "cache = none" for disk 3, it seems doesn't take effect without >>> set >>> rbd cache = false in ceph.conf. >> >> >> It is very strange and contradictive to what it should be. Could you >> post a resulting qemu argument string, by a chance? Also please share >> a method which you are using to determine if disk uses emulator cache >> or not. >> > > Here my qemu arguments strings for the related VM: > > /usr/bin/qemu-system-x86_64 -name www-pa2-01 -S -machine > pc-i440fx-1.6,accel=kvm,usb=off -m 2048 -realtime mlock=off -smp > 2,sockets=2,cores=1,threads=1 -uuid 3542c57c-dd47-44cd-933f-7dae0b949012 > -no-user-config -nodefaults -chardev > socket,id=charmonitor,path=/var/lib/libvirt/qemu/www-pa2-01.monitor,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown > -boot order=c,menu=on,strict=on -device > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive > file=rbd:libvirt-pool/www-pa2-01:id=libvirt:key=XXXX:auth_supported=cephx\;none:mon_host=1.1.1.1\:6789\;1.1.1.2\:6789\;1.1.1.3\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback > -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 > -drive > file=rbd:libvirt-pool/www-pa2-01-data:id=libvirt:key=XXX:auth_supported=cephx\;none:mon_host=1.1.1.1\:6789\;1.1.1.2\:6789\;1.1.1.3\:6789,if=none,id=drive-virtio-disk1,format=raw,cache=writeback > -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1 > -drive > file=rbd:libvirt-pool/www-pa2-webmutu:id=libvirt:key=XXX:auth_supported=cephx\;none:mon_host=1.1.1.1\:6789\;1.1.1.2\:6789\;1.1.1.3\:6789,if=none,id=drive-virtio-disk2,format=raw,cache=none > -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk2,id=virtio-disk2 > -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev > tap,fd=29,id=hostnet0,vhost=on,vhostfd=34 -device > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:90:53:6b,bus=pci.0,addr=0x3 > -netdev tap,fd=35,id=hostnet1,vhost=on,vhostfd=36 -device > virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:7b:b9:85,bus=pci.0,addr=0x8 > -netdev tap,fd=37,id=hostnet2,vhost=on,vhostfd=38 -device > virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:2e:ce:f6,bus=pci.0,addr=0xa > -chardev pty,id=charserial0 -device > isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:5 -device > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on > > For the disk without cache: > > -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1 > -drive > file=rbd:libvirt-pool/www-pa2-webmutu:id=libvirt:key=XXX:auth_supported=cephx\;none:mon_host=1.1.1.1\:6789\;1.1.1.2\:6789\;1.1.1.3\:6789,if=none,id=drive-virtio-disk2,format=raw,cache=none > > > > I don't really have method to determine if disk use emulator's cache or not. > When I was testing if my ocfs2 cluster work correctly, I realized that if > rbd "cache = true" in ceph.conf and "cache=none" in XML file, my ocfs2 > cluster doesn't work. Cluster's member doesn't see if members join or leave > the cluster. > But if "rbd cache = false" in ceph.conf and "cache = none" in XML. OCFS2 > cluster work, cluster's members see the other members when they join or > leave. > Thanks, can you please also add a description about how your ocfs cluster set up? Honestly, there is no chance to intersect with any kind of software bug because of different entities you are using (userspace storage emulation versus kernel file system). _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |