Ran wrote:
The line that you need to add is the one with "writeback" in it.
If you are running qemu-kvm manually, you'll need to add the "cache=writeback" > to your list of -drive option parameters.
All of this, of course, doesn't preclude applying ionice to the qemu container processes.
ionice has no affect on network mounts just local disks
so basicly its useless to ionice the kvm proccess wich takes its IO
from gluster rather then local disk .
So you are saying that the client gets bogged down by the glfs server(s)
being slow?
My original understanding was that you were saying that the glfs server
was consuming all of the disk I/O, which is clearly an entirely
different issue.
I agree with gorden , the sulotion in this particular cases need to be
IO improvments on the OS(gluster servers) level and on gluster
application level .
Are you using caching performance translators?
The setup is as follow :
2 gluster servers wich DRBD to 1 another so basicly there is only 1
active storage server with 4 disks --> 2 for os wich is md raid 1 and
2 for storage wich is also md raid 1 but over drbd replication as well
Which sync protocol are you using for DRBD? What fs are you using for
backing glusterfs? Are you mounting with noatime?
The applicatons that uses this storage are :
1) mail storage - nfs over gluster (about 3000 mail accounts)
Ouch. Just - ouch. What version of gluster are you using, what kernel,
and are you using knfsd with the patched fuse module from
fuse-2.7.4glfs11 or unfsd?
In general, my experience of performane of unfsd over gluster has bee
terrible, but knfsd isn't that much better, either (even if we ignore
the buggyness off that setup - I could never get it to work sufficiently
well to use it).
2) statistic logs of web servers - samba over gluster(some NT servers uses this)
3) kvm images - for now im testing with only 1 KVM host and 3 virtual
win2k servers)
Hmm... If your host and glfs servers are UPS backed, make sure you
enable write-caching on qemu. Note: libvirt doesn't yet support this,
even though qemu-kvm that ships with RHEL/CentOS 5.4 does. Weirdly,
virt-install supports the caching parameters for creating the qemu xml
configs, but starting up the VM, you'll find that cache=writeback isn't
there on the running qemu command line. This means you'll have to start
up your VMs manually with the suitable qemu command line. But
performance difference is worth it. Also make sure you are using the
deadline I/O scheduler for the KVM host.
what happen is that if say 1 KVM virtual win2k (not host) run for
example high IO test
with stress IO tool(inside the win2k virt) , the entire gluster
storage crowl to not functioning
including mail etc... the gluster server load is at 3 to 4
the storage barily function , this is with only 1 virtual machine(win2k kvm)
so im just wondering what will happen with say 10 virtual machines
nothing will work .
Right, I follow now what you're saying.
I agree that the md raid is affecting the all thing but i didnt think
it will be crucial .
I don't think MD RAID1 has anything to do with your problems here. The
problem you are seeing is ultimately no different to any SAN/NAS. If one
client's I/O overwhelms the capabilities of your SAN/NAS, then yes, all
the other clients will grind to a halt with it. But if you are aware of
any other SAN or NAS solutions that have resource throttling features
along the lines of what would be required to address this, I'd rather
like to hear about it - I've not seen anything with such functionality
in the wild.
In the meantime, you may find that aggressive caching on all levels
(write caching in the guest, writeback on the qemu hypervisor, cache and
write-behind on glfs) will help quite a lot. The downside is that fsync
guarantees will go out the window, so you'll have to make sure that the
machines in question are all UPS backed if in a live environment.
Gordan
