|
Am 08.12.2015 um 02:59 schrieb Lindsay
Mathieson:
Hi Udo, thanks for posting your
volume info settings. Please note for the following, I am not
one of the devs, just a user, so unfortunately I have no
authoritative answers :(
I am running a very similar setup - Proxmox 4.0, three nodes,
but using ceph for our production storage. Am heavily testing
gluster 3.7 on the side. We find the performance of ceph slow on
these small setups and management of it a PITA.
Some more questions
- how are your VM images being accessed by Proxmox? gfapi?
(Proxmox Gluster storage type) or by using the fuse mount?
Sorry, forgot to say that: I'm accessing the Gluster Storage via NFS
since (at least in version 3.4 of Proxmox) the gfapi method has some
problems with sockets.
- whats your underlying filesystem
(ext4, zfs etc)
a dedicated ext4 partition
- Are you using the HA/Watchdog system in Proxmox?
I am now (watchdog HA), but Proxmox was running in non-HA mode at
the time of failure.
On 07/12/15 21:03, Udo Giacomozzi wrote:
esterday
I had a strange situation where Gluster healing corrupted *all* my VM images.
In detail:
I had about 15 VMs running (in Proxmox 4.0) totaling about 600
GB of qcow2 images. Gluster is used as storage for those images
in replicate 3 setup (ie. 3 physical servers replicating all
data).
All VMs were running on machine #1 - the two other machines (#2
and #3) were *idle*.
Gluster was fully operating (no healing) when I rebooted machine
#2.
For other reasons I had to reboot machines #2 and #3 a few
times, but since all VMs were running on machine #1 and nothing
on the other machines was accessing Gluster files, I was
confident that this wouldn't disturb Gluster.
But anyway this means that I rebootet Gluster nodes during a
healing process.
After a few minutes, Gluster files began showing corruption - up
to the point that the qcow2 files became unreadable and all VMs
stopped working.
:( sounds painful - my sympathies.
You're running 3.5.2 - thats getting rather old. I use the gluster
debian repos:
3.6.7 : http://download.gluster.org/pub/gluster/glusterfs/3.6/LATEST/Debian/
3.7.6 : http://download.gluster.org/pub/gluster/glusterfs/LATEST/Debian/jessie/
3.6.x is the latest stable, 3.7 is close to stable(?) 3.7 has some
nice new features such as sharding, which is very useful for VM
hosting - it enables much faster heal times.
I'm using the most current version provided by Proxmox APT sources.
Can 3.6 or even 3.7 Gluster nodes work together with 3.5 node? If
not I'm wondering how I could upgrade...
You might understand that I hesitate a bit to upgrade Gluster
without having some certainty that it won't make things even worse.
I mean, this is a production system..
Regards what happened with your VM's, I'm not sure. Having two
servers down should have disabled the entire store making it not
readable or writable.
I'm not sure if both servers were down at the same time (could be,
though). I'm just sure that I rebootet them rather quickly in
sequence.
Right now my credo is "never ever reboot/shutdown more than 1
node at a time and most importantly, always make sure that no
Gluster healing is in progress". For sure, I did not respect
that when I crashed my storage.
I note
that you are missing some settings that need to be set for VM
stores - there will be corruption problems if you live migrate
without them.
quick-read=off
read-ahead=off
io-cache=off
stat-prefetch=off
eager-lock=enable
remote-dio=enable
quorum-type=auto
server-quorum-type=server
"stat-prefetch=off" is particularly important.
Thanks. Is there a document that explains the reasoning behind this
config?
Does this apply to volumes for virtual HDD images only? My
"docker-repo" is still "replicate 3" type but is used by the VMs
themselves, not by the hypervisor - I guess other settings apply
there..
Thanks a lot,
Udo
|
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
