|
Good luck!
On 25/04/2016 10:01 PM, Kevin Lemonnier wrote: Hi, So I'm trying that now. I installed 3.7.11 on two nodes and put a few VMs on it, same config as before but with 64MB shards and the heal algo to full. As expected, if I poweroff one of the nodes, everything is dead, which is fine. Now I'm adding a third node, a big heal was started after the add-brick of everything (7000+ shards), and for now everything seems to be working fine on the VMs. Last time I tried adding a brick, all those VM died for the duration of the heal, so that's already pretty good. I'm gonna let it finish to copy everything on the new nodes, then I'll try to simulate nodes going down to see if my original problem of freezing and low heal time is solved with this config. For reference, here is the volume info, if someone sees something I should change : Volume Name: gluster Type: Replicate Volume ID: e4f01509-beaf-447d-821f-957cc5c20c0a Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ipvr2.client_name:/mnt/storage/gluster Brick2: ipvr3.client_name:/mnt/storage/gluster Brick3: ipvr50.client_name:/mnt/storage/gluster Options Reconfigured: cluster.quorum-type: auto cluster.server-quorum-type: server network.remote-dio: enable cluster.eager-lock: enable performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off features.shard: on features.shard-block-size: 64MB cluster.data-self-heal-algorithm: full performance.readdir-ahead: on It starts at 2 and jumps to 50 because the first server is doing something else for now, and I use 50 to be the temporary third node. If everything goes well, I'll migrate the production on the cluster, re-install the first server and do a replace-brick, which I hope will work just as well as the add-brick I'm doing now. Last replace-brick also brought everything down, but I guess that was the joy of 3.7.6 :). Thanks ! On Mon, Apr 18, 2016 at 08:17:05PM +0530, Krutika Dhananjay wrote:On Mon, Apr 18, 2016 at 8:02 PM, Kevin Lemonnier <lemonnierk@xxxxxxxxx> wrote:I will try migrating to 3.7.10, is it considered stable yet ?Oops, just realized 3.7.10 had a regression. Then 3.7.11 it is. :)Should I change the self heal algorithm even if I move to 3.7.10, or is that not necessary ? Not sure what that change might do.So the other algorithm is 'diff' which computes rolling checksum on chunks of the src(es) and sink(s), compares them and heals upon mismatch. This is known to consume lot of CPU. 'full' algo on the other hand simply copies the src into sink in chunks. With sharding, it shouldn't be all that bad copying a 256MB file (in your case) from src to sink. We've used double the block size and had no issues reported. So you could change self heal algo to full even in the upgraded cluster. -KrutikaAnyway, I'll try to create a 3.7.10 cluster in the week end slowly move the VMs on it then, Thanks a lot for your help, Regards On Mon, Apr 18, 2016 at 07:58:44PM +0530, Krutika Dhananjay wrote:Hi, Yeah, so the fuse mount log didn't convey much information. So one of the reasons heal may have taken so long (and also consumed resources) is because of a bug in self-heal where it would do heal from both source bricks in 3-way replication. With such a bug, heal would take twice the amount of time and consume resources both the times by the same amount. This issue is fixed at http://review.gluster.org/#/c/14008/ and will be available in 3.7.12. The other thing you could do is to set cluster.data-self-heal-algorithmto'full', for better heal performance and more regulated resourceconsumptionby the same. #gluster volume set <VOL> cluster.data-self-heal-algorithm full As far as sharding is concerned, some critical caching issues were fixedin3.7.7 and 3.7.8. And my guess is that the vm crash/unbootable state could be because ofthisissue, which exists in 3.7.6. 3.7.10 saw the introduction of throttled client side heals which alsomovessuch heals to the background, which is all the more helpful forpreventingstarvation of vms during client heal. Considering these factors, I think it would be better if you upgradedyourmachines to 3.7.10. Do let me know if migrating to 3.7.10 solves your issues. -Krutika On Mon, Apr 18, 2016 at 12:40 PM, Kevin Lemonnier <lemonnierk@xxxxxxxxx> wrote:Yes, but as I was saying I don't believe KVM is using a mount point, I think it uses the API (http://www.gluster.org/community/documentation/index.php/Libgfapi_with_qemu_libvirt). Might be mistaken ofcourse. Proxmox does have a mountpoint for conveniance, I'll attach those logs, hoping they contain the informations you need. They do seem to contain a lot of errors for the 15. For reference, there was a disconnect of the first brick (10.10.0.1) in the morning and then a successfull heal that caused about 40 minutes downtime of the VMs. Right after that heal finished (if my memory is correct it was about noon or close) the second brick (10.10.0.2)rebooted,and that's the one I disconnected to prevent the heal from causing another downtime. I reconnected it one at the end of the afternoon, hoping the healwould gowell but everything went down like in the morning so I disconnected it again, and waited 11pm(23:00) toreconnect it and let it finish. Thanks for your help, On Mon, Apr 18, 2016 at 12:28:28PM +0530, Krutika Dhananjay wrote:Sorry, I was referring to the glusterfs client logs. Assuming you are using FUSE mount, your log file will be in /var/log/glusterfs/<hyphenated-mount-point-path>.log -Krutika On Sun, Apr 17, 2016 at 9:37 PM, Kevin Lemonnier <lemonnierk@xxxxxxxxx>wrote:I believe Proxmox is just an interface to KVM that uses the lib,so ifI'mnot mistaken there isn't client logs ? It's not the first time I have the issue, it happens on every healonthe2 clusters I have. I did let the heal finish that night and the VMs are working now,butitis pretty scarry for future crashes or brick replacement. Should I maybe lower the shard size ? Won't solve the fact that 2brickson 3 aren't keeping the filesystem usable but might make thehealingquicker right ? Thanks Le 17 avril 2016 17:56:37 GMT+02:00, Krutika Dhananjay < kdhananj@xxxxxxxxxx> a écrit :Could you share the client logs and information about the approx time/day when you saw this issue? -Krutika On Sat, Apr 16, 2016 at 12:57 AM, Kevin Lemonnier <lemonnierk@xxxxxxxxx> wrote:Hi, We have a small glusterFS 3.7.6 cluster with 3 nodes runningwithproxmoxVM's on it. I did set up the different recommended option likethevirtgroup, but by hand since it's on debian. The shards are 256MB, if thatmatters.This morning the second node crashed, and as it came back upstartedaheal, but that basically froze all the VM's running on thatvolume.Sincewe really really can't have 40 minutes down time in the middle of the day, I justremovedthe node from the network and that stopped the heal, allowingtheVM's toaccess their disks again. The plan was to re-connecte the node in acoupleofhours to let it heal at night. But a VM crashed now, and it can't boot up again : seems tofreeztryingto access the disks. Looking at the heal info for the volume, it has gone way upsincethismorning, it looks like the VM's aren't writing to both nodes,justthe onethey are on. It seems pretty bad, we have 2 nodes on 3 up, I would expect thevolume towork just fine since it has quorum. What am I missing ? It is still too early to start the heal, is there a way tostart theVManyway right now ? I mean, it was running a moment ago so thedataisthere, it just needs to let the VM access it. Volume Name: vm-storage Type: Replicate Volume ID: a5b19324-f032-4136-aaac-5e9a4c88aaef Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: first_node:/mnt/vg1-storage Brick2: second_node:/mnt/vg1-storage Brick3: third_node:/mnt/vg1-storage Options Reconfigured: cluster.quorum-type: auto cluster.server-quorum-type: server network.remote-dio: enable cluster.eager-lock: enable performance.readdir-ahead: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off features.shard: on features.shard-block-size: 256MB cluster.server-quorum-ratio: 51% Thanks for your help -- Kevin Lemonnier PGP Fingerprint : 89A5 2283 04A0 E6E9 0111 _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users-- Envoyé de mon appareil Android avec K-9 Mail. Veuillez excuser mabrièveté._______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users-- Kevin Lemonnier PGP Fingerprint : 89A5 2283 04A0 E6E9 0111 _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users-- Kevin Lemonnier PGP Fingerprint : 89A5 2283 04A0 E6E9 0111 _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users -- Lindsay Mathieson |
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
