Ok, that's the output after the VM has been halted: [root@networker ~]# getfattr -d -m. -e hex /glustexp/pri1/brick/alfresco.qc2 getfattr: Removing leading '/' from absolute path names # file: glustexp/pri1/brick/alfresco.qc2 security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.gv_pri-client-0=0x000001390000000000000000 trusted.afr.gv_pri-client-1=0x000000000000000000000000 trusted.gfid=0x298c76de7c8643a3909f7ef77dc294fe [root@networker2 ~]# getfattr -d -m. -e hex /glustexp/pri1/brick/alfresco.qc2 getfattr: Removing leading '/' from absolute path names # file: glustexp/pri1/brick/alfresco.qc2 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.gv_pri-client-0=0x000001390000000000000000 trusted.afr.gv_pri-client-1=0x000000000000000000000000 trusted.gfid=0x298c76de7c8643a3909f7ef77dc294fe When "heal info" stops reporting alfresco.qc2 I get: [root@networker glusterfs]# getfattr -d -m. -e hex /glustexp/pri1/brick/alfresco.qc2 getfattr: Removing leading '/' from absolute path names # file: glustexp/pri1/brick/alfresco.qc2 security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.gv_pri-client-0=0x000000000000000000000000 trusted.afr.gv_pri-client-1=0x000000000000000000000000 trusted.gfid=0x298c76de7c8643a3909f7ef77dc294fe [root@networker2 ~]# getfattr -d -m. -e hex /glustexp/pri1/brick/alfresco.qc2 getfattr: Removing leading '/' from absolute path names # file: glustexp/pri1/brick/alfresco.qc2 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.gv_pri-client-0=0x000000000000000000000000 trusted.afr.gv_pri-client-1=0x000000000000000000000000 trusted.gfid=0x298c76de7c8643a3909f7ef77dc294fe Fabio ----- Messaggio originale ----- > Da: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> > A: "Fabio Rosati" <fabio.rosati@xxxxxxxxxxxxxxxxx> > Cc: "Gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx> > Inviato: Venerdì, 24 gennaio 2014 11:36:12 > Oggetto: Re: Replication delay > > This time when you stop the VM, could you get the output of "getfattr -d -m. > -e hex <file-path-on-brick>" on both the bricks to debug further. > > Pranith > ----- Original Message ----- > > From: "Fabio Rosati" <fabio.rosati@xxxxxxxxxxxxxxxxx> > > To: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> > > Cc: "Gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx> > > Sent: Friday, January 24, 2014 3:58:38 PM > > Subject: Re: Replication delay > > > > > > > > > > ----- Messaggio originale ----- > > > Da: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> > > > A: "Fabio Rosati" <fabio.rosati@xxxxxxxxxxxxxxxxx> > > > Cc: "Gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx> > > > Inviato: Venerdì, 24 gennaio 2014 11:02:15 > > > Oggetto: Re: Replication delay > > > > > > > > > > > > ----- Original Message ----- > > > > From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> > > > > To: "Fabio Rosati" <fabio.rosati@xxxxxxxxxxxxxxxxx> > > > > Cc: "Gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx> > > > > Sent: Friday, January 24, 2014 3:29:19 PM > > > > Subject: Re: Replication delay > > > > > > > > Hi Fabio, > > > > This is a known issue that has been addressed on master. It may be > > > > backported to 3.5. When a file is undergoing changes, it may > > > > appear > > > > in > > > > 'gluster volume heal <volname> info' output even when it doesn't > > > > need > > > > any self-heal. > > > > > > > > Pranith > > > > > > Sorry, I just saw that there is a self-heal happening for 15 minutes when > > > you > > > stop the VMs. How are you checking that the self-heal is happening? > > > > When I stop the VM for alfresco.qc2, "heal info" still reports alfresco.qc2 > > as in need for healing for about 15min. > > It seems this is a real out-of-sync situation because if I check the two > > bricks I get different modification times up until they are healed (no more > > reported by "heal info"). This is the bricks' status for alfresco.qc2 while > > the VM is halted: > > > > [root@networker ~]# ll /glustexp/pri1/brick/ > > totale 27769492 > > -rw-------. 2 qemu qemu 8212709376 24 gen 11:16 alfresco.qc2 > > [...] > > > > [root@networker2 ~]# ll /glustexp/pri1/brick/ > > totale 27769384 > > -rw-------. 2 qemu qemu 8212709376 24 gen 11:05 alfresco.qc2 > > [...] > > > > Bricks' status AFTER "heal info" doesn't report alfresco.qc2 anymore: > > > > [root@networker ~]# ll /glustexp/pri1/brick/ > > totale 27769492 > > -rw-------. 2 qemu qemu 8212709376 24 gen 11:05 alfresco.qc2 > > > > [root@networker2 ~]# ll /glustexp/pri1/brick/ > > totale 27769384 > > -rw-------. 2 qemu qemu 8212709376 24 gen 11:05 alfresco.qc2 > > > > Thanks for helping! > > > > Fabio > > > > > > > > Pranith > > > > > > > > ----- Original Message ----- > > > > > From: "Fabio Rosati" <fabio.rosati@xxxxxxxxxxxxxxxxx> > > > > > To: "Gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx> > > > > > Sent: Friday, January 24, 2014 3:17:27 PM > > > > > Subject: Replication delay > > > > > > > > > > Hi All, > > > > > > > > > > in a distributed-replicated volume hosting some VMs disk images > > > > > (GlusterFS > > > > > 3.4.2 on CentOS 6.5, qemu-kvm with glusterfs native support, no fuse > > > > > mount), > > > > > I always get the same two files that need healing: > > > > > > > > > > > > > > > > > > > > [root@networker ~]# gluster volume heal gv_pri info > > > > > Gathering Heal info on volume gv_pri has been successful > > > > > > > > > > > > > > > > > > > > > > > > > Brick nw1glus.gem.local:/glustexp/pri1/brick > > > > > Number of entries: 2 > > > > > /alfresco.qc2 > > > > > /remlog.qc2 > > > > > > > > > > > > > > > > > > > > > > > > > Brick nw2glus.gem.local:/glustexp/pri1/brick > > > > > Number of entries: 2 > > > > > /alfresco.qc2 > > > > > /remlog.qc2 > > > > > > > > > > > > > > > > > > > > > > > > > Brick nw3glus.gem.local:/glustexp/pri2/brick > > > > > Number of entries: 0 > > > > > > > > > > > > > > > > > > > > > > > > > Brick nw4glus.gem.local:/glustexp/pri2/brick > > > > > Number of entries: 0 > > > > > > > > > > This is not a split-brain situation (I checked) and If I stop the two > > > > > VMs > > > > > that use these images, I get the two files healed/synced in about > > > > > 15min. > > > > > This is too much time, IMHO. > > > > > In this volume there are other VM images with (smaller) disk images > > > > > replicated on the same bricks and they get synced "in real-time". > > > > > > > > > > These are the volume's details, the host "networker" is > > > > > nw1glus.gem.local > > > > > : > > > > > > > > > > > > > > > > > > > > [root@networker ~]# gluster volume info gv_pri > > > > > > > > > > Volume Name: gv_pri > > > > > Type: Distributed-Replicate > > > > > Volume ID: 3d91b91e-4d72-484f-8655-e5ed8d38bb28 > > > > > Status: Started > > > > > Number of Bricks: 2 x 2 = 4 > > > > > Transport-type: tcp > > > > > Bricks: > > > > > Brick1: nw1glus.gem.local:/glustexp/pri1/brick > > > > > Brick2: nw2glus.gem.local:/glustexp/pri1/brick > > > > > Brick3: nw3glus.gem.local:/glustexp/pri2/brick > > > > > Brick4: nw4glus.gem.local:/glustexp/pri2/brick > > > > > Options Reconfigured: > > > > > server.allow-insecure: on > > > > > storage.owner-uid: 107 > > > > > storage.owner-gid: 107 > > > > > > > > > > [root@networker ~]# gluster volume status gv_pri detail > > > > > > > > > > > > > > > Status of volume: gv_pri > > > > > ------------------------------------------------------------------------------ > > > > > Brick : Brick nw1glus.gem.local:/glustexp/pri1/brick > > > > > Port : 50178 > > > > > Online : Y > > > > > Pid : 25721 > > > > > File System : xfs > > > > > Device : /dev/mapper/vg_guests-lv_brick1 > > > > > Mount Options : rw,noatime > > > > > Inode Size : 512 > > > > > Disk Space Free : 168.4GB > > > > > Total Disk Space : 194.9GB > > > > > Inode Count : 102236160 > > > > > Free Inodes : 102236130 > > > > > ------------------------------------------------------------------------------ > > > > > Brick : Brick nw2glus.gem.local:/glustexp/pri1/brick > > > > > Port : 50178 > > > > > Online : Y > > > > > Pid : 27832 > > > > > File System : xfs > > > > > Device : /dev/mapper/vg_guests-lv_brick1 > > > > > Mount Options : rw,noatime > > > > > Inode Size : 512 > > > > > Disk Space Free : 168.4GB > > > > > Total Disk Space : 194.9GB > > > > > Inode Count : 102236160 > > > > > Free Inodes : 102236130 > > > > > ------------------------------------------------------------------------------ > > > > > Brick : Brick nw3glus.gem.local:/glustexp/pri2/brick > > > > > Port : 50182 > > > > > Online : Y > > > > > Pid : 14571 > > > > > File System : xfs > > > > > Device : /dev/mapper/vg_guests-lv_brick2 > > > > > Mount Options : rw,noatime > > > > > Inode Size : 512 > > > > > Disk Space Free : 418.3GB > > > > > Total Disk Space : 433.8GB > > > > > Inode Count : 227540992 > > > > > Free Inodes : 227540973 > > > > > ------------------------------------------------------------------------------ > > > > > Brick : Brick nw4glus.gem.local:/glustexp/pri2/brick > > > > > Port : 50181 > > > > > Online : Y > > > > > Pid : 21942 > > > > > File System : xfs > > > > > Device : /dev/mapper/vg_guests-lv_brick2 > > > > > Mount Options : rw,noatime > > > > > Inode Size : 512 > > > > > Disk Space Free : 418.3GB > > > > > Total Disk Space : 433.8GB > > > > > Inode Count : 227540992 > > > > > Free Inodes : 227540973 > > > > > > > > > > fuse-mount of the gv_pri volume: > > > > > > > > > > > > > > > > > > > > [root@networker ~]# ll -h /mnt/gluspri/ > > > > > totale 37G > > > > > -rw-------. 1 qemu qemu 7,7G 24 gen 10:21 alfresco.qc2 > > > > > -rw-------. 1 qemu qemu 4,2G 24 gen 10:22 check_mk-salmo.qc2 > > > > > -rw-------. 1 qemu qemu 27M 23 gen 16:42 newnxserver.qc2 > > > > > -rw-------. 1 qemu qemu 1,1G 23 gen 13:38 newubutest1.qc2 > > > > > -rw-------. 1 qemu qemu 11G 24 gen 10:17 nxserver.qc2 > > > > > -rw-------. 1 qemu qemu 8,1G 24 gen 10:17 remlog.qc2 > > > > > -rw-------. 1 qemu qemu 5,6G 24 gen 10:19 ubutest1.qc2 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you think this is the expected behaviour, maybe due to caching? > > > > > What > > > > > if > > > > > the most updated node goes down while the VMs are running? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks a lot, > > > > > > > > > > > > > > > > > > > > > > > > > Fabio Rosati > > > > > > > > > > _______________________________________________ > > > > > Gluster-users mailing list > > > > > Gluster-users@xxxxxxxxxxx > > > > > http://supercolony.gluster.org/mailman/listinfo/gluster-users > > > > _______________________________________________ > > > > Gluster-users mailing list > > > > Gluster-users@xxxxxxxxxxx > > > > http://supercolony.gluster.org/mailman/listinfo/gluster-users > > > > > > > > > > _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
