Re: Healing issue

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 





On 08/16/2015 04:22 PM, Miloš Kozák wrote:
Hi, I have been running an glusterfs for a while, and everything works just fine even after one node failure.. However, I went for brick replacement due to my bricks were not thin-provisioned and I wanted to use snapshots. In short, whole volume went down due to heal daemon which tool all IO and all VMs running on top of that volume started to be unresponsive.

In short, I am rebuilding the volume from scratch. I created new thinly provisioned bricks:
lvs:
  brick_s3-sata-10k                  vg_s3-sata-10k Vwi-aotz 931,25g s3-sata-10k_pool                     2,95
  s3-sata-10k_pool                   vg_s3-sata-10k twi-a-tz 931,25g

vgs:
  vg_s3-sata-10k   1   3   0 wz--n- 931,51g 148,00m

df:
  /dev/mapper/vg_s3--sata--10k-brick_s3--sata--10k 976009600  28383480 947626120   3% /gfs/s3-sata-10k

and mounted. When I uploaded two images onto it I found there might be a problem. For the time being I run the volume in replica 2 mode on top of two servers. The files were copied from node1, and I think the files are OK on node1 only. However, the volume heal indicates everything is OK.
Hi,
If I understand you correctly, are you saying that when you created a new replica 2 volume with thin LVs for the bricks , mounted the volume (fuse?) and copied 2 files (VM images?) via this mount, the files were being written to only node1?

-Ravi
My symptoms are as follows:

df information from both servers:
/dev/mapper/vg_s3--sata--10k-brick_s3--sata--10k 976009600 30754296 945255304   4% /gfs/s3-sata-10k
/dev/mapper/vg_s3--sata--10k-brick_s3--sata--10k 976009600  28383480 947626120   3% /gfs/s3-sata-10k



[root@nodef01i ~]#  du /gfs/s3-sata-10k/
0    /gfs/s3-sata-10k/fs/.glusterfs/indices/xattrop
0    /gfs/s3-sata-10k/fs/.glusterfs/indices
0    /gfs/s3-sata-10k/fs/.glusterfs/changelogs/htime
0    /gfs/s3-sata-10k/fs/.glusterfs/changelogs/csnap
0    /gfs/s3-sata-10k/fs/.glusterfs/changelogs
0    /gfs/s3-sata-10k/fs/.glusterfs/00/00
0    /gfs/s3-sata-10k/fs/.glusterfs/00
0    /gfs/s3-sata-10k/fs/.glusterfs/landfill
20480004    /gfs/s3-sata-10k/fs/.glusterfs/84/26
20480004    /gfs/s3-sata-10k/fs/.glusterfs/84
10240000    /gfs/s3-sata-10k/fs/.glusterfs/d0/ff
10240000    /gfs/s3-sata-10k/fs/.glusterfs/d0
30720008    /gfs/s3-sata-10k/fs/.glusterfs
30720008    /gfs/s3-sata-10k/fs
30720008    /gfs/s3-sata-10k/

[root@nodef02i ~]# du /gfs/s3-sata-10k/
0    /gfs/s3-sata-10k/fs/.glusterfs/indices/xattrop
0    /gfs/s3-sata-10k/fs/.glusterfs/indices
0    /gfs/s3-sata-10k/fs/.glusterfs/changelogs/htime
0    /gfs/s3-sata-10k/fs/.glusterfs/changelogs/csnap
0    /gfs/s3-sata-10k/fs/.glusterfs/changelogs
0    /gfs/s3-sata-10k/fs/.glusterfs/00/00
0    /gfs/s3-sata-10k/fs/.glusterfs/00
0    /gfs/s3-sata-10k/fs/.glusterfs/landfill
18727172    /gfs/s3-sata-10k/fs/.glusterfs/84/26
18727172    /gfs/s3-sata-10k/fs/.glusterfs/84
9622016    /gfs/s3-sata-10k/fs/.glusterfs/d0/ff
9622016    /gfs/s3-sata-10k/fs/.glusterfs/d0
28349192    /gfs/s3-sata-10k/fs/.glusterfs
28349192    /gfs/s3-sata-10k/fs
28349192    /gfs/s3-sata-10k/


[root@nodef01i ~]# du /gfs/s3-sata-10k/fs/*
20480004    /gfs/s3-sata-10k/fs/f1607f25aa52f4fb6f98f20ef0f3f9d7
10240000    /gfs/s3-sata-10k/fs/3706a2cb0bb27ba5787b3c12388f4ebb

[root@nodef02i ~]# du /gfs/s3-sata-10k/fs/*
18727172    /gfs/s3-sata-10k/fs/f1607f25aa52f4fb6f98f20ef0f3f9d7
9622016        /gfs/s3-sata-10k/fs/3706a2cb0bb27ba5787b3c12388f4ebb


[root@nodef01i ~]# ll /gfs/s3-sata-10k/fs/
celkem 30720004
-rw-r----- 2 oneadmin oneadmin 20971520512  3. srp 23.53 f1607f25aa52f4fb6f98f20ef0f3f9d7
-rw-r----- 2 oneadmin oneadmin 10485760000 16. srp 11.23 3706a2cb0bb27ba5787b3c12388f4ebb

[root@nodef02i ~]# ll /gfs/s3-sata-10k/fs/
celkem 28349188
-rw-r----- 2 oneadmin oneadmin 20971520512  3. srp 23.53 f1607f25aa52f4fb6f98f20ef0f3f9d7
-rw-r----- 2 oneadmin oneadmin 10485760000 16. srp 11.22 3706a2cb0bb27ba5787b3c12388f4ebb


[root@nodef01i ~]# gluster volume heal ph-fs-0 info split-brain
Gathering list of split brain entries on volume ph-fs-0 has been successful

Brick 10.11.100.1:/gfs/s3-sata-10k/fs
Number of entries: 0

Brick 10.11.100.2:/gfs/s3-sata-10k/fs
Number of entries: 0


[root@nodef01i ~]# gluster volume heal ph-fs-0 info
Brick nodef01i.czprg:/gfs/s3-sata-10k/fs/
Number of entries: 0

Brick nodef02i.czprg:/gfs/s3-sata-10k/fs/
Number of entries: 0


[root@nodef01i ~]# gluster volume status
Status of volume: ph-fs-0
Gluster process                        Port    Online    Pid
------------------------------------------------------------------------------
Brick 10.11.100.1:/gfs/s3-sata-10k/fs            49152    Y    3733
Brick 10.11.100.2:/gfs/s3-sata-10k/fs            49152    Y    64711
NFS Server on localhost                    2049    Y    3747
Self-heal Daemon on localhost                N/A    Y    3752
NFS Server on 10.11.100.2                2049    Y    64725
Self-heal Daemon on 10.11.100.2                N/A    Y    64730

Task Status of Volume ph-fs-0
------------------------------------------------------------------------------
There are no active volume tasks

[root@nodef02i ~]# gluster volume status
Status of volume: ph-fs-0
Gluster process                        Port    Online    Pid
------------------------------------------------------------------------------
Brick 10.11.100.1:/gfs/s3-sata-10k/fs            49152    Y    3733
Brick 10.11.100.2:/gfs/s3-sata-10k/fs            49152    Y    64711
NFS Server on localhost                    2049    Y    64725
Self-heal Daemon on localhost                N/A    Y    64730
NFS Server on 10.11.100.1                2049    Y    3747
Self-heal Daemon on 10.11.100.1                N/A    Y    3752

Task Status of Volume ph-fs-0
------------------------------------------------------------------------------
There are no active volume tasks


[root@nodef02i ~]# rpm -qa | grep gluster
glusterfs-server-3.6.2-1.el6.x86_64
glusterfs-3.6.2-1.el6.x86_64
glusterfs-api-3.6.2-1.el6.x86_64
glusterfs-libs-3.6.2-1.el6.x86_64
glusterfs-cli-3.6.2-1.el6.x86_64
glusterfs-fuse-3.6.2-1.el6.x86_64

What other information should I provide?

Thanks Milos


_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux