Hi,
I simplified the config in my first email, but I actually have 2x4 servers in replicate-distribute with each 4 bricks for 6 of them and 2 bricks for the remaining 2. Full healing will just take ages... for a just single brick to resync !
> gluster v status home
volume status home
Status of volume: home
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick server1:/data/glusterfs/home/brick1 49157 0 Y 5003
Brick server1:/data/glusterfs/home/brick2 49153 0 Y 5023
Brick server1:/data/glusterfs/home/brick3 49154 0 Y 5004
Brick server1:/data/glusterfs/home/brick4 49155 0 Y 5011
Brick server3:/data/glusterfs/home/brick1 49152 0 Y 5422
Brick server4:/data/glusterfs/home/brick1 49152 0 Y 5019
Brick server3:/data/glusterfs/home/brick2 49153 0 Y 5429
Brick server4:/data/glusterfs/home/brick2 49153 0 Y 5033
Brick server3:/data/glusterfs/home/brick3 49154 0 Y 5437
Brick server4:/data/glusterfs/home/brick3 49154 0 Y 5026
Brick server3:/data/glusterfs/home/brick4 49155 0 Y 5444
Brick server4:/data/glusterfs/home/brick4 N/A N/A N N/A
Brick server5:/data/glusterfs/home/brick1 49152 0 Y 5275
Brick server6:/data/glusterfs/home/brick1 49152 0 Y 5786
Brick server5:/data/glusterfs/home/brick2 49153 0 Y 5276
Brick server6:/data/glusterfs/home/brick2 49153 0 Y 5792
Brick server5:/data/glusterfs/home/brick3 49154 0 Y 5282
Brick server6:/data/glusterfs/home/brick3 49154 0 Y 5794
Brick server5:/data/glusterfs/home/brick4 49155 0 Y 5293
Brick server6:/data/glusterfs/home/brick4 49155 0 Y 5806
Brick server7:/data/glusterfs/home/brick1 49156 0 Y 22339
Brick server8:/data/glusterfs/home/brick1 49153 0 Y 17992
Brick server7:/data/glusterfs/home/brick2 49157 0 Y 22347
Brick server8:/data/glusterfs/home/brick2 49154 0 Y 18546
NFS Server on localhost 2049 0 Y 683
Self-heal Daemon on localhost N/A N/A Y 693
NFS Server on server8 2049 0 Y 18553
Self-heal Daemon on server8 N/A N/A Y 18566
NFS Server on server5 2049 0 Y 23115
Self-heal Daemon on server5 N/A N/A Y 23121
NFS Server on server7 2049 0 Y 4201
Self-heal Daemon on server7 N/A N/A Y 4210
NFS Server on server3 2049 0 Y 5460
Self-heal Daemon on server3 N/A N/A Y 5469
NFS Server on server6 2049 0 Y 22709
Self-heal Daemon on server6 N/A N/A Y 22718
NFS Server on server4 2049 0 Y 6044
Self-heal Daemon on server4 N/A N/A Y 6243
server 2 is currently powered off as we are waiting a replacement RAID controller, as well as for
server4:/data/glusterfs/home/brick4
And as I said, there is a rebalance in progress
> gluster rebalance home status
Node Rebalanced-files size scanned failures skipped status run time in h:m:s
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 42083 23.3GB 1568065 1359 303734 in progress 16:49:31
server5 35698 23.8GB 1027934 0 240748 in progress 16:49:23
server4 35096 23.4GB 899491 0 229064 in progress 16:49:18
server3 27031 18.0GB 701759 8 182592 in progress 16:49:27
server8 0 0Bytes 327602 0 805 in progress 16:49:18
server6 35672 23.9GB 1028469 0 240810 in progress 16:49:17
server7 1 45Bytes 53 0 0 completed 0:03:53
Estimated time left for rebalance to complete : 359739:51:24
volume rebalance: home: success
Thanks,
A.
On Thursday, 1 February 2018 18:57:17 CET Serkan Çoban wrote:
> What is server4? You just mentioned server1 and server2 previously.
> Can you post the output of gluster v status volname
>
> On Thu, Feb 1, 2018 at 8:13 PM, Alessandro Ipe <Alessandro.Ipe@xxxxxxxx> wrote:
> > Hi,
> >
> >
> > Thanks. However "gluster v heal volname full" returned the following error
> > message
> > Commit failed on server4. Please check log file for details.
> >
> > I have checked the log files in /var/log/glusterfs on server4 (by grepping
> > heal), but did not get any match. What should I be looking for and in
> > which
> > log file, please ?
> >
> > Note that there is currently a rebalance process running on the volume.
> >
> >
> > Many thanks,
> >
> >
> > A.
> >
> > On Thursday, 1 February 2018 17:32:19 CET Serkan Çoban wrote:
> >> You do not need to reset brick if brick path does not change. Replace
> >> the brick format and mount, then gluster v start volname force.
> >> To start self heal just run gluster v heal volname full.
> >>
> >> On Thu, Feb 1, 2018 at 6:39 PM, Alessandro Ipe <Alessandro.Ipe@xxxxxxxx>
> >
> > wrote:
> >> > Hi,
> >> >
> >> >
> >> > My volume home is configured in replicate mode (version 3.12.4) with
> >> > the
> >> > bricks server1:/data/gluster/brick1
> >> > server2:/data/gluster/brick1
> >> >
> >> > server2:/data/gluster/brick1 was corrupted, so I killed gluster daemon
> >> > for
> >> > that brick on server2, umounted it, reformated it, remounted it and did
> >> > a>
> >> >
> >> >> gluster volume reset-brick home server2:/data/gluster/brick1
> >> >> server2:/data/gluster/brick1 commit force>
> >> >
> >> > I was expecting that the self-heal daemon would start copying data from
> >> > server1:/data/gluster/brick1 (about 7.4 TB) to the empty
> >> > server2:/data/gluster/brick1, which it only did for directories, but
> >> > not
> >> > for files.
> >> >
> >> > For the moment, I launched on the fuse mount point
> >> >
> >> >> find . | xargs stat
> >> >
> >> > but crawling the whole volume (100 TB) to trigger self-healing of a
> >> > single
> >> > brick of 7.4 TB is unefficient.
> >> >
> >> > Is there any trick to only self-heal a single brick, either by setting
> >> > some attributes to its top directory, for example ?
> >> >
> >> >
> >> > Many thanks,
> >> >
> >> >
> >> > Alessandro
> >> >
> >> >
> >> > _______________________________________________
> >> > Gluster-users mailing list
> >> > Gluster-users@xxxxxxxxxxx
> >> > http://lists.gluster.org/mailman/listinfo/gluster-users
> >
> > --
> >
> > Dr. Ir. Alessandro Ipe
> > Department of Observations Tel. +32 2 373 06 31
> > Remote Sensing from Space
> > Royal Meteorological Institute
> > Avenue Circulaire 3 Email:
> > B-1180 Brussels Belgium Alessandro.Ipe@xxxxxxxx
> > Web: http://gerb.oma.be
--
Dr. Ir. Alessandro Ipe
Department of Observations Tel. +32 2 373 06 31
Remote Sensing from Space
Royal Meteorological Institute
Avenue Circulaire 3 Email:
B-1180 Brussels Belgium Alessandro.Ipe@xxxxxxxx
Web: http://gerb.oma.be
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
