Re: sometimes entry remains in "gluster v heal vol-name info" until visit it from mnt

["Zhou, Cynthia (NSB - CN/Hangzhou)" <cynthia.zhou@xxxxxxxxxxxxxxx>]

Hi, I meet this issue again, from my env on sn-1 mstate volume brick dir there is additional fstest_49e9d4767e95a3eda0a599437ef6ce8f, but on other sn nodes(sn-0, sn-2) there is no such directory.

 

 

[root@sn-1:/mnt/bricks/mstate/brick]

# stat fstest_49e9d4767e95a3eda0a599437ef6ce8f

  File: fstest_49e9d4767e95a3eda0a599437ef6ce8f

  Size: 4096           Blocks: 16         IO Block: 4096   directory

Device: fd61h/64865d Inode: 45450       Links: 2

Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)

Access: 2018-10-10 06:51:09.045660991 +0300

Modify: 2018-10-10 06:51:09.045660991 +0300

Change: 2018-10-10 07:06:05.060278140 +0300

Birth: -

[root@sn-1:/mnt/bricks/mstate/brick]

# gluster v heal mstate info

Brick sn-0.local:/mnt/bricks/mstate/brick

Status: Connected

Number of entries: 0

 

Brick sn-2.local:/mnt/bricks/mstate/brick

Status: Connected

Number of entries: 0

 

Brick sn-1.local:/mnt/bricks/mstate/brick

/fstest_49e9d4767e95a3eda0a599437ef6ce8f

Status: Connected

Number of entries: 1

 

[root@sn-1:/mnt/bricks/mstate/brick]

# cd .glusterfs/indices/xattrop/

[root@sn-1:/mnt/bricks/mstate/brick/.glusterfs/indices/xattrop]

# ls

0c40b296-a4d6-46c9-b7a5-037105de84aa  xattrop-90daa317-04db-4704-9f34-6f571eb3691c

[root@sn-1:/mnt/bricks/mstate/brick/.glusterfs/indices/xattrop]

# cd ../../..

[root@sn-1:/mnt/bricks/mstate/brick]

# getfattr -m . -d -e hex fstest_49e9d4767e95a3eda0a599437ef6ce8f

# file: fstest_49e9d4767e95a3eda0a599437ef6ce8f

trusted.afr.mstate-client-1=0x000000000000000100000001

trusted.gfid=0x0c40b296a4d646c9b7a5037105de84aa

trusted.glusterfs.dht=0x000000010000000000000000ffffffff

 

[root@sn-1:/mnt/bricks/mstate/brick]

#

Enclosed is the glustershd log on sn-1,from which I see following log related to fstest_49e9d4767e95a3eda0a599437ef6ce8f

 

[2018-10-10 06:22:12.976487] I [MSGID: 108026] [afr-self-heald.c:341:afr_shd_selfheal] 0-mstate-replicate-0: entry: path /fstest_49e9d4767e95a3eda0a599437ef6ce8f, gfid: 0c40b296-a4d6-46c9-b7a5-037105de84aa  //this is added in the entrance of function afr_shd_selfheal.

 

 

 

 

From: Karthik Subrahmanya <ksubrahm@xxxxxxxxxx>
Sent: Saturday, September 29, 2018 12:33 AM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou@xxxxxxxxxxxxxxx>
Cc: gluster-users <gluster-users@xxxxxxxxxxx>; Ravishankar N <ravishankar@xxxxxxxxxx>
Subject: Re: sometimes entry remains in "gluster v heal vol-name info" until visit it from mnt

 

Hey,

 

Please provide the glustershd log from all the nodes and client logs on the node from where you did the lookup on the file to resolve this issue.

 

Regards,

Karthik

 

On Fri, Sep 28, 2018 at 5:27 PM Ravishankar N <ravishankar@xxxxxxxxxx> wrote:

+ gluster-users.

Adding Karthik to see if he has some cycles to look into this.

-Ravi

 

On 09/28/2018 12:07 PM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:

Hi, glusterfs expert

When I test with glusterfs version 3.12.3 I find it quite often that sometimes, there are entry remains in gluster volume heal <vol-name>info output for long time, it does not disappear until you visit it from the mount point, is this normal?

 

 

[root@sn-0:/root]

# gluster v heal services info

Brick sn-0.local:/mnt/bricks/services/brick

Status: Connected

Number of entries: 0

 

Brick sn-1.local:/mnt/bricks/services/brick

Status: Connected

Number of entries: 0

 

Brick sn-2.local:/mnt/bricks/services/brick

/fstest_88402c989256d6e39e50208c90c1e85d          //this entry remains in the output until you touch /mnt/services/ fstest_88402c989256d6e39e50208c90c1e85d

Status: Connected

Number of entries: 1

 

[root@sn-0:/root]

# ssh sn-2.local

Warning: Permanently added 'sn-2.local' (RSA) to the list of known hosts.

 

USAGE OF THE ROOT ACCOUNT AND THE FULL BASH IS RECOMMENDED ONLY FOR LIMITED USE. PLEASE USE A NON-ROOT ACCOUNT AND THE SCLI SHELL (fsclish) AND/OR LIMITED BASH SHELL.

 

Read /opt/nokia/share/security/readme_root.txt for more details.

 

[root@sn-2:/root]

# cd /mnt/bricks/services/brick/.glusterfs/indices/xattrop/

[root@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop]

# ls

9138e315-efd6-46e0-8a3a-db535078c781  xattrop-dfcd7e67-8c2d-4ef1-93e2-c180073c8d87

[root@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop]

# getfattr -m . -d -e hex /mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/

trusted.afr.services-client-1=0x000000000000000100000001

trusted.gfid=0x9138e315efd646e08a3adb535078c781

trusted.glusterfs.dht=0x000000010000000000000000ffffffff

 

[root@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop]

# getfattr -m . -d -e hex /mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/fstest_4cf1be62e0b12d3d65fac8eacb523ef3/

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/fstest_4cf1be62e0b12d3d65fac8eacb523ef3/

trusted.gfid=0x0ccb5c1f96064e699f62fdc72cf036f5

 

 

 

“fstest_88402c989256d6e39e50208c90c1e85d” is only seen from sn-2 mount point and sn-2 service brick, there is no such entry if you ls /mnt/services on sn-0 or sn-1.

[root@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop]

# cd /mnt/services/

[root@sn-2:/mnt/services]

# ls

backup           db                      fstest_88402c989256d6e39e50208c90c1e85d  LCM  NE3SAgent  _nokrcpautoremoteuser  PM9  RCP_Backup  SS_AlLightProcessor  SymptomDataUpl

commoncollector  EventCorrelationEngine  hypertracer                              Log  netserv    ODS                    ptp  rcpha       SWM

[root@sn-2:/mnt/services]

 

 

<<attachment: glustershd-sn1.zip>>

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users