Re: remaining entry in gluster volume heal info command even after reboot

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

In real world this also happened, and when this happened, I notice that the getfattr result of the remaining entry(from gluster v heal info command) of sn-0 blame sn-2(trusted.afr.mstate-client-2=0x00000000000000000000001c) and sn-2 blame sn-0(trusted.afr.mstate-client-0=0x000000010000000000000003), sn-1 does not blame anyone. But the file within the remaining entry does not have any changelog blaming others on 3 sn nodes.!

Since this issue just appear occasionally, So my test step is to simulate this situation in real world, I think it is better that glusterfs could also consider this situation. Do you have any comments?

 

From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Thursday, September 06, 2018 9:08 AM
To: 'Pranith Kumar Karampuri' <pkarampu@xxxxxxxxxx>
Cc: Gluster Devel <gluster-devel@xxxxxxxxxxx>; Ravishankar N <ravishankar@xxxxxxxxxx>
Subject: RE: remaining entry in gluster volume heal info command even after reboot

 

This test steps is only to simulate the real test case(reboot+ftest), in real case it also happened occasionally.

 

From: Pranith Kumar Karampuri <pkarampu@xxxxxxxxxx>
Sent: Wednesday, September 05, 2018 5:32 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou@xxxxxxxxxxxxxxx>
Cc: Gluster Devel <gluster-devel@xxxxxxxxxxx>; Ravishankar N <ravishankar@xxxxxxxxxx>
Subject: Re: remaining entry in gluster volume heal info command even after reboot

 

Looks like the test case is a bit involved and also has modifications directly on the brick. Could you let us know if there is any reason to touch the brick directly?

 

On Wed, Sep 5, 2018 at 2:53 PM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou@xxxxxxxxxxxxxxx> wrote:

I will try to reproduce (reboot + ftest)and tell you later, but in following steps you can also simulate this issue locally, at least the remaining entry happened and the entry heal quit also because of all thee empty heald_sinks.(not sure if this is exactly the same from reboot+ftest reproduced one but I guess it should be the same)

 

 

1>   Stop client quorum by command “gluster v set <volname> cluster.quorum-type none”

2>   Isolate sn-0 from sn-1 and sn-2

iptables -I OUTPUT -d sn-1.local -j DROP

iptables -I OUTPUT -d sn-2.local -j DROP

iptables -I INPUT -s sn-2.local -j DROP

iptables -I INPUT -s sn-1.local -j DROP

3>   Touch /mnt/export/testdir/common.txt on sn-0

4>   Touch /mnt/export/testidir/common.txt on sn-1

5>   On sn-1 node,Delete all  /mnt/bricks/export/brick/testdir/common.txt metadata until getfattr returns empty,

setfattr -x  trusted.afr.dirty /mnt/bricks/export/brick/testdir/common.txt 

setfattr -x  trusted.afr.export-client-0  /mnt/bricks/export/brick/testdir/common.txt

setfattr -x  trusted.gfid  /mnt/bricks/export/brick/testdir/common.txt

setfattr -x  trusted.gfid2path.53be37be7f01389d /mnt/bricks/export/brick/testdir/common.txt

then getfattr returns empty

[root@sn-1:/home/robot]

# getfattr -m . -d -e hex  /mnt/bricks/export/brick/testdir/common.txt

[root@sn-1:/home/robot]

6>   then delete the corresponding entry(common.txt) in /mnt/bricks/export/brick/.glusterfs/indices/xattrop/

[root@sn-1:/home/robot]

# rm -rf /mnt/bricks/export/brick/.glusterfs/indices/xattrop/d0d237f7-0c43-4828-8720-dfb3792fe5fb

7>Restore network on sn-0 node.

iptables -D OUTPUT -d  sn-1.local  -j DROP

      iptables -D OUTPUT -d sn-2.local  -j DROP

       iptables -D INPUT -s sn-1.local -j DROP

iptables -D INPUT -s sn-2.local  -j DROP

7>   Do touch /mnt/export/testdir/common.txt on sn-0 node

8>   Gluster v heal export info will show following and keep for long time

# gluster v heal export info

Brick sn-0.local:/mnt/bricks/export/brick

/testdir

Status: Connected

Number of entries: 1

 

Brick sn-1.local:/mnt/bricks/export/brick

Status: Connected

Number of entries: 0

 

Brick sn-2.local:/mnt/bricks/export/brick

/testdir

Status: Connected

Number of entries: 1

 

 

 

From: Pranith Kumar Karampuri <pkarampu@xxxxxxxxxx>
Sent: Wednesday, September 05, 2018 4:56 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou@xxxxxxxxxxxxxxx>
Cc: Gluster Devel <gluster-devel@xxxxxxxxxxx>; Ravishankar N <ravishankar@xxxxxxxxxx>
Subject: Re: remaining entry in gluster volume heal info command even after reboot

 

 

On Wed, Sep 5, 2018 at 1:27 PM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou@xxxxxxxxxxxxxxx> wrote:

Hi glusterfs experts:

       Good day!

       Recently when I do some test on my gluster env, I found that there are some remaining entries in command “gluster v heal mstate info” even after reboot.

    fstest_035ffc492ec43551a64087f9280ffe3e is a folder in /mnt/mstate and in this folder only one file(fstest_458bb82d8884ed5c9dadec4ed93bec4e) exists.

When I dbg by gdb on sn-0 I found that:

Parent dir changelog(fstest_035ffc492ec43551a64087f9280ffe3e) says need to heal entry, but the changelog/gfid/filetype of the only entry in parent dir shows there is nothing to be healed, so glustershd does nothing every round of heal. And this entry will remain.

My gdb shows that each round of heal on sn-0 , it exits in function __afr_selfheal_entry (if (AFR_COUNT(healed_sinks, priv->child_count) == 0)), because in this case all three healed_sinks are zero.

 

What is the return value of this function in gdb?

 

   Have you any idea how to solve this issue from glusterfs pov?
     Thanks!

 

[test steps]

       Reboot three sn nodes( sn-0, sn-1, sn-2(arbiter)) sequentially, and on another node (with glusterfs clients) run fstest.

 

[problem description]

 

Remaining entries in “gluster v heal mstate info” command even after reboot sn-0 many times, the entries are still there!

 

[root@sn-0:/home/robot]

#  gluster v heal mstate info

Brick sn-0.local:/mnt/bricks/mstate/brick

/fstest_035ffc492ec43551a64087f9280ffe3e

Status: Connected

Number of entries: 1

 

Brick sn-1.local:/mnt/bricks/mstate/brick

Status: Connected

Number of entries: 0

 

Brick sn-2.local:/mnt/bricks/mstate/brick

/fstest_035ffc492ec43551a64087f9280ffe3e

Status: Connected

Number of entries: 1

 

 

 

////////////////////////////////////////////////////////////////////some env informations///////////////////////////////////////////////////////

# gluster v info mstate

Volume Name: mstate

Type: Replicate

Volume ID: 1d896674-17a2-4ae7-aa7c-c6e22013df99

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x (2 + 1) = 3

Transport-type: tcp

Bricks:

Brick1: sn-0.local:/mnt/bricks/mstate/brick

Brick2: sn-1.local:/mnt/bricks/mstate/brick

Brick3: sn-2.local:/mnt/bricks/mstate/brick (arbiter)

Options Reconfigured:

performance.client-io-threads: off

nfs.disable: on

transport.address-family: inet

cluster.server-quorum-type: none

cluster.quorum-reads: no

cluster.favorite-child-policy: mtime

cluster.consistent-metadata: on

network.ping-timeout: 42

cluster.quorum-type: auto

server.allow-insecure: on

cluster.server-quorum-ratio: 51%

[root@sn-1:/home/robot]

 

 

[root@sn-2:/root]

# getfattr -m . -d -e hex /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/

trusted.afr.dirty=0x000000000000000000000000

trusted.afr.mstate-client-0=0x000000010000000000000003

trusted.gfid=0xa0975560eaef4cb299467101de00446a

trusted.glusterfs.dht=0x000000010000000000000000ffffffff

 

[root@sn-2:/root]

# getfattr -m . -d -e hex /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e

trusted.afr.mstate-client-0=0x000000000000000000000000

trusted.gfid=0x9fc20f587f094182816390f056f7370f

trusted.gfid2path.864159d77373ad5f=0x61303937353536302d656165662d346362322d393934362d3731303164653030343436612f6673746573745f3435386262383264383838346564356339646164656334656439336265633465

 

# cd /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e

[root@sn-2:/mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e]

# ls

fstest_458bb82d8884ed5c9dadec4ed93bec4e

[root@sn-2:/mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e]

# stat fstest_458bb82d8884ed5c9dadec4ed93bec4e

  File: fstest_458bb82d8884ed5c9dadec4ed93bec4e

  Size: 0               Blocks: 8          IO Block: 4096   fifo

Device: fd31h/64817d Inode: 22086       Links: 2

Access: (0644/prw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)

Access: 2018-08-30 04:33:17.552870661 +0300

Modify: 2018-08-30 04:33:17.552870661 +0300

Change: 2018-08-30 04:33:17.553870661 +0300

Birth: -

[root@sn-2:/mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e]

[root@sn-2:/root]

# exit

logout

Connection to sn-2.local closed.

[root@sn-0:/home/robot]

# getfattr -m . -d -e hex /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/

trusted.afr.dirty=0x000000000000000000000000

trusted.afr.mstate-client-2=0x00000000000000000000001c

trusted.gfid=0xa0975560eaef4cb299467101de00446a

trusted.glusterfs.dht=0x000000010000000000000000ffffffff

 

[root@sn-0:/home/robot]

# getfattr -m . -d -e hex /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e

trusted.gfid=0x9fc20f587f094182816390f056f7370f

trusted.gfid2path.864159d77373ad5f=0x61303937353536302d656165662d346362322d393934362d3731303164653030343436612f6673746573745f3435386262383264383838346564356339646164656334656439336265633465

 

# ls /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e

fstest_458bb82d8884ed5c9dadec4ed93bec4e

[root@sn-0:/home/robot]

# stat /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e

  File: /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e

  Size: 0               Blocks: 8          IO Block: 4096   fifo

Device: fd31h/64817d Inode: 21899       Links: 2

Access: (0644/prw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)

Access: 2018-08-30 04:33:17.552870661 +0300

Modify: 2018-08-30 04:33:17.552870661 +0300

Change: 2018-08-30 04:33:17.263809048 +0300

Birth: -

[root@sn-0:/home/robot]

[root@sn-0:/home/robot]

# ssh sn-1.local

 

USAGE OF THE ROOT ACCOUNT AND THE FULL BASH IS RECOMMENDED ONLY FOR LIMITED USE. PLEASE USE A NON-ROOT ACCOUNT AND THE SCLI SHELL (fsclish) AND/OR LIMITED BASH SHELL.

 

Read /opt/nokia/share/security/readme_root.txt for more details.

 

[root@sn-1:/root]

# getfattr -m . -d -e hex /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/

trusted.afr.dirty=0x000000000000000000000000

trusted.afr.mstate-client-0=0x000000000000000000000000

trusted.afr.mstate-client-2=0x000000000000000000000000

trusted.gfid=0xa0975560eaef4cb299467101de00446a

trusted.glusterfs.dht=0x000000010000000000000000ffffffff

 

[root@sn-1:/root]

# getfattr -m . -d -e hex /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e

trusted.afr.mstate-client-0=0x000000000000000000000000

trusted.gfid=0x9fc20f587f094182816390f056f7370f

trusted.gfid2path.864159d77373ad5f=0x61303937353536302d656165662d346362322d393934362d3731303164653030343436612f6673746573745f3435386262383264383838346564356339646164656334656439336265633465

 

[root@sn-1:/root]

[root@sn-1:/root]

# ls /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e

fstest_458bb82d8884ed5c9dadec4ed93bec4e

[root@sn-1:/root]

# stat /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e

  File: /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e

  Size: 0               Blocks: 8          IO Block: 4096   fifo

Device: fd31h/64817d Inode: 22168       Links: 2

Access: (0644/prw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)

Access: 2018-08-30 04:33:17.552870661 +0300

Modify: 2018-08-30 04:33:17.552870661 +0300

Change: 2018-08-30 04:33:17.037673648 +0300

Birth: -

[root@sn-1:/root]

 

 

 



--

Pranith



--

Pranith

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-devel
[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux