Re: split-brain

[Ravishankar N <ravishankar@xxxxxxxxxx>]

SFILE_CONTAINER_080 is the one which seems to be in split-brain. SFILE_CONTAINER_046, for which you have provided the getfattr output, hard links etc doesn't seem to be in split-brain.  We do see that the fops on SFILE_CONTAINER_046 are failing on the client translator itself due to EIO:

[2017-03-17 19:49:56.088867] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 (bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]

[2017-03-17 19:49:56.089012] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 (bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]

which is  why the sha256sum on the mount gave EIO.  And that is because the file seems to be corrupt on both bricks because the 'trusted.bit-rot.bad-file' xattr is set.

Did you write to the files directly on the backend? What is interesting is that the sha256sum is same on both the bricks despite being both marked as bad by bitrot.

-Ravi


On 03/18/2017 03:20 AM, Bernhard Dübi wrote:

Hi,

I have a situation

the volume logfile reports a possible split-brain but when I try to heal it fails because the file is not in split-brain. Any ideas?

root@chastcvtprd04:~# uname -a

Linux chastcvtprd04 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

root@chastcvtprd04:~# dpkg -l | grep gluster

ii  glusterfs-client                    3.8.8-ubuntu1~xenial1              amd64        clustered file-system (client package)

ii  glusterfs-common                    3.8.8-ubuntu1~xenial1              amd64        GlusterFS common libraries and translator modules

ii  glusterfs-server                    3.8.8-ubuntu1~xenial1              amd64        clustered file-system (server package)

 

 

root@chastcvtprd04:~# gluster volume info Server_Legal_01

 

Volume Name: Server_Legal_01

Type: Replicate

Volume ID: ce6426a8-6fd2-4f34-bb04-7817cea598fb

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: chastcvtprd04:/data/glusterfs/Server_Legal/2I-1-32/brick

Brick2: chglbcvtprd04:/data/glusterfs/Server_Legal/2I-1-32/brick

Options Reconfigured:

auth.allow: 127.0.0.1,10.30.28.43,10.30.28.44

nfs.disable: on

performance.readdir-ahead: on

diagnostics.latency-measurement: on

diagnostics.count-fop-hits: on

features.bitrot: on

features.scrub: Active

 

 

 

root@chastcvtprd04:~# ls -l /data/simpana/Data/Server_Legal_01/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

-rw-rw-rw- 1 root simpana 27357050 Jan 11 21:21 /data/simpana/Data/Server_Legal_01/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

 

 

root@chastcvtprd04:~# sha256sum /data/simpana/Data/Server_Legal_01/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

sha256sum: /data/simpana/Data/Server_Legal_01/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046: Input/output error

 

 

root@chastcvtprd04:~# gluster volume heal Server_Legal_01 info

Brick chastcvtprd04:/data/glusterfs/Server_Legal/2I-1-32/brick

Status: Connected

Number of entries: 0

 

Brick chglbcvtprd04:/data/glusterfs/Server_Legal/2I-1-32/brick

Status: Connected

Number of entries: 0

 

 

 

root@chastcvtprd04:~# getfattr -d -e hex -m .  /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

getfattr: Removing leading '/' from absolute path names

# file: data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

trusted.afr.dirty=0x000000000000000000000000

trusted.bit-rot.bad-file=0x3100

trusted.bit-rot.signature=0x010500000000000000164947a36f9e22b5dcdfb4cf69d809396302de9d6a9b3f266532e40ddc45db04

trusted.bit-rot.version=0x0500000000000000585818d80008b694

trusted.gfid=0xbfdfe21a1af3474ba6a4bc0e17edb529

 

 

 

root@chglbcvtprd04:~# getfattr -d -e hex -m .  /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

getfattr: Removing leading '/' from absolute path names

# file: data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

trusted.afr.dirty=0x000000000000000000000000

trusted.bit-rot.bad-file=0x3100

trusted.bit-rot.signature=0x010500000000000000164947a36f9e22b5dcdfb4cf69d809396302de9d6a9b3f266532e40ddc45db04

trusted.bit-rot.version=0x050000000000000058580d5f0001543b

trusted.gfid=0xbfdfe21a1af3474ba6a4bc0e17edb529

 

 

 

root@chastcvtprd04:~# sha256sum  /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

6334395ab5eae6c9956ca82b76e2f0124aff3eb3a50ef6ebccbdccad65525f73  /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

 

 

root@chglbcvtprd04:~# sha256sum  /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

6334395ab5eae6c9956ca82b76e2f0124aff3eb3a50ef6ebccbdccad65525f73  /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

 

 

 

root@chastcvtprd04:~# ls -li  /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

1163752 -rw-rw-rw- 2 root simpana 27357050 Jan 11 21:21 /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

root@chastcvtprd04:~# find /data/glusterfs/Server_Legal/2I-1-32/brick -inum 1163752

/data/glusterfs/Server_Legal/2I-1-32/brick/.glusterfs/bf/df/bfdfe21a-1af3-474b-a6a4-bc0e17edb529

/data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

 

 

root@chglbcvtprd04:~# ls -li /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

12886039523 -rw-rw-rw- 2 root simpana 27357050 Jan 11 21:21 /data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

root@chglbcvtprd04:~# find /data/glusterfs/Server_Legal/2I-1-32/brick -inum  12886039523

/data/glusterfs/Server_Legal/2I-1-32/brick/.glusterfs/bf/df/bfdfe21a-1af3-474b-a6a4-bc0e17edb529

/data/glusterfs/Server_Legal/2I-1-32/brick/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046

 

 

 

root@chastcvtprd04:/var/log/glusterfs# ls -lrt *Server_Legal_01*.log

-rw------- 1 root root 2024748 Mar 17 20:50 data-simpana-Data-Server_Legal_01.log

-rw------- 1 root root  251700 Mar 17 20:54 glfsheal-Server_Legal_01.log

root@chastcvtprd04:/var/log/glusterfs#

root@chastcvtprd04:/var/log/glusterfs# tail -100 data-simpana-Data-Server_Legal_01.log

[2017-03-17 19:12:36.122616] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid c5a0afdf-3114-4b2f-9664-b00255a08b1b: split-brain observed. [Input/output error]

[2017-03-17 19:12:36.122949] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid c5a0afdf-3114-4b2f-9664-b00255a08b1b. (Possible split-brain)

[2017-03-17 19:12:36.123099] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:12:36.123385] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:12:36.123407] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid c5a0afdf-3114-4b2f-9664-b00255a08b1b: split-brain observed. [Input/output error]

[2017-03-17 19:12:36.137285] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:12:36.137498] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080 (c5a0afdf-3114-4b2f-9664-b00255a08b1b) [Input/output error]

[2017-03-17 19:12:36.137837] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080 (c5a0afdf-3114-4b2f-9664-b00255a08b1b) [Input/output error]

[2017-03-17 19:12:36.137875] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9440, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:12:36.137917] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 321134: READ => -1 gfid=c5a0afdf-3114-4b2f-9664-b00255a08b1b fd=0x7f9dfb0f9440 (Input/output error)

[2017-03-17 19:12:36.137994] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid c5a0afdf-3114-4b2f-9664-b00255a08b1b. (Possible split-brain)

[2017-03-17 19:12:36.138172] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:12:36.138484] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:12:36.138505] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid c5a0afdf-3114-4b2f-9664-b00255a08b1b: split-brain observed. [Input/output error]

[2017-03-17 19:12:36.138802] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid c5a0afdf-3114-4b2f-9664-b00255a08b1b. (Possible split-brain)

[2017-03-17 19:12:36.138989] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:12:36.139241] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:12:36.139246] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid c5a0afdf-3114-4b2f-9664-b00255a08b1b: split-brain observed. [Input/output error]

[2017-03-17 19:12:36.139793] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:12:36.139966] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080 (c5a0afdf-3114-4b2f-9664-b00255a08b1b) [Input/output error]

[2017-03-17 19:12:36.140245] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080 (c5a0afdf-3114-4b2f-9664-b00255a08b1b) [Input/output error]

[2017-03-17 19:12:36.140274] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9440, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9283240/SFILE_CONTAINER_080 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:12:36.140316] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 321135: READ => -1 gfid=c5a0afdf-3114-4b2f-9664-b00255a08b1b fd=0x7f9dfb0f9440 (Input/output error)

[2017-03-17 19:12:47.802878] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 4619a256-3787-4c06-ac57-33dfb174456e. (Possible split-brain)

[2017-03-17 19:12:47.803128] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:12:47.803342] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:12:47.803373] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid 4619a256-3787-4c06-ac57-33dfb174456e: split-brain observed. [Input/output error]

[2017-03-17 19:12:47.803524] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 4619a256-3787-4c06-ac57-33dfb174456e. (Possible split-brain)

[2017-03-17 19:12:47.803692] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:12:47.803999] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:12:47.804020] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 4619a256-3787-4c06-ac57-33dfb174456e: split-brain observed. [Input/output error]

[2017-03-17 19:12:47.804576] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:12:47.804746] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070 (4619a256-3787-4c06-ac57-33dfb174456e) [Input/output error]

[2017-03-17 19:12:47.805046] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070 (4619a256-3787-4c06-ac57-33dfb174456e) [Input/output error]

[2017-03-17 19:12:47.805078] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f94cc, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:12:47.805120] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 332890: READ => -1 gfid=4619a256-3787-4c06-ac57-33dfb174456e fd=0x7f9dfb0f94cc (Input/output error)

[2017-03-17 19:12:47.805177] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 4619a256-3787-4c06-ac57-33dfb174456e. (Possible split-brain)

[2017-03-17 19:12:47.805353] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:12:47.805690] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:12:47.805712] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid 4619a256-3787-4c06-ac57-33dfb174456e: split-brain observed. [Input/output error]

[2017-03-17 19:12:47.806462] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 4619a256-3787-4c06-ac57-33dfb174456e. (Possible split-brain)

[2017-03-17 19:12:47.806663] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:12:47.806944] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:12:47.806950] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 4619a256-3787-4c06-ac57-33dfb174456e: split-brain observed. [Input/output error]

[2017-03-17 19:12:47.807486] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:12:47.807651] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070 (4619a256-3787-4c06-ac57-33dfb174456e) [Input/output error]

[2017-03-17 19:12:47.807993] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070 (4619a256-3787-4c06-ac57-33dfb174456e) [Input/output error]

[2017-03-17 19:12:47.808044] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f94cc, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9286373/SFILE_CONTAINER_070 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:12:47.808086] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 332891: READ => -1 gfid=4619a256-3787-4c06-ac57-33dfb174456e fd=0x7f9dfb0f94cc (Input/output error)

[2017-03-17 19:11:20.346125] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid 6de87045-ece9-40e8-bdf3-7dad12de561f: split-brain observed. [Input/output error]

The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 6de87045-ece9-40e8-bdf3-7dad12de561f. (Possible split-brain)" repeated 3 times between [2017-03-17 19:11:20.343014] and [2017-03-17 19:11:20.346996]

[2017-03-17 19:12:59.771690] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc. (Possible split-brain)

[2017-03-17 19:12:59.771978] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:12:59.772209] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:12:59.772228] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid 1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc: split-brain observed. [Input/output error]

[2017-03-17 19:12:59.772958] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc: split-brain observed. [Input/output error]

[2017-03-17 19:12:59.773509] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:12:59.773677] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116 (1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc) [Input/output error]

[2017-03-17 19:12:59.774045] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116 (1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc) [Input/output error]

[2017-03-17 19:12:59.774132] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9184, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:12:59.774218] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 345004: READ => -1 gfid=1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc fd=0x7f9dfb0f9184 (Input/output error)

The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]" repeated 3 times between [2017-03-17 19:12:59.772209] and [2017-03-17 19:12:59.775988]

[2017-03-17 19:12:59.775996] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc: split-brain observed. [Input/output error]

[2017-03-17 19:12:59.776538] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:12:59.776814] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116 (1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc) [Input/output error]

[2017-03-17 19:12:59.777067] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116 (1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc) [Input/output error]

[2017-03-17 19:12:59.777131] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 345006: READ => -1 gfid=1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc fd=0x7f9dfb0f9184 (Input/output error)

[2017-03-17 19:12:59.774854] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid 1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc: split-brain observed. [Input/output error]

The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 1b7d36fc-fe3f-4e4c-91a5-4a33dd7884bc. (Possible split-brain)" repeated 3 times between [2017-03-17 19:12:59.771690] and [2017-03-17 19:12:59.775548]

The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]" repeated 3 times between [2017-03-17 19:12:59.771978] and [2017-03-17 19:12:59.775719]

[2017-03-17 19:12:59.777105] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9184, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944459/CHUNK_9288450/SFILE_CONTAINER_116 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:26:51.275897] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid bfdfe21a-1af3-474b-a6a4-bc0e17edb529. (Possible split-brain)

[2017-03-17 19:26:51.276390] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing GETXATTR on gfid bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. [Input/output error]

[2017-03-17 19:26:51.276427] W [fuse-bridge.c:3441:fuse_xattr_cbk] 0-glusterfs-fuse: 1154312: GETXATTR((null)) /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 => -1 (Input/output error)

[2017-03-17 19:49:56.076750] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid bfdfe21a-1af3-474b-a6a4-bc0e17edb529. (Possible split-brain)

[2017-03-17 19:49:56.077000] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:49:56.077259] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:49:56.077294] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. [Input/output error]

[2017-03-17 19:49:56.078283] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. [Input/output error]

[2017-03-17 19:49:56.078337] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. [Input/output error]

[2017-03-17 19:49:56.088463] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:49:56.088539] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:49:56.088830] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 (bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]

[2017-03-17 19:49:56.088867] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 (bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]

[2017-03-17 19:49:56.089012] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 (bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]

[2017-03-17 19:49:56.089060] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9184, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:49:56.089081] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 (bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]

[2017-03-17 19:49:56.089107] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 2491526: READ => -1 gfid=bfdfe21a-1af3-474b-a6a4-bc0e17edb529 fd=0x7f9dfb0f9184 (Input/output error)

[2017-03-17 19:49:56.089124] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9184, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:49:56.089165] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 2491525: READ => -1 gfid=bfdfe21a-1af3-474b-a6a4-bc0e17edb529 fd=0x7f9dfb0f9184 (Input/output error)

The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]" repeated 5 times between [2017-03-17 19:49:56.077259] and [2017-03-17 19:49:56.090419]

[2017-03-17 19:49:56.090426] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. [Input/output error]

[2017-03-17 19:49:56.091034] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:49:56.091201] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 (bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]

[2017-03-17 19:49:56.091519] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 (bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]

[2017-03-17 19:49:56.091578] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 2491534: READ => -1 gfid=bfdfe21a-1af3-474b-a6a4-bc0e17edb529 fd=0x7f9dfb0f9184 (Input/output error)

The message "E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid bfdfe21a-1af3-474b-a6a4-bc0e17edb529: split-brain observed. [Input/output error]" repeated 2 times between [2017-03-17 19:49:56.077294] and [2017-03-17 19:49:56.089723]

The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid bfdfe21a-1af3-474b-a6a4-bc0e17edb529. (Possible split-brain)" repeated 5 times between [2017-03-17 19:49:56.076750] and [2017-03-17 19:49:56.089951]

The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]" repeated 5 times between [2017-03-17 19:49:56.077000] and [2017-03-17 19:49:56.090086]

[2017-03-17 19:49:56.091548] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f9dfb0f9184, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046 @ Server_Legal_01-replicate-0 [Input/output error]

root@chastcvtprd04:/var/log/glusterfs#

root@chastcvtprd04:/var/log/glusterfs# tail -100 glfsheal-Server_Legal_01.log

[2017-03-17 19:19:32.393846] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:19:32.561091] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-1: Server lk version = 1

[2017-03-17 19:19:32.561904] I [MSGID: 108031] [afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0: selecting local read_child Server_Legal_01-client-0

[2017-03-17 19:19:32.562494] I [MSGID: 104041] [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to graph 63686173-7463-7674-7072-6430342d3334 (0)

[2017-03-17 19:23:55.983126] I [MSGID: 104045] [glfs-master.c:91:notify] 0-gfapi: New graph 63686173-7463-7674-7072-6430342d3939 (0) coming up

[2017-03-17 19:23:55.983162] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-0: parent translators are ready, attempting connect on transport

[2017-03-17 19:23:56.003121] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-1: parent translators are ready, attempting connect on transport

[2017-03-17 19:23:56.003303] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-0: changing port to 49635 (from 0)

[2017-03-17 19:23:56.004036] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:23:56.004251] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:23:56.004269] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:23:56.004307] I [MSGID: 108005] [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume 'Server_Legal_01-client-0' came back up; going online.

[2017-03-17 19:23:56.004329] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-0: Server lk version = 1

[2017-03-17 19:23:56.004367] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-1: changing port to 49658 (from 0)

[2017-03-17 19:23:56.005341] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:23:56.005862] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:23:56.005877] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:23:56.099082] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-1: Server lk version = 1

[2017-03-17 19:23:56.099922] I [MSGID: 108031] [afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0: selecting local read_child Server_Legal_01-client-0

[2017-03-17 19:23:56.100670] I [MSGID: 104041] [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to graph 63686173-7463-7674-7072-6430342d3939 (0)

[2017-03-17 19:34:07.952797] I [MSGID: 104045] [glfs-master.c:91:notify] 0-gfapi: New graph 63686173-7463-7674-7072-6430342d3331 (0) coming up

[2017-03-17 19:34:07.957727] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-0: parent translators are ready, attempting connect on transport

[2017-03-17 19:34:07.958670] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-1: parent translators are ready, attempting connect on transport

[2017-03-17 19:34:07.958853] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-0: changing port to 49635 (from 0)

[2017-03-17 19:34:07.959549] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:34:07.959819] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:34:07.959832] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:34:07.959865] I [MSGID: 108005] [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume 'Server_Legal_01-client-0' came back up; going online.

[2017-03-17 19:34:07.959885] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-0: Server lk version = 1

[2017-03-17 19:34:07.960019] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-1: changing port to 49658 (from 0)

[2017-03-17 19:34:07.960957] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:34:07.961445] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:34:07.961457] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:34:08.130678] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-1: Server lk version = 1

[2017-03-17 19:34:08.131439] I [MSGID: 108031] [afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0: selecting local read_child Server_Legal_01-client-0

[2017-03-17 19:34:08.132110] I [MSGID: 104041] [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to graph 63686173-7463-7674-7072-6430342d3331 (0)

[2017-03-17 19:36:40.730984] I [MSGID: 104045] [glfs-master.c:91:notify] 0-gfapi: New graph 63686173-7463-7674-7072-6430342d3234 (0) coming up

[2017-03-17 19:36:40.731023] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-0: parent translators are ready, attempting connect on transport

[2017-03-17 19:36:40.732963] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-1: parent translators are ready, attempting connect on transport

[2017-03-17 19:36:40.733160] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-0: changing port to 49635 (from 0)

[2017-03-17 19:36:40.733704] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:36:40.733952] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:36:40.733974] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:36:40.734012] I [MSGID: 108005] [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume 'Server_Legal_01-client-0' came back up; going online.

[2017-03-17 19:36:40.734034] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-0: Server lk version = 1

[2017-03-17 19:36:40.734119] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-1: changing port to 49658 (from 0)

[2017-03-17 19:36:40.735092] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:36:40.735594] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:36:40.735607] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:36:40.935215] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-1: Server lk version = 1

[2017-03-17 19:36:40.936135] I [MSGID: 108031] [afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0: selecting local read_child Server_Legal_01-client-0

[2017-03-17 19:36:40.936856] I [MSGID: 104041] [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to graph 63686173-7463-7674-7072-6430342d3234 (0)

[2017-03-17 19:44:08.745876] I [MSGID: 104045] [glfs-master.c:91:notify] 0-gfapi: New graph 63686173-7463-7674-7072-6430342d3335 (0) coming up

[2017-03-17 19:44:08.745911] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-0: parent translators are ready, attempting connect on transport

[2017-03-17 19:44:08.748918] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-1: parent translators are ready, attempting connect on transport

[2017-03-17 19:44:08.749122] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-0: changing port to 49635 (from 0)

[2017-03-17 19:44:08.749677] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:44:08.749980] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:44:08.750000] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:44:08.750047] I [MSGID: 108005] [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume 'Server_Legal_01-client-0' came back up; going online.

[2017-03-17 19:44:08.750071] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-0: Server lk version = 1

[2017-03-17 19:44:08.750145] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-1: changing port to 49658 (from 0)

[2017-03-17 19:44:08.751171] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:44:08.751669] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:44:08.751682] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:44:08.871739] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-1: Server lk version = 1

[2017-03-17 19:44:08.872568] I [MSGID: 108031] [afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0: selecting local read_child Server_Legal_01-client-0

[2017-03-17 19:44:08.873269] I [MSGID: 104041] [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to graph 63686173-7463-7674-7072-6430342d3335 (0)

[2017-03-17 19:51:04.343728] I [MSGID: 104045] [glfs-master.c:91:notify] 0-gfapi: New graph 63686173-7463-7674-7072-6430342d3231 (0) coming up

[2017-03-17 19:51:04.343761] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-0: parent translators are ready, attempting connect on transport

[2017-03-17 19:51:04.345547] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-1: parent translators are ready, attempting connect on transport

[2017-03-17 19:51:04.345702] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-0: changing port to 49635 (from 0)

[2017-03-17 19:51:04.346288] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:51:04.346666] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:51:04.346681] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:51:04.346724] I [MSGID: 108005] [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume 'Server_Legal_01-client-0' came back up; going online.

[2017-03-17 19:51:04.346756] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-0: Server lk version = 1

[2017-03-17 19:51:04.346788] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-1: changing port to 49658 (from 0)

[2017-03-17 19:51:04.347909] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:51:04.348419] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:51:04.348432] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:51:04.528253] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-1: Server lk version = 1

[2017-03-17 19:51:04.529286] I [MSGID: 108031] [afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0: selecting local read_child Server_Legal_01-client-0

[2017-03-17 19:51:04.530379] I [MSGID: 104041] [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to graph 63686173-7463-7674-7072-6430342d3231 (0)

[2017-03-17 19:54:04.941563] I [MSGID: 104045] [glfs-master.c:91:notify] 0-gfapi: New graph 63686173-7463-7674-7072-6430342d3232 (0) coming up

[2017-03-17 19:54:04.941606] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-0: parent translators are ready, attempting connect on transport

[2017-03-17 19:54:04.942498] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-1: parent translators are ready, attempting connect on transport

[2017-03-17 19:54:04.942753] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-0: changing port to 49635 (from 0)

[2017-03-17 19:54:04.943498] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:54:04.943737] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-1: changing port to 49658 (from 0)

[2017-03-17 19:54:04.943823] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:54:04.943848] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:54:04.943893] I [MSGID: 108005] [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume 'Server_Legal_01-client-0' came back up; going online.

[2017-03-17 19:54:04.943914] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-0: Server lk version = 1

[2017-03-17 19:54:04.944870] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:54:04.945439] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:54:04.945460] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:54:05.292434] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-1: Server lk version = 1

[2017-03-17 19:54:05.293381] I [MSGID: 108031] [afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0: selecting local read_child Server_Legal_01-client-0

[2017-03-17 19:54:05.294255] I [MSGID: 104041] [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to graph 63686173-7463-7674-7072-6430342d3232 (0)

root@chastcvtprd04:/var/log/glusterfs#

 

 

 

 

 

root@chglbcvtprd04:/var/log/glusterfs# ls -lrt *Server_Legal_01*.log

-rw------- 1 root root  240660 Mar 17 20:52 glfsheal-Server_Legal_01.log

-rw------- 1 root root 7788358 Mar 17 20:53 data-simpana-Data-Server_Legal_01.log

root@chglbcvtprd04:/var/log/glusterfs# tail -100 data-simpana-Data-Server_Legal_01.log

[2017-03-17 19:48:39.804078] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:48:39.804249] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:48:39.804253] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 53f03480-7281-48c1-8daf-51cd3099b47f: split-brain observed. [Input/output error]

[2017-03-17 19:48:39.804716] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:48:39.804900] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9292716/SFILE_CONTAINER_030 (53f03480-7281-48c1-8daf-51cd3099b47f) [Input/output error]

[2017-03-17 19:48:39.805156] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9292716/SFILE_CONTAINER_030 (53f03480-7281-48c1-8daf-51cd3099b47f) [Input/output error]

[2017-03-17 19:48:39.805185] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002328, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9292716/SFILE_CONTAINER_030 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:48:39.805224] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 7535988: READ => -1 gfid=53f03480-7281-48c1-8daf-51cd3099b47f fd=0x7f1a94002328 (Input/output error)

[2017-03-17 19:47:45.522943] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid ddc01be9-1c4a-4535-adc3-91a61e326e3e: split-brain observed. [Input/output error]

The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid ddc01be9-1c4a-4535-adc3-91a61e326e3e. (Possible split-brain)" repeated 3 times between [2017-03-17 19:47:45.520414] and [2017-03-17 19:47:45.523127]

[2017-03-17 19:49:44.398208] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid e09bffec-f6f4-432f-8f68-ec3520c6f211. (Possible split-brain)

[2017-03-17 19:49:44.398446] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:49:44.398693] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:49:44.398720] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid e09bffec-f6f4-432f-8f68-ec3520c6f211: split-brain observed. [Input/output error]

[2017-03-17 19:49:44.399501] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid e09bffec-f6f4-432f-8f68-ec3520c6f211: split-brain observed. [Input/output error]

[2017-03-17 19:49:44.400061] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:49:44.400258] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044 (e09bffec-f6f4-432f-8f68-ec3520c6f211) [Input/output error]

[2017-03-17 19:49:44.400507] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044 (e09bffec-f6f4-432f-8f68-ec3520c6f211) [Input/output error]

[2017-03-17 19:49:44.400537] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002328, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:49:44.400585] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 7538116: READ => -1 gfid=e09bffec-f6f4-432f-8f68-ec3520c6f211 fd=0x7f1a94002328 (Input/output error)

The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]" repeated 3 times between [2017-03-17 19:49:44.398693] and [2017-03-17 19:49:44.401700]

[2017-03-17 19:49:44.401709] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid e09bffec-f6f4-432f-8f68-ec3520c6f211: split-brain observed. [Input/output error]

[2017-03-17 19:49:44.402194] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:49:44.402372] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044 (e09bffec-f6f4-432f-8f68-ec3520c6f211) [Input/output error]

[2017-03-17 19:49:44.402627] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044 (e09bffec-f6f4-432f-8f68-ec3520c6f211) [Input/output error]

[2017-03-17 19:49:44.402678] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 7538117: READ => -1 gfid=e09bffec-f6f4-432f-8f68-ec3520c6f211 fd=0x7f1a94002328 (Input/output error)

[2017-03-17 19:49:44.402654] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002328, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_044 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:50:47.215179] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid e71812cf-725d-4e95-a471-59cf099055e5. (Possible split-brain)

The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]" repeated 4 times between [2017-03-17 19:49:44.398446] and [2017-03-17 19:50:47.215449]

[2017-03-17 19:50:47.215623] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:50:47.215659] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid e71812cf-725d-4e95-a471-59cf099055e5: split-brain observed. [Input/output error]

[2017-03-17 19:50:47.215974] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid e71812cf-725d-4e95-a471-59cf099055e5. (Possible split-brain)

[2017-03-17 19:50:47.216138] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:50:47.216339] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:50:47.216355] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid e71812cf-725d-4e95-a471-59cf099055e5: split-brain observed. [Input/output error]

[2017-03-17 19:50:47.216802] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:50:47.217006] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005 (e71812cf-725d-4e95-a471-59cf099055e5) [Input/output error]

[2017-03-17 19:50:47.217221] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005 (e71812cf-725d-4e95-a471-59cf099055e5) [Input/output error]

[2017-03-17 19:50:47.217254] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002440, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:50:47.217298] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 7538180: READ => -1 gfid=e71812cf-725d-4e95-a471-59cf099055e5 fd=0x7f1a94002440 (Input/output error)

[2017-03-17 19:50:47.217382] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid e71812cf-725d-4e95-a471-59cf099055e5. (Possible split-brain)

[2017-03-17 19:50:47.217572] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:50:47.217779] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:50:47.217799] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid e71812cf-725d-4e95-a471-59cf099055e5: split-brain observed. [Input/output error]

[2017-03-17 19:50:47.217966] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid e71812cf-725d-4e95-a471-59cf099055e5. (Possible split-brain)

[2017-03-17 19:50:47.218181] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:50:47.218389] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:50:47.218394] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid e71812cf-725d-4e95-a471-59cf099055e5: split-brain observed. [Input/output error]

[2017-03-17 19:50:47.218842] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:50:47.219020] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005 (e71812cf-725d-4e95-a471-59cf099055e5) [Input/output error]

[2017-03-17 19:50:47.219200] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005 (e71812cf-725d-4e95-a471-59cf099055e5) [Input/output error]

[2017-03-17 19:50:47.219231] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002440, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293741/SFILE_CONTAINER_005 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:50:47.219270] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 7538181: READ => -1 gfid=e71812cf-725d-4e95-a471-59cf099055e5 fd=0x7f1a94002440 (Input/output error)

[2017-03-17 19:49:44.401142] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid e09bffec-f6f4-432f-8f68-ec3520c6f211: split-brain observed. [Input/output error]

The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid e09bffec-f6f4-432f-8f68-ec3520c6f211. (Possible split-brain)" repeated 3 times between [2017-03-17 19:49:44.398208] and [2017-03-17 19:49:44.401301]

[2017-03-17 19:51:13.064446] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 2532bb1c-6241-4fd0-92df-30059be2be17. (Possible split-brain)

[2017-03-17 19:51:13.064691] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:51:13.065358] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:51:13.065389] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid 2532bb1c-6241-4fd0-92df-30059be2be17: split-brain observed. [Input/output error]

[2017-03-17 19:51:13.066393] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 2532bb1c-6241-4fd0-92df-30059be2be17: split-brain observed. [Input/output error]

[2017-03-17 19:51:13.066873] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:51:13.067147] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110 (2532bb1c-6241-4fd0-92df-30059be2be17) [Input/output error]

[2017-03-17 19:51:13.067281] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110 (2532bb1c-6241-4fd0-92df-30059be2be17) [Input/output error]

[2017-03-17 19:51:13.067366] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002440, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:51:13.067416] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 7538199: READ => -1 gfid=2532bb1c-6241-4fd0-92df-30059be2be17 fd=0x7f1a94002440 (Input/output error)

The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]" repeated 3 times between [2017-03-17 19:51:13.065358] and [2017-03-17 19:51:13.068580]

[2017-03-17 19:51:13.068586] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 2532bb1c-6241-4fd0-92df-30059be2be17: split-brain observed. [Input/output error]

[2017-03-17 19:51:13.069092] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:51:13.069276] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110 (2532bb1c-6241-4fd0-92df-30059be2be17) [Input/output error]

[2017-03-17 19:51:13.069538] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110 (2532bb1c-6241-4fd0-92df-30059be2be17) [Input/output error]

[2017-03-17 19:51:13.069624] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 7538200: READ => -1 gfid=2532bb1c-6241-4fd0-92df-30059be2be17 fd=0x7f1a94002440 (Input/output error)

[2017-03-17 19:51:13.069597] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f1a94002440, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9294213/SFILE_CONTAINER_110 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:52:14.972489] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 5752e22b-5e89-4543-b7d7-da2a22af09df. (Possible split-brain)

The message "W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]" repeated 4 times between [2017-03-17 19:51:13.064691] and [2017-03-17 19:52:14.972716]

[2017-03-17 19:52:14.972967] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:52:14.973008] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid 5752e22b-5e89-4543-b7d7-da2a22af09df: split-brain observed. [Input/output error]

[2017-03-17 19:52:14.973861] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 5752e22b-5e89-4543-b7d7-da2a22af09df. (Possible split-brain)

[2017-03-17 19:52:14.974022] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:52:14.974239] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:52:14.974263] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 5752e22b-5e89-4543-b7d7-da2a22af09df: split-brain observed. [Input/output error]

[2017-03-17 19:52:14.974724] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:52:14.974893] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062 (5752e22b-5e89-4543-b7d7-da2a22af09df) [Input/output error]

[2017-03-17 19:52:14.975112] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062 (5752e22b-5e89-4543-b7d7-da2a22af09df) [Input/output error]

[2017-03-17 19:52:14.975146] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f1a940020f8, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:52:14.975192] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 7538225: READ => -1 gfid=5752e22b-5e89-4543-b7d7-da2a22af09df fd=0x7f1a940020f8 (Input/output error)

[2017-03-17 19:52:14.975266] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 5752e22b-5e89-4543-b7d7-da2a22af09df. (Possible split-brain)

[2017-03-17 19:52:14.975463] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:52:14.975654] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:52:14.975685] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid 5752e22b-5e89-4543-b7d7-da2a22af09df: split-brain observed. [Input/output error]

[2017-03-17 19:52:14.975860] W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 5752e22b-5e89-4543-b7d7-da2a22af09df. (Possible split-brain)

[2017-03-17 19:52:14.976067] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-1: remote operation failed [Input/output error]

[2017-03-17 19:52:14.976240] W [MSGID: 114031] [client-rpc-fops.c:1494:client3_3_fstat_cbk] 0-Server_Legal_01-client-0: remote operation failed [Input/output error]

[2017-03-17 19:52:14.976248] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing FGETXATTR on gfid 5752e22b-5e89-4543-b7d7-da2a22af09df: split-brain observed. [Input/output error]

[2017-03-17 19:52:14.976742] W [MSGID: 108027] [afr-common.c:2491:afr_discover_done] 0-Server_Legal_01-replicate-0: no read subvols for (null)

[2017-03-17 19:52:14.976933] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-1: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062 (5752e22b-5e89-4543-b7d7-da2a22af09df) [Input/output error]

[2017-03-17 19:52:14.977122] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-Server_Legal_01-client-0: remote operation failed. Path: /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062 (5752e22b-5e89-4543-b7d7-da2a22af09df) [Input/output error]

[2017-03-17 19:52:14.977152] E [MSGID: 109089] [dht-helper.c:1297:dht_migration_complete_check_task] 0-Server_Legal_01-dht: failed to open the fd (0x7f1a940020f8, flags=0100000) on file /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9292929/SFILE_CONTAINER_062 @ Server_Legal_01-replicate-0 [Input/output error]

[2017-03-17 19:52:14.977193] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 7538226: READ => -1 gfid=5752e22b-5e89-4543-b7d7-da2a22af09df fd=0x7f1a940020f8 (Input/output error)

[2017-03-17 19:51:13.067915] E [MSGID: 108008] [afr-read-txn.c:80:afr_read_txn_refresh_done] 0-Server_Legal_01-replicate-0: Failing READ on gfid 2532bb1c-6241-4fd0-92df-30059be2be17: split-brain observed. [Input/output error]

The message "W [MSGID: 108008] [afr-read-txn.c:238:afr_read_txn] 0-Server_Legal_01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 2532bb1c-6241-4fd0-92df-30059be2be17. (Possible split-brain)" repeated 3 times between [2017-03-17 19:51:13.064446] and [2017-03-17 19:51:13.068212]

root@chglbcvtprd04:/var/log/glusterfs#

root@chglbcvtprd04:/var/log/glusterfs# tail -100 glfsheal-Server_Legal_01.log

[2017-03-17 19:02:46.687131] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:02:46.778952] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-0: Server lk version = 1

[2017-03-17 19:02:46.779801] I [MSGID: 108031] [afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0: selecting local read_child Server_Legal_01-client-1

[2017-03-17 19:02:46.780649] I [MSGID: 104041] [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to graph 6368676c-6263-7674-7072-6430342d3334 (0)

[2017-03-17 19:12:50.450406] I [MSGID: 104045] [glfs-master.c:91:notify] 0-gfapi: New graph 6368676c-6263-7674-7072-6430342d3139 (0) coming up

[2017-03-17 19:12:50.450442] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-0: parent translators are ready, attempting connect on transport

[2017-03-17 19:12:50.451271] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-1: parent translators are ready, attempting connect on transport

[2017-03-17 19:12:50.451783] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-1: changing port to 49658 (from 0)

[2017-03-17 19:12:50.452408] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-0: changing port to 49635 (from 0)

[2017-03-17 19:12:50.453418] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:12:50.454094] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:12:50.454109] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:12:50.454151] I [MSGID: 108005] [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume 'Server_Legal_01-client-0' came back up; going online.

[2017-03-17 19:12:50.454510] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-0: Server lk version = 1

[2017-03-17 19:12:50.459877] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:12:50.460109] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:12:50.460123] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:12:50.594044] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-1: Server lk version = 1

[2017-03-17 19:12:50.594904] I [MSGID: 108031] [afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0: selecting local read_child Server_Legal_01-client-1

[2017-03-17 19:12:50.595728] I [MSGID: 104041] [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to graph 6368676c-6263-7674-7072-6430342d3139 (0)

[2017-03-17 19:22:39.620877] I [MSGID: 104045] [glfs-master.c:91:notify] 0-gfapi: New graph 6368676c-6263-7674-7072-6430342d3337 (0) coming up

[2017-03-17 19:22:39.620916] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-0: parent translators are ready, attempting connect on transport

[2017-03-17 19:22:39.621777] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-1: parent translators are ready, attempting connect on transport

[2017-03-17 19:22:39.622293] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-1: changing port to 49658 (from 0)

[2017-03-17 19:22:39.622741] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:22:39.622912] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-0: changing port to 49635 (from 0)

[2017-03-17 19:22:39.622991] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:22:39.623011] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:22:39.623057] I [MSGID: 108005] [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume 'Server_Legal_01-client-1' came back up; going online.

[2017-03-17 19:22:39.623080] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-1: Server lk version = 1

[2017-03-17 19:22:39.623953] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:22:39.625582] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:22:39.625599] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:22:39.721866] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-0: Server lk version = 1

[2017-03-17 19:22:39.722617] I [MSGID: 108031] [afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0: selecting local read_child Server_Legal_01-client-1

[2017-03-17 19:22:39.723187] I [MSGID: 104041] [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to graph 6368676c-6263-7674-7072-6430342d3337 (0)

[2017-03-17 19:32:36.816124] I [MSGID: 104045] [glfs-master.c:91:notify] 0-gfapi: New graph 6368676c-6263-7674-7072-6430342d3130 (0) coming up

[2017-03-17 19:32:36.827326] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-0: parent translators are ready, attempting connect on transport

[2017-03-17 19:32:36.877385] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-1: parent translators are ready, attempting connect on transport

[2017-03-17 19:32:36.877896] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-1: changing port to 49658 (from 0)

[2017-03-17 19:32:36.878285] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:32:36.878345] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-0: changing port to 49635 (from 0)

[2017-03-17 19:32:36.878567] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:32:36.878587] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:32:36.878626] I [MSGID: 108005] [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume 'Server_Legal_01-client-1' came back up; going online.

[2017-03-17 19:32:36.878646] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-1: Server lk version = 1

[2017-03-17 19:32:36.879552] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:32:36.892670] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:32:36.892690] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:32:37.030054] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-0: Server lk version = 1

[2017-03-17 19:32:37.030893] I [MSGID: 108031] [afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0: selecting local read_child Server_Legal_01-client-1

[2017-03-17 19:32:37.031642] I [MSGID: 104041] [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to graph 6368676c-6263-7674-7072-6430342d3130 (0)

[2017-03-17 19:42:49.185410] I [MSGID: 104045] [glfs-master.c:91:notify] 0-gfapi: New graph 6368676c-6263-7674-7072-6430342d3339 (0) coming up

[2017-03-17 19:42:49.193109] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-0: parent translators are ready, attempting connect on transport

[2017-03-17 19:42:49.194028] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-1: parent translators are ready, attempting connect on transport

[2017-03-17 19:42:49.194544] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-1: changing port to 49658 (from 0)

[2017-03-17 19:42:49.194997] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:42:49.195080] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-0: changing port to 49635 (from 0)

[2017-03-17 19:42:49.195233] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:42:49.195249] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:42:49.195287] I [MSGID: 108005] [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume 'Server_Legal_01-client-1' came back up; going online.

[2017-03-17 19:42:49.195307] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-1: Server lk version = 1

[2017-03-17 19:42:49.196118] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:42:49.196576] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:42:49.196592] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:42:49.359040] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-0: Server lk version = 1

[2017-03-17 19:42:49.359995] I [MSGID: 108031] [afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0: selecting local read_child Server_Legal_01-client-1

[2017-03-17 19:42:49.360629] I [MSGID: 104041] [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to graph 6368676c-6263-7674-7072-6430342d3339 (0)

[2017-03-17 19:52:42.886903] I [MSGID: 104045] [glfs-master.c:91:notify] 0-gfapi: New graph 6368676c-6263-7674-7072-6430342d3439 (0) coming up

[2017-03-17 19:52:42.887194] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-0: parent translators are ready, attempting connect on transport

[2017-03-17 19:52:42.888075] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-1: parent translators are ready, attempting connect on transport

[2017-03-17 19:52:42.888592] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-1: changing port to 49658 (from 0)

[2017-03-17 19:52:42.888974] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-0: changing port to 49635 (from 0)

[2017-03-17 19:52:42.889131] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:52:42.889475] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:52:42.889499] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:52:42.889542] I [MSGID: 108005] [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume 'Server_Legal_01-client-1' came back up; going online.

[2017-03-17 19:52:42.889574] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-1: Server lk version = 1

[2017-03-17 19:52:42.889891] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 19:52:42.896888] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 19:52:42.896926] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 19:52:43.024347] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-0: Server lk version = 1

[2017-03-17 19:52:43.025326] I [MSGID: 108031] [afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0: selecting local read_child Server_Legal_01-client-1

[2017-03-17 19:52:43.025934] I [MSGID: 104041] [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to graph 6368676c-6263-7674-7072-6430342d3439 (0)

[2017-03-17 20:02:39.027898] I [MSGID: 104045] [glfs-master.c:91:notify] 0-gfapi: New graph 6368676c-6263-7674-7072-6430342d3337 (0) coming up

[2017-03-17 20:02:39.028098] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-0: parent translators are ready, attempting connect on transport

[2017-03-17 20:02:39.029060] I [MSGID: 114020] [client.c:2356:notify] 0-Server_Legal_01-client-1: parent translators are ready, attempting connect on transport

[2017-03-17 20:02:39.029750] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-1: changing port to 49658 (from 0)

[2017-03-17 20:02:39.029976] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-Server_Legal_01-client-0: changing port to 49635 (from 0)

[2017-03-17 20:02:39.030502] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 20:02:39.030780] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-1: Connected to Server_Legal_01-client-1, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 20:02:39.030801] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-1: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 20:02:39.030841] I [MSGID: 108005] [afr-common.c:4382:afr_notify] 0-Server_Legal_01-replicate-0: Subvolume 'Server_Legal_01-client-1' came back up; going online.

[2017-03-17 20:02:39.030873] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-1: Server lk version = 1

[2017-03-17 20:02:39.030900] I [MSGID: 114057] [client-handshake.c:1447:select_server_supported_programs] 0-Server_Legal_01-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)

[2017-03-17 20:02:39.031378] I [MSGID: 114046] [client-handshake.c:1223:client_setvolume_cbk] 0-Server_Legal_01-client-0: Connected to Server_Legal_01-client-0, attached to remote volume '/data/glusterfs/Server_Legal/2I-1-32/brick'.

[2017-03-17 20:02:39.031392] I [MSGID: 114047] [client-handshake.c:1234:client_setvolume_cbk] 0-Server_Legal_01-client-0: Server and Client lk-version numbers are not same, reopening the fds

[2017-03-17 20:02:39.147618] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-Server_Legal_01-client-0: Server lk version = 1

[2017-03-17 20:02:39.148469] I [MSGID: 108031] [afr-common.c:2152:afr_local_discovery_cbk] 0-Server_Legal_01-replicate-0: selecting local read_child Server_Legal_01-client-1

[2017-03-17 20:02:39.148990] I [MSGID: 104041] [glfs-resolve.c:885:__glfs_active_subvol] 0-Server_Legal_01: switched to graph 6368676c-6263-7674-7072-6430342d3337 (0)

root@chglbcvtprd04:/var/log/glusterfs#

 

 

 

Healing /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9282627/SFILE_CONTAINER_066 failed: File not in split-brain.

Volume heal failed.

Healing /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9282627/SFILE_CONTAINER_034 failed: File not in split-brain.

Volume heal failed.

Healing /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293457/SFILE_CONTAINER_021 failed: File not in split-brain.

Volume heal failed.

Healing /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9282627/SFILE_CONTAINER_046 failed: File not in split-brain.

Volume heal failed.

Healing /Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9293559/SFILE_CONTAINER_005 failed: File not in split-brain.

Volume heal failed.

Healing /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9288763/SFILE_CONTAINER_030 failed: File not in split-brain.

Volume heal failed.

Healing /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9282627/SFILE_CONTAINER_014 failed: File not in split-brain.

Volume heal failed.

Healing /Server_Legal/CV_MAGNETIC/V_944456/CHUNK_9288068/SFILE_CONTAINER_004 failed: File not in split-brain.

Volume heal failed.

 

Regards

Bernhard

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users