Re: transport endpoint not connected on just 2 files

Kingsley Tart <gluster@xxxxxxxxxxxxxxxxxxx> · Fri, 27 May 2022 09:45:28 +0100

Hi, thanks.

OK that's interesting. Picking one of the files, on bricks A and B I see this (and all of the values are identical between bricks A and B):

trusted.afr.dirty=0x000000000000000000000000
trusted.afr.gw-runqueues-client-2=0x000000010000000200000000
trusted.gfid=0xa40bb83ff3784ae09c997d272296a7a9
trusted.gfid2path.06eddbe9be9c7c75=0x30323665396561652d613661662d346365642d623863632d6261353037333339646364372f677733
trusted.glusterfs.mdata=0x01000000000000000000000000628ec57700000000007168bb00000000628ec576000000000000000000000000628ec5760000000000000000

and on brick C I see this:

trusted.gfid=0xd73992aee03e4021824b1baced973df3
trusted.gfid2path.06eddbe9be9c7c75=0x30323665396561652d613661662d346365642d623863632d6261353037333339646364372f677733
trusted.glusterfs.mdata=0x01000000000000000000000000628ec5230000000030136ca000000000628ec523000000000000000000000000628ec5230000000000000000

So brick C is missing the trusted.afr attributes and the trusted.gfid and mdata differ.

What do I need to do to fix this?

Cheers,
Kingsley.

On Fri, 2022-05-27 at 03:59 +0000, Strahil Nikolov wrote:
Check the file attributes on all bricks:

getfattr -d -e hex -m. /data/brick/gw-runqueues/<path to file>

Best Regards,
Strahil Nikolov

On Thu, May 26, 2022 at 16:05, Kingsley Tart
<gluster@xxxxxxxxxxxxxxxxxxx> wrote:
Hi,

I've got a strange issue where on all clients I've tested on (tested on
4) I have "transport endpoint is not connected" on two files in a
directory, whereas other files can be read fine.

Any ideas?

On one of the servers (all same version):

# gluster --version
glusterfs 9.1

On one of the clients (same thing with all of them) - problem with
files "gw3" and "gw11":

[root@gw6 btl]# cd /mnt/runqueues/runners/
[root@gw6 runners]# ls -la
ls: cannot access gw11: Transport endpoint is not connected
ls: cannot access gw3: Transport endpoint is not connected
total 8
drwxr-xr-x  2 root root 4096 May 26 09:48 .
drwxr-xr-x 13 root root 4096 Apr 12  2021 ..
-rw-r--r--  1 root root    0 May 26 09:49 gw1
-rw-r--r--  1 root root    0 May 26 09:49 gw10
-?????????  ? ?    ?      ?            ? gw11
-rw-r--r--  1 root root    0 May 26 09:49 gw2
-?????????  ? ?    ?      ?            ? gw3
-rw-r--r--  1 root root    0 May 26 09:49 gw4
-rw-r--r--  1 root root    0 May 26 09:49 gw6
-rw-r--r--  1 root root    0 May 26 09:49 gw7
[root@gw6 runners]# cat *
cat: gw11: Transport endpoint is not connected
cat: gw3: Transport endpoint is not connected
[root@gw6 runners]#

Querying on a server shows those two problematic files:

# gluster volume heal gw-runqueues info
Brick gluster9a:/data/brick/gw-runqueues
/runners
/runners/gw11
/runners/gw3
Status: Connected
Number of entries: 3

Brick gluster9b:/data/brick/gw-runqueues
/runners
/runners/gw11
/runners/gw3
Status: Connected
Number of entries: 3

Brick gluster9c:/data/brick/gw-runqueues
Status: Connected
Number of entries: 0

However several hours later there's no obvious change. The servers have
hardly any load and the volume is tiny. From a client:

# find /mnt/runqueues | wc -l
35

glfsheal-gw-runqueues.log from server gluster9a:
https://pastebin.com/7mPszBBM

glfsheal-gw-runqueues.log from server gluster9b:
https://pastebin.com/rxXs5Tcv

Any pointers would be much appreciated!

Cheers,
Kingsley.

________

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

________

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users