‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Thursday, November 8, 2018 11:05 AM, Ravishankar N <ravishankar@xxxxxxxxxx> wrote: > It is not a split-brain. Nodes 1 and 3 have xattrs indicating a pending > entry heal on node2 , so heal must have happened ideally. Can you check > a few things? > - Is there any disconnects between each of the shds and the brick > processes (check via statedump or look for disconnect messages in > glustershd.log). Does restarting shd via a `volume start force` solve > the problem? Yes there is one disconnect at 14:21 (UTC 13:21) because node2 ran out of memory (although it has 32 GB of RAM) and I had to reboot it. Here are the relevant log entries taken from glustershd.log on node1: [2018-11-05 13:21:16.284239] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-myvol-pro-client-1: server 192.168.10.33:49154 has not responded in the last 42 seconds, disconnecting. [2018-11-05 13:21:16.284385] I [MSGID: 114018] [client.c:2254:client_rpc_notify] 0-myvol-pro-client-1: disconnected from myvol-pro-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2018-11-05 13:21:16.284889] W [rpc-clnt-ping.c:222:rpc_clnt_ping_cbk] 0-myvol-pro-client-1: socket disconnected I also just ran a "volume start force" and saw that the glustershd processes got restarted on all 3 nodes but that did not trigger any healing. There are still the same amount of files/dirs pending heal... > - Is the symlink pointing to oc_dir present inside .glusterfs/25/e2 in > all 3 bricks? They are yes for node1 and node3 but node2 there is no such symlink... I hope that helps to debug the issue further, else please let me know if you need more info _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
