Problem: Although there are indications of trouble in the log files, gluster appears normal on boot. Clients can copy files to the distributed file system, but disconnect when an application tries to open an existing file and write over it.
I hope this is a newbie installation error, but I'm tired of pounding my head against the wall. I started with a much more complex application that had file-access problems and came up with this simple example. Thanks in advance for any guidance about how to address this problem.
Here is a short list of my observations, followed by more details for each one:
1) At boot, lots of complaints in the file system log
2) At boot, glusterd log warns of errors through RPC layer and peers being disconnected
3) Client A can copy a directory to the gluster file system
4) Client A can write a gluster file from an application
5) Client B verifies the write
6) Client A tries to open the file and overwrite, but crashes
7) Client A is now disconnected from gluster; sees core dump in root
8) After rebooting client A, file system log provides some details of the crash.
9) glusterd log doesn't provide any additional details
10) Gluster was installed via apt, not compiled from source
11) Running Raspbian GNU/Linux 10 (buster). 32 RPI-4 clients. Testing with 2 RPI-3b+ servers.
12) Clients see the 4 Tb distributed storage under /gfdata
13) At boot, clusterd isn't always ready when the clients get to the point of mounting disks, so I forcibly mount them with sudo mount -a
14) The gluster volume looks sensible at boot time
15) The pool looks okay too
=================================Details=================================
1) At boot, lots of complaints in the file system log:
pi@pi00:~ $ sudo cat /var/log/glusterfs/gfdata.log
[2020-09-30 11:57:58.596738] I [glusterfsd-mgmt.c:2005:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing
[2020-09-30 11:57:59.049660] I [glusterfsd-mgmt.c:2005:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing
[2020-09-30 11:57:59.051730] I [glusterfsd-mgmt.c:2005:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing
The message "I [MSGID: 100011] [glusterfsd.c:1570:reincarnate] 0-glusterfsd: Fetching the volume file from server..." repeated 2 times between [2020-09-30 11:57:57.312930] and [2020-09-30 11:57:57.685434]
[2020-09-30 12:08:07.427162] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001
[2020-09-30 12:08:07.427319] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001
[2020-09-30 12:08:07.429373] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
[2020-09-30 12:08:17.161867] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001
[2020-09-30 12:08:17.161877] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001
[2020-09-30 12:08:17.163889] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
[2020-09-30 12:10:54.285700] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001
[2020-09-30 12:10:54.285853] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001
[2020-09-30 12:10:54.287936] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
[2020-09-30 12:10:54.701882] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in (null) (gfid = 4202cc06-0ec3-45f7-914a-b7d9fc872088). Holes=1 overlaps=0
[2020-09-30 12:10:54.702033] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in (null) (gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6). Holes=1 overlaps=0
[2020-09-30 12:10:54.702176] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in (null) (gfid = e81dc1be-b7f7-4baf-8049-1122c5844019). Holes=1 overlaps=0
[2020-09-30 12:10:55.081494] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /examples (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
[2020-09-30 12:10:55.106417] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
[2020-09-30 12:10:55.167731] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test3 (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
[2020-09-30 12:11:06.349404] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
[2020-09-30 12:11:06.361029] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /test3: Disk layout missing, gfid = e81dc1be-b7f7-4baf-8049-1122c5844019
[2020-09-30 12:11:06.361248] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /test3, gfid = e81dc1be-b7f7-4baf-8049-1122c5844019
[2020-09-30 12:11:06.364042] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test3 (gfid = e81dc1be-b7f7-4baf-8049-1122c5844019). Holes=1 overlaps=0
[2020-09-30 12:11:06.963973] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:11:06.964619] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:11:07.014836] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:11:07.014970] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:11:07.188570] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:11:07.189452] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:11:07.192062] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:11:07.192346] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:11:07.222268] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:11:07.222880] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:11:07.225861] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:11:07.227039] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:11:07.227411] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:11:07.227699] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:11:07.228019] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:11:07.231825] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
The message "I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001" repeated 2 times between [2020-09-30 12:10:54.285853] and [2020-09-30 12:11:07.410603]
[2020-09-30 12:11:07.412422] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
The message "I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001" repeated 3 times between [2020-09-30 12:10:54.285700] and [2020-09-30 12:11:10.533318]
[2020-09-30 12:11:10.533328] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001
[2020-09-30 12:11:10.551494] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
[2020-09-30 12:11:10.574455] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /examples: Disk layout missing, gfid = 4202cc06-0ec3-45f7-914a-b7d9fc872088
[2020-09-30 12:11:10.574620] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /examples, gfid = 4202cc06-0ec3-45f7-914a-b7d9fc872088
[2020-09-30 12:11:10.577251] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /examples (gfid = 4202cc06-0ec3-45f7-914a-b7d9fc872088). Holes=1 overlaps=0
[2020-09-30 12:11:10.592811] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /test: Disk layout missing, gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6
[2020-09-30 12:11:10.592927] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /test, gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6
[2020-09-30 12:11:10.595673] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test (gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6). Holes=1 overlaps=0
[2020-09-30 12:14:35.163945] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001
[2020-09-30 12:14:35.164099] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001
[2020-09-30 12:14:35.166219] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
[2020-09-30 12:14:35.178003] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /test: Disk layout missing, gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6
[2020-09-30 12:14:35.178111] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /test, gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6
[2020-09-30 12:14:35.180824] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test (gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6). Holes=1 overlaps=0
[2020-09-30 12:14:35.694281] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:14:35.694455] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:14:35.695229] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:14:35.698612] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:14:35.698766] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:14:35.699413] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:14:36.184500] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:14:36.191364] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:14:36.192896] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:14:36.193355] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:14:36.193607] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:14:36.195633] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:17:39.677016] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001
[2020-09-30 12:17:39.677160] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001
[2020-09-30 12:17:39.679343] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
2) On boot, glusterd log warns of errors through RPC layer and peers being disconnected
pi@pi00:~ $ sudo cat /var/log/glusterfs/glusterd.log
[2020-09-30 11:57:57.678768] E [MSGID: 106167] [glusterd-handshake.c:1996:__glusterd_mgmt_hndsk_version_cbk] 0-management: Error through RPC layer, retry again later
[2020-09-30 11:57:57.678999] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(2)), xid = 0x5, unique = 0, sent = 2020-09-29 21:30:38.833820, timeout = 600 for 192.168.1.79:24007
[2020-09-30 11:57:57.679015] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 8004fcb7-2a9f-481a-8e2d-79409ea39375
[2020-09-30 11:57:58.276231] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:57:59.341029] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Gluster MGMT Handshake), op(MGMT-VERS-ACK(2)), xid = 0x4, unique = 0, sent = 2020-09-29 21:30:47.473226, timeout = 600 for 192.168.1.35:24007
[2020-09-30 11:57:59.341224] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 83800431-f92d-4c2b-a402-160bdd4a4935
[2020-09-30 11:57:59.341328] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400
[2020-09-30 11:57:59.379583] E [MSGID: 106167] [glusterd-handshake.c:1885:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Error through RPC layer, retry again later
[2020-09-30 11:57:59.379769] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(2)), xid = 0x5, unique = 0, sent = 2020-09-29 21:30:45.579517, timeout = 600 for 192.168.1.19:24007
[2020-09-30 11:57:59.379886] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 03d20361-606d-4040-8b70-76473821706a
[2020-09-30 11:57:59.379945] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (5)
[2020-09-30 11:57:59.380004] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.19:24007
[2020-09-30 11:57:59.380154] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.19> (<95e08fe4-534e-4388-94f9-c4db97b51795>), in state <Peer in Cluster>, has disconnected from glusterd.
[2020-09-30 11:57:59.380374] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held
[2020-09-30 11:57:59.380419] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data
[2020-09-30 11:58:00.137761] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: ab97b2b5-2e15-4358-86d0-62324b93007a
[2020-09-30 11:58:00.379959] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x7, unique = 0, sent = 2020-09-29 21:30:36.588912, timeout = 600 for 192.168.1.61:24007
[2020-09-30 11:58:04.189199] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error
[2020-09-30 11:58:04.189357] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000
[2020-09-30 11:58:04.189444] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x6, unique = 0, sent = 2020-09-29 21:30:36.557644, timeout = 600 for 192.168.1.61:24007
[2020-09-30 11:58:04.189506] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error
[2020-09-30 11:58:04.189574] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000
[2020-09-30 11:58:04.189686] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x6, unique = 0, sent = 2020-09-29 21:30:42.471874, timeout = 600 for 192.168.1.13:24007
[2020-09-30 11:58:04.189744] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error
[2020-09-30 11:58:04.189863] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000
[2020-09-30 11:58:04.189956] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x6, unique = 0, sent = 2020-09-29 21:30:48.137287, timeout = 600 for 192.168.1.52:24007
[2020-09-30 11:58:04.190015] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error
[2020-09-30 11:58:04.190083] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000
[2020-09-30 11:58:04.190213] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x7, unique = 0, sent = 2020-09-29 21:30:33.681570, timeout = 600 for 192.168.1.48:24007
[2020-09-30 11:58:04.190271] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error
[2020-09-30 11:58:04.190339] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000
[2020-09-30 11:58:04.190881] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x6, unique = 0, sent = 2020-09-29 21:30:47.454357, timeout = 600 for 192.168.1.15:24007
[2020-09-30 11:58:04.190946] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error
[2020-09-30 11:58:04.191018] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000
[2020-09-30 11:58:04.191150] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(2)), xid = 0x5, unique = 0, sent = 2020-09-29 21:30:49.384128, timeout = 600 for 192.168.1.69:24007
[2020-09-30 11:58:04.191239] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x6, unique = 0, sent = 2020-09-29 21:30:49.399827, timeout = 600 for 192.168.1.20:24007
[2020-09-30 11:58:04.191295] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error
[2020-09-30 11:58:04.191362] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000
[2020-09-30 11:58:04.191614] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (6)
[2020-09-30 11:58:04.192070] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.20:24007
[2020-09-30 11:58:04.192233] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.20> (<ae87a0cd-ee9a-43e5-99e6-40fdbb5b79fd>), in state <Peer in Cluster>, has disconnected from glusterd.
[2020-09-30 11:58:04.192628] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held
[2020-09-30 11:58:04.192718] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data
[2020-09-30 11:58:04.193072] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.27> (<1e90f804-99fa-43b8-8df3-07d5da49c0da>), in state <Peer in Cluster>, has disconnected from glusterd.
[2020-09-30 11:58:04.193768] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held
[2020-09-30 11:58:04.194105] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data
[2020-09-30 11:58:04.194398] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (7)
[2020-09-30 11:58:04.194964] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.48:24007
[2020-09-30 11:58:04.195226] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.48> (<40e7d4fe-702d-4b5d-93f0-cf2ff4880f27>), in state <Peer in Cluster>, has disconnected from glusterd.
[2020-09-30 11:58:04.195604] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held
[2020-09-30 11:58:04.195686] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data
[2020-09-30 11:58:04.196108] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e
[2020-09-30 11:58:04.223503] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:04.223944] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.79 (0), ret: 0, op_ret: 0
[2020-09-30 11:58:04.267435] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:04.268218] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (5)
[2020-09-30 11:58:04.268377] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.79:24007
[2020-09-30 11:58:04.268513] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.79> (<1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e>), in state <Peer in Cluster>, has disconnected from glusterd.
[2020-09-30 11:58:04.268914] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held
[2020-09-30 11:58:04.268994] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data
[2020-09-30 11:58:04.269128] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:04.269189] E [rpc-clnt.c:346:saved_frames_unwind] (null) 0-management: forced unwinding frame type(Peer mgmt) op(--(4)) called at 2020-09-30 11:58:04.246959 (xid=0x6)
[2020-09-30 11:58:04.269251] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error
[2020-09-30 11:58:04.269321] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000
[2020-09-30 11:58:04.269608] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (6)
[2020-09-30 11:58:04.269754] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.52:24007
[2020-09-30 11:58:04.269920] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.52> (<6beeff80-a573-4571-b83e-2c14e3add80a>), in state <Peer in Cluster>, has disconnected from glusterd.
[2020-09-30 11:58:04.270294] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held
[2020-09-30 11:58:04.270376] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data
[2020-09-30 11:58:04.271032] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c
[2020-09-30 11:58:04.292492] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:04.292896] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.53 (0), ret: 0, op_ret: 0
[2020-09-30 11:58:04.329487] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9
[2020-09-30 11:58:04.346693] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:04.347000] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.13 (0), ret: 0, op_ret: 0
[2020-09-30 11:58:04.420122] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400
[2020-09-30 11:58:04.455643] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (6)
[2020-09-30 11:58:04.455779] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.15:24007
[2020-09-30 11:58:04.455902] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.15> (<8b612b17-7f2b-47b5-a27a-c1e87b845241>), in state <Peer in Cluster>, has disconnected from glusterd.
[2020-09-30 11:58:04.456220] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held
[2020-09-30 11:58:04.456293] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data
[2020-09-30 11:58:04.456643] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419
[2020-09-30 11:58:04.472699] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:04.473023] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.70 (0), ret: 0, op_ret: 0
[2020-09-30 11:58:04.528589] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.70> (<8f357d63-f96f-4a09-b3a7-b128e4780419>), in state <Peer in Cluster>, has disconnected from glusterd.
[2020-09-30 11:58:04.528934] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held
[2020-09-30 11:58:04.528993] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data
[2020-09-30 11:58:04.529256] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.35> (<99f71a38-0935-46a3-853e-ab09a1a4903a>), in state <Peer in Cluster>, has disconnected from glusterd.
[2020-09-30 11:58:04.529477] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held
[2020-09-30 11:58:04.529642] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data
[2020-09-30 11:58:04.530071] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (6)
[2020-09-30 11:58:04.531818] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.61:24007
[2020-09-30 11:58:04.531933] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.61> (<75306501-6238-4f4e-b562-74c6b10a614a>), in state <Peer in Cluster>, has disconnected from glusterd.
[2020-09-30 11:58:04.532228] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held
[2020-09-30 11:58:04.532293] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data
[2020-09-30 11:58:04.532875] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: ab97b2b5-2e15-4358-86d0-62324b93007a
[2020-09-30 11:58:04.552980] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:04.553292] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.65 (0), ret: 0, op_ret: 0
[2020-09-30 11:58:04.599958] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (5)
[2020-09-30 11:58:04.600235] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.69:24007
[2020-09-30 11:58:04.601589] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.69> (<d98ebd55-514c-4c00-ab5b-7d06ba7b402d>), in state <Peer in Cluster>, has disconnected from glusterd.
[2020-09-30 11:58:04.601987] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held
[2020-09-30 11:58:04.602067] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data
[2020-09-30 11:58:04.602464] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400
[2020-09-30 11:58:04.669632] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c, host: 192.168.1.53, port: 0
[2020-09-30 11:58:04.728815] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (6)
[2020-09-30 11:58:04.728949] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.13:24007
[2020-09-30 11:58:04.729039] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.13> (<4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9>), in state <Peer in Cluster>, has disconnected from glusterd.
[2020-09-30 11:58:04.729308] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held
[2020-09-30 11:58:04.729357] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data
[2020-09-30 11:58:04.729445] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:04.729481] E [rpc-clnt.c:346:saved_frames_unwind] (null) 0-management: forced unwinding frame type(Peer mgmt) op(--(4)) called at 2020-09-30 11:58:04.402849 (xid=0x7)
[2020-09-30 11:58:04.729519] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error
[2020-09-30 11:58:04.729562] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000
[2020-09-30 11:58:04.729844] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 6ee2eb12-592f-486c-83dd-b9508066fe51, host: 192.168.1.47, port: 0
[2020-09-30 11:58:04.754469] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 6ee2eb12-592f-486c-83dd-b9508066fe51
[2020-09-30 11:58:04.959493] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:05.586162] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e
[2020-09-30 11:58:05.869649] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:06.643933] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c
[2020-09-30 11:58:06.975254] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:14.481239] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: b93ce138-cce4-4a95-994a-319a0387efb3
[2020-09-30 11:58:14.544391] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:14.544614] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.68 (0), ret: 0, op_ret: 0
[2020-09-30 11:58:14.614575] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 8004fcb7-2a9f-481a-8e2d-79409ea39375
[2020-09-30 11:58:14.614718] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 6ee2eb12-592f-486c-83dd-b9508066fe51
[2020-09-30 11:58:14.614809] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400
[2020-09-30 11:58:14.663601] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9
[2020-09-30 11:58:14.903834] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:15.739950] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: ab97b2b5-2e15-4358-86d0-62324b93007a
[2020-09-30 11:58:16.030240] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:16.564846] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: d98ebd55-514c-4c00-ab5b-7d06ba7b402d
[2020-09-30 11:58:16.565899] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: ab97b2b5-2e15-4358-86d0-62324b93007a
[2020-09-30 11:58:16.566003] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c
[2020-09-30 11:58:16.566134] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c
[2020-09-30 11:58:16.777107] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:17.278005] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:17.278249] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.69 (0), ret: 0, op_ret: 0
[2020-09-30 11:58:18.515292] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: b93ce138-cce4-4a95-994a-319a0387efb3
[2020-09-30 11:58:18.666299] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:18.949557] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795
[2020-09-30 11:58:18.956347] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:18.956620] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.19 (0), ret: 0, op_ret: 0
[2020-09-30 11:58:19.011957] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: b93ce138-cce4-4a95-994a-319a0387efb3
[2020-09-30 11:58:19.014177] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c
[2020-09-30 11:58:19.014935] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795
[2020-09-30 11:58:19.132556] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:19.464173] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400
[2020-09-30 11:58:19.516527] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400
[2020-09-30 11:58:19.543397] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 75306501-6238-4f4e-b562-74c6b10a614a
[2020-09-30 11:58:19.549901] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:19.550132] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.61 (0), ret: 0, op_ret: 0
[2020-09-30 11:58:19.576012] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9, host: 192.168.1.13, port: 0
[2020-09-30 11:58:19.623547] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 75306501-6238-4f4e-b562-74c6b10a614a, host: 192.168.1.61, port: 0
[2020-09-30 11:58:19.659552] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241, host: 192.168.1.15, port: 0
[2020-09-30 11:58:19.669103] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9
[2020-09-30 11:58:19.759223] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:21.934219] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241
[2020-09-30 11:58:22.112225] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:22.516139] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: ae87a0cd-ee9a-43e5-99e6-40fdbb5b79fd, host: 192.168.1.20, port: 0
[2020-09-30 11:58:22.532749] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 75306501-6238-4f4e-b562-74c6b10a614a
[2020-09-30 11:58:22.729363] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:43.939426] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: ae87a0cd-ee9a-43e5-99e6-40fdbb5b79fd
[2020-09-30 11:58:43.986065] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:44.452170] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 1e90f804-99fa-43b8-8df3-07d5da49c0da, host: 192.168.1.27, port: 0
[2020-09-30 11:58:44.463225] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 6beeff80-a573-4571-b83e-2c14e3add80a, host: 192.168.1.52, port: 0
[2020-09-30 11:58:44.474079] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1e90f804-99fa-43b8-8df3-07d5da49c0da
[2020-09-30 11:58:44.539423] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:45.327726] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 6beeff80-a573-4571-b83e-2c14e3add80a
[2020-09-30 11:58:45.569036] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:45.810147] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795
[2020-09-30 11:58:45.810337] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 99f71a38-0935-46a3-853e-ab09a1a4903a, host: 192.168.1.35, port: 0
[2020-09-30 11:58:45.820901] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:45.821070] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400
[2020-09-30 11:58:45.821073] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:45.830232] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9
[2020-09-30 11:58:45.830418] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241
[2020-09-30 11:58:45.830615] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d98ebd55-514c-4c00-ab5b-7d06ba7b402d
[2020-09-30 11:58:45.907496] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:46.092804] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 75306501-6238-4f4e-b562-74c6b10a614a
[2020-09-30 11:58:46.093769] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241
[2020-09-30 11:58:46.101145] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:46.101300] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.15 (0), ret: 0, op_ret: 0
[2020-09-30 11:58:46.120581] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 75306501-6238-4f4e-b562-74c6b10a614a
[2020-09-30 11:58:46.188794] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:46.404964] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: ae87a0cd-ee9a-43e5-99e6-40fdbb5b79fd
[2020-09-30 11:58:46.405549] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 6beeff80-a573-4571-b83e-2c14e3add80a
[2020-09-30 11:58:46.405686] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 1e90f804-99fa-43b8-8df3-07d5da49c0da
[2020-09-30 11:58:46.406132] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 75306501-6238-4f4e-b562-74c6b10a614a
[2020-09-30 11:58:46.406493] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400
[2020-09-30 11:58:46.415743] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241
[2020-09-30 11:58:46.492945] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:46.681793] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241
[2020-09-30 11:58:46.682456] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419
[2020-09-30 11:58:46.690024] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:46.690306] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.70 (0), ret: 0, op_ret: 0
[2020-09-30 11:58:46.747693] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 99f71a38-0935-46a3-853e-ab09a1a4903a
[2020-09-30 11:58:46.747909] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 99f71a38-0935-46a3-853e-ab09a1a4903a
[2020-09-30 11:58:46.756824] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:46.757107] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.35 (0), ret: 0, op_ret: 0
[2020-09-30 11:58:46.778487] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419
[2020-09-30 11:58:46.849136] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:47.063765] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795, host: 192.168.1.19, port: 0
[2020-09-30 11:58:47.073740] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 40e7d4fe-702d-4b5d-93f0-cf2ff4880f27, host: 192.168.1.48, port: 0
[2020-09-30 11:58:47.084813] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419, host: 192.168.1.70, port: 0
[2020-09-30 11:58:47.095474] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795
[2020-09-30 11:58:47.183420] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:47.396429] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: d98ebd55-514c-4c00-ab5b-7d06ba7b402d, host: 192.168.1.69, port: 0
[2020-09-30 11:58:47.407953] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 99f71a38-0935-46a3-853e-ab09a1a4903a
[2020-09-30 11:58:47.439796] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:47.674369] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 40e7d4fe-702d-4b5d-93f0-cf2ff4880f27
[2020-09-30 11:58:47.780995] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:48.012688] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419
[2020-09-30 11:58:48.076704] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:48.267824] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d98ebd55-514c-4c00-ab5b-7d06ba7b402d
[2020-09-30 11:58:48.343564] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:48.548045] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e, host: 192.168.1.79, port: 0
[2020-09-30 11:58:48.559020] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 99f71a38-0935-46a3-853e-ab09a1a4903a
[2020-09-30 11:58:48.559193] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419
[2020-09-30 11:58:48.559320] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 40e7d4fe-702d-4b5d-93f0-cf2ff4880f27
[2020-09-30 11:58:48.559441] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795
[2020-09-30 11:58:48.559670] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e
[2020-09-30 11:58:48.631854] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:48.879914] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: d98ebd55-514c-4c00-ab5b-7d06ba7b402d
[2020-09-30 11:58:48.880121] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400
[2020-09-30 11:58:48.912996] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c
[2020-09-30 11:58:48.939447] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:48.939731] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.53 (0), ret: 0, op_ret: 0
[2020-09-30 11:58:49.049668] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c
[2020-09-30 11:58:49.263217] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:49.728404] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c
[2020-09-30 11:58:50.950398] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e
[2020-09-30 11:58:50.951296] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400
[2020-09-30 11:58:50.969707] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e
[2020-09-30 11:58:50.985716] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.
[2020-09-30 11:58:50.986057] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.79 (0), ret: 0, op_ret: 0
[2020-09-30 11:58:51.017655] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e
[2020-09-30 11:58:51.233916] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2020-09-30 11:58:51.724591] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e
[2020-09-30 12:09:07.963638] I [MSGID: 106488] [glusterd-handler.c:1558:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-09-30 12:09:07.964701] I [MSGID: 106488] [glusterd-handler.c:1558:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-09-30 12:09:37.014817] I [MSGID: 106487] [glusterd-handler.c:1497:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
3) The bit about peers disconnecting from glusterd seems to be a red herring:
pi@pi00:~ $ ssh 192.168.1.15 'df '
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 30491968 2517040 26703808 9% /
devtmpfs 829976 0 829976 0% /dev
tmpfs 962072 0 962072 0% /dev/shm
tmpfs 962072 9564 952508 1% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 962072 0 962072 0% /sys/fs/cgroup
/dev/mmcblk0p1 258096 54605 203491 22% /boot
192.168.1.41:/data 3907024444 39370916 3867653528 2% /gfdata
tmpfs 192412 0 192412 0% /run/user/1000
3) Client A can copy a directory to the gluster file system
pi@pi00:~ $ ls test3
a.out filetest.f95 filetest.f95~
pi@pi00:~ $ cat test3/filetest.f95
program main
implicit none
integer:: i, n, nmax
real:: x(1024)
!
! Try banging on a gluster file.
!
nmax = 1
do n=1, nmax
do i=1, 1024
x(i) = float(n)*float(i)
enddo
open(10, file='testfile', form='unformatted')
write(10) x
close(10)
write(6,*) 'n = ', n
enddo
stop
end
pi@pi00:~ $ cp -r test3 /gfdata
4) Client A can write a gluster file from an application
pi@pi00:~ $ cd /gfdata/test3
pi@pi00:/gfdata/test3 $ date
Wed 30 Sep 07:31:07 CDT 2020
pi@pi00:/gfdata/test3 $ ./a.out
n = 1
5) Client B verifies the write
pi@pi10:~ $ ls -altr /gfdata/test3
total 22
-rwxrwxrwx 1 root root 8684 Sep 30 07:29 a.out
-rwxrwxrwx 1 root root 291 Sep 30 07:29 filetest.f95
-rwxrwxrwx 1 root root 265 Sep 30 07:29 filetest.f95~
-rwxrwxrwx 1 root root 4104 Sep 30 07:31 testfile
drwxrwxrwx 1 root root 4096 Sep 30 07:31 ..
drwxrwxrwx 1 root root 4096 Sep 30 07:31 .
pi@pi10:~ $ od -f /gfdata/test3/testfile
0000000 5.74e-42 1 2 3
0000020 4 5 6 7
...
0007760 1020 1021 1022 1023
0010000 1024 5.74e-42
0010010
pi@pi10:~ $
6) Client A tries to open the file and write again, but crashes
pi@pi00:/gfdata/test3 $ date
Wed 30 Sep 07:31:46 CDT 2020
pi@pi00:/gfdata/test3 $ ./a.out
At line 14 of file filetest.f95 (unit = 10, file = 'testfile')
Fortran runtime error: Software caused connection abort
Error termination. Backtrace:
7) Client A is now disconnected from gluster. Sees core dump in root
pi@pi00:/gfdata/test3 $ ls /
ls: cannot access '/gfdata': Transport endpoint is not connected
bin core etc home lost+found mnt proc run srv sys usr
boot dev gfdata lib media opt root sbin ssh tmp var
pi@pi00:/gfdata/test3 $ df
df: /gfdata: Transport endpoint is not connected
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 30491968 3982908 25237940 14% /
devtmpfs 829976 0 829976 0% /dev
tmpfs 962072 0 962072 0% /dev/shm
tmpfs 962072 9580 952492 1% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 962072 0 962072 0% /sys/fs/cgroup
/dev/mmcblk0p1 258096 54612 203484 22% /boot
tmpfs 192412 0 192412 0% /run/user/1000
8) After reboot, file system log provides some details of the crash
pi@pi00:~ $ sudo cat /var/log/glusterfs/gfdata.log
...
[2020-09-30 12:29:44.427052] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001
[2020-09-30 12:29:44.427212] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001
[2020-09-30 12:29:44.429319] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
[2020-09-30 12:29:44.472259] E [MSGID: 114031] [client-rpc-fops_v2.c:216:client4_0_mkdir_cbk] 0-data-client-1: remote operation failed. Path: /test3 [No data available]
[2020-09-30 12:29:44.480524] W [MSGID: 114031] [client-rpc-fops_v2.c:1985:client4_0_setattr_cbk] 0-data-client-1: remote operation failed [Stale file handle]
The message "I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0" repeated 2 times between [2020-09-30 12:29:44.429319] and [2020-09-30 12:31:02.580946]
[2020-09-30 12:31:02.592738] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /test3: Disk layout missing, gfid = 9873ee11-e2e6-4ee8-9322-3401863dfa8f
[2020-09-30 12:31:02.592966] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /test3, gfid = 9873ee11-e2e6-4ee8-9322-3401863dfa8f
[2020-09-30 12:31:02.595817] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test3 (gfid = 9873ee11-e2e6-4ee8-9322-3401863dfa8f). Holes=1 overlaps=0
[2020-09-30 12:31:14.008297] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:31:14.009026] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
The message "I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001" repeated 2 times between [2020-09-30 12:29:44.427052] and [2020-09-30 12:31:02.578949]
The message "I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001" repeated 2 times between [2020-09-30 12:29:44.427212] and [2020-09-30 12:31:02.578960]
[2020-09-30 12:31:48.616479] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:31:48.616780] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:31:48.627752] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:31:48.628042] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size
[2020-09-30 12:31:48.635271] E [stripe.c:2949:stripe_ftruncate] 0-data-stripe-0: no stripe count
pending frames:
frame : type(1) op(FTRUNCATE)
frame : type(1) op(FTRUNCATE)
frame : type(1) op(OPEN)
frame : type(0) op(0)
patchset: git://git.gluster.org/glusterfs.git
signal received: 6
time of crash:
2020-09-30 12:31:48
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 5.5
---------
[2020-09-30 12:42:21.300134] I [MSGID: 100030] [glusterfsd.c:2715:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 5.5 (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=192.168.1.41 --volfile-id=/data /gfdata)
[2020-09-30 12:42:21.398194] I [MSGID: 101190] [event-epoll.c:621:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-09-30 12:42:21.694558] I [MSGID: 101190] [event-epoll.c:621:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2020-09-30 12:42:21.704186] I [MSGID: 114020] [client.c:2358:notify] 0-data-client-0: parent translators are ready, attempting connect on transport
[2020-09-30 12:42:21.708976] I [MSGID: 114020] [client.c:2358:notify] 0-data-client-1: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume data-client-0
2: type protocol/client
3: option ping-timeout 42
4: option remote-host 192.168.1.41
5: option remote-subvolume /mnt/ArmorA60/data
6: option transport-type socket
7: option transport.address-family inet
8: option transport.tcp-user-timeout 0
9: option transport.socket.keepalive-time 20
10: option transport.socket.keepalive-interval 2
11: option transport.socket.keepalive-count 9
12: option send-gids true
13: end-volume
14:
15: volume data-client-1
16: type protocol/client
17: option ping-timeout 42
18: option remote-host 192.168.1.20
19: option remote-subvolume /mnt/ArmorA60/data
20: option transport-type socket
21: option transport.address-family inet
22: option transport.tcp-user-timeout 0
23: option transport.socket.keepalive-time 20
24: option transport.socket.keepalive-interval 2
25: option transport.socket.keepalive-count 9
26: option send-gids true
27: end-volume
28:
29: volume data-stripe-0
30: type cluster/stripe
31: subvolumes data-client-0 data-client-1
32: end-volume
33:
34: volume data-dht
35: type cluster/distribute
36: option lock-migration off
37: option force-migration off
38: subvolumes data-stripe-0
39: end-volume
40:
41: volume data-write-behind
42: type performance/write-behind
43: subvolumes data-dht
44: end-volume
45:
46: volume data-read-ahead
47: type performance/read-ahead
48: subvolumes data-write-behind
49: end-volume
50:
51: volume data-readdir-ahead
52: type performance/readdir-ahead
53: option parallel-readdir off
54: option rda-request-size 131072
55: option rda-cache-limit 10MB
56: subvolumes data-read-ahead
57: end-volume
58:
59: volume data-io-cache
60: type performance/io-cache
61: subvolumes data-readdir-ahead
62: end-volume
63:
64: volume data-quick-read
65: type performance/quick-read
66: subvolumes data-io-cache
67: end-volume
68:
69: volume data-open-behind
70: type performance/open-behind
71: subvolumes data-quick-read
72: end-volume
73:
74: volume data-md-cache
75: type performance/md-cache
76: subvolumes data-open-behind
77: end-volume
78:
79: volume data-io-threads
80: type performance/io-threads
81: subvolumes data-md-cache
82: end-volume
83:
84: volume data
85: type debug/io-stats
86: option log-level INFO
87: option latency-measurement off
88: option count-fop-hits off
89: subvolumes data-io-threads
90: end-volume
91:
92: volume meta-autoload
93: type meta
94: subvolumes data
95: end-volume
96:
+------------------------------------------------------------------------------+
[2020-09-30 12:42:21.718355] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-data-client-1: changing port to 49152 (from 0)
[2020-09-30 12:42:21.718726] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-data-client-0: changing port to 49152 (from 0)
[2020-09-30 12:42:21.725344] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-data-client-1: Connected to data-client-1, attached to remote volume '/mnt/ArmorA60/data'.
[2020-09-30 12:42:21.731976] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-data-client-0: Connected to data-client-0, attached to remote volume '/mnt/ArmorA60/data'.
[2020-09-30 12:42:21.747464] I [fuse-bridge.c:4267:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.31
[2020-09-30 12:42:21.747593] I [fuse-bridge.c:4878:fuse_graph_sync] 0-fuse: switched to graph 0
[2020-09-30 12:42:21.760574] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
[2020-09-30 12:42:21.776484] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001
[2020-09-30 12:42:21.776598] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001
[2020-09-30 12:42:21.778843] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
9) glusterd log doesn't provide any additional details
...
[2020-09-30 12:09:07.963638] I [MSGID: 106488] [glusterd-handler.c:1558:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-09-30 12:09:07.964701] I [MSGID: 106488] [glusterd-handler.c:1558:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2020-09-30 12:09:37.014817] I [MSGID: 106487] [glusterd-handler.c:1497:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2020-09-30 12:41:50.915448] W [MSGID: 100032] [glusterfsd.c:1500:cleanup_and_exit] 0-: received signum (15), shutting down
[2020-09-30 12:42:10.042885] I [MSGID: 100030] [glusterfsd.c:2715:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 5.5 (args: /usr/sbin/glusterd -p /run/glusterd.pid --log-level INFO)
[2020-09-30 12:42:10.247943] I [MSGID: 106478] [glusterd.c:1435:init] 0-management: Maximum allowed open file descriptors set to 65536
[2020-09-30 12:42:10.248270] I [MSGID: 106479] [glusterd.c:1491:init] 0-management: Using /var/lib/glusterd as working directory
[2020-09-30 12:42:10.248356] I [MSGID: 106479] [glusterd.c:1497:init] 0-management: Using /var/run/gluster as pid file working directory
[2020-09-30 12:42:10.313231] W [MSGID: 103071] [rdma.c:4475:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2020-09-30 12:42:10.313405] W [MSGID: 103055] [rdma.c:4774:init] 0-rdma.management: Failed to initialize IB Device
[2020-09-30 12:42:10.313500] W [rpc-transport.c:339:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2020-09-30 12:42:10.314354] W [rpcsvc.c:1789:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2020-09-30 12:42:10.314450] E [MSGID: 106244] [glusterd.c:1798:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2020-09-30 12:42:20.719035] I [MSGID: 106513] [glusterd-store.c:2282:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 50400
[2020-09-30 12:42:20.734156] I [MSGID: 106544] [glusterd.c:152:glusterd_uuid_init] 0-management: retrieved UUID: 315aa9d5-76a8-4221-9187-4727b283406a
[2020-09-30 12:42:20.783134] I [MSGID: 106498] [glusterd-handler.c:3647:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
The message "I [MSGID: 106498] [glusterd-handler.c:3647:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0" repeated 32 times between [2020-09-30 12:42:20.783134] and [2020-09-30 12:42:20.817711]
[2020-09-30 12:42:20.817930] W [MSGID: 106061] [glusterd-handler.c:3453:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
10) Gluster was installed via apt, not compiled from source
11) Running Raspbian GNU/Linux 10 (buster). 32 RPI-4 clients, 2
RPI-3b+ servers. I'll upgrade to 4 RPI-4 servers when I'm confident
gluster is working correctly.
12) Clients see the 4 Tb distributed storage under /gfdata
pi@pi00:~ $ ls /
bin dev gfdata lib media opt root sbin ssh tmp var
boot etc home lost+found mnt proc run srv sys usr
pi@pi00:~ $ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 30491968 3982860 25237988 14% /
devtmpfs 829976 0 829976 0% /dev
tmpfs 962072 0 962072 0% /dev/shm
tmpfs 962072 9580 952492 1% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 962072 0 962072 0% /sys/fs/cgroup
/dev/mmcblk0p1 258096 54612 203484 22% /boot
192.168.1.41:/data 3907024444 39371016 3867653428 2% /gfdata
tmpfs 192412 0 192412 0% /run/user/1000
pi@pi00:~ $
13) At boot, clusterd isn't always ready when the clients get to the point of mounting disks, so I forcibly mount them with sudo mount -a
pi@pi00:~/gluster-build $ head check1
echo pi00 ; ls /gfdata/test
ssh pi@192.168.1.79 'echo pi01 ; ls /gfdata/test'
ssh pi@192.168.1.69 'echo pi02 ; ls /gfdata/test'
ssh pi@192.168.1.70 'echo pi03 ; ls /gfdata/test'
ssh pi@192.168.1.65 'echo pi04 ; ls /gfdata/test'
ssh pi@192.168.1.66 'echo pi05 ; ls /gfdata/test'
ssh pi@192.168.1.64 'echo pi06 ; ls /gfdata/test'
...
pi@pi00:~/gluster-build $ ./check1
pi00
a.out checkout.txt reset_aout test_mkdir testmpi.f testmpi.f90
...
pi10
ls: cannot access '/gfdata/test': No such file or directory
pi11
ls: cannot access '/gfdata/test': No such file or directory
pi12
...
14) The gluster volume looks sensible
pi@pi00:~ $ sudo gluster volume info
Volume Name: data
Type: Stripe
Volume ID: ee3fc5a6-edae-4896-b3af-a7bf2a2d33c1
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 192.168.1.41:/mnt/ArmorA60/data
Brick2: 192.168.1.20:/mnt/ArmorA60/data
Options Reconfigured:
nfs.disable: on
transport.address-family: inet
15) The pool looks okay too
pi@pi00:~ $ sudo gluster pool list
UUID Hostname State
9709d800-4920-4e1b-b531-b952b2520323 192.168.1.14 Connected
75306501-6238-4f4e-b562-74c6b10a614a 192.168.1.61 Connected
4c1e76fd-0458-46b4-bd4a-cbb8dd9c508b 192.168.1.66 Connected
e29d6f3a-8a32-4d20-9388-e2392054557b 192.168.1.34 Connected
03d20361-606d-4040-8b70-76473821706a 192.168.1.43 Connected
dc400830-fef2-4039-85d3-92a3085d528e 192.168.1.41 Connected
ab97b2b5-2e15-4358-86d0-62324b93007a 192.168.1.65 Connected
c56df059-0f40-4138-91be-e410a9b776db 192.168.1.32 Connected
8f357d63-f96f-4a09-b3a7-b128e4780419 192.168.1.70 Connected
1e90f804-99fa-43b8-8df3-07d5da49c0da 192.168.1.27 Connected
8004fcb7-2a9f-481a-8e2d-79409ea39375 192.168.1.33 Connected
4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9 192.168.1.13 Connected
b3d4b8f9-9a56-4bed-b10b-a9bb895ab746 192.168.1.54 Connected
6beeff80-a573-4571-b83e-2c14e3add80a 192.168.1.52 Connected
a40cc7c3-ee91-4ea9-9059-a89a3bd4fad2 192.168.1.55 Connected
b93ce138-cce4-4a95-994a-319a0387efb3 192.168.1.68 Connected
83800431-f92d-4c2b-a402-160bdd4a4935 192.168.1.16 Connected
8b612b17-7f2b-47b5-a27a-c1e87b845241 192.168.1.15 Connected
e71a2ffe-0463-492c-a367-e6dc7a95845c 192.168.1.81 Connected
603f4f57-e270-4a7d-a29f-6e2376aee8a1 192.168.1.29 Connected
95e08fe4-534e-4388-94f9-c4db97b51795 192.168.1.19 Connected
d98ebd55-514c-4c00-ab5b-7d06ba7b402d 192.168.1.69 Connected
0fac620b-761c-49c9-a019-9c996dd5c4a8 192.168.1.64 Connected
6ee2eb12-592f-486c-83dd-b9508066fe51 192.168.1.47 Connected
e239fc30-5f0b-447a-9742-fd5512fc7eac 192.168.1.60 Connected
1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e 192.168.1.79 Connected
99f71a38-0935-46a3-853e-ab09a1a4903a 192.168.1.35 Connected
096a50a2-2d86-450f-9f7d-929e91444e2e 192.168.1.28 Connected
ae87a0cd-ee9a-43e5-99e6-40fdbb5b79fd 192.168.1.20 Connected
3d64bd36-4078-49af-b251-07d9cbfddf9c 192.168.1.53 Connected
daf3673f-91e6-4b11-b119-8a3be918cca7 192.168.1.44 Connected
92728258-7422-41f3-b095-b733f858d49c 192.168.1.23 Connected
40e7d4fe-702d-4b5d-93f0-cf2ff4880f27 192.168.1.48 Connected
315aa9d5-76a8-4221-9187-4727b283406a localhost Connected
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
