I've been seeing the same thing happen, and in our case, it's because of running a script that checks gluster from time to time (https://github.com/jtopjian/scripts/blob/master/gluster/gluster-status.sh in our case).
Do you have a job that runs and
periodically
checks for gluster health?
On Fri, Feb 14, 2020 at 3:10 AM Christian Reiss <email@xxxxxxxxxxxxxxxxxx> wrote:
Hey folks,
my logs are constantly (every few secs, continuously) swamped with
[2020-02-14 11:05:20.258542] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-0:
Connected to ssd_storage-client-0, attached to remote volume
'/gluster_bricks/node01.company.com/gluster'.
[2020-02-14 11:05:20.258559] I [MSGID: 108005]
[afr-common.c:5280:__afr_handle_child_up_event]
0-ssd_storage-replicate-0: Subvolume 'ssd_storage-client-0' came back
up; going online.
[2020-02-14 11:05:20.258920] I [rpc-clnt.c:1963:rpc_clnt_reconfig]
0-ssd_storage-client-2: changing port to 49152 (from 0)
[2020-02-14 11:05:20.259132] I [socket.c:864:__socket_shutdown]
0-ssd_storage-client-2: intentional socket shutdown(11)
[2020-02-14 11:05:20.260010] I [MSGID: 114057]
[client-handshake.c:1376:select_server_supported_programs]
0-ssd_storage-client-1: Using Program GlusterFS 4.x v1, Num (1298437),
Version (400)
[2020-02-14 11:05:20.261077] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-1:
Connected to ssd_storage-client-1, attached to remote volume
'/gluster_bricks/node02.company.com/gluster'.
[2020-02-14 11:05:20.261089] I [MSGID: 108002]
[afr-common.c:5647:afr_notify] 0-ssd_storage-replicate-0: Client-quorum
is met
[2020-02-14 11:05:20.262005] I [MSGID: 114057]
[client-handshake.c:1376:select_server_supported_programs]
0-ssd_storage-client-2: Using Program GlusterFS 4.x v1, Num (1298437),
Version (400)
[2020-02-14 11:05:20.262685] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-2:
Connected to ssd_storage-client-2, attached to remote volume
'/gluster_bricks/node03.company.com/gluster'.
[2020-02-14 11:05:20.263909] I [MSGID: 108031]
[afr-common.c:2580:afr_local_discovery_cbk] 0-ssd_storage-replicate-0:
selecting local read_child ssd_storage-client-0
[2020-02-14 11:05:20.264124] I [MSGID: 104041]
[glfs-resolve.c:954:__glfs_active_subvol] 0-ssd_storage: switched to
graph 6e6f6465-3031-2e64-632d-6475732e6461 (0)
[2020-02-14 11:05:22.407851] I [MSGID: 114007]
[client.c:2478:client_check_remote_host] 0-ssd_storage-snapd-client:
Remote host is not set. Assuming the volfile server as remote host
[Invalid argument]
[2020-02-14 11:05:22.409711] I [MSGID: 104045] [glfs-master.c:80:notify]
0-gfapi: New graph 6e6f6465-3031-2e64-632d-6475732e6461 (0) coming up
[2020-02-14 11:05:22.409738] I [MSGID: 114020] [client.c:2436:notify]
0-ssd_storage-client-0: parent translators are ready, attempting connect
on transport
[2020-02-14 11:05:22.412949] I [MSGID: 114020] [client.c:2436:notify]
0-ssd_storage-client-1: parent translators are ready, attempting connect
on transport
[2020-02-14 11:05:22.413130] I [rpc-clnt.c:1963:rpc_clnt_reconfig]
0-ssd_storage-client-0: changing port to 49152 (from 0)
[2020-02-14 11:05:22.413154] I [socket.c:864:__socket_shutdown]
0-ssd_storage-client-0: intentional socket shutdown(10)
[2020-02-14 11:05:22.415534] I [MSGID: 114020] [client.c:2436:notify]
0-ssd_storage-client-2: parent translators are ready, attempting connect
on transport
[2020-02-14 11:05:22.417836] I [MSGID: 114057]
[client-handshake.c:1376:select_server_supported_programs]
0-ssd_storage-client-0: Using Program GlusterFS 4.x v1, Num (1298437),
Version (400)
[2020-02-14 11:05:22.418036] I [rpc-clnt.c:1963:rpc_clnt_reconfig]
0-ssd_storage-client-1: changing port to 49152 (from 0)
[2020-02-14 11:05:22.418095] I [socket.c:864:__socket_shutdown]
0-ssd_storage-client-1: intentional socket shutdown(12)
[2020-02-14 11:05:22.420029] I [MSGID: 114020] [client.c:2436:notify]
0-ssd_storage-snapd-client: parent translators are ready, attempting
connect on transport
[2020-02-14 11:05:22.420533] E [MSGID: 101075]
[common-utils.c:505:gf_resolve_ip6] 0-resolver: getaddrinfo failed
(family:2) (Name or service not known)
[2020-02-14 11:05:22.420545] E
[name.c:266:af_inet_client_get_remote_sockaddr]
0-ssd_storage-snapd-client: DNS resolution failed on host
/var/run/glusterd.socket
Final graph:
+------------------------------------------------------------------------------+
1: volume ssd_storage-client-0
2: type protocol/client
3: option opversion 70000
4: option clnt-lk-version 1
5: option volfile-checksum 0
6: option volfile-key ssd_storage
7: option client-version 7.0
8: option process-name gfapi.glfsheal
9: option process-uuid
CTX_ID:50cec79e-6028-4e6f-b8ed-dda9db36b2d0-GRAPH_ID:0-PID:24926-HOST:node01.company.com-PC_NAME:ssd_storage-client-0-RECON_NO:-0
10: option fops-version 1298437
11: option ping-timeout 42
12: option remote-host node01.company.com
13: option remote-subvolume /gluster_bricks/node01.company.com/gluster
14: option transport-type socket
15: option transport.address-family inet
16: option username 96bcf4d4-932f-4654-86c3-470a081d5021
17: option password 069e7ee9-b17d-4228-a612-b0f33588a9ec
18: option transport.socket.ssl-enabled off
19: option transport.tcp-user-timeout 0
20: option transport.socket.keepalive-time 20
21: option transport.socket.keepalive-interval 2
22: option transport.socket.keepalive-count 9
23: option send-gids true
24: end-volume
25:
26: volume ssd_storage-client-1
27: type protocol/client
28: option ping-timeout 42
29: option remote-host node02.company.com
30: option remote-subvolume /gluster_bricks/node02.company.com/gluster
31: option transport-type socket
32: option transport.address-family inet
33: option username 96bcf4d4-932f-4654-86c3-470a081d5021
34: option password 069e7ee9-b17d-4228-a612-b0f33588a9ec
35: option transport.socket.ssl-enabled off
36: option transport.tcp-user-timeout 0
37: option transport.socket.keepalive-time 20
38: option transport.socket.keepalive-interval 2
39: option transport.socket.keepalive-count 9
40: option send-gids true
41: end-volume
42:
43: volume ssd_storage-client-2
44: type protocol/client
45: option ping-timeout 42
46: option remote-host node03.company.com
47: option remote-subvolume /gluster_bricks/node03.company.com/gluster
48: option transport-type socket
49: option transport.address-family inet
50: option username 96bcf4d4-932f-4654-86c3-470a081d5021
51: option password 069e7ee9-b17d-4228-a612-b0f33588a9ec
52: option transport.socket.ssl-enabled off
53: option transport.tcp-user-timeout 0
54: option transport.socket.keepalive-time 20
55: option transport.socket.keepalive-interval 2
56: option transport.socket.keepalive-count 9
57: option send-gids true
58: end-volume
59:
60: volume ssd_storage-replicate-0
61: type cluster/replicate
62: option background-self-heal-count 0
63: option afr-pending-xattr
ssd_storage-client-0,ssd_storage-client-1,ssd_storage-client-2
64: option metadata-self-heal on
65: option data-self-heal on
66: option entry-self-heal on
67: option data-self-heal-algorithm full
68: option use-compound-fops off
69: subvolumes ssd_storage-client-0 ssd_storage-client-1
ssd_storage-client-2
70: end-volume
71:
72: volume ssd_storage-dht
73: type cluster/distribute
74: option readdir-optimize on
75: option lock-migration off
76: option force-migration off
77: subvolumes ssd_storage-replicate-0
78: end-volume
79:
80: volume ssd_storage-utime
81: type features/utime
82: option noatime on
83: subvolumes ssd_storage-dht
84: end-volume
85:
86: volume ssd_storage-write-behind
87: type performance/write-behind
88: subvolumes ssd_storage-utime
89: end-volume
90:
91: volume ssd_storage-read-ahead
92: type performance/read-ahead
93: subvolumes ssd_storage-write-behind
94: end-volume
95:
96: volume ssd_storage-readdir-ahead
97: type performance/readdir-ahead
98: option parallel-readdir off
99: option rda-request-size 131072
100: option rda-cache-limit 10MB
101: subvolumes ssd_storage-read-ahead
102: end-volume
103:
104: volume ssd_storage-io-cache
105: type performance/io-cache
106: subvolumes ssd_storage-readdir-ahead
107: end-volume
108:
109: volume ssd_storage-open-behind
110: type performance/open-behind
111: subvolumes ssd_storage-io-cache
112: end-volume
113:
114: volume ssd_storage-quick-read
115: type performance/quick-read
116: subvolumes ssd_storage-open-behind
117: end-volume
118:
119: volume ssd_storage-md-cache
120: type performance/md-cache
121: subvolumes ssd_storage-quick-read
122: end-volume
123:
124: volume ssd_storage-snapd-client
125: type protocol/client
126: option remote-host /var/run/glusterd.socket
127: option ping-timeout 42
128: option remote-subvolume snapd-ssd_storage
129: option transport-type socket
130: option transport.address-family inet
131: option username 96bcf4d4-932f-4654-86c3-470a081d5021
132: option password 069e7ee9-b17d-4228-a612-b0f33588a9ec
133: option transport.socket.ssl-enabled off
134: option transport.tcp-user-timeout 0
135: option transport.socket.keepalive-time 20
136: option transport.socket.keepalive-interval 2
137: option transport.socket.keepalive-count 9
138: option send-gids true
139: end-volume
140:
141: volume ssd_storage-snapview-client
142: type features/snapview-client
143: option snapshot-directory .snaps
144: option show-snapshot-directory on
145: subvolumes ssd_storage-md-cache ssd_storage-snapd-client
146: end-volume
147:
148: volume ssd_storage
149: type debug/io-stats
150: option log-level INFO
151: option threads 16
152: option latency-measurement off
153: option count-fop-hits off
154: option global-threading off
155: subvolumes ssd_storage-snapview-client
156: end-volume
157:
158: volume meta-autoload
159: type meta
160: subvolumes ssd_storage
161: end-volume
162:
+------------------------------------------------------------------------------+
[2020-02-14 11:05:22.421366] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-0:
Connected to ssd_storage-client-0, attached to remote volume
'/gluster_bricks/node01.company.com/gluster'.
[2020-02-14 11:05:22.421379] I [MSGID: 108005]
[afr-common.c:5280:__afr_handle_child_up_event]
0-ssd_storage-replicate-0: Subvolume 'ssd_storage-client-0' came back
up; going online.
[2020-02-14 11:05:22.421669] I [rpc-clnt.c:1963:rpc_clnt_reconfig]
0-ssd_storage-client-2: changing port to 49152 (from 0)
[2020-02-14 11:05:22.421686] I [socket.c:864:__socket_shutdown]
0-ssd_storage-client-2: intentional socket shutdown(11)
[2020-02-14 11:05:22.422460] I [MSGID: 114057]
[client-handshake.c:1376:select_server_supported_programs]
0-ssd_storage-client-1: Using Program GlusterFS 4.x v1, Num (1298437),
Version (400)
[2020-02-14 11:05:22.423377] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-1:
Connected to ssd_storage-client-1, attached to remote volume
'/gluster_bricks/node02.company.com/gluster'.
[2020-02-14 11:05:22.423391] I [MSGID: 108002]
[afr-common.c:5647:afr_notify] 0-ssd_storage-replicate-0: Client-quorum
is met
[2020-02-14 11:05:22.424586] I [MSGID: 114057]
[client-handshake.c:1376:select_server_supported_programs]
0-ssd_storage-client-2: Using Program GlusterFS 4.x v1, Num (1298437),
Version (400)
[2020-02-14 11:05:22.425323] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-2:
Connected to ssd_storage-client-2, attached to remote volume
'/gluster_bricks/node03.company.com/gluster'.
[2020-02-14 11:05:22.426613] I [MSGID: 108031]
[afr-common.c:2580:afr_local_discovery_cbk] 0-ssd_storage-replicate-0:
selecting local read_child ssd_storage-client-0
[2020-02-14 11:05:22.426758] I [MSGID: 104041]
[glfs-resolve.c:954:__glfs_active_subvol] 0-ssd_storage: switched to
graph 6e6f6465-3031-2e64-632d-6475732e6461 (0)
Can you guys make any sense out of this? 5 unsynced entries remain.
--
with kind regards,
mit freundlichen Gruessen,
Christian Reiss
________
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
