healing - but does it really? - remote operation failed

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



hi guys

something wrong with my gluster, it saysthere are files healing but it does not seem like it actually heals anything. Here is, apologies for biggish snippet, a bit of log from one volume. I cannot decode it but have a felling that can expert/devel spot something is not completely okey there.
(gluster does not show vol is in split-brain)

many thanks, L.

log:
...
[2018-06-30 18:55:56.420785] W [MSGID: 101174] [graph.c:363:_log_if_unknown_option] 0-GROUP-WORK-readdir-ahead: option 'parallel-readdir' is not recognized [2018-06-30 18:55:56.421105] I [MSGID: 104045] [glfs-master.c:91:notify] 0-gfapi: New graph 7768616c-652e-7072-6976-6174652e6363 (0) coming up [2018-06-30 18:55:56.421144] I [MSGID: 114020] [client.c:2360:notify] 0-GROUP-WORK-client-7: parent translators are ready, attempting connect on transport [2018-06-30 18:55:56.433472] I [MSGID: 114020] [client.c:2360:notify] 0-GROUP-WORK-client-8: parent translators are ready, attempting connect on transport [2018-06-30 18:55:56.437464] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-GROUP-WORK-client-7: changing port to 49154 (from 0) [2018-06-30 18:55:56.438162] I [MSGID: 114020] [client.c:2360:notify] 0-GROUP-WORK-client-9: parent translators are ready, attempting connect on transport [2018-06-30 18:55:56.446455] I [MSGID: 114057] [client-handshake.c:1478:select_server_supported_programs] 0-GROUP-WORK-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
Final graph:
+------------------------------------------------------------------------------+
  1: volume GROUP-WORK-client-7
  2:     type protocol/client
  3:     option opversion 31202
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key GROUP-WORK
  7:     option client-version 3.12.9
  8:     option process-uuid whale.private-3684460-2018/06/30-18:55:56:400699-GROUP-WORK-client-7-0-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host 10.5.6.49
 12:     option remote-subvolume /__.aLocalStorages/0/0-GLUSTERs/0GLUSTER-GROUP-WORK
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option username 7e319da6-d30c-4885-bfac-aaa3ddbe2725
 16:     option password ca697271-8219-4b58-b03b-698ff1901d0e
 17:     option transport.tcp-user-timeout 0
 18:     option transport.socket.keepalive-time 20
 19:     option transport.socket.keepalive-interval 2
 20:     option transport.socket.keepalive-count 9
 21:     option send-gids true
 22: end-volume
 23:
 24: volume GROUP-WORK-client-8
 25:     type protocol/client
 26:     option ping-timeout 42
 27:     option remote-host 10.5.6.100
 28:     option remote-subvolume /__.aLocalStorages/0/0-GLUSTERs/0GLUSTER-GROUP-WORK
 29:     option transport-type socket
 30:     option transport.address-family inet
 31:     option username 7e319da6-d30c-4885-bfac-aaa3ddbe2725
 32:     option password ca697271-8219-4b58-b03b-698ff1901d0e
 33:     option transport.tcp-user-timeout 0
 34:     option transport.socket.keepalive-time 20
 35:     option transport.socket.keepalive-interval 2
[2018-06-30 18:55:56.446995] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-GROUP-WORK-client-8: changing port to 49154 (from 0)
 36:     option transport.socket.keepalive-count 9
 37:     option send-gids true
 38: end-volume
 39:
 40: volume GROUP-WORK-client-9
 41:     type protocol/client
 42:     option ping-timeout 42
 43:     option remote-host 10.5.6.81
 44:     option remote-subvolume /__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.GROUP-WORK
 45:     option transport-type socket
 46:     option transport.address-family inet
 47:     option username 7e319da6-d30c-4885-bfac-aaa3ddbe2725
 48:     option password ca697271-8219-4b58-b03b-698ff1901d0e
 49:     option transport.tcp-user-timeout 0
 50:     option transport.socket.keepalive-time 20
 51:     option transport.socket.keepalive-interval 2
 52:     option transport.socket.keepalive-count 9
 53:     option send-gids true
 54: end-volume
 55:
 56: volume GROUP-WORK-replicate-0
 57:     type cluster/replicate
 58:     option background-self-heal-count 0
 59:     option afr-pending-xattr GROUP-WORK-client-7,GROUP-WORK-client-8,GROUP-WORK-client-9
 60:     option use-compound-fops off
 61:     subvolumes GROUP-WORK-client-7 GROUP-WORK-client-8 GROUP-WORK-client-9
 62: end-volume
 63:
 64: volume GROUP-WORK-dht
 65:     type cluster/distribute
 66:     option lock-migration off
 67:     subvolumes GROUP-WORK-replicate-0
 68: end-volume
 69:
 70: volume GROUP-WORK-write-behind
 71:     type performance/write-behind
 72:     subvolumes GROUP-WORK-dht
 73: end-volume
 74:
 75: volume GROUP-WORK-read-ahead
 76:     type performance/read-ahead
 77:     subvolumes GROUP-WORK-write-behind
 78: end-volume
 79:
 80: volume GROUP-WORK-readdir-ahead
 81:     type performance/readdir-ahead
 82:     option parallel-readdir off
 83:     option rda-request-size 131072
 84:     option rda-cache-limit 10MB
 85:     subvolumes GROUP-WORK-read-ahead
 86: end-volume
 87:
 88: volume GROUP-WORK-io-cache
 89:     type performance/io-cache
 90:     option cache-size 128MB
 91:     subvolumes GROUP-WORK-readdir-ahead
 92: end-volume
 93:
 94: volume GROUP-WORK-quick-read
 95:     type performance/quick-read
 96:     option cache-size 128MB
 97:     subvolumes GROUP-WORK-io-cache
 98: end-volume
 99:
100: volume GROUP-WORK-open-behind
101:     type performance/open-behind
102:     subvolumes GROUP-WORK-quick-read
103: end-volume
104:
105: volume GROUP-WORK-md-cache
106:     type performance/md-cache
107:     option md-cache-timeout 600
108:     option cache-samba-metadata on
109:     option cache-invalidation on
110:     subvolumes GROUP-WORK-open-behind
111: end-volume
112:
113: volume GROUP-WORK
114:     type debug/io-stats
115:     option log-level INFO
116:     option latency-measurement off
117:     option count-fop-hits off
118:     subvolumes GROUP-WORK-md-cache
119: end-volume
120:
121: volume meta-autoload
122:     type meta
123:     subvolumes GROUP-WORK
124: end-volume
125:
+------------------------------------------------------------------------------+
[2018-06-30 18:55:56.448438] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 0-GROUP-WORK-client-7: Connected to GROUP-WORK-client-7, attached to remote volume '/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER-GROUP-WORK'. [2018-06-30 18:55:56.448473] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 0-GROUP-WORK-client-7: Server and Client lk-version numbers are not same, reopening the fds [2018-06-30 18:55:56.448625] I [MSGID: 108005] [afr-common.c:5015:__afr_handle_child_up_event] 0-GROUP-WORK-replicate-0: Subvolume 'GROUP-WORK-client-7' came back up; going online. [2018-06-30 18:55:56.452916] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-GROUP-WORK-client-9: changing port to 49156 (from 0) [2018-06-30 18:55:56.453000] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-GROUP-WORK-client-7: Server lk version = 1 [2018-06-30 18:55:56.456971] I [MSGID: 114057] [client-handshake.c:1478:select_server_supported_programs] 0-GROUP-WORK-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-06-30 18:55:56.458254] I [MSGID: 114057] [client-handshake.c:1478:select_server_supported_programs] 0-GROUP-WORK-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-06-30 18:55:56.459241] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 0-GROUP-WORK-client-9: Connected to GROUP-WORK-client-9, attached to remote volume '/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.GROUP-WORK'. [2018-06-30 18:55:56.459282] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 0-GROUP-WORK-client-9: Server and Client lk-version numbers are not same, reopening the fds [2018-06-30 18:55:56.459353] I [MSGID: 108002] [afr-common.c:5312:afr_notify] 0-GROUP-WORK-replicate-0: Client-quorum is met [2018-06-30 18:55:56.459535] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-GROUP-WORK-client-9: Server lk version = 1 [2018-06-30 18:55:56.459860] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 0-GROUP-WORK-client-8: Connected to GROUP-WORK-client-8, attached to remote volume '/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER-GROUP-WORK'. [2018-06-30 18:55:56.459888] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 0-GROUP-WORK-client-8: Server and Client lk-version numbers are not same, reopening the fds [2018-06-30 18:55:56.461806] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-GROUP-WORK-client-8: Server lk version = 1 [2018-06-30 18:55:56.481552] I [MSGID: 108031] [afr-common.c:2458:afr_local_discovery_cbk] 0-GROUP-WORK-replicate-0: selecting local read_child GROUP-WORK-client-7 [2018-06-30 18:55:56.482490] I [MSGID: 104041] [glfs-resolve.c:971:__glfs_active_subvol] 0-GROUP-WORK: switched to graph 7768616c-652e-7072-6976-6174652e6363 (0) [2018-06-30 18:55:56.582941] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-GROUP-WORK-client-9: remote operation failed. Path: <gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2> (ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or directory] The message "W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-GROUP-WORK-client-9: remote operation failed. Path: <gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2> (ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or directory]" repeated 2 times between [2018-06-30 18:55:56.582941] and [2018-06-30 18:55:56.586687] [2018-06-30 18:55:56.773663] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-GROUP-WORK-client-9: remote operation failed. Path: <gfid:3e550eb9-33df-42b4-8fd2-29dce852e371> (3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or directory] The message "W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-GROUP-WORK-client-9: remote operation failed. Path: <gfid:3e550eb9-33df-42b4-8fd2-29dce852e371> (3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or directory]" repeated 2 times between [2018-06-30 18:55:56.773663] and [2018-06-30 18:55:56.780197] [2018-06-30 18:55:58.475889] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-GROUP-WORK-client-9: remote operation failed. Path: <gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2> (ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or directory] The message "W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-GROUP-WORK-client-9: remote operation failed. Path: <gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2> (ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or directory]" repeated 2 times between [2018-06-30 18:55:58.475889] and [2018-06-30 18:55:58.479828] [2018-06-30 18:55:58.685911] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-GROUP-WORK-client-9: remote operation failed. Path: <gfid:3e550eb9-33df-42b4-8fd2-29dce852e371> (3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or directory] The message "W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-GROUP-WORK-client-9: remote operation failed. Path: <gfid:3e550eb9-33df-42b4-8fd2-29dce852e371> (3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or directory]" repeated 2 times between [2018-06-30 18:55:58.685911] and [2018-06-30 18:55:58.691170] [2018-06-30 18:56:00.317702] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-GROUP-WORK-client-9: remote operation failed. Path: <gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2> (ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or directory] The message "W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-GROUP-WORK-client-9: remote operation failed. Path: <gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2> (ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or directory]" repeated 2 times between [2018-06-30 18:56:00.317702] and [2018-06-30 18:56:00.322284] [2018-06-30 18:56:00.324742] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-GROUP-WORK-client-9: remote operation failed. Path: <gfid:3e550eb9-33df-42b4-8fd2-29dce852e371> (3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or directory] The message "W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-GROUP-WORK-client-9: remote operation failed. Path: <gfid:3e550eb9-33df-42b4-8fd2-29dce852e371> (3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or directory]" repeated 2 times between [2018-06-30 18:56:00.324742] and [2018-06-30 18:56:00.329691] [2018-06-30 18:56:00.334014] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-GROUP-WORK-client-9: remote operation failed. Path: <gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2> (ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or directory] The message "W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-GROUP-WORK-client-9: remote operation failed. Path: <gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2> (ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or directory]" repeated 2 times between [2018-06-30 18:56:00.334014] and [2018-06-30 18:56:00.339246] [2018-06-30 18:56:00.341721] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-GROUP-WORK-client-9: remote operation failed. Path: <gfid:3e550eb9-33df-42b4-8fd2-29dce852e371> (3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or directory]

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users




[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux