Hi, I have big problem after network outage I think. I have a volume which is corrupted I think. Here are details: [root@hv-01 ~]# gluster --version glusterfs 5.3 1. volume info: [root@hv-01 ~]# gluster volume info engine Volume Name: engine Type: Replicate Volume ID: ca7c567a-1512-4c6c-964b-e9a2ca45b872 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: hv-01.cencert.local:/gluster_bricks/engine/engine Brick2: hv-02.cencert.local:/gluster_bricks/engine/engine Brick3: arbiter.cencert.local:/gluster_bricks/engine/engine (arbiter) Options Reconfigured: cluster.eager-lock: enable performance.io-cache: off performance.read-ahead: off performance.quick-read: off user.cifs: off network.ping-timeout: 30 network.remote-dio: off performance.strict-o-direct: on performance.low-prio-threads: 32 features.shard: on storage.owner-gid: 36 storage.owner-uid: 36 transport.address-family: inet nfs.disable: on performance.client-io-threads: off 2. Volume heal info and summary [root@hv-01 ~]# gluster volume heal engine info Brick hv-01.cencert.local:/gluster_bricks/engine/engine /7396183b-825b-4670-8646-5eb8d69f3480/images/02d6cf5e-179c-4730-8191- a1ee7b5a7737/30b0a24e-01cb-4d53-9cf1-422f8f217e33 Status: Connected Number of entries: 1 Brick hv-02.cencert.local:/gluster_bricks/engine/engine /7396183b-825b-4670-8646-5eb8d69f3480/images/02d6cf5e-179c-4730-8191- a1ee7b5a7737/30b0a24e-01cb-4d53-9cf1-422f8f217e33 Status: Connected Number of entries: 1 Brick arbiter.cencert.local:/gluster_bricks/engine/engine /7396183b-825b-4670-8646-5eb8d69f3480/images/02d6cf5e-179c-4730-8191- a1ee7b5a7737/30b0a24e-01cb-4d53-9cf1-422f8f217e33 Status: Connected Number of entries: 1 [root@hv-01 ~]# gluster volume heal engine info summary Brick hv-01.cencert.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 1 Number of entries in heal pending: 1 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick hv-02.cencert.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 2 Number of entries in heal pending: 2 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick arbiter.cencert.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 2 Number of entries in heal pending: 2 Number of entries in split-brain: 0 Number of entries possibly healing: 0 3. Details about corrupted file [root@hv-01 ~]# stat /gluster_bricks/engine/engine/7396183b-825b-4670- 8646-5eb8d69f3480/images/02d6cf5e-179c-4730-8191-a1ee7b5a7737/30b0a24e- 01cb-4d53-9cf1-422f8f217e33 Plik: „/gluster_bricks/engine/engine/7396183b-825b-4670-8646- 5eb8d69f3480/images/02d6cf5e-179c-4730-8191-a1ee7b5a7737/30b0a24e-01cb- 4d53-9cf1-422f8f217e33” rozmiar: 67108864 bloków: 122968 bloki I/O: 4096 zwykły plik Urządzenie: fd0fh/64783d inody: 134349053 dowiązań: 2 Dostęp: (0660/-rw-rw----) Uid: ( 36/ vdsm) Gid: ( 36/ kvm) Kontekst: system_u:object_r:unlabeled_t:s0 Dostęp: 2019-02-17 15:00:33.130284553 +0100 Modyfikacja: 2019-02-20 13:15:08.090856642 +0100 Zmiana: 2019-10-25 11:13:15.174957824 +0200 Utworzenie: - [root@hv-02 ~]# stat /gluster_bricks/engine/engine/7396183b-825b-4670- 8646-5eb8d69f3480/images/02d6cf5e-179c-4730-8191-a1ee7b5a7737/30b0a24e- 01cb-4d53-9cf1-422f8f217e33 Plik: „/gluster_bricks/engine/engine/7396183b-825b-4670-8646- 5eb8d69f3480/images/02d6cf5e-179c-4730-8191-a1ee7b5a7737/30b0a24e-01cb- 4d53-9cf1-422f8f217e33” rozmiar: 67108864 bloków: 109320 bloki I/O: 4096 zwykły plik Urządzenie: fd14h/64788d inody: 201326687 dowiązań: 2 Dostęp: (0660/-rw-rw----) Uid: ( 36/ vdsm) Gid: ( 36/ kvm) Kontekst: system_u:object_r:unlabeled_t:s0 Dostęp: 2019-02-17 15:00:33.130284553 +0100 Modyfikacja: 2019-02-20 13:15:08.090856642 +0100 Zmiana: 2019-10-25 11:13:06.424507311 +0200 Utworzenie: - [root@hv-01 ~]# getfattr -d -e hex -m . /gluster_bricks/engine/engine/7396183b-825b-4670-8646- 5eb8d69f3480/images/02d6cf5e-179c-4730-8191-a1ee7b5a7737/30b0a24e-01cb- 4d53-9cf1-422f8f217e33 getfattr: Usunięcie wiodącego '/' ze ścieżek bezwzględnych # file: gluster_bricks/engine/engine/7396183b-825b-4670-8646- 5eb8d69f3480/images/02d6cf5e-179c-4730-8191-a1ee7b5a7737/30b0a24e-01cb- 4d53-9cf1-422f8f217e33 security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65 645f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.afr.engine-client-1=0x000000010000000000000000 trusted.afr.engine-client-2=0x000000000000000000000000 trusted.gfid=0xf5af72751c1d498a97b7cee251b7f5a7 trusted.gfid2path.448e0d5596fdc4d9=0x38623630363364642d306562322d346362 392d383038342d3435653132656362353336652f33306230613234652d303163622d346 435332d396366312d343232663866323137653333 trusted.glusterfs.shard.block-size=0x0000000004000000 trusted.glusterfs.shard.file- size=0x000000200000000000000000000000000000000002bf8cdb0000000000000000 [root@hv-02 ~]# getfattr -d -e hex -m . /gluster_bricks/engine/engine/7396183b-825b-4670-8646- 5eb8d69f3480/images/02d6cf5e-179c-4730-8191-a1ee7b5a7737/30b0a24e-01cb- 4d53-9cf1-422f8f217e33 getfattr: Usunięcie wiodącego '/' ze ścieżek bezwzględnych # file: gluster_bricks/engine/engine/7396183b-825b-4670-8646- 5eb8d69f3480/images/02d6cf5e-179c-4730-8191-a1ee7b5a7737/30b0a24e-01cb- 4d53-9cf1-422f8f217e33 security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65 645f743a733000 trusted.afr.dirty=0x000000010000000000000000 trusted.afr.engine-client-0=0x000000010000000000000000 trusted.afr.engine-client-1=0x000000000000000000000000 trusted.gfid=0xf5af72751c1d498a97b7cee251b7f5a7 trusted.gfid2path.448e0d5596fdc4d9=0x38623630363364642d306562322d346362 392d383038342d3435653132656362353336652f33306230613234652d303163622d346 435332d396366312d343232663866323137653333 trusted.glusterfs.shard.block-size=0x0000000004000000 trusted.glusterfs.shard.file- size=0x000000200000000000000000000000000000000002bf8cdb0000000000000000 I'm also attaching shd log info. Can it be fixed? I really, really need to repair it. regards grzegorz
[2019-10-27 08:42:48.835802] I [MSGID: 104045] [glfs-master.c:86:notify] 0-gfapi: New graph 68762d30-312e-6365-6e63-6572742e6c6f (0) coming up [2019-10-27 08:42:48.835837] I [MSGID: 114020] [client.c:2354:notify] 0-engine-client-0: parent translators are ready, attempting connect on transport [2019-10-27 08:42:48.839588] I [MSGID: 114020] [client.c:2354:notify] 0-engine-client-1: parent translators are ready, attempting connect on transport [2019-10-27 08:42:48.839907] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-engine-client-0: changing port to 49153 (from 0) [2019-10-27 08:42:48.841706] I [MSGID: 114020] [client.c:2354:notify] 0-engine-client-2: parent translators are ready, attempting connect on transport [2019-10-27 08:42:48.844374] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-10-27 08:42:48.844861] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-engine-client-1: changing port to 49153 (from 0) Final graph: +------------------------------------------------------------------------------+ 1: volume engine-client-0 2: type protocol/client 3: option opversion 50000 4: option clnt-lk-version 1 5: option volfile-checksum 0 6: option volfile-key engine 7: option client-version 5.3 8: option process-name gfapi.glfsheal 9: option process-uuid CTX_ID:0d1d8887-b668-4eb9-8799-98e2d2090a0d-GRAPH_ID:0-PID:84125-HOST:hv-01.cencert.local-PC_NAME:engine-client-0-RECON_NO:-0 10: option fops-version 1298437 11: option ping-timeout 30 12: option remote-host hv-01.cencert.local 13: option remote-subvolume /gluster_bricks/engine/engine 14: option transport-type socket 15: option transport.address-family inet 16: option username 6b4cd6f3-2acc-4dbf-a7fb-b68b48315ea6 17: option password 827a2404-4d1f-4fea-9399-9338389e7f49 18: option filter-O_DIRECT off 19: option transport.tcp-user-timeout 0 20: option transport.socket.keepalive-time 20 21: option transport.socket.keepalive-interval 2 22: option transport.socket.keepalive-count 9 23: option send-gids true 24: end-volume 25: 26: volume engine-client-1 27: type protocol/client 28: option ping-timeout 30 29: option remote-host hv-02.cencert.local 30: option remote-subvolume /gluster_bricks/engine/engine 31: option transport-type socket 32: option transport.address-family inet 33: option username 6b4cd6f3-2acc-4dbf-a7fb-b68b48315ea6 34: option password 827a2404-4d1f-4fea-9399-9338389e7f49 35: option filter-O_DIRECT off 36: option transport.tcp-user-timeout 0 37: option transport.socket.keepalive-time 20 38: option transport.socket.keepalive-interval 2 39: option transport.socket.keepalive-count 9 40: option send-gids true 41: end-volume 42: 43: volume engine-client-2 44: type protocol/client 45: option ping-timeout 30 46: option remote-host arbiter.cencert.local 47: option remote-subvolume /gluster_bricks/engine/engine 48: option transport-type socket 49: option transport.address-family inet 50: option username 6b4cd6f3-2acc-4dbf-a7fb-b68b48315ea6 51: option password 827a2404-4d1f-4fea-9399-9338389e7f49 52: option filter-O_DIRECT off 53: option transport.tcp-user-timeout 0 54: option transport.socket.keepalive-time 20 55: option transport.socket.keepalive-interval 2 56: option transport.socket.keepalive-count 9 57: option send-gids true 58: end-volume 59: 60: volume engine-replicate-0 61: type cluster/replicate 62: option background-self-heal-count 0 63: option afr-pending-xattr engine-client-0,engine-client-1,engine-client-2 64: option arbiter-count 1 65: option eager-lock enable 66: option use-compound-fops off 67: subvolumes engine-client-0 engine-client-1 engine-client-2 68: end-volume 69: 70: volume engine-dht 71: type cluster/distribute 72: option lock-migration off 73: option force-migration off 74: subvolumes engine-replicate-0 75: end-volume 76: 77: volume engine-shard 78: type features/shard 79: subvolumes engine-dht 80: end-volume 81: 82: volume engine-write-behind 83: type performance/write-behind 84: option strict-O_DIRECT on 85: subvolumes engine-shard 86: end-volume 87: 88: volume engine-readdir-ahead 89: type performance/readdir-ahead 90: option parallel-readdir off 91: option rda-request-size 131072 92: option rda-cache-limit 10MB 93: subvolumes engine-write-behind 94: end-volume 95: 96: volume engine-open-behind 97: type performance/open-behind 98: subvolumes engine-readdir-ahead 99: end-volume 100: 101: volume engine-md-cache 102: type performance/md-cache 103: subvolumes engine-open-behind 104: end-volume 105: 106: volume engine 107: type debug/io-stats 108: option log-level INFO 109: option latency-measurement off 110: option count-fop-hits off 111: subvolumes engine-md-cache 112: end-volume 113: 114: volume meta-autoload 115: type meta 116: subvolumes engine 117: end-volume 118: +------------------------------------------------------------------------------+ [2019-10-27 08:42:48.846239] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-engine-client-0: Connected to engine-client-0, attached to remote volume '/gluster_bricks/engine/engine'. [2019-10-27 08:42:48.846263] I [MSGID: 108005] [afr-common.c:5237:__afr_handle_child_up_event] 0-engine-replicate-0: Subvolume 'engine-client-0' came back up; going online. [2019-10-27 08:42:48.847894] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-10-27 08:42:48.848173] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-engine-client-2: changing port to 49153 (from 0) [2019-10-27 08:42:48.850407] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-10-27 08:42:48.852230] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-engine-client-2: Connected to engine-client-2, attached to remote volume '/gluster_bricks/engine/engine'. [2019-10-27 08:42:48.881877] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-engine-client-1: Connected to engine-client-1, attached to remote volume '/gluster_bricks/engine/engine'. [2019-10-27 08:42:48.881883] I [MSGID: 108002] [afr-common.c:5588:afr_notify] 0-engine-replicate-0: Client-quorum is met [2019-10-27 08:42:48.918925] I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 0-engine-replicate-0: selecting local read_child engine-client-0 [2019-10-27 08:42:48.920095] I [MSGID: 104041] [glfs-resolve.c:954:__glfs_active_subvol] 0-engine: switched to graph 68762d30-312e-6365-6e63-6572742e6c6f (0) [2019-10-27 08:43:06.618738] I [MSGID: 104045] [glfs-master.c:86:notify] 0-gfapi: New graph 68762d30-312e-6365-6e63-6572742e6c6f (0) coming up [2019-10-27 08:43:06.618771] I [MSGID: 114020] [client.c:2354:notify] 0-engine-client-0: parent translators are ready, attempting connect on transport [2019-10-27 08:43:06.621421] I [MSGID: 114020] [client.c:2354:notify] 0-engine-client-1: parent translators are ready, attempting connect on transport [2019-10-27 08:43:06.621737] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-engine-client-0: changing port to 49153 (from 0) [2019-10-27 08:43:06.623542] I [MSGID: 114020] [client.c:2354:notify] 0-engine-client-2: parent translators are ready, attempting connect on transport [2019-10-27 08:43:06.625870] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-10-27 08:43:06.626353] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-engine-client-1: changing port to 49153 (from 0) Final graph: +------------------------------------------------------------------------------+ 1: volume engine-client-0 2: type protocol/client 3: option opversion 50000 4: option clnt-lk-version 1 5: option volfile-checksum 0 6: option volfile-key engine 7: option client-version 5.3 8: option process-name gfapi.glfsheal 9: option process-uuid CTX_ID:6384c002-dbcc-4da0-a588-afc9ff64aae2-GRAPH_ID:0-PID:84161-HOST:hv-01.cencert.local-PC_NAME:engine-client-0-RECON_NO:-0 10: option fops-version 1298437 11: option ping-timeout 30 12: option remote-host hv-01.cencert.local 13: option remote-subvolume /gluster_bricks/engine/engine 14: option transport-type socket 15: option transport.address-family inet 16: option username 6b4cd6f3-2acc-4dbf-a7fb-b68b48315ea6 17: option password 827a2404-4d1f-4fea-9399-9338389e7f49 18: option filter-O_DIRECT off 19: option transport.tcp-user-timeout 0 20: option transport.socket.keepalive-time 20 21: option transport.socket.keepalive-interval 2 22: option transport.socket.keepalive-count 9 23: option send-gids true 24: end-volume 25: 26: volume engine-client-1 27: type protocol/client 28: option ping-timeout 30 29: option remote-host hv-02.cencert.local 30: option remote-subvolume /gluster_bricks/engine/engine 31: option transport-type socket 32: option transport.address-family inet 33: option username 6b4cd6f3-2acc-4dbf-a7fb-b68b48315ea6 34: option password 827a2404-4d1f-4fea-9399-9338389e7f49 35: option filter-O_DIRECT off 36: option transport.tcp-user-timeout 0 37: option transport.socket.keepalive-time 20 38: option transport.socket.keepalive-interval 2 39: option transport.socket.keepalive-count 9 40: option send-gids true 41: end-volume 42: 43: volume engine-client-2 44: type protocol/client 45: option ping-timeout 30 46: option remote-host arbiter.cencert.local 47: option remote-subvolume /gluster_bricks/engine/engine 48: option transport-type socket 49: option transport.address-family inet 50: option username 6b4cd6f3-2acc-4dbf-a7fb-b68b48315ea6 51: option password 827a2404-4d1f-4fea-9399-9338389e7f49 52: option filter-O_DIRECT off 53: option transport.tcp-user-timeout 0 54: option transport.socket.keepalive-time 20 55: option transport.socket.keepalive-interval 2 56: option transport.socket.keepalive-count 9 57: option send-gids true 58: end-volume 59: 60: volume engine-replicate-0 61: type cluster/replicate 62: option background-self-heal-count 0 63: option afr-pending-xattr engine-client-0,engine-client-1,engine-client-2 64: option arbiter-count 1 65: option eager-lock enable 66: option use-compound-fops off 67: subvolumes engine-client-0 engine-client-1 engine-client-2 68: end-volume 69: 70: volume engine-dht 71: type cluster/distribute 72: option lock-migration off 73: option force-migration off 74: subvolumes engine-replicate-0 75: end-volume 76: 77: volume engine-shard 78: type features/shard 79: subvolumes engine-dht 80: end-volume 81: 82: volume engine-write-behind 83: type performance/write-behind 84: option strict-O_DIRECT on 85: subvolumes engine-shard 86: end-volume 87: 88: volume engine-readdir-ahead 89: type performance/readdir-ahead 90: option parallel-readdir off 91: option rda-request-size 131072 92: option rda-cache-limit 10MB 93: subvolumes engine-write-behind 94: end-volume 95: 96: volume engine-open-behind 97: type performance/open-behind 98: subvolumes engine-readdir-ahead 99: end-volume 100: 101: volume engine-md-cache 102: type performance/md-cache 103: subvolumes engine-open-behind 104: end-volume 105: 106: volume engine 107: type debug/io-stats 108: option log-level INFO 109: option latency-measurement off 110: option count-fop-hits off 111: subvolumes engine-md-cache 112: end-volume 113: 114: volume meta-autoload 115: type meta 116: subvolumes engine 117: end-volume 118: +------------------------------------------------------------------------------+ [2019-10-27 08:43:06.628148] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-engine-client-0: Connected to engine-client-0, attached to remote volume '/gluster_bricks/engine/engine'. [2019-10-27 08:43:06.628170] I [MSGID: 108005] [afr-common.c:5237:__afr_handle_child_up_event] 0-engine-replicate-0: Subvolume 'engine-client-0' came back up; going online. [2019-10-27 08:43:06.629869] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-10-27 08:43:06.630435] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-engine-client-2: changing port to 49153 (from 0) [2019-10-27 08:43:06.632688] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-10-27 08:43:06.632714] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-engine-client-1: Connected to engine-client-1, attached to remote volume '/gluster_bricks/engine/engine'. [2019-10-27 08:43:06.663730] I [MSGID: 108002] [afr-common.c:5588:afr_notify] 0-engine-replicate-0: Client-quorum is met [2019-10-27 08:43:06.664964] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-engine-client-2: Connected to engine-client-2, attached to remote volume '/gluster_bricks/engine/engine'. [2019-10-27 08:43:06.701769] I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 0-engine-replicate-0: selecting local read_child engine-client-0 [2019-10-27 08:43:06.702898] I [MSGID: 104041] [glfs-resolve.c:954:__glfs_active_subvol] 0-engine: switched to graph 68762d30-312e-6365-6e63-6572742e6c6f (0) [2019-10-27 08:46:25.724736] I [MSGID: 104045] [glfs-master.c:86:notify] 0-gfapi: New graph 68762d30-312e-6365-6e63-6572742e6c6f (0) coming up [2019-10-27 08:46:25.724772] I [MSGID: 114020] [client.c:2354:notify] 0-engine-client-0: parent translators are ready, attempting connect on transport [2019-10-27 08:46:25.729066] I [MSGID: 114020] [client.c:2354:notify] 0-engine-client-1: parent translators are ready, attempting connect on transport [2019-10-27 08:46:25.729488] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-engine-client-0: changing port to 49153 (from 0) [2019-10-27 08:46:25.731329] I [MSGID: 114020] [client.c:2354:notify] 0-engine-client-2: parent translators are ready, attempting connect on transport [2019-10-27 08:46:25.734003] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-10-27 08:46:25.734458] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-engine-client-1: changing port to 49153 (from 0) Final graph: +------------------------------------------------------------------------------+ 1: volume engine-client-0 2: type protocol/client 3: option opversion 50000 4: option clnt-lk-version 1 5: option volfile-checksum 0 6: option volfile-key engine 7: option client-version 5.3 8: option process-name gfapi.glfsheal 9: option process-uuid CTX_ID:f282cb21-3352-4462-a66b-e72455f20345-GRAPH_ID:0-PID:84438-HOST:hv-01.cencert.local-PC_NAME:engine-client-0-RECON_NO:-0 10: option fops-version 1298437 11: option ping-timeout 30 12: option remote-host hv-01.cencert.local 13: option remote-subvolume /gluster_bricks/engine/engine 14: option transport-type socket 15: option transport.address-family inet 16: option username 6b4cd6f3-2acc-4dbf-a7fb-b68b48315ea6 17: option password 827a2404-4d1f-4fea-9399-9338389e7f49 18: option filter-O_DIRECT off 19: option transport.tcp-user-timeout 0 20: option transport.socket.keepalive-time 20 21: option transport.socket.keepalive-interval 2 22: option transport.socket.keepalive-count 9 23: option send-gids true 24: end-volume 25: 26: volume engine-client-1 27: type protocol/client 28: option ping-timeout 30 29: option remote-host hv-02.cencert.local 30: option remote-subvolume /gluster_bricks/engine/engine 31: option transport-type socket 32: option transport.address-family inet 33: option username 6b4cd6f3-2acc-4dbf-a7fb-b68b48315ea6 34: option password 827a2404-4d1f-4fea-9399-9338389e7f49 35: option filter-O_DIRECT off 36: option transport.tcp-user-timeout 0 37: option transport.socket.keepalive-time 20 38: option transport.socket.keepalive-interval 2 39: option transport.socket.keepalive-count 9 40: option send-gids true 41: end-volume 42: 43: volume engine-client-2 44: type protocol/client 45: option ping-timeout 30 46: option remote-host arbiter.cencert.local 47: option remote-subvolume /gluster_bricks/engine/engine 48: option transport-type socket 49: option transport.address-family inet 50: option username 6b4cd6f3-2acc-4dbf-a7fb-b68b48315ea6 51: option password 827a2404-4d1f-4fea-9399-9338389e7f49 52: option filter-O_DIRECT off 53: option transport.tcp-user-timeout 0 54: option transport.socket.keepalive-time 20 55: option transport.socket.keepalive-interval 2 56: option transport.socket.keepalive-count 9 57: option send-gids true 58: end-volume 59: 60: volume engine-replicate-0 61: type cluster/replicate 62: option background-self-heal-count 0 63: option afr-pending-xattr engine-client-0,engine-client-1,engine-client-2 64: option arbiter-count 1 65: option eager-lock enable 66: option use-compound-fops off 67: subvolumes engine-client-0 engine-client-1 engine-client-2 68: end-volume 69: 70: volume engine-dht 71: type cluster/distribute 72: option lock-migration off 73: option force-migration off 74: subvolumes engine-replicate-0 75: end-volume 76: 77: volume engine-shard 78: type features/shard 79: subvolumes engine-dht 80: end-volume 81: 82: volume engine-write-behind 83: type performance/write-behind 84: option strict-O_DIRECT on 85: subvolumes engine-shard 86: end-volume 87: 88: volume engine-readdir-ahead 89: type performance/readdir-ahead 90: option parallel-readdir off 91: option rda-request-size 131072 92: option rda-cache-limit 10MB 93: subvolumes engine-write-behind 94: end-volume 95: 96: volume engine-open-behind 97: type performance/open-behind 98: subvolumes engine-readdir-ahead 99: end-volume 100: 101: volume engine-md-cache 102: type performance/md-cache 103: subvolumes engine-open-behind 104: end-volume 105: 106: volume engine 107: type debug/io-stats 108: option log-level INFO 109: option latency-measurement off 110: option count-fop-hits off 111: subvolumes engine-md-cache 112: end-volume 113: 114: volume meta-autoload 115: type meta 116: subvolumes engine 117: end-volume 118: +------------------------------------------------------------------------------+ [2019-10-27 08:46:25.736426] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-engine-client-0: Connected to engine-client-0, attached to remote volume '/gluster_bricks/engine/engine'. [2019-10-27 08:46:25.736460] I [MSGID: 108005] [afr-common.c:5237:__afr_handle_child_up_event] 0-engine-replicate-0: Subvolume 'engine-client-0' came back up; going online. [2019-10-27 08:46:25.738176] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-10-27 08:46:25.738722] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-engine-client-2: changing port to 49153 (from 0) [2019-10-27 08:46:25.741020] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-10-27 08:46:25.741041] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-engine-client-1: Connected to engine-client-1, attached to remote volume '/gluster_bricks/engine/engine'. [2019-10-27 08:46:25.772107] I [MSGID: 108002] [afr-common.c:5588:afr_notify] 0-engine-replicate-0: Client-quorum is met [2019-10-27 08:46:25.773417] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-engine-client-2: Connected to engine-client-2, attached to remote volume '/gluster_bricks/engine/engine'. [2019-10-27 08:46:25.808914] I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 0-engine-replicate-0: selecting local read_child engine-client-0 [2019-10-27 08:46:25.809809] I [MSGID: 104041] [glfs-resolve.c:954:__glfs_active_subvol] 0-engine: switched to graph 68762d30-312e-6365-6e63-6572742e6c6f (0) [2019-10-27 08:46:29.909106] I [MSGID: 104045] [glfs-master.c:86:notify] 0-gfapi: New graph 68762d30-312e-6365-6e63-6572742e6c6f (0) coming up [2019-10-27 08:46:29.909138] I [MSGID: 114020] [client.c:2354:notify] 0-engine-client-0: parent translators are ready, attempting connect on transport [2019-10-27 08:46:29.911866] I [MSGID: 114020] [client.c:2354:notify] 0-engine-client-1: parent translators are ready, attempting connect on transport [2019-10-27 08:46:29.912218] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-engine-client-0: changing port to 49153 (from 0) [2019-10-27 08:46:29.913915] I [MSGID: 114020] [client.c:2354:notify] 0-engine-client-2: parent translators are ready, attempting connect on transport [2019-10-27 08:46:29.916012] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-10-27 08:46:29.916483] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-engine-client-1: changing port to 49153 (from 0) Final graph: +------------------------------------------------------------------------------+ 1: volume engine-client-0 2: type protocol/client 3: option opversion 50000 4: option clnt-lk-version 1 5: option volfile-checksum 0 6: option volfile-key engine 7: option client-version 5.3 8: option process-name gfapi.glfsheal 9: option process-uuid CTX_ID:472ea483-d15b-4734-8325-7f6445044ae7-GRAPH_ID:0-PID:84465-HOST:hv-01.cencert.local-PC_NAME:engine-client-0-RECON_NO:-0 10: option fops-version 1298437 11: option ping-timeout 30 12: option remote-host hv-01.cencert.local 13: option remote-subvolume /gluster_bricks/engine/engine 14: option transport-type socket 15: option transport.address-family inet 16: option username 6b4cd6f3-2acc-4dbf-a7fb-b68b48315ea6 17: option password 827a2404-4d1f-4fea-9399-9338389e7f49 18: option filter-O_DIRECT off 19: option transport.tcp-user-timeout 0 20: option transport.socket.keepalive-time 20 21: option transport.socket.keepalive-interval 2 22: option transport.socket.keepalive-count 9 23: option send-gids true 24: end-volume 25: 26: volume engine-client-1 27: type protocol/client 28: option ping-timeout 30 29: option remote-host hv-02.cencert.local 30: option remote-subvolume /gluster_bricks/engine/engine 31: option transport-type socket 32: option transport.address-family inet 33: option username 6b4cd6f3-2acc-4dbf-a7fb-b68b48315ea6 34: option password 827a2404-4d1f-4fea-9399-9338389e7f49 35: option filter-O_DIRECT off 36: option transport.tcp-user-timeout 0 37: option transport.socket.keepalive-time 20 38: option transport.socket.keepalive-interval 2 39: option transport.socket.keepalive-count 9 40: option send-gids true 41: end-volume 42: 43: volume engine-client-2 44: type protocol/client 45: option ping-timeout 30 46: option remote-host arbiter.cencert.local 47: option remote-subvolume /gluster_bricks/engine/engine 48: option transport-type socket 49: option transport.address-family inet 50: option username 6b4cd6f3-2acc-4dbf-a7fb-b68b48315ea6 51: option password 827a2404-4d1f-4fea-9399-9338389e7f49 52: option filter-O_DIRECT off 53: option transport.tcp-user-timeout 0 54: option transport.socket.keepalive-time 20 55: option transport.socket.keepalive-interval 2 56: option transport.socket.keepalive-count 9 57: option send-gids true 58: end-volume 59: 60: volume engine-replicate-0 61: type cluster/replicate 62: option background-self-heal-count 0 63: option afr-pending-xattr engine-client-0,engine-client-1,engine-client-2 64: option arbiter-count 1 65: option eager-lock enable 66: option use-compound-fops off 67: subvolumes engine-client-0 engine-client-1 engine-client-2 68: end-volume 69: 70: volume engine-dht 71: type cluster/distribute 72: option lock-migration off 73: option force-migration off 74: subvolumes engine-replicate-0 75: end-volume 76: 77: volume engine-shard 78: type features/shard 79: subvolumes engine-dht 80: end-volume 81: 82: volume engine-write-behind 83: type performance/write-behind 84: option strict-O_DIRECT on 85: subvolumes engine-shard 86: end-volume 87: 88: volume engine-readdir-ahead 89: type performance/readdir-ahead 90: option parallel-readdir off 91: option rda-request-size 131072 92: option rda-cache-limit 10MB 93: subvolumes engine-write-behind 94: end-volume 95: 96: volume engine-open-behind 97: type performance/open-behind 98: subvolumes engine-readdir-ahead 99: end-volume 100: 101: volume engine-md-cache 102: type performance/md-cache 103: subvolumes engine-open-behind 104: end-volume 105: 106: volume engine 107: type debug/io-stats 108: option log-level INFO 109: option latency-measurement off 110: option count-fop-hits off 111: subvolumes engine-md-cache 112: end-volume 113: 114: volume meta-autoload 115: type meta 116: subvolumes engine 117: end-volume 118: +------------------------------------------------------------------------------+ [2019-10-27 08:46:29.918714] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-engine-client-0: Connected to engine-client-0, attached to remote volume '/gluster_bricks/engine/engine'. [2019-10-27 08:46:29.918733] I [MSGID: 108005] [afr-common.c:5237:__afr_handle_child_up_event] 0-engine-replicate-0: Subvolume 'engine-client-0' came back up; going online. [2019-10-27 08:46:29.920381] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-10-27 08:46:29.921017] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-engine-client-2: changing port to 49153 (from 0) [2019-10-27 08:46:29.923325] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-10-27 08:46:29.923340] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-engine-client-1: Connected to engine-client-1, attached to remote volume '/gluster_bricks/engine/engine'. [2019-10-27 08:46:29.953936] I [MSGID: 108002] [afr-common.c:5588:afr_notify] 0-engine-replicate-0: Client-quorum is met [2019-10-27 08:46:29.955110] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-engine-client-2: Connected to engine-client-2, attached to remote volume '/gluster_bricks/engine/engine'. [2019-10-27 08:46:29.991155] I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 0-engine-replicate-0: selecting local read_child engine-client-0 [2019-10-27 08:46:29.992234] I [MSGID: 104041] [glfs-resolve.c:954:__glfs_active_subvol] 0-engine: switched to graph 68762d30-312e-6365-6e63-6572742e6c6f (0)
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/118564314 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/118564314 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
