Okay, got it attached :) On Fri, Nov 18, 2016 at 11:00 AM, Krutika Dhananjay <kdhananj@xxxxxxxxxx> wrote: > Assuming you're using FUSE, if your gluster volume is mounted at /some/dir, > for example, > then its corresponding logs will be at /var/log/glusterfs/some-dir.log > > -Krutika > > On Fri, Nov 18, 2016 at 7:13 AM, Olivier Lambert <lambert.olivier@xxxxxxxxx> > wrote: >> >> Attached, bricks log. Where could I find the fuse client log? >> >> On Fri, Nov 18, 2016 at 2:22 AM, Krutika Dhananjay <kdhananj@xxxxxxxxxx> >> wrote: >> > Could you attach the fuse client and brick logs? >> > >> > -Krutika >> > >> > On Fri, Nov 18, 2016 at 6:12 AM, Olivier Lambert >> > <lambert.olivier@xxxxxxxxx> >> > wrote: >> >> >> >> Okay, used the exact same config you provided, and adding an arbiter >> >> node (node3) >> >> >> >> After halting node2, VM continues to work after a small "lag"/freeze. >> >> I restarted node2 and it was back online: OK >> >> >> >> Then, after waiting few minutes, halting node1. And **just** at this >> >> moment, the VM is corrupted (segmentation fault, /var/log folder empty >> >> etc.) >> >> >> >> dmesg of the VM: >> >> >> >> [ 1645.852905] EXT4-fs error (device xvda1): >> >> htree_dirblock_to_tree:988: inode #19: block 8286: comm bash: bad >> >> entry in directory: rec_len is smaller than minimal - offset=0(0), >> >> inode=0, rec_len=0, name_len=0 >> >> [ 1645.854509] Aborting journal on device xvda1-8. >> >> [ 1645.855524] EXT4-fs (xvda1): Remounting filesystem read-only >> >> >> >> And got a lot of " comm bash: bad entry in directory" messages then... >> >> >> >> Here is the current config with all Node back online: >> >> >> >> # gluster volume info >> >> >> >> Volume Name: gv0 >> >> Type: Replicate >> >> Volume ID: 5f15c919-57e3-4648-b20a-395d9fe3d7d6 >> >> Status: Started >> >> Snapshot Count: 0 >> >> Number of Bricks: 1 x (2 + 1) = 3 >> >> Transport-type: tcp >> >> Bricks: >> >> Brick1: 10.0.0.1:/bricks/brick1/gv0 >> >> Brick2: 10.0.0.2:/bricks/brick1/gv0 >> >> Brick3: 10.0.0.3:/bricks/brick1/gv0 (arbiter) >> >> Options Reconfigured: >> >> nfs.disable: on >> >> performance.readdir-ahead: on >> >> transport.address-family: inet >> >> features.shard: on >> >> features.shard-block-size: 16MB >> >> network.remote-dio: enable >> >> cluster.eager-lock: enable >> >> performance.io-cache: off >> >> performance.read-ahead: off >> >> performance.quick-read: off >> >> performance.stat-prefetch: on >> >> performance.strict-write-ordering: off >> >> cluster.server-quorum-type: server >> >> cluster.quorum-type: auto >> >> cluster.data-self-heal: on >> >> >> >> >> >> # gluster volume status >> >> Status of volume: gv0 >> >> Gluster process TCP Port RDMA Port Online >> >> Pid >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> Brick 10.0.0.1:/bricks/brick1/gv0 49152 0 Y >> >> 1331 >> >> Brick 10.0.0.2:/bricks/brick1/gv0 49152 0 Y >> >> 2274 >> >> Brick 10.0.0.3:/bricks/brick1/gv0 49152 0 Y >> >> 2355 >> >> Self-heal Daemon on localhost N/A N/A Y >> >> 2300 >> >> Self-heal Daemon on 10.0.0.3 N/A N/A Y >> >> 10530 >> >> Self-heal Daemon on 10.0.0.2 N/A N/A Y >> >> 2425 >> >> >> >> Task Status of Volume gv0 >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> There are no active volume tasks >> >> >> >> >> >> >> >> On Thu, Nov 17, 2016 at 11:35 PM, Olivier Lambert >> >> <lambert.olivier@xxxxxxxxx> wrote: >> >> > It's planned to have an arbiter soon :) It was just preliminary >> >> > tests. >> >> > >> >> > Thanks for the settings, I'll test this soon and I'll come back to >> >> > you! >> >> > >> >> > On Thu, Nov 17, 2016 at 11:29 PM, Lindsay Mathieson >> >> > <lindsay.mathieson@xxxxxxxxx> wrote: >> >> >> On 18/11/2016 8:17 AM, Olivier Lambert wrote: >> >> >>> >> >> >>> gluster volume info gv0 >> >> >>> >> >> >>> Volume Name: gv0 >> >> >>> Type: Replicate >> >> >>> Volume ID: 2f8658ed-0d9d-4a6f-a00b-96e9d3470b53 >> >> >>> Status: Started >> >> >>> Snapshot Count: 0 >> >> >>> Number of Bricks: 1 x 2 = 2 >> >> >>> Transport-type: tcp >> >> >>> Bricks: >> >> >>> Brick1: 10.0.0.1:/bricks/brick1/gv0 >> >> >>> Brick2: 10.0.0.2:/bricks/brick1/gv0 >> >> >>> Options Reconfigured: >> >> >>> nfs.disable: on >> >> >>> performance.readdir-ahead: on >> >> >>> transport.address-family: inet >> >> >>> features.shard: on >> >> >>> features.shard-block-size: 16MB >> >> >> >> >> >> >> >> >> >> >> >> When hosting VM's its essential to set these options: >> >> >> >> >> >> network.remote-dio: enable >> >> >> cluster.eager-lock: enable >> >> >> performance.io-cache: off >> >> >> performance.read-ahead: off >> >> >> performance.quick-read: off >> >> >> performance.stat-prefetch: on >> >> >> performance.strict-write-ordering: off >> >> >> cluster.server-quorum-type: server >> >> >> cluster.quorum-type: auto >> >> >> cluster.data-self-heal: on >> >> >> >> >> >> Also with replica two and quorum on (required) your volume will >> >> >> become >> >> >> read-only when one node goes down to prevent the possibility of >> >> >> split-brain >> >> >> - you *really* want to avoid that :) >> >> >> >> >> >> I'd recommend a replica 3 volume, that way 1 node can go down, but >> >> >> the >> >> >> other >> >> >> two still form a quorum and will remain r/w. >> >> >> >> >> >> If the extra disks are not possible, then a Arbiter volume can be >> >> >> setup >> >> >> - >> >> >> basically dummy files on the third node. >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Lindsay Mathieson >> >> >> >> >> >> _______________________________________________ >> >> >> Gluster-users mailing list >> >> >> Gluster-users@xxxxxxxxxxx >> >> >> http://www.gluster.org/mailman/listinfo/gluster-users >> >> _______________________________________________ >> >> Gluster-users mailing list >> >> Gluster-users@xxxxxxxxxxx >> >> http://www.gluster.org/mailman/listinfo/gluster-users >> > >> > > >
[2016-11-17 23:41:50.277122] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.5 (args: /usr/sbin/glusterfs --volfile-server=10.0.0.1 --volfile-id=/gv0 /mnt) [2016-11-17 23:41:50.307314] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-11-17 23:41:50.314910] I [MSGID: 114020] [client.c:2356:notify] 0-gv0-client-0: parent translators are ready, attempting connect on transport [2016-11-17 23:41:50.317030] I [MSGID: 114020] [client.c:2356:notify] 0-gv0-client-1: parent translators are ready, attempting connect on transport [2016-11-17 23:41:50.317087] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2016-11-17 23:41:50.317284] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-gv0-client-0: changing port to 49152 (from 0) [2016-11-17 23:41:50.319026] I [MSGID: 114020] [client.c:2356:notify] 0-gv0-client-2: parent translators are ready, attempting connect on transport [2016-11-17 23:41:50.321355] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-gv0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-17 23:41:50.321859] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-gv0-client-0: Connected to gv0-client-0, attached to remote volume '/bricks/brick1/gv0'. [2016-11-17 23:41:50.321870] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-gv0-client-0: Server and Client lk-version numbers are not same, reopening the fds [2016-11-17 23:41:50.321957] I [MSGID: 108005] [afr-common.c:4295:afr_notify] 0-gv0-replicate-0: Subvolume 'gv0-client-0' came back up; going online. [2016-11-17 23:41:50.322050] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-gv0-client-0: Server lk version = 1 [2016-11-17 23:41:50.322377] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-gv0-client-1: changing port to 49152 (from 0) Final graph: +------------------------------------------------------------------------------+ 1: volume gv0-client-0 2: type protocol/client 3: option clnt-lk-version 1 4: option volfile-checksum 0 5: option volfile-key /gv0 6: option client-version 3.8.5 7: option process-uuid xosan1-2765-2016/11/17-23:41:50:274723-gv0-client-0-0-0 8: option fops-version 1298437 9: option ping-timeout 42 10: option remote-host 10.0.0.1 11: option remote-subvolume /bricks/brick1/gv0 12: option transport-type socket 13: option transport.address-family inet 14: option username d9b06a73-97bc-45a4-a7a7-18285b048f4a 15: option password 0c6d4cdf-bbf0-4900-aefb-3d75c41d4a23 16: option filter-O_DIRECT enable 17: option send-gids true 18: end-volume 19: 20: volume gv0-client-1 21: type protocol/client 22: option ping-timeout 42 23: option remote-host 10.0.0.2 24: option remote-subvolume /bricks/brick1/gv0 25: option transport-type socket 26: option transport.address-family inet 27: option username d9b06a73-97bc-45a4-a7a7-18285b048f4a 28: option password 0c6d4cdf-bbf0-4900-aefb-3d75c41d4a23 29: option filter-O_DIRECT enable 30: option send-gids true 31: end-volume 32: 33: volume gv0-client-2 34: type protocol/client 35: option ping-timeout 42 36: option remote-host 10.0.0.3 37: option remote-subvolume /bricks/brick1/gv0 38: option transport-type socket 39: option transport.address-family inet 40: option username d9b06a73-97bc-45a4-a7a7-18285b048f4a 41: option password 0c6d4cdf-bbf0-4900-aefb-3d75c41d4a23 42: option filter-O_DIRECT enable 43: option send-gids true 44: end-volume 45: 46: volume gv0-replicate-0 47: type cluster/replicate 48: option arbiter-count 1 49: option data-self-heal on 50: option eager-lock enable 51: option quorum-type auto 52: subvolumes gv0-client-0 gv0-client-1 gv0-client-2 53: end-volume 54: 55: volume gv0-dht 56: type cluster/distribute 57: option lock-migration off 58: subvolumes gv0-replicate-0 59: end-volume 60: 61: volume gv0-shard 62: type features/shard 63: option shard-block-size 16MB 64: subvolumes gv0-dht 65: end-volume 66: 67: volume gv0-write-behind 68: type performance/write-behind 69: option strict-write-ordering off 70: subvolumes gv0-shard 71: end-volume 72: 73: volume gv0-readdir-ahead 74: type performance/readdir-ahead 75: subvolumes gv0-write-behind 76: end-volume 77: 78: volume gv0-open-behind 79: type performance/open-behind 80: subvolumes gv0-readdir-ahead 81: end-volume 82: 83: volume gv0-md-cache 84: type performance/md-cache 85: subvolumes gv0-open-behind 86: end-volume 87: 88: volume gv0 89: type debug/io-stats 90: option log-level INFO 91: option latency-measurement off 92: option count-fop-hits off 93: subvolumes gv0-md-cache 94: end-volume 95: 96: volume meta-autoload 97: type meta 98: subvolumes gv0 99: end-volume 100: +------------------------------------------------------------------------------+ [2016-11-17 23:41:50.324573] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-gv0-client-2: changing port to 49152 (from 0) [2016-11-17 23:41:50.326256] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-gv0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-17 23:41:50.326902] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-gv0-client-1: Connected to gv0-client-1, attached to remote volume '/bricks/brick1/gv0'. [2016-11-17 23:41:50.326916] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-gv0-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-11-17 23:41:50.327174] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-gv0-client-1: Server lk version = 1 [2016-11-17 23:41:50.327608] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-gv0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-17 23:41:50.327993] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-gv0-client-2: Connected to gv0-client-2, attached to remote volume '/bricks/brick1/gv0'. [2016-11-17 23:41:50.328006] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-gv0-client-2: Server and Client lk-version numbers are not same, reopening the fds [2016-11-17 23:41:50.330179] I [fuse-bridge.c:5241:fuse_graph_setup] 0-fuse: switched to graph 0 [2016-11-17 23:41:50.330280] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-gv0-client-2: Server lk version = 1 [2016-11-17 23:41:50.330329] I [fuse-bridge.c:4153:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22 [2016-11-17 23:41:50.331308] I [MSGID: 108031] [afr-common.c:2067:afr_local_discovery_cbk] 0-gv0-replicate-0: selecting local read_child gv0-client-0 [2016-11-17 23:41:50.332328] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-gv0-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0 [2016-11-17 23:44:01.154233] I [MSGID: 100011] [glusterfsd.c:1396:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2016-11-17 23:44:01.154437] W [glusterfsd.c:1327:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f52e3090dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f52e4724cd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f52e4724b4b] ) 0-: received signum (15), shutting down [2016-11-17 23:44:01.154468] I [fuse-bridge.c:5793:fini] 0-fuse: Unmounting '/mnt'. [2016-11-17 23:44:23.896688] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.5 (args: /usr/sbin/glusterfs --volfile-server=10.0.0.1 --volfile-id=/gv0 /mnt) [2016-11-17 23:44:23.955158] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-11-17 23:44:24.030058] I [MSGID: 114020] [client.c:2356:notify] 0-gv0-client-0: parent translators are ready, attempting connect on transport [2016-11-17 23:44:24.032718] I [MSGID: 114020] [client.c:2356:notify] 0-gv0-client-1: parent translators are ready, attempting connect on transport [2016-11-17 23:44:24.034852] I [MSGID: 114020] [client.c:2356:notify] 0-gv0-client-2: parent translators are ready, attempting connect on transport Final graph: [2016-11-17 23:44:24.037218] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 +------------------------------------------------------------------------------+ 1: volume gv0-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host 10.0.0.1 5: option remote-subvolume /bricks/brick1/gv0 6: option transport-type socket 7: option transport.address-family inet 8: option username d9b06a73-97bc-45a4-a7a7-18285b048f4a 9: option password 0c6d4cdf-bbf0-4900-aefb-3d75c41d4a23 10: option filter-O_DIRECT enable 11: option send-gids true 12: end-volume 13: 14: volume gv0-client-1 15: type protocol/client 16: option ping-timeout 42 17: option remote-host 10.0.0.2 18: option remote-subvolume /bricks/brick1/gv0 19: option transport-type socket 20: option transport.address-family inet 21: option username d9b06a73-97bc-45a4-a7a7-18285b048f4a 22: option password 0c6d4cdf-bbf0-4900-aefb-3d75c41d4a23 23: option filter-O_DIRECT enable 24: option send-gids true 25: end-volume 26: 27: volume gv0-client-2 28: type protocol/client 29: option ping-timeout 42 30: option remote-host 10.0.0.3 31: option remote-subvolume /bricks/brick1/gv0 32: option transport-type socket 33: option transport.address-family inet 34: option username d9b06a73-97bc-45a4-a7a7-18285b048f4a 35: option password 0c6d4cdf-bbf0-4900-aefb-3d75c41d4a23 36: option filter-O_DIRECT enable 37: option send-gids true 38: end-volume 39: 40: volume gv0-replicate-0 41: type cluster/replicate 42: option arbiter-count 1 43: option data-self-heal on 44: option eager-lock enable 45: option quorum-type auto 46: subvolumes gv0-client-0 gv0-client-1 gv0-client-2 47: end-volume 48: 49: volume gv0-dht 50: type cluster/distribute 51: option lock-migration off 52: subvolumes gv0-replicate-0 53: end-volume 54: 55: volume gv0-shard 56: type features/shard 57: option shard-block-size 16MB 58: subvolumes gv0-dht 59: end-volume 60: 61: volume gv0-write-behind 62: type performance/write-behind 63: option strict-write-ordering off 64: subvolumes gv0-shard 65: end-volume 66: 67: volume gv0-readdir-ahead 68: type performance/readdir-ahead 69: subvolumes gv0-write-behind 70: end-volume 71: 72: volume gv0-open-behind 73: type performance/open-behind 74: subvolumes gv0-readdir-ahead 75: end-volume 76: 77: volume gv0-md-cache 78: type performance/md-cache 79: subvolumes gv0-open-behind 80: end-volume 81: 82: volume gv0 83: type debug/io-stats 84: option log-level INFO 85: option latency-measurement off 86: option count-fop-hits off 87: subvolumes gv0-md-cache 88: end-volume 89: 90: volume meta-autoload 91: type meta 92: subvolumes gv0 93: end-volume 94: +------------------------------------------------------------------------------+ [2016-11-17 23:44:24.037929] E [MSGID: 114058] [client-handshake.c:1533:client_query_portmap_cbk] 0-gv0-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2016-11-17 23:44:24.038090] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-gv0-client-0: disconnected from gv0-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2016-11-17 23:44:24.054517] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-gv0-client-2: changing port to 49152 (from 0) [2016-11-17 23:44:24.054584] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-gv0-client-1: changing port to 49152 (from 0) [2016-11-17 23:44:24.059959] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-gv0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-17 23:44:24.061703] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-gv0-client-2: Connected to gv0-client-2, attached to remote volume '/bricks/brick1/gv0'. [2016-11-17 23:44:24.061727] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-gv0-client-2: Server and Client lk-version numbers are not same, reopening the fds [2016-11-17 23:44:24.061793] I [MSGID: 108005] [afr-common.c:4295:afr_notify] 0-gv0-replicate-0: Subvolume 'gv0-client-2' came back up; going online. [2016-11-17 23:44:24.061819] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-gv0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-17 23:44:24.062019] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-gv0-client-2: Server lk version = 1 [2016-11-17 23:44:24.063084] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-gv0-client-1: Connected to gv0-client-1, attached to remote volume '/bricks/brick1/gv0'. [2016-11-17 23:44:24.063101] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-gv0-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-11-17 23:44:24.065434] I [fuse-bridge.c:5241:fuse_graph_setup] 0-fuse: switched to graph 0 [2016-11-17 23:44:24.065623] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-gv0-client-1: Server lk version = 1 [2016-11-17 23:44:24.065747] I [fuse-bridge.c:4153:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22 [2016-11-17 23:44:27.960385] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-gv0-client-0: changing port to 49152 (from 0) [2016-11-17 23:44:27.963070] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-gv0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-17 23:44:27.965700] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-gv0-client-0: Connected to gv0-client-0, attached to remote volume '/bricks/brick1/gv0'. [2016-11-17 23:44:27.965725] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-gv0-client-0: Server and Client lk-version numbers are not same, reopening the fds [2016-11-17 23:44:27.965853] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-gv0-client-0: Server lk version = 1 [2016-11-17 23:45:00.619447] I [MSGID: 108031] [afr-common.c:2067:afr_local_discovery_cbk] 0-gv0-replicate-0: selecting local read_child gv0-client-0 [2016-11-17 23:45:27.119084] W [socket.c:590:__socket_rwv] 0-gv0-client-1: readv on 10.0.0.2:49152 failed (Connection reset by peer) [2016-11-17 23:45:27.119138] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-gv0-client-1: disconnected from gv0-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2016-11-17 23:45:37.974069] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-gv0-client-1: changing port to 49152 (from 0) [2016-11-17 23:45:37.977587] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-gv0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-17 23:45:37.978205] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-gv0-client-1: Connected to gv0-client-1, attached to remote volume '/bricks/brick1/gv0'. [2016-11-17 23:45:37.978216] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-gv0-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-11-17 23:45:37.978493] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-gv0-client-1: Server lk version = 1 [2016-11-17 23:46:47.791910] I [MSGID: 108031] [afr-common.c:2067:afr_local_discovery_cbk] 0-gv0-replicate-0: selecting local read_child gv0-client-0 [2016-11-17 23:46:47.793210] I [MSGID: 108031] [afr-common.c:2067:afr_local_discovery_cbk] 0-gv0-replicate-0: selecting local read_child gv0-client-1 [2016-11-17 23:54:31.849347] W [MSGID: 109033] [dht-diskusage.c:261:dht_is_subvol_filled] 0-gv0-dht: disk space on subvolume 'gv0-replicate-0' is getting full (99.00 %), consider adding more bricks [2016-11-17 23:57:17.697231] W [MSGID: 109033] [dht-diskusage.c:261:dht_is_subvol_filled] 0-gv0-dht: disk space on subvolume 'gv0-replicate-0' is getting full (99.00 %), consider adding more bricks [2016-11-18 00:08:51.602132] W [MSGID: 109033] [dht-diskusage.c:261:dht_is_subvol_filled] 0-gv0-dht: disk space on subvolume 'gv0-replicate-0' is getting full (99.00 %), consider adding more bricks The message "W [MSGID: 109033] [dht-diskusage.c:261:dht_is_subvol_filled] 0-gv0-dht: disk space on subvolume 'gv0-replicate-0' is getting full (99.00 %), consider adding more bricks" repeated 2 times between [2016-11-18 00:08:51.602132] and [2016-11-18 00:10:23.248235] [2016-11-18 00:10:36.967841] W [MSGID: 109033] [dht-diskusage.c:261:dht_is_subvol_filled] 0-gv0-dht: disk space on subvolume 'gv0-replicate-0' is getting full (99.00 %), consider adding more bricks [2016-11-18 00:14:34.778339] W [MSGID: 109033] [dht-diskusage.c:261:dht_is_subvol_filled] 0-gv0-dht: disk space on subvolume 'gv0-replicate-0' is getting full (99.00 %), consider adding more bricks The message "W [MSGID: 109033] [dht-diskusage.c:261:dht_is_subvol_filled] 0-gv0-dht: disk space on subvolume 'gv0-replicate-0' is getting full (99.00 %), consider adding more bricks" repeated 2 times between [2016-11-18 00:14:34.778339] and [2016-11-18 00:15:07.352589] [2016-11-18 00:26:06.396624] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gv0-client-1: server 10.0.0.2:49152 has not responded in the last 42 seconds, disconnecting. [2016-11-18 00:26:06.402199] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f3e5db98002] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f3e5d95f84e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f3e5d95f95e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f3e5d9610b4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f3e5d961990] ))))) 0-gv0-client-1: forced unwinding frame type(GlusterFS 3.3) op(STAT(1)) called at 2016-11-18 00:24:44.677464 (xid=0x925ea) [2016-11-18 00:26:06.402223] W [MSGID: 114031] [client-rpc-fops.c:504:client3_3_stat_cbk] 0-gv0-client-1: remote operation failed [Transport endpoint is not connected] [2016-11-18 00:26:06.404425] I [socket.c:3391:socket_submit_request] 0-gv0-client-1: not connected (priv->connected = 0) [2016-11-18 00:26:06.404436] W [rpc-clnt.c:1640:rpc_clnt_submit] 0-gv0-client-1: failed to submit rpc-request (XID: 0x925ed Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv0-client-1) [2016-11-18 00:26:06.404446] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-gv0-client-1: remote operation failed. Path: /block.img (90548459-452d-4129-80f5-b5670b073440) [Transport endpoint is not connected] [2016-11-18 00:26:06.404644] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f3e5db98002] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f3e5d95f84e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f3e5d95f95e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f3e5d9610b4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f3e5d961990] ))))) 0-gv0-client-1: forced unwinding frame type(GlusterFS 3.3) op(STAT(1)) called at 2016-11-18 00:25:04.246650 (xid=0x925eb) [2016-11-18 00:26:06.404712] W [rpc-clnt.c:1640:rpc_clnt_submit] 0-gv0-client-1: failed to submit rpc-request (XID: 0x925ee Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv0-client-1) [2016-11-18 00:26:06.404857] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f3e5db98002] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f3e5d95f84e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f3e5d95f95e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f3e5d9610b4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f3e5d961990] ))))) 0-gv0-client-1: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2016-11-18 00:25:24.390977 (xid=0x925ec) [2016-11-18 00:26:06.404872] W [rpc-clnt-ping.c:203:rpc_clnt_ping_cbk] 0-gv0-client-1: socket disconnected [2016-11-18 00:26:06.404881] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-gv0-client-1: disconnected from gv0-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2016-11-18 00:26:07.056556] E [socket.c:2309:socket_connect_finish] 0-gv0-client-1: connection to 10.0.0.2:24007 failed (No route to host) [2016-11-18 00:26:06.404660] W [MSGID: 114031] [client-rpc-fops.c:504:client3_3_stat_cbk] 0-gv0-client-1: remote operation failed [Transport endpoint is not connected] [2016-11-18 00:26:06.404722] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-gv0-client-1: remote operation failed. Path: /block.img (90548459-452d-4129-80f5-b5670b073440) [Transport endpoint is not connected] [2016-11-18 00:27:18.182694] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-gv0-client-1: changing port to 49152 (from 0) [2016-11-18 00:27:18.185815] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-gv0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-18 00:27:18.188008] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-gv0-client-1: Connected to gv0-client-1, attached to remote volume '/bricks/brick1/gv0'. [2016-11-18 00:27:18.188024] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-gv0-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-11-18 00:27:18.188031] I [MSGID: 114042] [client-handshake.c:1053:client_post_handshake] 0-gv0-client-1: 1 fds open - Delaying child_up until they are re-opened [2016-11-18 00:27:18.189054] I [MSGID: 114041] [client-handshake.c:675:client_child_up_reopen_done] 0-gv0-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP [2016-11-18 00:27:18.189302] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-gv0-client-1: Server lk version = 1 [2016-11-18 00:27:18.621292] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-gv0-replicate-0: Completed data selfheal on 486c2b6a-592b-4288-84da-1544386ed1c3. sources=[0] 2 sinks=1 [2016-11-18 00:27:19.773140] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-gv0-replicate-0: Completed data selfheal on 2a74c4cb-18e5-40bc-b2fa-b644f763223e. sources=[0] 2 sinks=1 [2016-11-18 00:27:20.893199] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-gv0-replicate-0: Completed data selfheal on 44fa5152-b1d2-46d6-bfb0-9bc81de13b43. sources=[0] 2 sinks=1 [2016-11-18 00:27:20.984878] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-gv0-replicate-0: Completed data selfheal on d9589199-d39e-4051-8e8b-ff6613c4de89. sources=[0] 2 sinks=1 [2016-11-18 00:27:21.085415] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-gv0-replicate-0: Completed data selfheal on 7b81f06a-096d-48f0-975a-e161baf3bf92. sources=[0] 2 sinks=1 [2016-11-18 00:27:21.730057] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-gv0-replicate-0: Completed data selfheal on 9ef0cddb-4944-4aee-a586-3b284d6d0907. sources=[0] 2 sinks=1 [2016-11-18 00:37:11.790941] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.5 (args: /usr/sbin/glusterfs --volfile-server=10.0.0.1 --volfile-id=/gv0 /mnt) [2016-11-18 00:37:11.829173] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-11-18 00:37:11.910951] I [MSGID: 114020] [client.c:2356:notify] 0-gv0-client-0: parent translators are ready, attempting connect on transport [2016-11-18 00:37:11.913312] I [MSGID: 114020] [client.c:2356:notify] 0-gv0-client-1: parent translators are ready, attempting connect on transport [2016-11-18 00:37:11.915274] I [MSGID: 114020] [client.c:2356:notify] 0-gv0-client-2: parent translators are ready, attempting connect on transport Final graph: +------------------------------------------------------------------------------+ 1: volume gv0-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host 10.0.0.1 5: option remote-subvolume /bricks/brick1/gv0 6: option transport-type socket 7: option transport.address-family inet 8: option username d9b06a73-97bc-45a4-a7a7-18285b048f4a 9: option password 0c6d4cdf-bbf0-4900-aefb-3d75c41d4a23 10: option filter-O_DIRECT enable 11: option send-gids true 12: end-volume 13: 14: volume gv0-client-1 15: type protocol/client 16: option ping-timeout 42 17: option remote-host 10.0.0.2 18: option remote-subvolume /bricks/brick1/gv0 19: option transport-type socket 20: option transport.address-family inet 21: option username d9b06a73-97bc-45a4-a7a7-18285b048f4a 22: option password 0c6d4cdf-bbf0-4900-aefb-3d75c41d4a23 23: option filter-O_DIRECT enable 24: option send-gids true 25: end-volume 26: 27: volume gv0-client-2 28: type protocol/client 29: option ping-timeout 42 30: option remote-host 10.0.0.3 31: option remote-subvolume /bricks/brick1/gv0 32: option transport-type socket 33: option transport.address-family inet 34: option username d9b06a73-97bc-45a4-a7a7-18285b048f4a 35: option password 0c6d4cdf-bbf0-4900-aefb-3d75c41d4a23 36: option filter-O_DIRECT enable 37: option send-gids true 38: end-volume 39: 40: volume gv0-replicate-0 41: type cluster/replicate 42: option arbiter-count 1 43: option data-self-heal on 44: option eager-lock enable 45: option quorum-type auto 46: subvolumes gv0-client-0 gv0-client-1 gv0-client-2 47: end-volume 48: 49: volume gv0-dht 50: type cluster/distribute 51: option lock-migration off 52: subvolumes gv0-replicate-0 53: end-volume 54: 55: volume gv0-shard 56: type features/shard 57: option shard-block-size 16MB 58: subvolumes gv0-dht 59: end-volume 60: 61: volume gv0-write-behind 62: type performance/write-behind 63: option strict-write-ordering off 64: subvolumes gv0-shard 65: end-volume 66: 67: volume gv0-readdir-ahead 68: type performance/readdir-ahead 69: subvolumes gv0-write-behind 70: end-volume 71: 72: volume gv0-open-behind 73: type performance/open-behind 74: subvolumes gv0-readdir-ahead 75: end-volume 76: 77: volume gv0-md-cache 78: type performance/md-cache 79: subvolumes gv0-open-behind 80: end-volume 81: 82: volume gv0 83: type debug/io-stats 84: option log-level INFO 85: option latency-measurement off 86: option count-fop-hits off 87: subvolumes gv0-md-cache 88: end-volume 89: 90: volume meta-autoload 91: type meta 92: subvolumes gv0 93: end-volume 94: +------------------------------------------------------------------------------+ [2016-11-18 00:37:11.924977] E [MSGID: 114058] [client-handshake.c:1533:client_query_portmap_cbk] 0-gv0-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2016-11-18 00:37:11.925128] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-gv0-client-0: disconnected from gv0-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2016-11-18 00:37:11.925215] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-gv0-client-2: changing port to 49152 (from 0) [2016-11-18 00:37:11.927544] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-gv0-client-1: changing port to 49152 (from 0) [2016-11-18 00:37:11.930312] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-gv0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-18 00:37:11.931737] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-gv0-client-2: Connected to gv0-client-2, attached to remote volume '/bricks/brick1/gv0'. [2016-11-18 00:37:11.931756] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-gv0-client-2: Server and Client lk-version numbers are not same, reopening the fds [2016-11-18 00:37:11.931801] I [MSGID: 108005] [afr-common.c:4295:afr_notify] 0-gv0-replicate-0: Subvolume 'gv0-client-2' came back up; going online. [2016-11-18 00:37:11.931835] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-gv0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-18 00:37:11.933838] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-gv0-client-2: Server lk version = 1 [2016-11-18 00:37:11.934306] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-gv0-client-1: Connected to gv0-client-1, attached to remote volume '/bricks/brick1/gv0'. [2016-11-18 00:37:11.934317] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-gv0-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-11-18 00:37:11.936534] I [fuse-bridge.c:5241:fuse_graph_setup] 0-fuse: switched to graph 0 [2016-11-18 00:37:11.936597] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-gv0-client-1: Server lk version = 1 [2016-11-18 00:37:11.947641] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2016-11-18 00:37:11.949055] I [fuse-bridge.c:4153:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22 [2016-11-18 00:37:11.954148] I [MSGID: 108031] [afr-common.c:2067:afr_local_discovery_cbk] 0-gv0-replicate-0: selecting local read_child gv0-client-1 [2016-11-18 00:37:15.836464] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-gv0-client-0: changing port to 49152 (from 0) [2016-11-18 00:37:15.838952] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-gv0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-18 00:37:15.839318] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-gv0-client-0: Connected to gv0-client-0, attached to remote volume '/bricks/brick1/gv0'. [2016-11-18 00:37:15.839330] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-gv0-client-0: Server and Client lk-version numbers are not same, reopening the fds [2016-11-18 00:37:15.839460] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-gv0-client-0: Server lk version = 1 [2016-11-18 01:09:18.782941] I [MSGID: 108031] [afr-common.c:2067:afr_local_discovery_cbk] 0-gv0-replicate-0: selecting local read_child gv0-client-0 [2016-11-18 01:09:18.783978] I [MSGID: 108031] [afr-common.c:2067:afr_local_discovery_cbk] 0-gv0-replicate-0: selecting local read_child gv0-client-1 [2016-11-18 01:21:02.104079] W [MSGID: 109033] [dht-diskusage.c:261:dht_is_subvol_filled] 0-gv0-dht: disk space on subvolume 'gv0-replicate-0' is getting full (99.00 %), consider adding more bricks [2016-11-18 01:24:19.228533] W [MSGID: 109033] [dht-diskusage.c:261:dht_is_subvol_filled] 0-gv0-dht: disk space on subvolume 'gv0-replicate-0' is getting full (99.00 %), consider adding more bricks The message "W [MSGID: 109033] [dht-diskusage.c:261:dht_is_subvol_filled] 0-gv0-dht: disk space on subvolume 'gv0-replicate-0' is getting full (99.00 %), consider adding more bricks" repeated 3 times between [2016-11-18 01:24:19.228533] and [2016-11-18 01:25:01.084667] [2016-11-18 01:25:16.931711] W [MSGID: 109033] [dht-diskusage.c:261:dht_is_subvol_filled] 0-gv0-dht: disk space on subvolume 'gv0-replicate-0' is getting full (99.00 %), consider adding more bricks The message "W [MSGID: 109033] [dht-diskusage.c:261:dht_is_subvol_filled] 0-gv0-dht: disk space on subvolume 'gv0-replicate-0' is getting full (99.00 %), consider adding more bricks" repeated 7 times between [2016-11-18 01:25:16.931711] and [2016-11-18 01:27:03.236469] [2016-11-18 01:34:11.335731] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gv0-client-1: server 10.0.0.2:49152 has not responded in the last 42 seconds, disconnecting. [2016-11-18 01:34:11.338481] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f87d74f2002] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f87d72b984e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f87d72b995e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f87d72bb0b4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f87d72bb990] ))))) 0-gv0-client-1: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2016-11-18 01:33:17.450515 (xid=0x51b56) [2016-11-18 01:34:11.338498] E [MSGID: 114031] [client-rpc-fops.c:1600:client3_3_finodelk_cbk] 0-gv0-client-1: remote operation failed [Transport endpoint is not connected] [2016-11-18 01:34:11.338670] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f87d74f2002] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f87d72b984e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f87d72b995e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f87d72bb0b4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f87d72bb990] ))))) 0-gv0-client-1: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2016-11-18 01:33:32.825463 (xid=0x51b58) [2016-11-18 01:34:11.338874] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f87d74f2002] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f87d72b984e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f87d72b995e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f87d72bb0b4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f87d72bb990] ))))) 0-gv0-client-1: forced unwinding frame type(GlusterFS 3.3) op(STAT(1)) called at 2016-11-18 01:33:14.924037 (xid=0x51b55) [2016-11-18 01:34:11.338886] W [MSGID: 114031] [client-rpc-fops.c:504:client3_3_stat_cbk] 0-gv0-client-1: remote operation failed [Transport endpoint is not connected] [2016-11-18 01:34:11.341083] I [socket.c:3391:socket_submit_request] 0-gv0-client-1: not connected (priv->connected = 0) [2016-11-18 01:34:11.341096] W [rpc-clnt.c:1640:rpc_clnt_submit] 0-gv0-client-1: failed to submit rpc-request (XID: 0x51b59 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gv0-client-1) [2016-11-18 01:34:11.341112] W [MSGID: 114031] [client-rpc-fops.c:2937:client3_3_lookup_cbk] 0-gv0-client-1: remote operation failed. Path: /block.img (90548459-452d-4129-80f5-b5670b073440) [Transport endpoint is not connected] [2016-11-18 01:34:11.341120] W [rpc-clnt.c:1640:rpc_clnt_submit] 0-gv0-client-1: failed to submit rpc-request (XID: 0x51b5a Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (gv0-client-1) [2016-11-18 01:34:11.341280] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f87d74f2002] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f87d72b984e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f87d72b995e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f87d72bb0b4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f87d72bb990] ))))) 0-gv0-client-1: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2016-11-18 01:33:29.330546 (xid=0x51b57) [2016-11-18 01:34:11.341315] W [rpc-clnt.c:1640:rpc_clnt_submit] 0-gv0-client-1: failed to submit rpc-request (XID: 0x51b5b Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (gv0-client-1) [2016-11-18 01:34:11.341332] W [rpc-clnt-ping.c:203:rpc_clnt_ping_cbk] 0-gv0-client-1: socket disconnected [2016-11-18 01:34:11.341343] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-gv0-client-1: disconnected from gv0-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2016-11-18 01:34:11.987581] E [socket.c:2309:socket_connect_finish] 0-gv0-client-1: connection to 10.0.0.2:24007 failed (No route to host) The message "E [MSGID: 114031] [client-rpc-fops.c:1600:client3_3_finodelk_cbk] 0-gv0-client-1: remote operation failed [Transport endpoint is not connected]" repeated 3 times between [2016-11-18 01:34:11.338498] and [2016-11-18 01:34:11.341326] [2016-11-18 01:36:01.424328] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-gv0-client-1: changing port to 49152 (from 0) [2016-11-18 01:36:01.427516] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-gv0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-18 01:36:01.429506] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-gv0-client-1: Connected to gv0-client-1, attached to remote volume '/bricks/brick1/gv0'. [2016-11-18 01:36:01.429521] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-gv0-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-11-18 01:36:01.429528] I [MSGID: 114042] [client-handshake.c:1053:client_post_handshake] 0-gv0-client-1: 1 fds open - Delaying child_up until they are re-opened [2016-11-18 01:36:01.432136] I [MSGID: 114041] [client-handshake.c:675:client_child_up_reopen_done] 0-gv0-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP [2016-11-18 01:36:01.432440] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-gv0-client-1: Server lk version = 1 [2016-11-18 01:36:03.710599] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-gv0-replicate-0: Completed data selfheal on 365516d9-417e-41d3-92ce-35aec99315ba. sources=[0] 2 sinks=1 [2016-11-18 01:36:03.721445] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-gv0-replicate-0: Completed data selfheal on a5ee7ba3-107d-434b-8c2c-bc5c360f5ccd. sources=[0] 2 sinks=1 [2016-11-18 01:36:03.781527] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-gv0-replicate-0: Completed data selfheal on 41db25c8-7519-4259-986f-e5e1e972f9d8. sources=[0] 2 sinks=1 [2016-11-18 01:36:04.694491] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-gv0-replicate-0: Completed data selfheal on fd41840d-0736-4563-900e-f4b98eb59b48. sources=[0] 2 sinks=1 [2016-11-18 01:36:04.758497] I [MSGID: 108026] [afr-self-heal-common.c:1076:afr_log_selfheal] 0-gv0-replicate-0: Completed data selfheal on bc906f6d-083e-4b0c-b7d2-f8eba99fbab2. sources=[0] 2 sinks=1 [2016-11-18 01:39:09.924679] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.5 (args: /usr/sbin/glusterfs --volfile-server=10.0.0.1 --volfile-id=/gv0 /mnt) [2016-11-18 01:39:09.943910] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-11-18 01:39:09.986063] I [MSGID: 114020] [client.c:2356:notify] 0-gv0-client-0: parent translators are ready, attempting connect on transport [2016-11-18 01:39:09.988691] I [MSGID: 114020] [client.c:2356:notify] 0-gv0-client-1: parent translators are ready, attempting connect on transport [2016-11-18 01:39:09.990915] I [MSGID: 114020] [client.c:2356:notify] 0-gv0-client-2: parent translators are ready, attempting connect on transport [2016-11-18 01:39:09.998039] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2016-11-18 01:39:09.998349] E [MSGID: 114058] [client-handshake.c:1533:client_query_portmap_cbk] 0-gv0-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2016-11-18 01:39:09.998520] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-gv0-client-0: disconnected from gv0-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2016-11-18 01:39:10.001911] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-gv0-client-1: changing port to 49152 (from 0) Final graph: +------------------------------------------------------------------------------+ 1: volume gv0-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host 10.0.0.1 5: option remote-subvolume /bricks/brick1/gv0 6: option transport-type socket 7: option transport.address-family inet 8: option username d9b06a73-97bc-45a4-a7a7-18285b048f4a 9: option password 0c6d4cdf-bbf0-4900-aefb-3d75c41d4a23 10: option filter-O_DIRECT enable 11: option send-gids true 12: end-volume 13: 14: volume gv0-client-1 15: type protocol/client 16: option ping-timeout 42 17: option remote-host 10.0.0.2 18: option remote-subvolume /bricks/brick1/gv0 19: option transport-type socket 20: option transport.address-family inet 21: option username d9b06a73-97bc-45a4-a7a7-18285b048f4a 22: option password 0c6d4cdf-bbf0-4900-aefb-3d75c41d4a23 23: option filter-O_DIRECT enable 24: option send-gids true 25: end-volume 26: 27: volume gv0-client-2 28: type protocol/client 29: option ping-timeout 42 30: option remote-host 10.0.0.3 31: option remote-subvolume /bricks/brick1/gv0 32: option transport-type socket 33: option transport.address-family inet 34: option username d9b06a73-97bc-45a4-a7a7-18285b048f4a 35: option password 0c6d4cdf-bbf0-4900-aefb-3d75c41d4a23 36: option filter-O_DIRECT enable 37: option send-gids true 38: end-volume 39: 40: volume gv0-replicate-0 41: type cluster/replicate 42: option arbiter-count 1 43: option data-self-heal on 44: option eager-lock enable 45: option quorum-type auto 46: subvolumes gv0-client-0 gv0-client-1 gv0-client-2 47: end-volume 48: 49: volume gv0-dht 50: type cluster/distribute 51: option lock-migration off 52: subvolumes gv0-replicate-0 53: end-volume 54: 55: volume gv0-shard 56: type features/shard 57: option shard-block-size 16MB 58: subvolumes gv0-dht 59: end-volume 60: 61: volume gv0-write-behind 62: type performance/write-behind 63: option strict-write-ordering off 64: subvolumes gv0-shard 65: end-volume 66: 67: volume gv0-readdir-ahead 68: type performance/readdir-ahead 69: subvolumes gv0-write-behind 70: end-volume 71: 72: volume gv0-open-behind 73: type performance/open-behind 74: subvolumes gv0-readdir-ahead 75: end-volume 76: 77: volume gv0-md-cache 78: type performance/md-cache 79: subvolumes gv0-open-behind 80: end-volume 81: 82: volume gv0 83: type debug/io-stats 84: option log-level INFO 85: option latency-measurement off 86: option count-fop-hits off 87: subvolumes gv0-md-cache 88: end-volume 89: 90: volume meta-autoload 91: type meta 92: subvolumes gv0 93: end-volume 94: +------------------------------------------------------------------------------+ [2016-11-18 01:39:10.006249] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-gv0-client-2: changing port to 49152 (from 0) [2016-11-18 01:39:10.008220] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-gv0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-18 01:39:10.022019] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-gv0-client-1: Connected to gv0-client-1, attached to remote volume '/bricks/brick1/gv0'. [2016-11-18 01:39:10.022044] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-gv0-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-11-18 01:39:10.022099] I [MSGID: 108005] [afr-common.c:4295:afr_notify] 0-gv0-replicate-0: Subvolume 'gv0-client-1' came back up; going online. [2016-11-18 01:39:10.024279] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-gv0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-18 01:39:10.024351] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-gv0-client-1: Server lk version = 1 [2016-11-18 01:39:10.034918] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-gv0-client-2: Connected to gv0-client-2, attached to remote volume '/bricks/brick1/gv0'. [2016-11-18 01:39:10.034954] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-gv0-client-2: Server and Client lk-version numbers are not same, reopening the fds [2016-11-18 01:39:10.037487] I [fuse-bridge.c:5241:fuse_graph_setup] 0-fuse: switched to graph 0 [2016-11-18 01:39:10.037757] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-gv0-client-2: Server lk version = 1 [2016-11-18 01:39:10.039343] I [fuse-bridge.c:4153:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22 [2016-11-18 01:39:10.055536] I [MSGID: 108031] [afr-common.c:2067:afr_local_discovery_cbk] 0-gv0-replicate-0: selecting local read_child gv0-client-1 [2016-11-18 01:39:13.985879] I [rpc-clnt.c:1947:rpc_clnt_reconfig] 0-gv0-client-0: changing port to 49152 (from 0) [2016-11-18 01:39:13.988237] I [MSGID: 114057] [client-handshake.c:1446:select_server_supported_programs] 0-gv0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-18 01:39:13.988480] I [MSGID: 114046] [client-handshake.c:1222:client_setvolume_cbk] 0-gv0-client-0: Connected to gv0-client-0, attached to remote volume '/bricks/brick1/gv0'. [2016-11-18 01:39:13.988491] I [MSGID: 114047] [client-handshake.c:1233:client_setvolume_cbk] 0-gv0-client-0: Server and Client lk-version numbers are not same, reopening the fds [2016-11-18 01:39:13.988589] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-gv0-client-0: Server lk version = 1
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
