Here is a tail from /var/log/glusterfs/firewall-scripts.log. The entries at 00:39:56 GMT must correspond to /var/log/messages at 19:39:56 USA Central time. There are no firewall rules and the default policy is ACCEPT. With no rc.local file, my application is out of the picture right now, so nobody is sleeping or taking themselves offline or anything like that. Just a straight up boot using fstab to mount the filesystems. . . . [2013-07-16 00:36:40.495183] I [afr-common.c:2120:afr_discovery_cbk] 0-firewall-scripts-replicate-0: selecting local read_child firewall-scripts-client-0 [2013-07-16 00:39:12.584585] W [socket.c:514:__socket_rwv] 0-glusterfs: readv failed (No data available) [2013-07-16 00:39:12.584853] W [socket.c:1962:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (No d ata available), peer (127.0.0.1:24007) [2013-07-16 00:39:12.714225] W [glusterfsd.c:970:cleanup_and_exit] (-->/usr/lib64/libc.so.6(clone+0x6d) [0x7fd5c27f913d] (-->/us r/lib64/libpthread.so.0(+0x33c1607c53) [0x7fd5c2e8fc53] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5) [0x7fd5c3b7de35]))) 0- : received signum (15), shutting down [2013-07-16 00:39:12.714289] I [fuse-bridge.c:5212:fini] 0-fuse: Unmounting '/firewall-scripts'. [2013-07-16 00:39:12.715170] I [fuse-bridge.c:4583:fuse_thread_proc] 0-fuse: unmounting /firewall-scripts [2013-07-16 00:39:56.667185] I [glusterfsd.c:1878:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0 beta4 (/usr/sbin/glusterfs --volfile-id=/firewall-scripts --volfile-server=localhost /firewall-scripts) [2013-07-16 00:39:56.726785] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled [2013-07-16 00:39:56.726993] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread [2013-07-16 00:40:05.794794] I [socket.c:3480:socket_init] 0-firewall-scripts-client-1: SSL support is NOT enabled [2013-07-16 00:40:05.794927] I [socket.c:3495:socket_init] 0-firewall-scripts-client-1: using system polling thread [2013-07-16 00:40:05.801351] I [socket.c:3480:socket_init] 0-firewall-scripts-client-0: SSL support is NOT enabled [2013-07-16 00:40:05.801486] I [socket.c:3495:socket_init] 0-firewall-scripts-client-0: using system polling thread [2013-07-16 00:40:05.801611] I [client.c:2154:notify] 0-firewall-scripts-client-0: parent translators are ready, attempting conn ect on transport [2013-07-16 00:40:05.817724] I [client.c:2154:notify] 0-firewall-scripts-client-1: parent translators are ready, attempting conn ect on transport Given volfile: +------------------------------------------------------------------------------+ 1: volume firewall-scripts-client-0 2: type protocol/client 3: option password fb3955b7-a6ca-49bb-b886-d4b6609392f8 4: option username de6eacd1-31bc-4bdb-a049-776cd840059e 5: option transport-type tcp 6: option remote-subvolume /gluster-fw1 7: option remote-host 192.168.253.1 8: end-volume 9: 10: volume firewall-scripts-client-1 11: type protocol/client 12: option password fb3955b7-a6ca-49bb-b886-d4b6609392f8 13: option username de6eacd1-31bc-4bdb-a049-776cd840059e 14: option transport-type tcp 15: option remote-subvolume /gluster-fw2 16: option remote-host 192.168.253.2 17: end-volume 18: 19: volume firewall-scripts-replicate-0 20: type cluster/replicate 21: subvolumes firewall-scripts-client-0 firewall-scripts-client-1 22: end-volume 23: 24: volume firewall-scripts-dht 25: type cluster/distribute 26: subvolumes firewall-scripts-replicate-0 27: end-volume 28: 29: volume firewall-scripts-write-behind 30: type performance/write-behind 31: subvolumes firewall-scripts-dht 32: end-volume 33: 34: volume firewall-scripts-read-ahead 35: type performance/read-ahead 36: subvolumes firewall-scripts-write-behind 37: end-volume 38: 39: volume firewall-scripts-io-cache 40: type performance/io-cache 41: subvolumes firewall-scripts-read-ahead 42: end-volume 43: 44: volume firewall-scripts-quick-read 45: type performance/quick-read 46: subvolumes firewall-scripts-io-cache 47: end-volume 48: 49: volume firewall-scripts-open-behind 50: type performance/open-behind 51: subvolumes firewall-scripts-quick-read 52: end-volume 53: 54: volume firewall-scripts-md-cache 55: type performance/md-cache 56: subvolumes firewall-scripts-open-behind 57: end-volume 58: 59: volume firewall-scripts 60: type debug/io-stats 61: option count-fop-hits off 62: option latency-measurement off 63: subvolumes firewall-scripts-md-cache 64: end-volume +------------------------------------------------------------------------------+ [2013-07-16 00:40:05.975356] E [client-handshake.c:1741:client_query_portmap_cbk] 0-firewall-scripts-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2013-07-16 00:40:05.975588] W [socket.c:514:__socket_rwv] 0-firewall-scripts-client-0: readv failed (No data available) [2013-07-16 00:40:05.975708] I [client.c:2097:client_rpc_notify] 0-firewall-scripts-client-0: disconnected [2013-07-16 00:40:06.027821] E [client-handshake.c:1741:client_query_portmap_cbk] 0-firewall-scripts-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2013-07-16 00:40:06.028010] W [socket.c:514:__socket_rwv] 0-firewall-scripts-client-1: readv failed (No data available) [2013-07-16 00:40:06.028103] I [client.c:2097:client_rpc_notify] 0-firewall-scripts-client-1: disconnected [2013-07-16 00:40:06.028148] E [afr-common.c:3735:afr_notify] 0-firewall-scripts-replicate-0: All subvolumes are down. Going off line until atleast one of them comes back up. [2013-07-16 00:40:06.048172] I [fuse-bridge.c:4723:fuse_graph_setup] 0-fuse: switched to graph 0 [2013-07-16 00:40:06.049068] I [fuse-bridge.c:3680:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7. 13 kernel 7.21 [2013-07-16 00:40:06.051158] W [fuse-bridge.c:665:fuse_attr_cbk] 0-glusterfs-fuse: 2: LOOKUP() / => -1 (No such file or director y) [2013-07-16 00:40:06.077420] I [fuse-bridge.c:4583:fuse_thread_proc] 0-fuse: unmounting /firewall-scripts [2013-07-16 00:40:06.078427] W [glusterfsd.c:970:cleanup_and_exit] (-->/usr/lib64/libc.so.6(clone+0x6d) [0x7f1f0078d13d] (-->/us r/lib64/libpthread.so.0(+0x33c1607c53) [0x7f1f00e23c53] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5) [0x7f1f01b11e35]))) 0- : received signum (15), shutting down [2013-07-16 00:40:06.078501] I [fuse-bridge.c:5212:fini] 0-fuse: Unmounting '/firewall-scripts'. [2013-07-16 00:53:39.844556] I [glusterfsd.c:1878:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0 beta4 (/usr/sbin/glusterfs --volfile-id=/firewall-scripts --volfile-server=localhost /firewall-scripts) [2013-07-16 00:53:39.858957] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled [2013-07-16 00:53:39.859117] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread [2013-07-16 00:53:39.907716] I [socket.c:3480:socket_init] 0-firewall-scripts-client-1: SSL support is NOT enabled [2013-07-16 00:53:39.907881] I [socket.c:3495:socket_init] 0-firewall-scripts-client-1: using system polling thread [2013-07-16 00:53:39.909563] I [socket.c:3480:socket_init] 0-firewall-scripts-client-0: SSL support is NOT enabled [2013-07-16 00:53:39.909655] I [socket.c:3495:socket_init] 0-firewall-scripts-client-0: using system polling thread [2013-07-16 00:53:39.909778] I [client.c:2154:notify] 0-firewall-scripts-client-0: parent translators are ready, attempting conn ect on transport [2013-07-16 00:53:39.920933] I [client.c:2154:notify] 0-firewall-scripts-client-1: parent translators are ready, attempting conn ect on transport Given volfile: +------------------------------------------------------------------------------+ 1: volume firewall-scripts-client-0 2: type protocol/client 3: option password fb3955b7-a6ca-49bb-b886-d4b6609392f8 4: option username de6eacd1-31bc-4bdb-a049-776cd840059e 5: option transport-type tcp 6: option remote-subvolume /gluster-fw1 7: option remote-host 192.168.253.1 8: end-volume 9: 10: volume firewall-scripts-client-1 11: type protocol/client 12: option password fb3955b7-a6ca-49bb-b886-d4b6609392f8 13: option username de6eacd1-31bc-4bdb-a049-776cd840059e 14: option transport-type tcp 15: option remote-subvolume /gluster-fw2 16: option remote-host 192.168.253.2 17: end-volume 18: 19: volume firewall-scripts-replicate-0 20: type cluster/replicate 21: subvolumes firewall-scripts-client-0 firewall-scripts-client-1 22: end-volume 23: 24: volume firewall-scripts-dht 25: type cluster/distribute 26: subvolumes firewall-scripts-replicate-0 27: end-volume 28: 29: volume firewall-scripts-write-behind 30: type performance/write-behind 31: subvolumes firewall-scripts-dht 32: end-volume 33: 34: volume firewall-scripts-read-ahead 35: type performance/read-ahead 36: subvolumes firewall-scripts-write-behind 37: end-volume 38: 39: volume firewall-scripts-io-cache 40: type performance/io-cache 41: subvolumes firewall-scripts-read-ahead 42: end-volume 43: 44: volume firewall-scripts-quick-read 45: type performance/quick-read 46: subvolumes firewall-scripts-io-cache 47: end-volume 48: 49: volume firewall-scripts-open-behind 50: type performance/open-behind 51: subvolumes firewall-scripts-quick-read 52: end-volume 53: 54: volume firewall-scripts-md-cache 55: type performance/md-cache 56: subvolumes firewall-scripts-open-behind 57: end-volume 58: 59: volume firewall-scripts 60: type debug/io-stats 61: option count-fop-hits off 62: option latency-measurement off 63: subvolumes firewall-scripts-md-cache 64: end-volume +------------------------------------------------------------------------------+ [2013-07-16 00:53:39.933009] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-firewall-scripts-client-0: changing port to 49152 (from 0) [2013-07-16 00:53:39.933178] W [socket.c:514:__socket_rwv] 0-firewall-scripts-client-0: readv failed (No data available) [2013-07-16 00:53:39.950457] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-firewall-scripts-client-1: changing port to 49152 (from 0) [2013-07-16 00:53:39.950621] W [socket.c:514:__socket_rwv] 0-firewall-scripts-client-1: readv failed (No data available) [2013-07-16 00:53:39.966646] I [client-handshake.c:1658:select_server_supported_programs] 0-firewall-scripts-client-0: Using Pro gram GlusterFS 3.3, Num (1298437), Version (330) [2013-07-16 00:53:39.966994] I [client-handshake.c:1658:select_server_supported_programs] 0-firewall-scripts-client-1: Using Pro gram GlusterFS 3.3, Num (1298437), Version (330) [2013-07-16 00:53:39.967417] I [client-handshake.c:1456:client_setvolume_cbk] 0-firewall-scripts-client-0: Connected to 192.168. 253.1:49152, attached to remote volume '/gluster-fw1'. [2013-07-16 00:53:39.967498] I [client-handshake.c:1468:client_setvolume_cbk] 0-firewall-scripts-client-0: Server and Client lk- version numbers are not same, reopening the fds [2013-07-16 00:53:39.967800] I [afr-common.c:3698:afr_notify] 0-firewall-scripts-replicate-0: Subvolume 'firewall-scripts-client -0' came back up; going online. [2013-07-16 00:53:39.967959] I [client-handshake.c:450:client_set_lk_version_cbk] 0-firewall-scripts-client-0: Server lk version = 1 [2013-07-16 00:53:39.968068] I [client-handshake.c:1456:client_setvolume_cbk] 0-firewall-scripts-client-1: Connected to 192.168. 253.2:49152, attached to remote volume '/gluster-fw2'. [2013-07-16 00:53:39.968114] I [client-handshake.c:1468:client_setvolume_cbk] 0-firewall-scripts-client-1: Server and Client lk- version numbers are not same, reopening the fds [2013-07-16 00:53:39.982434] I [fuse-bridge.c:4723:fuse_graph_setup] 0-fuse: switched to graph 0 [2013-07-16 00:53:39.982877] I [client-handshake.c:450:client_set_lk_version_cbk] 0-firewall-scripts-client-1: Server lk version = 1 [2013-07-16 00:53:39.983205] I [fuse-bridge.c:3680:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7. 13 kernel 7.21 [2013-07-16 00:53:39.984516] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 0-firewall-scripts-replicate-0: added root inode [2013-07-16 00:53:39.985412] I [afr-common.c:2120:afr_discovery_cbk] 0-firewall-scripts-replicate-0: selecting local read_child firewall-scripts-client-0 [root at chicago-fw1 ~]#
