What happens if you telnet to any of the GlusterFS ports - from another machine and from localhost? -JM ----- Original Message ----- > I am still unable to mount a new 3.3.1 glusterfs install. I have > tried from one of the actual machines in the cluster to itself, as > well as from various other clients. They all seem to be failing in > the same part of the process. > > From the client log, my comments in bold > > Initial contact with server, no errors > [2012-11-19 15:07:44.802826] I [glusterfsd.c:1666:main] > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version > 3.3.1 > [2012-11-19 15:07:44.811730] I [io-cache.c:1549:check_cache_size_ok] > 0-gdata-quick-read: Max cache size is 25332322304 > [2012-11-19 15:07:44.811797] I [io-cache.c:1549:check_cache_size_ok] > 0-gdata-io-cache: Max cache size is 25332322304 > [2012-11-19 15:07:44.815938] I [client.c:2142:notify] > 0-gdata-client-0: parent translators are ready, attempting connect > on transport > [2012-11-19 15:07:44.819667] I [client.c:2142:notify] > 0-gdata-client-1: parent translators are ready, attempting connect > on transport > [2012-11-19 15:07:44.823668] I [client.c:2142:notify] > 0-gdata-client-2: parent translators are ready, attempting connect > on transport > Data from server is returned, looks like proper responses > Given volfile: > +------------------------------------------------------------------------------+ > 1: volume gdata-client-0 > 2: type protocol/client > 3: option remote-host gluster-0-0 > 4: option remote-subvolume /mseas-data-0-0 > 5: option transport-type tcp > 6: end-volume > 7: > 8: volume gdata-client-1 > 9: type protocol/client > 10: option remote-host gluster-0-1 > 11: option remote-subvolume /mseas-data-0-1 > 12: option transport-type tcp > 13: end-volume > 14: > 15: volume gdata-client-2 > 16: type protocol/client > 17: option remote-host gluster-data > 18: option remote-subvolume /data > 19: option transport-type tcp > 20: end-volume > 21: > 22: volume gdata-dht > 23: type cluster/distribute > 24: subvolumes gdata-client-0 gdata-client-1 gdata-client-2 > 25: end-volume > 26: > 27: volume gdata-write-behind > 28: type performance/write-behind > 29: subvolumes gdata-dht > 30: end-volume > 31: > 32: volume gdata-read-ahead > 33: type performance/read-ahead > 34: subvolumes gdata-write-behind > 35: end-volume > 36: > 37: volume gdata-io-cache > 38: type performance/io-cache > 39: subvolumes gdata-read-ahead > 40: end-volume > 41: > 42: volume gdata-quick-read > 43: type performance/quick-read > 44: subvolumes gdata-io-cache > 45: end-volume > 46: > 47: volume gdata-stat-prefetch > 48: type performance/stat-prefetch > 49: subvolumes gdata-quick-read > 50: end-volume > 51: > 52: volume gdata > 53: type debug/io-stats > 54: subvolumes gdata-stat-prefetch > 55: end-volume > +------------------------------------------------------------------------------+ > > Port is changed and we can no longer connect. > [2012-11-19 15:07:44.828213] I [rpc-clnt.c:1657:rpc_clnt_reconfig] > 0-gdata-client-2: changing port to 24009 (from 0) > [2012-11-19 15:07:44.828285] I [rpc-clnt.c:1657:rpc_clnt_reconfig] > 0-gdata-client-0: changing port to 24009 (from 0) > [2012-11-19 15:07:44.828329] I [rpc-clnt.c:1657:rpc_clnt_reconfig] > 0-gdata-client-1: changing port to 24009 (from 0) > [2012-11-19 15:07:48.812899] W > [client-handshake.c:1819:client_dump_version_cbk] 0-gdata-client-2: > received RPC status error > [2012-11-19 15:07:48.812958] W > [socket.c:1512:__socket_proto_state_machine] 0-gdata-client-2: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.1.1.2:24009) > [2012-11-19 15:07:48.812985] I [client.c:2090:client_rpc_notify] > 0-gdata-client-2: disconnected > [2012-11-19 15:07:48.816284] W > [client-handshake.c:1819:client_dump_version_cbk] 0-gdata-client-0: > received RPC status error > [2012-11-19 15:07:48.816325] W > [socket.c:1512:__socket_proto_state_machine] 0-gdata-client-0: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.1.1.10:24009) > [2012-11-19 15:07:48.816347] I [client.c:2090:client_rpc_notify] > 0-gdata-client-0: disconnected > [2012-11-19 15:07:48.819636] W > [client-handshake.c:1819:client_dump_version_cbk] 0-gdata-client-1: > received RPC status error > [2012-11-19 15:07:48.819669] W > [socket.c:1512:__socket_proto_state_machine] 0-gdata-client-1: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.1.1.11:24009) > [2012-11-19 15:07:48.819688] I [client.c:2090:client_rpc_notify] > 0-gdata-client-1: disconnected > [2012-11-19 15:07:48.827123] I [fuse-bridge.c:4191:fuse_graph_setup] > 0-fuse: switched to graph 0 > [2012-11-19 15:07:48.827368] I [fuse-bridge.c:3376:fuse_init] > 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 > kernel 7.13 > [2012-11-19 15:07:48.827492] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for / > [2012-11-19 15:07:48.827607] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for / > [2012-11-19 15:07:48.827626] W [fuse-bridge.c:513:fuse_attr_cbk] > 0-glusterfs-fuse: 2: LOOKUP() / => -1 (Invalid argument) > [2012-11-19 15:07:48.827709] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for / > [2012-11-19 15:07:48.827779] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for / > [2012-11-19 15:07:48.827917] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for / > [2012-11-19 15:07:48.828063] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for / > [2012-11-19 15:07:48.841568] I [fuse-bridge.c:4091:fuse_thread_proc] > 0-fuse: unmounting /gdata > [2012-11-19 15:07:48.841928] W [glusterfsd.c:831:cleanup_and_exit] > (-->/lib64/libc.so.6(clone+0x6d) [0x3065ae5ccd] > (-->/lib64/libpthread.so.0() [0x30666077f1] > (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x405d4d]))) 0-: > received signum (15), shutting down > [2012-11-19 15:07:48.841965] I [fuse-bridge.c:4648:fini] 0-fuse: > Unmounting '/gdata'. > > > > netstat verifies that glusterfs is listening on 24009 > [root at mseas-data ~]# netstat --tcp --listening --programs > Active Internet connections (only servers) > Proto Recv-Q Send-Q Local Address Foreign Address > State PID/Program name > tcp 0 0 *:amanda *:* > LISTEN 13890/xinetd > tcp 0 0 *:sift-uft *:* > LISTEN 23324/rpc.rquotad > tcp 0 0 *:38465 *:* > LISTEN 26629/glusterfs > tcp 0 0 *:kamanda *:* > LISTEN 13890/xinetd > tcp 0 0 *:nfs *:* > LISTEN - > tcp 0 0 *:24007 *:* > LISTEN 27040/glusterd > tcp 0 0 *:sunrpc *:* > LISTEN 2682/portmap > tcp 0 0 *:51538 *:* > LISTEN - > tcp 0 0 *:ssh *:* > LISTEN 14045/sshd > tcp 0 0 *:ipcserver *:* > LISTEN 19498/rpc.statd > tcp 0 0 localhost.localdomain:smtp *:* > LISTEN 3268/sendmail > tcp 0 0 localhost.lo:x11-ssh-offset *:* > LISTEN 23982/sshd > tcp 0 0 *:794 *:* > LISTEN 23511/rpc.mountd > tcp 0 0 localhost.localdomain:6011 *:* > LISTEN 24022/sshd > tcp 0 0 localhost.localdomain:6012 *:* > LISTEN 9462/sshd > tcp 0 0 *:24009 *:* > LISTEN 26625/glusterfsd > tcp 0 0 *:ssh *:* > LISTEN 14045/sshd > tcp 0 0 localhost6.l:x11-ssh-offset *:* > LISTEN 23982/sshd > tcp 0 0 localhost6.localdomain:6011 *:* > LISTEN 24022/sshd > tcp 0 0 localhost6.localdomain:6012 *:* > LISTEN 9462/sshd > > IPTABLES service has been stopped on all machines. > > > > Any help you could give me would be greatly appreciated. > > Thanks, > Steve Postma > > > ________________________________ > From: Steve Postma > Sent: Friday, November 16, 2012 10:51 AM > To: gluster-users at gluster.org > Subject: cant mount gluster volume > > I have a 3 node gluster cluster running: > gluster --version > glusterfs 3.3.1 built on Oct 11 2012 22:01:05 > > It was previously running a 2.? version, uninstalled and installed > 3.3.1 > > It appears to be working correctly. > gluster volume info > > Volume Name: gdata > Type: Distribute > Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d > Status: Started > Number of Bricks: 3 > Transport-type: tcp > Bricks: > Brick1: gluster-0-0:/mseas-data-0-0 > Brick2: gluster-0-1:/mseas-data-0-1 > Brick3: gluster-data:/data > > The client previously > I have a client with rpm's installed: > rpm -qa | grep gluster > glusterfs-fuse-3.3.1-1.el6.x86_64 > glusterfs-3.3.1-1.el6.x86_64 > glusterfs-debuginfo-3.3.1-1.el6.x86_64 > > > I have a line in my fstab file of: > mseas-data:/gdata /gdata glusterfs defaults > 0 0 > This is copied from backups before client reinstall > > I have also tried: > mount.glusterfs mseas-data:/gdata \gdata > > > From /var/log/glusterfs/gdata.log > > cat gdata.log > [2012-11-16 10:43:36.462998] I [glusterfsd.c:1666:main] > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version > 3.3.1 > [2012-11-16 10:43:36.471280] I [io-cache.c:1549:check_cache_size_ok] > 0-gdata-quick-read: Max cache size is 25332322304 > [2012-11-16 10:43:36.471347] I [io-cache.c:1549:check_cache_size_ok] > 0-gdata-io-cache: Max cache size is 25332322304 > [2012-11-16 10:43:36.474518] I [client.c:2142:notify] > 0-gdata-client-0: parent translators are ready, attempting connect > on transport > [2012-11-16 10:43:36.478166] I [client.c:2142:notify] > 0-gdata-client-1: parent translators are ready, attempting connect > on transport > [2012-11-16 10:43:36.481604] I [client.c:2142:notify] > 0-gdata-client-2: parent translators are ready, attempting connect > on transport > Given volfile: > +------------------------------------------------------------------------------+ > 1: volume gdata-client-0 > 2: type protocol/client > 3: option remote-host gluster-0-0 > 4: option remote-subvolume /mseas-data-0-0 > 5: option transport-type tcp > 6: end-volume > 7: > 8: volume gdata-client-1 > 9: type protocol/client > 10: option remote-host gluster-0-1 > 11: option remote-subvolume /mseas-data-0-1 > 12: option transport-type tcp > 13: end-volume > 14: > 15: volume gdata-client-2 > 16: type protocol/client > 17: option remote-host gluster-data > 18: option remote-subvolume /data > 19: option transport-type tcp > 20: end-volume > 21: > 22: volume gdata-dht > 23: type cluster/distribute > 24: subvolumes gdata-client-0 gdata-client-1 gdata-client-2 > 25: end-volume > 26: > 27: volume gdata-write-behind > 28: type performance/write-behind > 29: subvolumes gdata-dht > 30: end-volume > 31: > 32: volume gdata-read-ahead > 33: type performance/read-ahead > 34: subvolumes gdata-write-behind > 35: end-volume > 36: > 37: volume gdata-io-cache > 38: type performance/io-cache > 39: subvolumes gdata-read-ahead > 40: end-volume > 41: > 42: volume gdata-quick-read > 43: type performance/quick-read > 44: subvolumes gdata-io-cache > 45: end-volume > 46: > 47: volume gdata-stat-prefetch > 48: type performance/stat-prefetch > 49: subvolumes gdata-quick-read > 50: end-volume > 51: > 52: volume gdata > 53: type debug/io-stats > 54: subvolumes gdata-stat-prefetch > 55: end-volume > > +------------------------------------------------------------------------------+ > [2012-11-16 10:43:36.485221] E [socket.c:1715:socket_connect_finish] > 0-gdata-client-2: connection to failed (No route to host) > [2012-11-16 10:43:36.485534] E > [client-handshake.c:1717:client_query_portmap_cbk] 0-gdata-client-1: > failed to get the port number for remote subvolume > [2012-11-16 10:43:36.485607] I [client.c:2090:client_rpc_notify] > 0-gdata-client-1: disconnected > [2012-11-16 10:43:36.485626] E > [client-handshake.c:1717:client_query_portmap_cbk] 0-gdata-client-0: > failed to get the port number for remote subvolume > [2012-11-16 10:43:36.485647] I [client.c:2090:client_rpc_notify] > 0-gdata-client-0: disconnected > [2012-11-16 10:43:36.492937] I [fuse-bridge.c:4191:fuse_graph_setup] > 0-fuse: switched to graph 0 > [2012-11-16 10:43:36.493966] I [fuse-bridge.c:3376:fuse_init] > 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 > kernel 7.13 > [2012-11-16 10:43:36.494088] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for / > [2012-11-16 10:43:36.494214] E [dht-common.c:1372:dht_lookup] > 0-gdata-dht: Failed to get hashed subvol for / > [2012-11-16 10:43:36.494231] W [fuse-bridge.c:513:fuse_attr_cbk] > 0-glusterfs-fuse: 2: LOOKUP() / => -1 (Invalid argument) > [2012-11-16 10:43:36.500651] I [fuse-bridge.c:4091:fuse_thread_proc] > 0-fuse: unmounting /var/log/glusterfs/gdata > [2012-11-16 10:43:36.501036] W [glusterfsd.c:831:cleanup_and_exit] > (-->/lib64/libc.so.6(clone+0x6d) [0x3065ae5ccd] > (-->/lib64/libpthread.so.0() [0x30666077f1] > (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x405d4d]))) 0-: > received signum (15), shutting down > [2012-11-16 10:43:36.501072] I [fuse-bridge.c:4648:fini] 0-fuse: > Unmounting '/var/log/glusterfs/gdata'. > > > > > It appears I am reaching the file system, but: > 0-gdata-client-0: failed to get the port number for remote subvolume > 0-gdata-dht: Failed to get hashed subvol for / > > I have googled/troubleshot but am unable to find solution. Your help > would be Greatly appreciated. > Thanks, > Steve Postma > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users >
