On Fri, 22 Jun 2012, Harry Mangalam wrote: > Were you automounting your glusterfs? > > This is extremely similar to what I described previously: > > <http://comments.gmane.org/gmane.comp.file-systems.gluster.user/9241> > > In my case the servers were running SL6.2; clients were running > CentOS 5.7 and were automounting the glusterfs with IPoIB. When we > switched to hard mounts, the problem seems to have gone away, but I'd > be interested in seeing this resolved. The "Client lk-version numbers > are not same, reopening the fds" strings in the logs were also > identical, even when I compiled new client versions that matched the > server version. > > If this does not have a trivial answer, I'll be happy to file an > official bug report. One report could be a twit :). second one could > be a bug. I was automounting, however I get the same problem when I manually mount with: mount -t glusterfs localhost:/share /share ><> Nathan Stratton nathan at robotics.net http://www.robotics.net > hjm > > On Fri, Jun 22, 2012 at 7:49 AM, Nathan Stratton <nathan at robotics.net> wrote: >> >> When I do a NFS mount and do a ls I get: >> >> [root at ovirt share]# ls >> ls: reading directory .: Too many levels of symbolic links >> >> [root at ovirt share]# ls -fl >> ls: reading directory .: Too many levels of symbolic links >> total 3636 >> drwxr-xr-x ? 3 root root 16384 Jun 21 19:34 . >> dr-xr-xr-x. 21 root root ?4096 Jun 21 19:29 .. >> drwxr-xr-x ? 3 root root 16384 Jun 21 19:34 . >> dr-xr-xr-x. 21 root root ?4096 Jun 21 19:29 .. >> drwxr-xr-x ? 3 root root 16384 Jun 21 19:34 . >> dr-xr-xr-x. 21 root root ?4096 Jun 21 19:29 .. >> drwxr-xr-x ? 3 root root 16384 Jun 21 19:34 . >> dr-xr-xr-x. 21 root root ?4096 Jun 21 19:29 .. >> drwxr-xr-x ? 3 root root 16384 Jun 21 19:34 . >> >> {and so on} >> >> When I try a fuse mount I get: >> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-0: Server and >> Client lk-version numbers are not same, reopening the fds >> >> I have tried the fedora 16 RPMs and also built new fedora 17 RPMs. >> >> Full Log: >> >> [2012-06-21 19:24:35.633510] I [glusterfsd.c:1666:main] >> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0 >> [2012-06-21 19:24:35.646832] I [io-cache.c:1549:check_cache_size_ok] >> 0-share-quick-read: Max cache size is 16825450496 >> [2012-06-21 19:24:35.646916] I [io-cache.c:1549:check_cache_size_ok] >> 0-share-io-cache: Max cache size is 16825450496 >> [2012-06-21 19:24:35.660807] I [client.c:2142:notify] 0-share-client-0: >> parent translators are ready, attempting connect on transport >> [2012-06-21 19:24:35.664026] I [client.c:2142:notify] 0-share-client-1: >> parent translators are ready, attempting connect on transport >> [2012-06-21 19:24:35.666801] I [client.c:2142:notify] 0-share-client-2: >> parent translators are ready, attempting connect on transport >> [2012-06-21 19:24:35.669385] I [client.c:2142:notify] 0-share-client-3: >> parent translators are ready, attempting connect on transport >> [2012-06-21 19:24:35.671951] I [client.c:2142:notify] 0-share-client-4: >> parent translators are ready, attempting connect on transport >> [2012-06-21 19:24:35.674514] I [client.c:2142:notify] 0-share-client-5: >> parent translators are ready, attempting connect on transport >> [2012-06-21 19:24:35.677093] I [client.c:2142:notify] 0-share-client-6: >> parent translators are ready, attempting connect on transport >> [2012-06-21 19:24:35.679652] I [client.c:2142:notify] 0-share-client-7: >> parent translators are ready, attempting connect on transport >> Given volfile: >> +------------------------------------------------------------------------------+ >> ?1: volume share-client-0 >> ?2: ? ? type protocol/client >> ?3: ? ? option remote-host virt01.casabi.net >> ?4: ? ? option remote-subvolume /export >> ?5: ? ? option transport-type tcp >> ?6: ? ? option username f0a71b3a-1cb2-4a38-8620-f3853aa90798 >> ?7: ? ? option password 9641cac2-de93-4621-b94b-9320ada20fd7 >> ?8: end-volume >> ?9: >> ?10: volume share-client-1 >> ?11: ? ? type protocol/client >> ?12: ? ? option remote-host virt02.casabi.net >> ?13: ? ? option remote-subvolume /export >> ?14: ? ? option transport-type tcp >> ?15: ? ? option username f0a71b3a-1cb2-4a38-8620-f3853aa90798 >> ?16: ? ? option password 9641cac2-de93-4621-b94b-9320ada20fd7 >> ?17: end-volume >> ?18: >> ?19: volume share-client-2 >> ?20: ? ? type protocol/client >> ?21: ? ? option remote-host virt03.casabi.net >> ?22: ? ? option remote-subvolume /export >> ?23: ? ? option transport-type tcp >> ?24: ? ? option username f0a71b3a-1cb2-4a38-8620-f3853aa90798 >> ?25: ? ? option password 9641cac2-de93-4621-b94b-9320ada20fd7 >> ?26: end-volume >> ?27: >> ?28: volume share-client-3 >> ?29: ? ? type protocol/client >> ?30: ? ? option remote-host virt04.casabi.net >> ?31: ? ? option remote-subvolume /export >> ?32: ? ? option transport-type tcp >> ?33: ? ? option username f0a71b3a-1cb2-4a38-8620-f3853aa90798 >> ?34: ? ? option password 9641cac2-de93-4621-b94b-9320ada20fd7 >> ?35: end-volume >> ?36: >> ?37: volume share-client-4 >> ?38: ? ? type protocol/client >> ?39: ? ? option remote-host virt05.casabi.net >> ?40: ? ? option remote-subvolume /export >> ?41: ? ? option transport-type tcp >> ?42: ? ? option username f0a71b3a-1cb2-4a38-8620-f3853aa90798 >> ?43: ? ? option password 9641cac2-de93-4621-b94b-9320ada20fd7 >> ?44: end-volume >> ?45: >> ?46: volume share-client-5 >> ?47: ? ? type protocol/client >> ?48: ? ? option remote-host virt06.casabi.net >> ?49: ? ? option remote-subvolume /export >> ?50: ? ? option transport-type tcp >> ?51: ? ? option username f0a71b3a-1cb2-4a38-8620-f3853aa90798 >> ?52: ? ? option password 9641cac2-de93-4621-b94b-9320ada20fd7 >> ?53: end-volume >> ?54: >> ?55: volume share-client-6 >> ?56: ? ? type protocol/client >> ?57: ? ? option remote-host virt07.casabi.net >> ?58: ? ? option remote-subvolume /export >> ?59: ? ? option transport-type tcp >> ?60: ? ? option username f0a71b3a-1cb2-4a38-8620-f3853aa90798 >> ?61: ? ? option password 9641cac2-de93-4621-b94b-9320ada20fd7 >> ?62: end-volume >> ?63: >> ?64: volume share-client-7 >> ?65: ? ? type protocol/client >> ?66: ? ? option remote-host virt08.casabi.net >> ?67: ? ? option remote-subvolume /export >> ?68: ? ? option transport-type tcp >> ?69: ? ? option username f0a71b3a-1cb2-4a38-8620-f3853aa90798 >> ?70: ? ? option password 9641cac2-de93-4621-b94b-9320ada20fd7 >> ?71: end-volume >> ?72: >> ?73: volume share-replicate-0 >> ?74: ? ? type cluster/replicate >> ?75: ? ? subvolumes share-client-0 share-client-1 >> ?76: end-volume >> ?77: >> ?78: volume share-replicate-1 >> ?79: ? ? type cluster/replicate >> ?80: ? ? subvolumes share-client-2 share-client-3 >> ?81: end-volume >> ?82: >> ?83: volume share-replicate-2 >> ?84: ? ? type cluster/replicate >> ?85: ? ? subvolumes share-client-4 share-client-5 >> ?86: end-volume >> ?87: >> ?88: volume share-replicate-3 >> ?89: ? ? type cluster/replicate >> ?90: ? ? subvolumes share-client-6 share-client-7 >> ?91: end-volume >> ?92: >> ?93: volume share-dht >> ?94: ? ? type cluster/distribute >> ?95: ? ? subvolumes share-replicate-0 share-replicate-1 share-replicate-2 >> share-replicate-3 >> ?96: end-volume >> ?97: >> ?98: volume share-write-behind >> ?99: ? ? type performance/write-behind >> 100: ? ? subvolumes share-dht >> 101: end-volume >> 102: >> 103: volume share-read-ahead >> 104: ? ? type performance/read-ahead >> 105: ? ? subvolumes share-write-behind >> 106: end-volume >> 107: >> 108: volume share-io-cache >> 109: ? ? type performance/io-cache >> 110: ? ? subvolumes share-read-ahead >> 111: end-volume >> 112: >> 113: volume share-quick-read >> 114: ? ? type performance/quick-read >> 115: ? ? subvolumes share-io-cache >> 116: end-volume >> 117: >> 118: volume share-md-cache >> 119: ? ? type performance/md-cache >> 120: ? ? subvolumes share-quick-read >> 121: end-volume >> 122: >> 123: volume share >> 124: ? ? type debug/io-stats >> 125: ? ? option latency-measurement off >> 126: ? ? option count-fop-hits off >> 127: ? ? subvolumes share-md-cache >> 128: end-volume >> >> +------------------------------------------------------------------------------+ >> [2012-06-21 19:24:35.682819] I [rpc-clnt.c:1660:rpc_clnt_reconfig] >> 0-share-client-0: changing port to 24009 (from 0) >> [2012-06-21 19:24:35.682973] I [rpc-clnt.c:1660:rpc_clnt_reconfig] >> 0-share-client-1: changing port to 24010 (from 0) >> [2012-06-21 19:24:35.683014] I [rpc-clnt.c:1660:rpc_clnt_reconfig] >> 0-share-client-2: changing port to 24010 (from 0) >> [2012-06-21 19:24:35.683039] I [rpc-clnt.c:1660:rpc_clnt_reconfig] >> 0-share-client-3: changing port to 24010 (from 0) >> [2012-06-21 19:24:35.683121] I [rpc-clnt.c:1660:rpc_clnt_reconfig] >> 0-share-client-5: changing port to 24010 (from 0) >> [2012-06-21 19:24:35.683162] I [rpc-clnt.c:1660:rpc_clnt_reconfig] >> 0-share-client-4: changing port to 24010 (from 0) >> [2012-06-21 19:24:35.683206] I [rpc-clnt.c:1660:rpc_clnt_reconfig] >> 0-share-client-7: changing port to 24010 (from 0) >> [2012-06-21 19:24:35.683246] I [rpc-clnt.c:1660:rpc_clnt_reconfig] >> 0-share-client-6: changing port to 24010 (from 0) >> [2012-06-21 19:24:39.644198] I >> [client-handshake.c:1636:select_server_supported_programs] 0-share-client-0: >> Using Program GlusterFS 3.3.0, Num (1298437), Version (330) >> [2012-06-21 19:24:39.644493] I >> [client-handshake.c:1433:client_setvolume_cbk] 0-share-client-0: Connected >> to 10.59.0.11:24009, attached to remote volume '/export'. >> [2012-06-21 19:24:39.644512] I >> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-0: Server and >> Client lk-version numbers are not same, reopening the fds >> [2012-06-21 19:24:39.644579] I [afr-common.c:3627:afr_notify] >> 0-share-replicate-0: Subvolume 'share-client-0' came back up; going online. >> [2012-06-21 19:24:39.644621] I >> [client-handshake.c:453:client_set_lk_version_cbk] 0-share-client-0: Server >> lk version = 1 >> [2012-06-21 19:24:39.646923] I >> [client-handshake.c:1636:select_server_supported_programs] 0-share-client-1: >> Using Program GlusterFS 3.3.0, Num (1298437), Version (330) >> [2012-06-21 19:24:39.647207] I >> [client-handshake.c:1433:client_setvolume_cbk] 0-share-client-1: Connected >> to 10.59.0.12:24010, attached to remote volume '/export'. >> [2012-06-21 19:24:39.647227] I >> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-1: Server and >> Client lk-version numbers are not same, reopening the fds >> [2012-06-21 19:24:39.647371] I >> [client-handshake.c:453:client_set_lk_version_cbk] 0-share-client-1: Server >> lk version = 1 >> [2012-06-21 19:24:39.649587] I >> [client-handshake.c:1636:select_server_supported_programs] 0-share-client-2: >> Using Program GlusterFS 3.3.0, Num (1298437), Version (330) >> [2012-06-21 19:24:39.649806] I >> [client-handshake.c:1433:client_setvolume_cbk] 0-share-client-2: Connected >> to 10.59.0.13:24010, attached to remote volume '/export'. >> [2012-06-21 19:24:39.649826] I >> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-2: Server and >> Client lk-version numbers are not same, reopening the fds >> [2012-06-21 19:24:39.649865] I [afr-common.c:3627:afr_notify] >> 0-share-replicate-1: Subvolume 'share-client-2' came back up; going online. >> [2012-06-21 19:24:39.649935] I >> [client-handshake.c:453:client_set_lk_version_cbk] 0-share-client-2: Server >> lk version = 1 >> [2012-06-21 19:24:39.652192] I >> [client-handshake.c:1636:select_server_supported_programs] 0-share-client-3: >> Using Program GlusterFS 3.3.0, Num (1298437), Version (330) >> [2012-06-21 19:24:39.652428] I >> [client-handshake.c:1433:client_setvolume_cbk] 0-share-client-3: Connected >> to 10.59.0.14:24010, attached to remote volume '/export'. >> [2012-06-21 19:24:39.652448] I >> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-3: Server and >> Client lk-version numbers are not same, reopening the fds >> [2012-06-21 19:24:39.652566] I >> [client-handshake.c:453:client_set_lk_version_cbk] 0-share-client-3: Server >> lk version = 1 >> [2012-06-21 19:24:39.654706] I >> [client-handshake.c:1636:select_server_supported_programs] 0-share-client-5: >> Using Program GlusterFS 3.3.0, Num (1298437), Version (330) >> [2012-06-21 19:24:39.654910] I >> [client-handshake.c:1433:client_setvolume_cbk] 0-share-client-5: Connected >> to 10.59.0.16:24010, attached to remote volume '/export'. >> [2012-06-21 19:24:39.654926] I >> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-5: Server and >> Client lk-version numbers are not same, reopening the fds >> [2012-06-21 19:24:39.654961] I [afr-common.c:3627:afr_notify] >> 0-share-replicate-2: Subvolume 'share-client-5' came back up; going online. >> [2012-06-21 19:24:39.655035] I >> [client-handshake.c:453:client_set_lk_version_cbk] 0-share-client-5: Server >> lk version = 1 >> [2012-06-21 19:24:39.657289] I >> [client-handshake.c:1636:select_server_supported_programs] 0-share-client-4: >> Using Program GlusterFS 3.3.0, Num (1298437), Version (330) >> [2012-06-21 19:24:39.657511] I >> [client-handshake.c:1433:client_setvolume_cbk] 0-share-client-4: Connected >> to 10.59.0.15:24010, attached to remote volume '/export'. >> [2012-06-21 19:24:39.657526] I >> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-4: Server and >> Client lk-version numbers are not same, reopening the fds >> [2012-06-21 19:24:39.657646] I >> [client-handshake.c:453:client_set_lk_version_cbk] 0-share-client-4: Server >> lk version = 1 >> [2012-06-21 19:24:39.659884] I >> [client-handshake.c:1636:select_server_supported_programs] 0-share-client-7: >> Using Program GlusterFS 3.3.0, Num (1298437), Version (330) >> [2012-06-21 19:24:39.660075] I >> [client-handshake.c:1433:client_setvolume_cbk] 0-share-client-7: Connected >> to 10.59.0.18:24010, attached to remote volume '/export'. >> [2012-06-21 19:24:39.660091] I >> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-7: Server and >> Client lk-version numbers are not same, reopening the fds >> [2012-06-21 19:24:39.660127] I [afr-common.c:3627:afr_notify] >> 0-share-replicate-3: Subvolume 'share-client-7' came back up; going online. >> [2012-06-21 19:24:39.660202] I >> [client-handshake.c:453:client_set_lk_version_cbk] 0-share-client-7: Server >> lk version = 1 >> [2012-06-21 19:24:39.662579] I >> [client-handshake.c:1636:select_server_supported_programs] 0-share-client-6: >> Using Program GlusterFS 3.3.0, Num (1298437), Version (330) >> [2012-06-21 19:24:39.662908] I >> [client-handshake.c:1433:client_setvolume_cbk] 0-share-client-6: Connected >> to 10.59.0.17:24010, attached to remote volume '/export'. >> [2012-06-21 19:24:39.662931] I >> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-6: Server and >> Client lk-version numbers are not same, reopening the fds >> [2012-06-21 19:24:39.668028] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: >> switched to graph 0 >> [2012-06-21 19:24:39.668234] I >> [client-handshake.c:453:client_set_lk_version_cbk] 0-share-client-6: Server >> lk version = 1 >> [2012-06-21 19:24:39.668287] I [fuse-bridge.c:3376:fuse_init] >> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel >> 7.18 >> [2012-06-21 19:24:39.668739] I >> [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 0-share-replicate-0: >> added root inode >> [2012-06-21 19:24:39.669232] I >> [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 0-share-replicate-1: >> added root inode >> [2012-06-21 19:24:39.669321] I >> [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 0-share-replicate-2: >> added root inode >> [2012-06-21 19:24:39.669379] I >> [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 0-share-replicate-3: >> added root inode >> >> >> >>> <> >> >> Nathan Stratton >> nathan at robotics.net >> http://www.robotics.net >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >
