I poke around at this today and figured it out - Sort of. When I did 'gluster volume set named nfs.register-with-portmap on' originally, I only had rpcbind running on two of my four servers. Gluster nfs started up on all four, but obviously only two correctly connected with rpcbind/portmap. Seems that if rpcbind is not running when you set 'register-with-portmap on', even when you cycle gluster it still doesn't work. So, I started up rpcbind, did a 'register-with-portmap off', followed by 'register-with-portmap on' and it works now. When I diffed the before and after nfs-server.vol files, I see this: [root at dresproddns01 ~]# diff nfs-vol-1 /etc/glusterd/nfs/nfs-server.vol 143c143 < option rpc.register-with-portmap off --- > option rpc.register-with-portmap on Apparently if rpcbind is not running, the option does not get enabled properly. There is an error in nfs.log, but it's hard to find especially if the node you manage the cluster from isn't the node with the issue. It isn't clear either that it's broken even if you cycle gluster (and even though the gluster volume configuration says 'register-with-portmap on'. Does the 'gluster volume set' command have the ability to get success/fail information back from each node? It also appears that 'register-with-portmap' is applied to all volumes, even if you just enable it on one - Is there a cluster-wide place to 'set' options? [2012-03-05 19:51:22.517368] E [rpcsvc.c:2771:nfs_rpcsvc_program_register_portmap] 0-nfsrpc: Could not register with portmap [2012-03-05 19:51:22.517420] E [rpcsvc.c:2861:nfs_rpcsvc_program_register] 0-nfsrpc: portmap registration of program failed [2012-03-05 19:51:22.517428] E [rpcsvc.c:2874:nfs_rpcsvc_program_register] 0-nfsrpc: Program registration failed: MOUNT3, Num: 100005, Ver: 3, Port: 38465 David On 3/5/12 2:05 PM, Bryan Whitehead wrote: > Is selinux running? iptables? > > Can you http://pastie.org/ the nfs.log in /var/log/glusterfs ? > > On Mon, Mar 5, 2012 at 3:59 AM, David Coulson <david at davidcoulson.net > <mailto:david at davidcoulson.net>> wrote: > > Yep. > > [root at dresproddns01 ~]# service glusterd stop > Stopping glusterd: [ OK ] > > [root at dresproddns01 ~]# ps ax | grep nfs > 120494 pts/0 S+ 0:00 grep nfs > > 2167119 ? S 0:00 [nfsiod] > [root at dresproddns01 ~]# service rpcbind stop > Stopping rpcbind: [ OK ] > > [root at dresproddns01 ~]# rpcinfo -p > rpcinfo: can't contact portmapper: RPC: Remote system error - No > such file or directory > [root at dresproddns01 ~]# service rpcbind start > Starting rpcbind: [ OK ] > [root at dresproddns01 ~]# service glusterd start > Starting glusterd: [ OK ] > > [root at dresproddns01 ~]# rpcinfo -p > program vers proto port service > 100000 4 tcp 111 portmapper > 100000 3 tcp 111 portmapper > 100000 2 tcp 111 portmapper > 100000 4 udp 111 portmapper > 100000 3 udp 111 portmapper > 100000 2 udp 111 portmapper > > Note that I waited a short while between the last two steps. FYI, > this is RHEL6 (the two systems that work are RHEL6 too, so I'm not > sure it matters much). > > > On 3/5/12 3:27 AM, Bryan Whitehead wrote: >> did you start portmap service before you started gluster? >> >> On Sun, Mar 4, 2012 at 11:53 AM, David Coulson >> <david at davidcoulson.net <mailto:david at davidcoulson.net>> wrote: >> >> I've four systems with multiple 4-way replica volumes. I'm >> migrating a number of volumes from Fuse to NFS for >> performance reasons. >> >> My first two hosts seem to work nicely, but the other two >> won't start the NFS services properly. I looked through the >> nfs.log, but it doesn't give any indication of why it did not >> register with rpcbind. I'm presuming I've got a >> misconfiguration on two of the systems, but there isn't a >> clear indication of what is not working. >> >> Here is an example from a host which does not work: >> >> [root at dresproddns01 ~]# rpcinfo -p >> program vers proto port service >> 100000 4 tcp 111 portmapper >> 100000 3 tcp 111 portmapper >> 100000 2 tcp 111 portmapper >> 100000 4 udp 111 portmapper >> 100000 3 udp 111 portmapper >> 100000 2 udp 111 portmapper >> [root at dresproddns01 ~]# ps ax | grep nfs >> 2167119 ? S 0:00 [nfsiod] >> 2738268 ? Ssl 0:00 >> /opt/glusterfs/3.2.5/sbin/glusterfs -f >> /etc/glusterd/nfs/nfs-server.vol -p >> /etc/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log >> 2934228 pts/0 S+ 0:00 grep nfs >> [root at dresproddns01 ~]# netstat -ntlp | grep 2738268 >> tcp 0 0 0.0.0.0:38465 >> <http://0.0.0.0:38465> >> 0.0.0.0:* LISTEN 2738268/glusterfs >> tcp 0 0 0.0.0.0:38466 >> <http://0.0.0.0:38466> >> 0.0.0.0:* LISTEN 2738268/glusterfs >> tcp 0 0 0.0.0.0:38467 >> <http://0.0.0.0:38467> >> 0.0.0.0:* LISTEN 2738268/glusterfs >> >> [root at dresproddns01 ~]# gluster volume info svn >> >> Volume Name: svn >> Type: Replicate >> Status: Started >> Number of Bricks: 4 >> Transport-type: tcp >> Bricks: >> Brick1: rhesproddns01:/gluster/svn >> Brick2: rhesproddns02:/gluster/svn >> Brick3: dresproddns01:/gluster/svn >> Brick4: dresproddns02:/gluster/svn >> Options Reconfigured: >> performance.client-io-threads: 1 >> performance.flush-behind: on >> network.ping-timeout: 5 >> performance.stat-prefetch: 1 >> nfs.disable: off >> nfs.register-with-portmap: on >> auth.allow: 10.250.53.*,10.252.248.*,169.254.*,127.0.0.1 >> performance.cache-size: 256Mb >> performance.write-behind-window-size: 128Mb >> >> Only obvious difference with a host which does work is this: >> >> [root at rhesproddns01 named]# rpcinfo -p >> program vers proto port service >> 100000 4 tcp 111 portmapper >> 100000 3 tcp 111 portmapper >> 100000 2 tcp 111 portmapper >> 100000 4 udp 111 portmapper >> 100000 3 udp 111 portmapper >> 100000 2 udp 111 portmapper >> 100005 3 tcp 38465 mountd >> 100005 1 tcp 38466 mountd >> 100003 3 tcp 38467 nfs >> >> >> Any ideas where to look for errors? >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20120305/c214d338/attachment-0001.htm>