Re: [Gluster-devel] v3.6.2

"David F. Robinson" <david.robinson@xxxxxxxxxxxxx> · Tue, 27 Jan 2015 15:35:57 +0000

After shutting down all NFS and gluster processes, there was still an 
NFS process.

[root@gfs01bkp ~]# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100005    3   tcp  38465  mountd
    100005    1   tcp  38466  mountd
    100003    3   tcp   2049  nfs
    100024    1   udp  34738  status
    100024    1   tcp  37269  status
[root@gfs01bkp ~]# netstat -anp | grep 2049
[root@gfs01bkp ~]# netstat -anp | grep 38465
[root@gfs01bkp ~]# netstat -anp | grep 38466

I killed off the processes using "rpcinfo -d"

[root@gfs01bkp ~]# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  34738  status
    100024    1   tcp  37269  status

Then I restarted the glusterd and did a 'mount -a'.  Worked perfectly.  
And the errors that were showing up in the logs every 3-seconds stopped.

Thanks for your help.  Greatly appreciated.

David

------ Original Message ------
From: "Xavier Hernandez" <xhernandez@xxxxxxxxxx>
To: "David F. Robinson" <david.robinson@xxxxxxxxxxxxx>; "Kaushal M" 
<kshlmster@xxxxxxxxx>
Cc: "Gluster Users" <gluster-users@xxxxxxxxxxx>; "Gluster Devel" 
<gluster-devel@xxxxxxxxxxx>
Sent: 1/27/2015 10:02:31 AM
Subject: Re: [Gluster-devel]  v3.6.2

Hi,

I had a similar problem once. It happened after doing some unrelated 
tests with NFS. I thought it was a problem I generated doing weird 
things, so I didn't investigate the cause further.

To see if this is the same case, try this:

* Unmount all NFS mounts and stop all gluster volumes
* Check that there are no gluster processes running (ps ax | grep 
gluster), specially any glusterfs. glusterd is ok.
* Check that there are no NFS processes running (ps ax | grep nfs)
* Check with 'rpcinfo -p' that there's no nfs service registered

The output should be similar to this:

   program vers proto port service
    100000 4 tcp 111 portmapper
    100000 3 tcp 111 portmapper
    100000 2 tcp 111 portmapper
    100000 4 udp 111 portmapper
    100000 3 udp 111 portmapper
    100000 2 udp 111 portmapper
    100024 1 udp 33482 status
    100024 1 tcp 37034 status

If there are more services registered, you can directly delete them or 
check if they correspond to an active process. For example, if the 
output is this:

   program vers proto port service
    100000 4 tcp 111 portmapper
    100000 3 tcp 111 portmapper
    100000 2 tcp 111 portmapper
    100000 4 udp 111 portmapper
    100000 3 udp 111 portmapper
    100000 2 udp 111 portmapper
    100021 3 udp 39618 nlockmgr
    100021 3 tcp 41067 nlockmgr
    100024 1 udp 33482 status
    100024 1 tcp 37034 status

You can do a "netstat -anp | grep 39618" to see if there is some 
process really listening at the nlockmgr port. You can repeat this for 
port 41067. If there is some process, you should stop it. If there is 
no process listening on that port, you should remove it with a command 
like this:

    rpcinfo -d 100021 3

You must execute this command for all stale ports for any services 
other than portmapper and status. Once done you should get the output 
shown before.

After that, you can try to start your volume and see if everything is 
registered (rpcinfo -p) and if gluster has started the nfs server 
(gluster volume status).

If everything is ok, you should be able to mount the volume using NFS.

Xavi

On 01/27/2015 03:18 PM, David F. Robinson wrote:
Turning off nfslock did not help. Also, still getting these messages
every 3-seconds:

[2015-01-27 14:16:12.921880] W [socket.c:611:__socket_rwv] 
0-management:
readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
(Invalid argument)
[2015-01-27 14:16:15.922431] W [socket.c:611:__socket_rwv] 
0-management:
readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
(Invalid argument)
[2015-01-27 14:16:18.923080] W [socket.c:611:__socket_rwv] 
0-management:
readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
(Invalid argument)
[2015-01-27 14:16:21.923748] W [socket.c:611:__socket_rwv] 
0-management:
readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
(Invalid argument)
[2015-01-27 14:16:24.924472] W [socket.c:611:__socket_rwv] 
0-management:
readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
(Invalid argument)
[2015-01-27 14:16:27.925192] W [socket.c:611:__socket_rwv] 
0-management:
readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
(Invalid argument)
[2015-01-27 14:16:30.925895] W [socket.c:611:__socket_rwv] 
0-management:
readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
(Invalid argument)
[2015-01-27 14:16:33.926563] W [socket.c:611:__socket_rwv] 
0-management:
readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
(Invalid argument)
[2015-01-27 14:16:36.927248] W [socket.c:611:__socket_rwv] 
0-management:
readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
(Invalid argument)
------ Original Message ------
From: "Kaushal M" <kshlmster@xxxxxxxxx <mailto:kshlmster@xxxxxxxxx>>
To: "David F. Robinson" <david.robinson@xxxxxxxxxxxxx
<mailto:david.robinson@xxxxxxxxxxxxx>>
Cc: "Joe Julian" <joe@xxxxxxxxxxxxxxxx <mailto:joe@xxxxxxxxxxxxxxxx>>;
"Gluster Users" <gluster-users@xxxxxxxxxxx
<mailto:gluster-users@xxxxxxxxxxx>>; "Gluster Devel"
<gluster-devel@xxxxxxxxxxx <mailto:gluster-devel@xxxxxxxxxxx>>
Sent: 1/27/2015 1:49:56 AM
Subject: Re: Re[2]: [Gluster-devel]  v3.6.2
Your nfs.log file has the following lines,
```
[2015-01-26 20:06:58.298078] E
[rpcsvc.c:1303:rpcsvc_program_register_portmap] 0-rpc-service: Could
not register with portmap 100021 4 38468
[2015-01-26 20:06:58.298108] E [nfs.c:331:nfs_init_versions] 0-nfs:
Program NLM4 registration failed
```

The Gluster NFS server has it's own NLM (nlockmgr) implementation. 
You
said that you have the nfslock service on. Can you turn that off and
try again?

~kaushal

On Tue, Jan 27, 2015 at 11:21 AM, David F. Robinson
<david.robinson@xxxxxxxxxxxxx <mailto:david.robinson@xxxxxxxxxxxxx>>
wrote:

    On a different system where gluster-NFS (not kernel-nfs) is
    working properly shows the following:
    [root@gfs01a glusterfs]# rpcinfo -p
       program vers proto port service
        100000 4 tcp 111 portmapper
        100000 3 tcp 111 portmapper
        100000 2 tcp 111 portmapper
        100000 4 udp 111 portmapper
        100000 3 udp 111 portmapper
        100000 2 udp 111 portmapper
        100005 3 tcp 38465 mountd
        100005 1 tcp 38466 mountd
        100003 3 tcp 2049 nfs
        100024 1 udp 42413 status
        100024 1 tcp 35424 status
        100021 4 tcp 38468 nlockmgr
        100021 1 udp 801 nlockmgr
        100227 3 tcp 2049 nfs_acl
        100021 1 tcp 804 nlockmgr

    [root@gfs01a glusterfs]# /etc/init.d/nfs status
    rpc.svcgssd is stopped
    rpc.mountd is stopped
    nfsd is stopped
    rpc.rquotad is stopped
    ------ Original Message ------
    From: "Joe Julian" <joe@xxxxxxxxxxxxxxxx
    <mailto:joe@xxxxxxxxxxxxxxxx>>
    To: "Kaushal M" <kshlmster@xxxxxxxxx
    <mailto:kshlmster@xxxxxxxxx>>; "David F. Robinson"
    <david.robinson@xxxxxxxxxxxxx 
<mailto:david.robinson@xxxxxxxxxxxxx>>
    Cc: "Gluster Users" <gluster-users@xxxxxxxxxxx
    <mailto:gluster-users@xxxxxxxxxxx>>; "Gluster Devel"
    <gluster-devel@xxxxxxxxxxx <mailto:gluster-devel@xxxxxxxxxxx>>
    Sent: 1/27/2015 12:48:49 AM
    Subject: Re: [Gluster-devel]  v3.6.2
    If that was true, wouldn't it not "Connection refused" because
    the kernel nfs is listening?

    On January 26, 2015 9:43:34 PM PST, Kaushal M
    <kshlmster@xxxxxxxxx <mailto:kshlmster@xxxxxxxxx>> wrote:

        Seems like you have the kernel NFS server running. The
        `rpcinfo -p` output you provided shows that there are other
        mountd, nfs and nlockmgr services running your system.
        Gluster NFS server requires that the kernel nfs services be
        disabled and not running.

        ~kaushal

        On Tue, Jan 27, 2015 at 10:56 AM, David F. Robinson
        <david.robinson@xxxxxxxxxxxxx
        <mailto:david.robinson@xxxxxxxxxxxxx>> wrote:

            [root@gfs01bkp ~]# gluster volume status homegfs_bkp
            Status of volume: homegfs_bkp
            Gluster process
             Port Online Pid

------------------------------__------------------------------__------------------
            Brick 
gfsib01bkp.corvidtec.com:/__data/brick01bkp/homegfs
            _bkp
            49152 Y 4087
            Brick 
gfsib01bkp.corvidtec.com:/__data/brick02bkp/homegfs
            _bkp
            49155 Y 4092
            NFS Server on localhost
             N/A N N/A

            Task Status of Volume homegfs_bkp

------------------------------__------------------------------__------------------
            Task : Rebalance
            ID : 6d4c6c4e-16da-48c9-9019-__dccb7d2cfd66
            Status : completed

            ------ Original Message ------
            From: "Atin Mukherjee" <amukherj@xxxxxxxxxx
            <mailto:amukherj@xxxxxxxxxx>>
            To: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx
            <mailto:pkarampu@xxxxxxxxxx>>; "Justin Clift"
            <justin@xxxxxxxxxxx <mailto:justin@xxxxxxxxxxx>>; "David
            F. Robinson" <david.robinson@xxxxxxxxxxxxx
            <mailto:david.robinson@xxxxxxxxxxxxx>>
            Cc: "Gluster Users" <gluster-users@xxxxxxxxxxx
            <mailto:gluster-users@xxxxxxxxxxx>>; "Gluster Devel"
            <gluster-devel@xxxxxxxxxxx
            <mailto:gluster-devel@xxxxxxxxxxx>>
            Sent: 1/26/2015 11:51:13 PM
            Subject: Re: [Gluster-devel] v3.6.2

                On 01/27/2015 07:33 AM, Pranith Kumar Karampuri 
wrote:

                     On 01/26/2015 09:41 PM, Justin Clift wrote:

                         On 26 Jan 2015, at 14:50, David F. Robinson
                         <david.robinson@xxxxxxxxxxxxx
                        <mailto:david.robinson@xxxxxxxxxxxxx>> 
wrote:

                             I have a server with v3.6.2 from which 
I
                            cannot mount using NFS. The
                             FUSE mount works, however, I cannot get
                            the NFS mount to work. From
                             /var/log/message:
                               Jan 26 09:27:28 gfs01bkp mount[2810]:
                            mount to NFS server
                             'gfsib01bkp.corvidtec.com
                            <http://gfsib01bkp.corvidtec.com/>'
                            failed: Connection refused, retrying
                             Jan 26 09:27:53 gfs01bkp mount[4456]:
                            mount to NFS server
                             'gfsib01bkp.corvidtec.com
                            <http://gfsib01bkp.corvidtec.com/>'
                            failed: Connection refused, retrying
                             Jan 26 09:29:28 gfs01bkp mount[2810]:
                            mount to NFS server
                             'gfsib01bkp.corvidtec.com
                            <http://gfsib01bkp.corvidtec.com/>'
                            failed: Connection refused, retrying
                             Jan 26 09:29:53 gfs01bkp mount[4456]:
                            mount to NFS server
                             'gfsib01bkp.corvidtec.com
                            <http://gfsib01bkp.corvidtec.com/>'
                            failed: Connection refused, retrying
                             Jan 26 09:31:28 gfs01bkp mount[2810]:
                            mount to NFS server
                             'gfsib01bkp.corvidtec.com
                            <http://gfsib01bkp.corvidtec.com/>'
                            failed: Connection refused, retrying
                             Jan 26 09:31:53 gfs01bkp mount[4456]:
                            mount to NFS server
                             'gfsib01bkp.corvidtec.com
                            <http://gfsib01bkp.corvidtec.com/>'
                            failed: Connection refused, retrying
                             Jan 26 09:33:28 gfs01bkp mount[2810]:
                            mount to NFS server
                             'gfsib01bkp.corvidtec.com
                            <http://gfsib01bkp.corvidtec.com/>'
                            failed: Connection refused, retrying
                             Jan 26 09:33:53 gfs01bkp mount[4456]:
                            mount to NFS server
                             'gfsib01bkp.corvidtec.com
                            <http://gfsib01bkp.corvidtec.com/>'
                            failed: Connection refused, retrying
                             Jan 26 09:35:28 gfs01bkp mount[2810]:
                            mount to NFS server
                             'gfsib01bkp.corvidtec.com
                            <http://gfsib01bkp.corvidtec.com/>'
                            failed: Connection refused, retrying
                             Jan 26 09:35:53 gfs01bkp mount[4456]:
                            mount to NFS server
                             'gfsib01bkp.corvidtec.com
                            <http://gfsib01bkp.corvidtec.com/>'
                            failed: Connection refused, retrying
                                 I also am continually getting the
                            following errors in
                             /var/log/glusterfs:
                               [root@gfs01bkp glusterfs]# tail -f
                            etc-glusterfs-glusterd.vol.log
                             [2015-01-26 14:41:51.260827] W
                            [socket.c:611:__socket_rwv]
                             0-management: readv on

/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
                            failed (Invalid
                             argument)
                             [2015-01-26 14:41:54.261240] W
                            [socket.c:611:__socket_rwv]
                             0-management: readv on

/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
                            failed (Invalid
                             argument)
                             [2015-01-26 14:41:57.261642] W
                            [socket.c:611:__socket_rwv]
                             0-management: readv on

/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
                            failed (Invalid
                             argument)
                             [2015-01-26 14:42:00.262073] W
                            [socket.c:611:__socket_rwv]
                             0-management: readv on

/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
                            failed (Invalid
                             argument)
                             [2015-01-26 14:42:03.262504] W
                            [socket.c:611:__socket_rwv]
                             0-management: readv on

/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
                            failed (Invalid
                             argument)
                             [2015-01-26 14:42:06.262935] W
                            [socket.c:611:__socket_rwv]
                             0-management: readv on

/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
                            failed (Invalid
                             argument)
                             [2015-01-26 14:42:09.263334] W
                            [socket.c:611:__socket_rwv]
                             0-management: readv on

/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
                            failed (Invalid
                             argument)
                             [2015-01-26 14:42:12.263761] W
                            [socket.c:611:__socket_rwv]
                             0-management: readv on

/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
                            failed (Invalid
                             argument)
                             [2015-01-26 14:42:15.264177] W
                            [socket.c:611:__socket_rwv]
                             0-management: readv on

/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
                            failed (Invalid
                             argument)
                             [2015-01-26 14:42:18.264623] W
                            [socket.c:611:__socket_rwv]
                             0-management: readv on

/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
                            failed (Invalid
                             argument)
                             [2015-01-26 14:42:21.265053] W
                            [socket.c:611:__socket_rwv]
                             0-management: readv on

/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
                            failed (Invalid
                             argument)
                             [2015-01-26 14:42:24.265504] W
                            [socket.c:611:__socket_rwv]
                             0-management: readv on

/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
                            failed (Invalid
                             argument)

                     I believe this error message comes when the
                    socket file is not present.
                     I see the following commit which changed the
                    location of the sockets.
                     May be Atin may know more. about this: +Atin.

                Can we get the output of 'gluster volume status' for
                the volume which
                you are trying to mount?

                ~Atin

                     Pranith

                             ^C
                               Also, when I try to NFS mount my
                            gluster volume, I am getting

                         Any chance there's a network or host based
                        firewall stopping some of
                         the ports?

                         + Justin

                         --
                         GlusterFS - http://www.gluster.org/

                         An open source, distributed file system
                        scaling to several
                         petabytes, and handling thousands of 
clients.

                         My personal twitter:
                        twitter.com/realjustinclift
                        <http://twitter.com/realjustinclift>

_________________________________________________
                         Gluster-devel mailing list
                        Gluster-devel@xxxxxxxxxxx
                        <mailto:Gluster-devel@xxxxxxxxxxx>

http://www.gluster.org/__mailman/listinfo/gluster-devel

<http://www.gluster.org/mailman/listinfo/gluster-devel>

            _________________________________________________
            Gluster-users mailing list
            Gluster-users@xxxxxxxxxxx 
<mailto:Gluster-users@xxxxxxxxxxx>
            http://www.gluster.org/__mailman/listinfo/gluster-users
            <http://www.gluster.org/mailman/listinfo/gluster-users>

------------------------------------------------------------------------

        Gluster-devel mailing list
        Gluster-devel@xxxxxxxxxxx <mailto:Gluster-devel@xxxxxxxxxxx>
        http://www.gluster.org/mailman/listinfo/gluster-devel

    --
    Sent from my Android device with K-9 Mail. Please excuse my 
brevity.

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users