Re: After upgrade from 3.5 to 3.7 gluster local NFS is not starting on one of the servers

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



In my original message I've mentioned that:
==========================================
root@PSC01SERV008:/var/lib/glusterd/nfs# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
==========================================
will try rpcbind in debug mode approach.

2015-09-16 4:37 GMT+08:00 Niels de Vos <ndevos@xxxxxxxxxx>:
On Sun, Sep 13, 2015 at 04:30:43PM +0530, Soumya Koduri wrote:
>
>
> On 09/13/2015 09:38 AM, Yaroslav Molochko wrote:
> >I wish this could be that simple:
> >root@PSC01SERV008:/var/lib# netstat -nap | grep 38465
> >root@PSC01SERV008:/var/lib# ss -n  | grep 38465
> >root@PSC01SERV008:/var/lib#
> >
> >2015-09-13 1:34 GMT+08:00 Atin Mukherjee <atin.mukherjee83@xxxxxxxxx
> ><mailto:atin.mukherjee83@xxxxxxxxx>>:
> >
> >    By any chance is your Gluster NFS server is already running? Output
> >    of netstat -nap | grep 38465 might give some clue?
> >
> >    -Atin
> >    Sent from one plus one
> >
> >    On Sep 12, 2015 10:54 PM, "Yaroslav Molochko" <onorua@xxxxxxxxx
> >    <mailto:onorua@xxxxxxxxx>> wrote:
> >
> >        Hello,
> >
> >        I have a problem reported in logs:
> >        ==================
> >        [2015-09-12 13:56:06.271644] I [MSGID: 100030]
> >        [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running
> >        /usr/sbin/glusterfs version 3.7.4 (args: /usr/sbin/glusterfs -s
> >        localhost --volfile-id gluster/nfs -p
> >        /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log
> >        -S /var/run/gluster/cb186678589f28e74c67da70fd06e736.socket)
> >        [2015-09-12 13:56:06.277921] I [MSGID: 101190]
> >        [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started
> >        thread with index 1
> >        [2015-09-12 13:56:07.284888] I
> >        [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service:
> >        Configured rpc.outstanding-rpc-limit with value 16
> >        [2015-09-12 13:56:07.292484] W [MSGID: 112153]
> >        [mount3.c:3910:mnt3svc_init] 0-nfs-mount: Exports auth has been
> >        disabled!
> >        [2015-09-12 13:56:07.294357] E
> >        [rpcsvc.c:1370:rpcsvc_program_register_portmap] 0-rpc-service:
> >        Could not register with portmap 100005 3 38465
>
> Port registration failed. Could you check '/var/log/messages' and dmesg to
> see if there are any errors logged? Is firewalld running on your system.
> Verify if the port is open to be used.

Registration of services at portmap can also fail because there is a
service with that program number and version registered already. Could
you check if that is the case?

    $ rpcinfo -p | grep 100005
        100005    3   tcp  38465  mountd
        100005    1   tcp  38466  mountd

If you see a similar output, check if you have standard NFS services
running. The rpc.mountd process will also register itself at the
portmapper, but it will conflict with the NFS-services that Gluster
provides. Make sure all NFS services (server and client) have been
disabled and stopped. After that, check with the 'rpcinfo' command if
any of nlockmgr, mount, status or nfs are registered. If that is the
case, you can unregister them one-by-one with commands like this:

    # rpcinfo -d 100005 1
    # rpcinfo -d 100005 3
    ...

After unregistering the services at the portmapper, you should be able
to start the Gluster-NFS service by restarting glusterd.

HTH,
Niels

>
> Thanks,
> Soumya
> >        [2015-09-12 13:56:07.294398] E [MSGID: 112088]
> >        [nfs.c:341:nfs_init_versions] 0-nfs: Required program  MOUNT3
> >        registration failed
> >        [2015-09-12 13:56:07.294413] E [MSGID: 112109] [nfs.c:1482:init]
> >        0-nfs: Failed to initialize protocols
> >        [2015-09-12 13:56:07.294426] E [MSGID: 101019]
> >        [xlator.c:428:xlator_init] 0-nfs-server: Initialization of
> >        volume 'nfs-server' failed, review your volfile again
> >        [2015-09-12 13:56:07.294438] E
> >        [graph.c:322:glusterfs_graph_init] 0-nfs-server: initializing
> >        translator failed
> >        [2015-09-12 13:56:07.294448] E
> >        [graph.c:661:glusterfs_graph_activate] 0-graph: init failed
> >        [2015-09-12 13:56:07.294781] W
> >        [glusterfsd.c:1219:cleanup_and_exit]
> >        (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x11a) [0x7fbe9c754b7a]
> >        -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x123)
> >        [0x7fbe9c74fcb3] -->/usr/sbin/glusterfs(cleanup_and_exit+0x59)
> >        [0x7fbe9c74f329] ) 0-: received signum (0), shutting down
> >        ===================
> >
> >        I've checked the page:
> >        http://www.gluster.org/community/documentation/index.php/Gluster_3.1:_NFS_Frequently_Asked_Questions
> >
> >        I've found report in RedHat that it's necessary to remove -w
> >        from rpcbind becuse some times it causes problems.
> >        I did all that but still no luck on one of the servers, what is
> >        interesting, the other server (peered) is working fine without
> >        any problems.
> >
> >        root@PSC01SERV008:/var/lib/glusterd/nfs# systemctl status nfs
> >        ● nfs.service
> >            Loaded: not-found (Reason: No such file or directory)
> >            Active: inactive (dead)
> >
> >        root@PSC01SERV008:/var/lib/glusterd/nfs# systemctl status rpcbind
> >        ● rpcbind.service - RPC bind portmap service
> >            Loaded: loaded (/etc/systemd/system/rpcbind.service;
> >        enabled; vendor preset: enabled)
> >           Drop-In: /run/systemd/generator/rpcbind.service.d
> >                    └─50-rpcbind-$portmap.conf
> >            Active: active (running) since Sat 2015-09-12 13:55:07 UTC;
> >        6min ago
> >          Main PID: 9796 (rpcbind)
> >            Memory: 428.0K
> >            CGroup: /system.slice/rpcbind.service
> >                    └─9796 /sbin/rpcbind
> >
> >        Sep 12 13:55:07 PSC01SERV008 systemd[1]: Starting RPC bind
> >        portmap service...
> >        Sep 12 13:55:07 PSC01SERV008 systemd[1]: Started RPC bind
> >        portmap service.
> >        root@PSC01SERV008:/var/lib/glusterd/nfs# rpcinfo -p
> >            program vers proto   port  service
> >             100000    4   tcp    111  portmapper
> >             100000    3   tcp    111  portmapper
> >             100000    2   tcp    111  portmapper
> >             100000    4   udp    111  portmapper
> >             100000    3   udp    111  portmapper
> >             100000    2   udp    111  portmapper
> >
> >        I've tried to reinstall it agaon and again - but there is no luck.
> >
> >        What I have:
> >        cat /etc/lsb-release
> >        DISTRIB_ID=Ubuntu
> >        DISTRIB_RELEASE=15.04
> >        DISTRIB_CODENAME=vivid
> >        DISTRIB_DESCRIPTION="Ubuntu 15.04"
> >
> >        ii  glusterfs-client
> >        3.7.4-ubuntu1~vivid1              amd64        clustered
> >        file-system (client package)
> >        ii  glusterfs-common
> >        3.7.4-ubuntu1~vivid1              amd64        GlusterFS common
> >        libraries and translator modules
> >        ii  glusterfs-server
> >        3.7.4-ubuntu1~vivid1              amd64        clustered
> >        file-system (server package)
> >
> >        What else can I check? How can I fix it, what is most important :)
> >        Thanks in advance!
> >
> >        _______________________________________________
> >        Gluster-users mailing list
> >        Gluster-users@xxxxxxxxxxx <mailto:Gluster-users@xxxxxxxxxxx>
> >        http://www.gluster.org/mailman/listinfo/gluster-users
> >
> >
> >
> >
> >_______________________________________________
> >Gluster-users mailing list
> >Gluster-users@xxxxxxxxxxx
> >http://www.gluster.org/mailman/listinfo/gluster-users
> >
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux