Re: NFS file locking with gluster 3.7.3

Anand Subramanian <ansubram@xxxxxxxxxx> · Mon, 17 Aug 2015 09:24:54 +0530

    Hi Thibault,

      There are a few tuneables that have helped boost ganesha
      performance and I suspect these tuneables on the OS side apply to
      improve performance for several workloads where ganesha is
      concerned. (And they don't seem to be necessary for gluster-nfs at
      all).

      Adding Manoj here, who may be able to point you to these
      configurables  (as he has experimented with the ganesha
      performance).

      Thanks,

      Anand

      On 08/13/2015 07:13 PM, Niels de Vos wrote:

      On Mon, Aug 10, 2015 at 09:19:25AM +0100, Thibault Godouet wrote:

        Thanks Niel for your helpful answer.

Regarding the locking, indeed that solves my issue. Now I'm wondering how
to monitor this. The best I have so far is get the list of RPC binds and
the TCP/UDP port in particular, and then run a lsof to find out if it is
Gluster.  Should work, but a bit indirect. If someone knows a better way
I'd be interested to know.

      That is almost how I do it as well. Instead of 'lsof' I use 'netstat' or
'ss', depending on the Linux distribution.

        As for Ganesha, I saw articles explaining that it effectively removes
layers, hence why I thought NFS v3 via Ganesha would be faster than native
Gluster NFS.  Given your answer I take it there are other moving parts /
differences.  Is there a general known guideline on which is best when?
E.g. does one handle better small files than the other one or something
like that?

      I am not aware of any guidelines for this. The difference in performance
is highly dependent on the workload and use-case. There is little
difference in the layers between Gluster/NFS and NFS-Ganesha, both are
userspace nfs-server implementations (neither has context switches for
the Linux-VFS like fuse mounts have).

If you need the best performance, you should problaby just try both
configurations, and run your intended workload against the servers.
Artificial/standard tests most often do not emulate a real workload.

HTH,
Niels

        On 5 Aug 2015 7:06 pm, "Niels de Vos" <ndevos@xxxxxxxxxx> wrote:

          On Wed, Aug 05, 2015 at 04:11:47PM +0100, Thibault Godouet wrote:

            Looking around I get the impression that file locking (NLM) may simply

          not

            be supported in glusterfs's built-in NFS server.

          This is actually supported. But note that you can not run a userspace
NLM implementation provided by a NFS-server (Gluster/NFS or NFS-Ganesha)
on a system that acts as an NFS-client. The Linux kernel NFS-client uses
the lockd kernel module, and there can be only one NLM implementation be
registered at rpcbind. Whichever service (nfs-client or nfs-server)
starts first, will be able to register itself, the 2nd one will (mostly
silently) fail.

            I get the impression that Ganesha is aimed at supporting NFS better, and
presumably supports locking well, so I should give it a try (If I
understand well the performance is also likely to be higher, which is a
nice bonus!)

          NFS-Ganesha offers more features than Gluster/NFS. The performance is
highly dependent on the workload, Gluster/NFS can be faster for many of
them.

Cheers,
Niels

            If someone could confirm this that would be useful to make sure I'm going
in the right direction.

Thanks,
Thibault.
On 4 Aug 2015 1:23 pm, "Thibault Godouet" <tibo92@xxxxxxxxxxx> wrote:

              Hi,

I have a cluster of 2 servers running 3.7.3 with replication, and

          standard

              NFS (no ganesha).  This in on CentOS 6.

I use CTDB with 2 virtual IPs (one for each server in a normal
situation) to share the volume over NFS and CIFS (samba).

fnctl() file locking doesn't seem to work when the volume is mounted

          over

              NFS.

This is apparent with a 'svn info' (svn 1.8 if it made any difference)

          in

              a local working copy:

$ svn info
svn: E200033: Another process is blocking the working copy database, or
the underlying filesystem does not support file locking; if the working
copy is on a network filesystem, make sure file locking has been

          enabled on

              the file server
svn: E200033: sqlite[S5]: database is locked, executing statement

          'PRAGMA

              synchronous=OFF;PRAGMA recursive_triggers=ON;PRAGMA

          foreign_keys=OFF;PRAGMA

              locking_mode = NORMAL;'

a strace shows:

$ svn info
svn: E200033: Another process is blocking the working copy database, or
the underlying filesystem does not support file locking; if the working
copy is on a network filesystem, make sure file locking has been

          enabled on

              the file server
svn: E200033: sqlite[S5]: database is locked, executing statement

          'PRAGMA

              synchronous=OFF;PRAGMA recursive_triggers=ON;PRAGMA

          foreign_keys=OFF;PRAGMA

              locking_mode = NORMAL;'

Everything seems to work fine on native Gluster (FUSE) mounts: the same
'svn info' works nicely.

I can't really use native mounts due to the performance hit (many small
files) and the fact I would need to install the gluster client

          software on

              every server.

Is fnctl() file locking supported in Gluster NFS mounts?  If so, any

          idea

              why it doesn't work for me?

Thanks,

Thibault.

            _______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

          _______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users