Re: Slow write times to gluster disk

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 





On 07/14/2017 06:40 AM, Pat Haley wrote:

Hi Soumya,

I just noticed some of the notes at the bottom.  In particular

  * Till glusterfs-3.7, gluster-NFS (gNFS) gets enabled by default. The
    only requirement is that kernel-NFS has to be disabled for
    gluster-NFS to come up. Please disable kernel-NFS server and restart
    glusterd to start gNFS. In case of any issues with starting gNFS
    server, please look at /var/log/glusterfs/nfs.log.

If we disable the kernel-NFS on our server and restart glusterd to start
gNFS will that affect the NFS file system also being served by that
server (i.e. the single server serves both a glusterFS area and an NFS
area)?   clients
Thats right. When you restart glusterd, it tries to spawn (provided nfs.disable option is set to off for any volume) a new glusterfs client process which acts like NFS-server as well.

Would we also have to disable the kernel-NFS for NFS-ganesha?

yes.


My second question concerns NFS-ganesha (v 2.3.x) for CentOS 6.8 and
gluster 3.7.11.  I think I see a couple of possibilities

 1. I see one possible rpm for version 2.3.3 in
    https://mirror.chpc.utah.edu/pub/vault.centos.org/centos/6.8/storage/Source/gluster-3.8/
    The other rpm's seem to be for gluster 3.8 packages, so I'm
    wondering if there is a concern for conflict

AFAIK, nfs-ganesha-2.3.3 should work with both 3.8 & 3.7 gluster.

 2. In one of the links you sent
    (https://buildlogs.centos.org/centos/6/storage/x86_64/gluster-3.7/)
    I see an rpm for glusterfs-ganesha-3.7.11 .  Is this a specific
    gluster package for compatibility with ganesha or a ganesha package
    for gluster?
This is to be compatible with gluster-3.7* package.


Does either possibility seem more likely to be what I need than the other?

The current stable/maintained/tested combination is nfs-ganesha2.4/2.5 + glusterfs-3.8/3.10. But however incase you cannot upgrade, you can still use nfs-ganesha-2.3* with glusterfs-3.8/3.7

Hope it is clear.

Thanks,
Soumya


Pat


On 07/07/2017 01:31 PM, Soumya Koduri wrote:
Hi,

On 07/07/2017 06:16 AM, Pat Haley wrote:

Hi All,

A follow-up question.  I've been looking at various pages on nfs-ganesha
& gluster.  Is there a version of nfs-ganesha that is recommended for
use with

glusterfs 3.7.11 built on Apr 27 2016 14:09:22
CentOS release 6.8 (Final)

For glusterfs 3.7, nfs-ganesha-2.3-* version can be used.

I see the packages built in centos7 storage sig [1] but not for
centos6. Request Niels to comment.



Thanks

Pat


On 07/05/2017 11:36 AM, Pat Haley wrote:

Hi Soumya,

(1) In http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/
I've placed the following 2 log files

etc-glusterfs-glusterd.vol.log
gdata.log

The first has repeated messages about nfs disconnects.  The second had
the <fuse_mnt_direcotry>.log name (but not much information).


Hmm yeah..weird ..there are not much logs in fuse mnt log file.

(2) About the gluster-NFS native server:  do you know where we can
find documentation on how to use/install it?  We haven't had success
in our searches.


Till glusterfs-3.7, gluster-NFS (gNFS) gets enabled by default. The
only requirement is that kernel-NFS has to be disabled for gluster-NFS
to come up. Please disable kernel-NFS server and restart glusterd to
start gNFS. In case of any issues with starting gNFS server, please
look at /var/log/glusterfs/nfs.log.

Thanks,
Soumya


[1] https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-3.7/
[2] https://buildlogs.centos.org/centos/6/storage/x86_64/gluster-3.7/

Thanks

Pat


On 07/04/2017 05:01 AM, Soumya Koduri wrote:


On 07/03/2017 09:01 PM, Pat Haley wrote:

Hi Soumya,

When I originally did the tests I ran tcpdump on the client.

I have rerun the tests, doing tcpdump on the server

tcpdump -i any -nnSs 0 host 172.16.1.121 -w
/root/capture_nfsfail.pcap

The results are in the same place

http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/

capture_nfsfail.pcap   has the results from the failed touch
experiment
capture_nfssucceed.pcap  has the results from the successful touch
experiment

The brick log files are there too.

Thanks for sharing. Looks like the error is not generated
@gluster-server side. The permission denied error was caused by
either kNFS or by fuse-mnt process or probably by the combination.

To check fuse-mnt logs, please look at
/var/log/glusterfs/<fuse_mnt_direcotry>.log

For eg.: if you have fuse mounted the gluster volume at /mnt/fuse-mnt
and exported it via kNFS, the log location for that fuse_mnt shall be
at /var/log/glusterfs/mnt-fuse-mnt.log


Also why not switch to either gluster-NFS native server or
NFS-Ganesha instead of using kNFS, as they are recommended NFS
servers to use with gluster?

Thanks,
Soumya


I believe we are using kernel-NFS exporting a fuse mounted gluster
volume.  I am having Steve confirm this.  I tried to find the
fuse-mnt
logs but failed.  Where should I look for them?

Thanks

Pat



On 07/03/2017 07:58 AM, Soumya Koduri wrote:


On 06/30/2017 07:56 PM, Pat Haley wrote:

Hi,

I was wondering if there were any additional test we could
perform to
help debug the group write-permissions issue?

Sorry for the delay. Please find response inline --


Thanks

Pat


On 06/27/2017 12:29 PM, Pat Haley wrote:

Hi Soumya,

One example, we have a common working directory dri_fleat in the
gluster volume

drwxrwsr-x 22 root dri_fleat     4.0K May  1 15:14 dri_fleat

my user (phaley) does not own that directory but is a member of
the
group  dri_fleat and should have write permissions. When I go
to the
nfs-mounted version and try to use the touch command I get the
following

ibfdr-compute-0-4(dri_fleat)% touch dum
touch: cannot touch `dum': Permission denied

One of the sub-directories under dri_fleat is "test" which phaley
owns

drwxrwsr-x  2 phaley   dri_fleat 4.0K May  1 15:16 test

Under this directory (mounted via nfs) user phaley can write

ibfdr-compute-0-4(test)% touch dum
ibfdr-compute-0-4(test)%

I have put the packet captures in

http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/

capture_nfsfail.pcap   has the results from the failed touch
experiment
capture_nfssucceed.pcap  has the results from the successful touch
experiment

The command I used for these was

tcpdump -i ib0 -nnSs 0 host 172.16.1.119 -w
/root/capture_nfstest.pcap

I hope these pkts were captured on the node where NFS server is
running. Could you please use '-i any' as I do not see glusterfs
traffic in the tcpdump.

Also looks like NFS v4 is used between client & nfs server. Are you
using kernel-NFS here (i.e, kernel-NFS exporting fuse mounted
gluster
volume)?
If that is the case please capture fuse-mnt logs as well. This error
may well be coming from kernel-NFS itself before the request is sent
to fuse-mnt process.

FWIW, we have below option -

Option: server.manage-gids
Default Value: off
Description: Resolve groups on the server-side.

I haven't looked into what this option exactly does. But it may
worth
testing with this option on.

Thanks,
Soumya



The brick log files are also in the above link.  If I read them
correctly they both funny times.  Specifically I see entries from
around 2017-06-27 14:02:37.404865  even though the system time was
2017-06-27 12:00:00.

One final item, another reply to my post had a link for possible
problems that could arise from users belonging to too many
group. We
have seen the above problem even with a user belonging to only 4
groups.

Let me know what additional information I can provide.


Thanks

Pat


On 06/27/2017 02:45 AM, Soumya Koduri wrote:


On 06/27/2017 10:17 AM, Pranith Kumar Karampuri wrote:
The only problem with using gluster mounted via NFS is that it
does not
respect the group write permissions which we need.

We have an exercise coming up in the a couple of weeks. It seems
to me
that in order to improve our write times before then, it
would be
good
to solve the group write permissions for gluster mounted via
NFS now.
We can then revisit gluster mounted via FUSE afterwards.

What information would you need to help us force gluster
mounted via
NFS
to respect the group write permissions?

Is this owning group or one of the auxiliary groups whose write
permissions are not considered? AFAIK, there are no special
permission checks done by gNFS server when compared to gluster
native
client.

Could you please provide simple steps to reproduce the issue and
collect pkt trace and nfs/brick logs as well.

Thanks,
Soumya






--

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley@xxxxxxx
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users



[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux