On 07/14/2017 06:40 AM, Pat Haley wrote:
Hi Soumya,
I just noticed some of the notes at the bottom. In particular
* Till glusterfs-3.7, gluster-NFS (gNFS) gets enabled by default. The
only requirement is that kernel-NFS has to be disabled for
gluster-NFS to come up. Please disable kernel-NFS server and restart
glusterd to start gNFS. In case of any issues with starting gNFS
server, please look at /var/log/glusterfs/nfs.log.
If we disable the kernel-NFS on our server and restart glusterd to start
gNFS will that affect the NFS file system also being served by that
server (i.e. the single server serves both a glusterFS area and an NFS
area)? clients
Thats right. When you restart glusterd, it tries to spawn (provided
nfs.disable option is set to off for any volume) a new glusterfs client
process which acts like NFS-server as well.
Would we also have to disable the kernel-NFS for NFS-ganesha?
yes.
My second question concerns NFS-ganesha (v 2.3.x) for CentOS 6.8 and
gluster 3.7.11. I think I see a couple of possibilities
1. I see one possible rpm for version 2.3.3 in
https://mirror.chpc.utah.edu/pub/vault.centos.org/centos/6.8/storage/Source/gluster-3.8/
The other rpm's seem to be for gluster 3.8 packages, so I'm
wondering if there is a concern for conflict
AFAIK, nfs-ganesha-2.3.3 should work with both 3.8 & 3.7 gluster.
2. In one of the links you sent
(https://buildlogs.centos.org/centos/6/storage/x86_64/gluster-3.7/)
I see an rpm for glusterfs-ganesha-3.7.11 . Is this a specific
gluster package for compatibility with ganesha or a ganesha package
for gluster?
This is to be compatible with gluster-3.7* package.
Does either possibility seem more likely to be what I need than the other?
The current stable/maintained/tested combination is nfs-ganesha2.4/2.5 +
glusterfs-3.8/3.10. But however incase you cannot upgrade, you can still
use nfs-ganesha-2.3* with glusterfs-3.8/3.7
Hope it is clear.
Thanks,
Soumya
Pat
On 07/07/2017 01:31 PM, Soumya Koduri wrote:
Hi,
On 07/07/2017 06:16 AM, Pat Haley wrote:
Hi All,
A follow-up question. I've been looking at various pages on nfs-ganesha
& gluster. Is there a version of nfs-ganesha that is recommended for
use with
glusterfs 3.7.11 built on Apr 27 2016 14:09:22
CentOS release 6.8 (Final)
For glusterfs 3.7, nfs-ganesha-2.3-* version can be used.
I see the packages built in centos7 storage sig [1] but not for
centos6. Request Niels to comment.
Thanks
Pat
On 07/05/2017 11:36 AM, Pat Haley wrote:
Hi Soumya,
(1) In http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/
I've placed the following 2 log files
etc-glusterfs-glusterd.vol.log
gdata.log
The first has repeated messages about nfs disconnects. The second had
the <fuse_mnt_direcotry>.log name (but not much information).
Hmm yeah..weird ..there are not much logs in fuse mnt log file.
(2) About the gluster-NFS native server: do you know where we can
find documentation on how to use/install it? We haven't had success
in our searches.
Till glusterfs-3.7, gluster-NFS (gNFS) gets enabled by default. The
only requirement is that kernel-NFS has to be disabled for gluster-NFS
to come up. Please disable kernel-NFS server and restart glusterd to
start gNFS. In case of any issues with starting gNFS server, please
look at /var/log/glusterfs/nfs.log.
Thanks,
Soumya
[1] https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-3.7/
[2] https://buildlogs.centos.org/centos/6/storage/x86_64/gluster-3.7/
Thanks
Pat
On 07/04/2017 05:01 AM, Soumya Koduri wrote:
On 07/03/2017 09:01 PM, Pat Haley wrote:
Hi Soumya,
When I originally did the tests I ran tcpdump on the client.
I have rerun the tests, doing tcpdump on the server
tcpdump -i any -nnSs 0 host 172.16.1.121 -w
/root/capture_nfsfail.pcap
The results are in the same place
http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/
capture_nfsfail.pcap has the results from the failed touch
experiment
capture_nfssucceed.pcap has the results from the successful touch
experiment
The brick log files are there too.
Thanks for sharing. Looks like the error is not generated
@gluster-server side. The permission denied error was caused by
either kNFS or by fuse-mnt process or probably by the combination.
To check fuse-mnt logs, please look at
/var/log/glusterfs/<fuse_mnt_direcotry>.log
For eg.: if you have fuse mounted the gluster volume at /mnt/fuse-mnt
and exported it via kNFS, the log location for that fuse_mnt shall be
at /var/log/glusterfs/mnt-fuse-mnt.log
Also why not switch to either gluster-NFS native server or
NFS-Ganesha instead of using kNFS, as they are recommended NFS
servers to use with gluster?
Thanks,
Soumya
I believe we are using kernel-NFS exporting a fuse mounted gluster
volume. I am having Steve confirm this. I tried to find the
fuse-mnt
logs but failed. Where should I look for them?
Thanks
Pat
On 07/03/2017 07:58 AM, Soumya Koduri wrote:
On 06/30/2017 07:56 PM, Pat Haley wrote:
Hi,
I was wondering if there were any additional test we could
perform to
help debug the group write-permissions issue?
Sorry for the delay. Please find response inline --
Thanks
Pat
On 06/27/2017 12:29 PM, Pat Haley wrote:
Hi Soumya,
One example, we have a common working directory dri_fleat in the
gluster volume
drwxrwsr-x 22 root dri_fleat 4.0K May 1 15:14 dri_fleat
my user (phaley) does not own that directory but is a member of
the
group dri_fleat and should have write permissions. When I go
to the
nfs-mounted version and try to use the touch command I get the
following
ibfdr-compute-0-4(dri_fleat)% touch dum
touch: cannot touch `dum': Permission denied
One of the sub-directories under dri_fleat is "test" which phaley
owns
drwxrwsr-x 2 phaley dri_fleat 4.0K May 1 15:16 test
Under this directory (mounted via nfs) user phaley can write
ibfdr-compute-0-4(test)% touch dum
ibfdr-compute-0-4(test)%
I have put the packet captures in
http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/
capture_nfsfail.pcap has the results from the failed touch
experiment
capture_nfssucceed.pcap has the results from the successful touch
experiment
The command I used for these was
tcpdump -i ib0 -nnSs 0 host 172.16.1.119 -w
/root/capture_nfstest.pcap
I hope these pkts were captured on the node where NFS server is
running. Could you please use '-i any' as I do not see glusterfs
traffic in the tcpdump.
Also looks like NFS v4 is used between client & nfs server. Are you
using kernel-NFS here (i.e, kernel-NFS exporting fuse mounted
gluster
volume)?
If that is the case please capture fuse-mnt logs as well. This error
may well be coming from kernel-NFS itself before the request is sent
to fuse-mnt process.
FWIW, we have below option -
Option: server.manage-gids
Default Value: off
Description: Resolve groups on the server-side.
I haven't looked into what this option exactly does. But it may
worth
testing with this option on.
Thanks,
Soumya
The brick log files are also in the above link. If I read them
correctly they both funny times. Specifically I see entries from
around 2017-06-27 14:02:37.404865 even though the system time was
2017-06-27 12:00:00.
One final item, another reply to my post had a link for possible
problems that could arise from users belonging to too many
group. We
have seen the above problem even with a user belonging to only 4
groups.
Let me know what additional information I can provide.
Thanks
Pat
On 06/27/2017 02:45 AM, Soumya Koduri wrote:
On 06/27/2017 10:17 AM, Pranith Kumar Karampuri wrote:
The only problem with using gluster mounted via NFS is that it
does not
respect the group write permissions which we need.
We have an exercise coming up in the a couple of weeks. It seems
to me
that in order to improve our write times before then, it
would be
good
to solve the group write permissions for gluster mounted via
NFS now.
We can then revisit gluster mounted via FUSE afterwards.
What information would you need to help us force gluster
mounted via
NFS
to respect the group write permissions?
Is this owning group or one of the auxiliary groups whose write
permissions are not considered? AFAIK, there are no special
permission checks done by gNFS server when compared to gluster
native
client.
Could you please provide simple steps to reproduce the issue and
collect pkt trace and nfs/brick logs as well.
Thanks,
Soumya
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley@xxxxxxx
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
