Hi Soumya,
I just noticed some of the notes at the bottom. In particular
- Till glusterfs-3.7, gluster-NFS (gNFS) gets enabled by
default. The only requirement is that kernel-NFS has to be
disabled for gluster-NFS to come up. Please disable kernel-NFS
server and restart glusterd to start gNFS. In case of any issues
with starting gNFS server, please look at
/var/log/glusterfs/nfs.log.
If we disable the kernel-NFS on our server and restart glusterd to
start gNFS will that affect the NFS file system also being served by
that server (i.e. the single server serves both a glusterFS area and
an NFS area)? Would we also have to disable the kernel-NFS for
NFS-ganesha?
My second question concerns NFS-ganesha (v 2.3.x) for CentOS 6.8 and
gluster 3.7.11. I think I see a couple of possibilities
- I see one possible rpm for version 2.3.3 in
https://mirror.chpc.utah.edu/pub/vault.centos.org/centos/6.8/storage/Source/gluster-3.8/
The other rpm's seem to be for gluster 3.8 packages, so I'm
wondering if there is a concern for conflict
- In one of the links you sent
(https://buildlogs.centos.org/centos/6/storage/x86_64/gluster-3.7/)
I see an rpm for glusterfs-ganesha-3.7.11 . Is this a specific
gluster package for compatibility with ganesha or a ganesha
package for gluster?
Does either possibility seem more likely to be what I need than the
other?
Pat
On 07/07/2017 01:31 PM, Soumya Koduri
wrote:
Hi,
On 07/07/2017 06:16 AM, Pat Haley wrote:
Hi All,
A follow-up question. I've been looking at various pages on
nfs-ganesha
& gluster. Is there a version of nfs-ganesha that is
recommended for
use with
glusterfs 3.7.11 built on Apr 27 2016 14:09:22
CentOS release 6.8 (Final)
For glusterfs 3.7, nfs-ganesha-2.3-* version can be used.
I see the packages built in centos7 storage sig [1] but not for
centos6. Request Niels to comment.
Thanks
Pat
On 07/05/2017 11:36 AM, Pat Haley wrote:
Hi Soumya,
(1) In
http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/
I've placed the following 2 log files
etc-glusterfs-glusterd.vol.log
gdata.log
The first has repeated messages about nfs disconnects. The
second had
the <fuse_mnt_direcotry>.log name (but not much
information).
Hmm yeah..weird ..there are not much logs in fuse mnt log file.
(2) About the gluster-NFS native
server: do you know where we can
find documentation on how to use/install it? We haven't had
success
in our searches.
Till glusterfs-3.7, gluster-NFS (gNFS) gets enabled by default.
The only requirement is that kernel-NFS has to be disabled for
gluster-NFS to come up. Please disable kernel-NFS server and
restart glusterd to start gNFS. In case of any issues with
starting gNFS server, please look at /var/log/glusterfs/nfs.log.
Thanks,
Soumya
[1]
https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-3.7/
[2]
https://buildlogs.centos.org/centos/6/storage/x86_64/gluster-3.7/
Thanks
Pat
On 07/04/2017 05:01 AM, Soumya Koduri wrote:
On 07/03/2017 09:01 PM, Pat Haley wrote:
Hi Soumya,
When I originally did the tests I ran tcpdump on the
client.
I have rerun the tests, doing tcpdump on the server
tcpdump -i any -nnSs 0 host 172.16.1.121 -w
/root/capture_nfsfail.pcap
The results are in the same place
http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/
capture_nfsfail.pcap has the results from the failed
touch experiment
capture_nfssucceed.pcap has the results from the
successful touch
experiment
The brick log files are there too.
Thanks for sharing. Looks like the error is not generated
@gluster-server side. The permission denied error was caused
by
either kNFS or by fuse-mnt process or probably by the
combination.
To check fuse-mnt logs, please look at
/var/log/glusterfs/<fuse_mnt_direcotry>.log
For eg.: if you have fuse mounted the gluster volume at
/mnt/fuse-mnt
and exported it via kNFS, the log location for that fuse_mnt
shall be
at /var/log/glusterfs/mnt-fuse-mnt.log
Also why not switch to either gluster-NFS native server or
NFS-Ganesha instead of using kNFS, as they are recommended
NFS
servers to use with gluster?
Thanks,
Soumya
I believe we are using kernel-NFS exporting a fuse mounted
gluster
volume. I am having Steve confirm this. I tried to find
the fuse-mnt
logs but failed. Where should I look for them?
Thanks
Pat
On 07/03/2017 07:58 AM, Soumya Koduri wrote:
On 06/30/2017 07:56 PM, Pat Haley wrote:
Hi,
I was wondering if there were any additional test we
could perform to
help debug the group write-permissions issue?
Sorry for the delay. Please find response inline --
Thanks
Pat
On 06/27/2017 12:29 PM, Pat Haley wrote:
Hi Soumya,
One example, we have a common working directory
dri_fleat in the
gluster volume
drwxrwsr-x 22 root dri_fleat 4.0K May 1 15:14
dri_fleat
my user (phaley) does not own that directory but is
a member of the
group dri_fleat and should have write permissions.
When I go to the
nfs-mounted version and try to use the touch command
I get the
following
ibfdr-compute-0-4(dri_fleat)% touch dum
touch: cannot touch `dum': Permission denied
One of the sub-directories under dri_fleat is "test"
which phaley
owns
drwxrwsr-x 2 phaley dri_fleat 4.0K May 1 15:16
test
Under this directory (mounted via nfs) user phaley
can write
ibfdr-compute-0-4(test)% touch dum
ibfdr-compute-0-4(test)%
I have put the packet captures in
http://mseas.mit.edu/download/phaley/GlusterUsers/TestNFSmount/
capture_nfsfail.pcap has the results from the
failed touch
experiment
capture_nfssucceed.pcap has the results from the
successful touch
experiment
The command I used for these was
tcpdump -i ib0 -nnSs 0 host 172.16.1.119 -w
/root/capture_nfstest.pcap
I hope these pkts were captured on the node where NFS
server is
running. Could you please use '-i any' as I do not see
glusterfs
traffic in the tcpdump.
Also looks like NFS v4 is used between client & nfs
server. Are you
using kernel-NFS here (i.e, kernel-NFS exporting fuse
mounted gluster
volume)?
If that is the case please capture fuse-mnt logs as
well. This error
may well be coming from kernel-NFS itself before the
request is sent
to fuse-mnt process.
FWIW, we have below option -
Option: server.manage-gids
Default Value: off
Description: Resolve groups on the server-side.
I haven't looked into what this option exactly does. But
it may worth
testing with this option on.
Thanks,
Soumya
The brick log files are also in the above link. If
I read them
correctly they both funny times. Specifically I see
entries from
around 2017-06-27 14:02:37.404865 even though the
system time was
2017-06-27 12:00:00.
One final item, another reply to my post had a link
for possible
problems that could arise from users belonging to
too many group. We
have seen the above problem even with a user
belonging to only 4
groups.
Let me know what additional information I can
provide.
Thanks
Pat
On 06/27/2017 02:45 AM, Soumya Koduri wrote:
On 06/27/2017 10:17 AM, Pranith Kumar Karampuri
wrote:
The only problem with
using gluster mounted via NFS is that it
does not
respect the group write permissions which we
need.
We have an exercise coming up in the a couple of
weeks. It seems
to me
that in order to improve our write times before
then, it would be
good
to solve the group write permissions for gluster
mounted via
NFS now.
We can then revisit gluster mounted via FUSE
afterwards.
What information would you need to help us force
gluster
mounted via
NFS
to respect the group write permissions?
Is this owning group or one of the auxiliary
groups whose write
permissions are not considered? AFAIK, there are
no special
permission checks done by gNFS server when
compared to gluster
native
client.
Could you please provide simple steps to reproduce
the issue and
collect pkt trace and nfs/brick logs as well.
Thanks,
Soumya
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley@xxxxxxx
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
|
