Re: GlusterFS 3.7.2 and ACL

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 


[2015-07-30 13:16:01.002296] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: ACL3 - SETACL for 10.1.1.32:742
[2015-07-30 13:16:01.002325] T [MSGID: 0] [acl3.c:672:acl3svc_setacl] 0-nfs-ACL: FH to Volume: acltest
[2015-07-30 13:16:01.004287] T [rpcsvc.c:1319:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x16185ddc, Program: ACL3, ProgVers: 3, Proc: 2) to rpc-transport (socket.nfs-server)
[2015-07-30 13:16:22.823894] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: ACL3 - GETACL for 10.1.1.32:742
[2015-07-30 13:16:22.823900] T [MSGID: 0] [acl3.c:532:acl3svc_getacl] 0-nfs-ACL: FH to Volume: acltest
[2015-07-30 13:16:22.824218] D [MSGID: 0] [client-rpc-fops.c:1156:client3_3_getxattr_cbk] 0-acltest-client-1: remote operation failed: No data available. Path: <gfid:12f02b4f-a181-47d4-9b5b-69e889483570> (12f02b4f-a181-47d4-9b5b-69e889483570). Key: system.posix_acl_default
[2015-07-30 13:16:22.825675] D [MSGID: 0] [client-rpc-fops.c:1156:client3_3_getxattr_cbk] 0-acltest-client-0: remote operation failed: No data available. Path: <gfid:12f02b4f-a181-47d4-9b5b-69e889483570> (12f02b4f-a181-47d4-9b5b-69e889483570). Key: system.posix_acl_default
[2015-07-30 13:16:22.825713] T [rpcsvc.c:1319:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x63815edc, Program: ACL3, ProgVers: 3, Proc: 1) to rpc-transport (socket.nfs-server)
[2015-07-30 13:16:22.828243] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: ACL3 - SETACL for 10.1.1.32:742
[2015-07-30 13:16:22.828266] T [MSGID: 0] [acl3.c:672:acl3svc_setacl] 0-nfs-ACL: FH to Volume: acltest
[2015-07-30 13:16:22.829931] T [rpcsvc.c:1319:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x75815edc, Program: ACL3, ProgVers: 3, Proc: 2) to rpc-transport (socket.nfs-server)

Enabled trace for few moments and tried to make any sense of it by searching for lines containing ‘acl’  according to this everything kind of works except lines which state that “remote operation failed” GlusterFS failed to replicate or commit acl changes? 



On 07/30/2015 06:22 PM, Jüri Palis wrote:
Hi,

Thanks Niels, your hints about those two options did the trick although
I had to enable both of them and I had to add nscd (sssd provides user
identities) to this mix as well.

Now back to the problem with ACL’s. Is your test setup something like
this: GlusterFS 3.7.2 replicated volume on Centos/RHEL 7 and client or
clients accessing GlusterFS volumes by NFS protocol, correct?

As Jiffin had suggested, did you try the same command on GlusterFS Native mount?

Log levels can be increased to TRACE/DEBUG mode using the command 'gluster vol set <volname> diagnostics.client-log-level [TRACE,DEBUG]'

Also please capture a packet trace on the server-side using the command - 'tcpdump -i any -s 0 -w /var/tmp/nfs-acl.pcap tcp and not port 22'

Verify the packets sent by Gluster-NFS process to the brick process to set the ACL.

Thanks,
Soumya

# gluster volume info acltest
Volume Name: acltest
Type: Replicate
Volume ID: 9e0de3f5-45ba-4612-a4f1-16bc5d1eb985
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: vfs-node-01:/data/gfs/acltest/brick0/brick
Brick2: vfs-node-02:/data/gfs/acltest/brick0/brick
Options Reconfigured:
server.manage-gids: on
nfs.server-aux-gids: on
performance.readdir-ahead: on
server.event-threads: 32
performance.cache-size: 2GB
storage.linux-aio: on
nfs.disable: off
performance.write-behind-window-size: 1GB
performance.nfs.io-cache: on
performance.nfs.write-behind-window-size: 250MB
performance.nfs.stat-prefetch: on
performance.nfs.read-ahead: on
performance.nfs.io-threads: on
cluster.readdir-optimize: on
network.remote-dio: on
auth.allow: 10.1.1.32,10.1.1.42
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on
nfs.rpc-auth-allow: 10.1.1.32,10.1.1.42
nfs.trusted-sync: on

Maybe there is a way to increase verbosity of nfs server which could
help me to trace this problem. I did not find any good hints for
increasing verbosity of nfs server in documentation.

Regards,
J.

On 30 Jul 2015, at 10:09, Jiffin Tony Thottan <jthottan@xxxxxxxxxx
<mailto:jthottan@xxxxxxxxxx>> wrote:



On 29/07/15 20:14, Niels de Vos wrote:
On Wed, Jul 29, 2015 at 05:22:31PM +0300, Jüri Palis wrote:
Hi,

Another issue with NFS and sec=sys mode. As we all know there is a
limit of 15 security ids involved when running NFS in sec=sys mode.
This limit makes effective and granular usage of ACL assigned through
groups almost unusable. One way to overcome this limit is to use
kerberised NFS but GlusterFS does not natively support this access
mode . Another option, at least  according to one email thread, states
that  GlusterFS has an option server.manage-gids which should mitigate
this limit and raise it to 90 something.  Is this the option, which
can be used for increasing sec=sys limit. Sadly documentation does not
have clear description about this option, what exactly this option
does and how it should be used.
server.manage-gids is an option to resolve the groups of a uid in the
brick process. You probably need to also use the nfs.server-aux-gids
option so that the NFS-server resolves the gids of the uid accessing the
NFS-server.

The nfs.server-aux-gids option is used to overcome the
AUTH_SYS/AUTH_UNIX limit of (I thought 32?) groups.

The server.manage-gids option is used to overcome the GlusterFS protocol
limit of ~93 groups.

If your users do not belong to 90+ groups, you would not need to set the
server.manage-gids option, and nfs.server-aux-gids might be sufficient.

HTH,
Niels

J.


On 29 Jul 2015, at 16:16, Jiffin Tony Thottan <jthottan@xxxxxxxxxx
<mailto:jthottan@xxxxxxxxxx>> wrote:


On 29/07/15 18:04, Jüri Palis wrote:
Hi,

setfacl for dir on local filesystem:

1. set acl setfacl -m g:x_meie_sec-test02:rx test
2. get acl

# getfacl test
user::rwx
group::r-x
group:x_meie_sec-test02:r-x
mask::r-x
other::r-x

setfacl for dir on GlusterFS volume which is NFS mounted to client
system

1. same command is used for setting ACE, no error is returned by
that command
2. get acl

#getfacl test
user::rwx
group::r-x
other::---


If I use ordinary file as a target on GlusterFS like this

setfacl -m g:x_meie_sec-test02:rw dummy

then ACE entry is set for file dummy stored on GlusterFS

# getfacl dummy
user::rw-
group::r--
group:x_meie_sec-test02:rw-
mask::rw-
other::—

So, as you can see setting ACLs for files works but does not work
for directories.

This all is happening on CentOS7, running GlusterFS 3.7.2
Hi Jyri,

It seems there are couple of issues ,

1.)  when u set a named group acl for file/directory, it clears the
permission of others too.
2.)  named group acl is not working properly for directories ,

I will try the same on my setup and share my findings.
--
Jiffin

In my setup (glusterfs 3.7.2 and RHEL 7.1 client) it worked properly

I followed the same steps mentioned by you.
#cd /mnt
# mkdir dir
# touch file
# getfacl file
# file: file
# owner: root
# group: root
user::rw-
group::r--
other::r--

# getfacl dir
# file: dir
# owner: root
# group: root
user::rwx
group::r-x
other::r-x

# setfacl -m g:gluster:rw file
# getfacl file
# file: file
# owner: root
# group: root
user::rw-
group::r--
group:gluster:rw-
mask::rw-
other::r--

setfacl -m g:gluster:r-x dir
getfacl dir
# file: dir
# owner: root
# group: root
user::rwx
group::r-x
group:gluster:r-x
mask::r-x
other::r-x


So can u share the following  information from the server.
1.) gluster vol info
2.) nfs.log (nfs-server log)
3.) brick logs

and also can u try the same on fuse mount(gluster native mount).

--
Jiffin

J.
On 29 Jul 2015, at 15:16, Jiffin Thottan <jthottan@xxxxxxxxxx
<mailto:jthottan@xxxxxxxxxx>> wrote:

----- Original Message -----
From: "Jüri Palis" <jyri.palis@xxxxxxxxx
<mailto:jyri.palis@xxxxxxxxx>>
To: gluster-users@xxxxxxxxxxx <mailto:gluster-users@xxxxxxxxxxx>
Sent: Wednesday, July 29, 2015 4:19:20 PM
Subject: GlusterFS 3.7.2 and ACL

Hi

Setup:
GFS 3.7.2, NFS is used for host access

Problem:
POSIX ACL work correctly when ACLs are applied to files but do
not work when ACLs are applied to directories on GFS volumes.

How can I debug this issue more deeply?

Can you please explain the issue with more details, i.e what
exactly not working properly , is it setting acl or any
functionality issue, in which client?
__
Jiffin

Regards,
Jyri
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx <mailto:Gluster-users@xxxxxxxxxxx>
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx <mailto:Gluster-users@xxxxxxxxxxx>
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx <mailto:Gluster-users@xxxxxxxxxxx>
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx <mailto:Gluster-users@xxxxxxxxxxx>
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx <mailto:Gluster-users@xxxxxxxxxxx>
http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx <mailto:Gluster-users@xxxxxxxxxxx>
http://www.gluster.org/mailman/listinfo/gluster-users



_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux