Hi, Hmm, I have almost the same setup except both client and server are running on RHEL 7.1. I’ll do packet capture and if I have spare time then I’ll test RHEL 6.x as a client, maybe I can replicate you result and the problem is somewhere in RHEL7’s NFS implementation. Regards, Jyri On 31 Jul 2015, at 09:35, Soumya Koduri <skoduri@xxxxxxxxxx> wrote: > I have tested it using the gluster-NFS server with GlusterFS version 3.7.* running on a RHEL7 machine and RHEL 6.7 as NFS client. ACLs with named groups got properly set on the directory. > > Could you please provide us the packet trace (better taken on the server side so that we can check Gluster operations too) while doing setfacl and getfacl ? > > Thanks, > Soumya > > On 07/30/2015 07:38 PM, Jüri Palis wrote: >> Hi, >> >> Mounted GlusterFS volume with native mount and ACL’s are working as >> expected, mounted same volume with nfs protocol and the result is >> exactly the same as I described below. ACL set to files work and ACL set >> to directory do not work as expected. Ohh, I’m out of ideas :( >> >> J. >> On 30 Jul 2015, at 16:38, Jüri Palis <jyri.palis@xxxxxxxxx >> <mailto:jyri.palis@xxxxxxxxx>> wrote: >> >>> >>> [2015-07-30 13:16:01.002296] T [rpcsvc.c:316:rpcsvc_program_actor] >>> 0-rpc-service: Actor found: ACL3 - SETACL for 10.1.1.32:742 >>> [2015-07-30 13:16:01.002325] T [MSGID: 0] [acl3.c:672:acl3svc_setacl] >>> 0-nfs-ACL: FH to Volume: acltest >>> [2015-07-30 13:16:01.004287] T [rpcsvc.c:1319:rpcsvc_submit_generic] >>> 0-rpc-service: submitted reply for rpc-message (XID: 0x16185ddc, >>> Program: ACL3, ProgVers: 3, Proc: 2) to rpc-transport (socket.nfs-server) >>> [2015-07-30 13:16:22.823894] T [rpcsvc.c:316:rpcsvc_program_actor] >>> 0-rpc-service: Actor found: ACL3 - GETACL for 10.1.1.32:742 >>> [2015-07-30 13:16:22.823900] T [MSGID: 0] [acl3.c:532:acl3svc_getacl] >>> 0-nfs-ACL: FH to Volume: acltest >>> [2015-07-30 13:16:22.824218] D [MSGID: 0] >>> [client-rpc-fops.c:1156:client3_3_getxattr_cbk] 0-acltest-client-1: >>> remote operation failed: No data available. Path: >>> <gfid:12f02b4f-a181-47d4-9b5b-69e889483570> >>> (12f02b4f-a181-47d4-9b5b-69e889483570). Key: system.posix_acl_default >>> [2015-07-30 13:16:22.825675] D [MSGID: 0] >>> [client-rpc-fops.c:1156:client3_3_getxattr_cbk] 0-acltest-client-0: >>> remote operation failed: No data available. Path: >>> <gfid:12f02b4f-a181-47d4-9b5b-69e889483570> >>> (12f02b4f-a181-47d4-9b5b-69e889483570). Key: system.posix_acl_default >>> [2015-07-30 13:16:22.825713] T [rpcsvc.c:1319:rpcsvc_submit_generic] >>> 0-rpc-service: submitted reply for rpc-message (XID: 0x63815edc, >>> Program: ACL3, ProgVers: 3, Proc: 1) to rpc-transport (socket.nfs-server) >>> [2015-07-30 13:16:22.828243] T [rpcsvc.c:316:rpcsvc_program_actor] >>> 0-rpc-service: Actor found: ACL3 - SETACL for 10.1.1.32:742 >>> [2015-07-30 13:16:22.828266] T [MSGID: 0] [acl3.c:672:acl3svc_setacl] >>> 0-nfs-ACL: FH to Volume: acltest >>> [2015-07-30 13:16:22.829931] T [rpcsvc.c:1319:rpcsvc_submit_generic] >>> 0-rpc-service: submitted reply for rpc-message (XID: 0x75815edc, >>> Program: ACL3, ProgVers: 3, Proc: 2) to rpc-transport (socket.nfs-server) >>> >>> Enabled trace for few moments and tried to make any sense of it by >>> searching for lines containing ‘acl’ according to this everything >>> kind of works except lines which state that “remote operation failed” >>> GlusterFS failed to replicate or commit acl changes? >>> >>>> >>>> >>>> On 07/30/2015 06:22 PM, Jüri Palis wrote: >>>>> Hi, >>>>> >>>>> Thanks Niels, your hints about those two options did the trick although >>>>> I had to enable both of them and I had to add nscd (sssd provides user >>>>> identities) to this mix as well. >>>>> >>>>> Now back to the problem with ACL’s. Is your test setup something like >>>>> this: GlusterFS 3.7.2 replicated volume on Centos/RHEL 7 and client or >>>>> clients accessing GlusterFS volumes by NFS protocol, correct? >>>>> >>>> As Jiffin had suggested, did you try the same command on GlusterFS >>>> Native mount? >>>> >>>> Log levels can be increased to TRACE/DEBUG mode using the command >>>> 'gluster vol set <volname> diagnostics.client-log-level [TRACE,DEBUG]' >>>> >>>> Also please capture a packet trace on the server-side using the >>>> command - 'tcpdump -i any -s 0 -w /var/tmp/nfs-acl.pcap tcp and not >>>> port 22' >>>> >>>> Verify the packets sent by Gluster-NFS process to the brick process >>>> to set the ACL. >>>> >>>> Thanks, >>>> Soumya >>>> >>>>> # gluster volume info acltest >>>>> Volume Name: acltest >>>>> Type: Replicate >>>>> Volume ID: 9e0de3f5-45ba-4612-a4f1-16bc5d1eb985 >>>>> Status: Started >>>>> Number of Bricks: 1 x 2 = 2 >>>>> Transport-type: tcp >>>>> Bricks: >>>>> Brick1: vfs-node-01:/data/gfs/acltest/brick0/brick >>>>> Brick2: vfs-node-02:/data/gfs/acltest/brick0/brick >>>>> Options Reconfigured: >>>>> server.manage-gids: on >>>>> nfs.server-aux-gids: on >>>>> performance.readdir-ahead: on >>>>> server.event-threads: 32 >>>>> performance.cache-size: 2GB >>>>> storage.linux-aio: on >>>>> nfs.disable: off >>>>> performance.write-behind-window-size: 1GB >>>>> performance.nfs.io-cache: on >>>>> performance.nfs.write-behind-window-size: 250MB >>>>> performance.nfs.stat-prefetch: on >>>>> performance.nfs.read-ahead: on >>>>> performance.nfs.io-threads: on >>>>> cluster.readdir-optimize: on >>>>> network.remote-dio: on >>>>> auth.allow: 10.1.1.32,10.1.1.42 >>>>> diagnostics.latency-measurement: on >>>>> diagnostics.count-fop-hits: on >>>>> nfs.rpc-auth-allow: 10.1.1.32,10.1.1.42 >>>>> nfs.trusted-sync: on >>>>> >>>>> Maybe there is a way to increase verbosity of nfs server which could >>>>> help me to trace this problem. I did not find any good hints for >>>>> increasing verbosity of nfs server in documentation. >>>>> >>>>> Regards, >>>>> J. >>>>> >>>>> On 30 Jul 2015, at 10:09, Jiffin Tony Thottan <jthottan@xxxxxxxxxx >>>>> <mailto:jthottan@xxxxxxxxxx> >>>>> <mailto:jthottan@xxxxxxxxxx>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On 29/07/15 20:14, Niels de Vos wrote: >>>>>>> On Wed, Jul 29, 2015 at 05:22:31PM +0300, Jüri Palis wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> Another issue with NFS and sec=sys mode. As we all know there is a >>>>>>>> limit of 15 security ids involved when running NFS in sec=sys mode. >>>>>>>> This limit makes effective and granular usage of ACL assigned through >>>>>>>> groups almost unusable. One way to overcome this limit is to use >>>>>>>> kerberised NFS but GlusterFS does not natively support this access >>>>>>>> mode . Another option, at least according to one email thread, >>>>>>>> states >>>>>>>> that GlusterFS has an option server.manage-gids which should >>>>>>>> mitigate >>>>>>>> this limit and raise it to 90 something. Is this the option, which >>>>>>>> can be used for increasing sec=sys limit. Sadly documentation >>>>>>>> does not >>>>>>>> have clear description about this option, what exactly this option >>>>>>>> does and how it should be used. >>>>>>> server.manage-gids is an option to resolve the groups of a uid in the >>>>>>> brick process. You probably need to also use the nfs.server-aux-gids >>>>>>> option so that the NFS-server resolves the gids of the uid >>>>>>> accessing the >>>>>>> NFS-server. >>>>>>> >>>>>>> The nfs.server-aux-gids option is used to overcome the >>>>>>> AUTH_SYS/AUTH_UNIX limit of (I thought 32?) groups. >>>>>>> >>>>>>> The server.manage-gids option is used to overcome the GlusterFS >>>>>>> protocol >>>>>>> limit of ~93 groups. >>>>>>> >>>>>>> If your users do not belong to 90+ groups, you would not need to >>>>>>> set the >>>>>>> server.manage-gids option, and nfs.server-aux-gids might be >>>>>>> sufficient. >>>>>>> >>>>>>> HTH, >>>>>>> Niels >>>>>>> >>>>>>>> J. >>>>>>>> >>>>>>>> >>>>>>>> On 29 Jul 2015, at 16:16, Jiffin Tony Thottan >>>>>>>> <jthottan@xxxxxxxxxx <mailto:jthottan@xxxxxxxxxx> >>>>>>>> <mailto:jthottan@xxxxxxxxxx>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> On 29/07/15 18:04, Jüri Palis wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> setfacl for dir on local filesystem: >>>>>>>>>> >>>>>>>>>> 1. set acl setfacl -m g:x_meie_sec-test02:rx test >>>>>>>>>> 2. get acl >>>>>>>>>> >>>>>>>>>> # getfacl test >>>>>>>>>> user::rwx >>>>>>>>>> group::r-x >>>>>>>>>> group:x_meie_sec-test02:r-x >>>>>>>>>> mask::r-x >>>>>>>>>> other::r-x >>>>>>>>>> >>>>>>>>>> setfacl for dir on GlusterFS volume which is NFS mounted to client >>>>>>>>>> system >>>>>>>>>> >>>>>>>>>> 1. same command is used for setting ACE, no error is returned by >>>>>>>>>> that command >>>>>>>>>> 2. get acl >>>>>>>>>> >>>>>>>>>> #getfacl test >>>>>>>>>> user::rwx >>>>>>>>>> group::r-x >>>>>>>>>> other::--- >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> If I use ordinary file as a target on GlusterFS like this >>>>>>>>>> >>>>>>>>>> setfacl -m g:x_meie_sec-test02:rw dummy >>>>>>>>>> >>>>>>>>>> then ACE entry is set for file dummy stored on GlusterFS >>>>>>>>>> >>>>>>>>>> # getfacl dummy >>>>>>>>>> user::rw- >>>>>>>>>> group::r-- >>>>>>>>>> group:x_meie_sec-test02:rw- >>>>>>>>>> mask::rw- >>>>>>>>>> other::— >>>>>>>>>> >>>>>>>>>> So, as you can see setting ACLs for files works but does not work >>>>>>>>>> for directories. >>>>>>>>>> >>>>>>>>>> This all is happening on CentOS7, running GlusterFS 3.7.2 >>>>>>>>> Hi Jyri, >>>>>>>>> >>>>>>>>> It seems there are couple of issues , >>>>>>>>> >>>>>>>>> 1.) when u set a named group acl for file/directory, it clears the >>>>>>>>> permission of others too. >>>>>>>>> 2.) named group acl is not working properly for directories , >>>>>>>>> >>>>>>>>> I will try the same on my setup and share my findings. >>>>>>>>> -- >>>>>>>>> Jiffin >>>>>> >>>>>> In my setup (glusterfs 3.7.2 and RHEL 7.1 client) it worked properly >>>>>> >>>>>> I followed the same steps mentioned by you. >>>>>> #cd /mnt >>>>>> # mkdir dir >>>>>> # touch file >>>>>> # getfacl file >>>>>> # file: file >>>>>> # owner: root >>>>>> # group: root >>>>>> user::rw- >>>>>> group::r-- >>>>>> other::r-- >>>>>> >>>>>> # getfacl dir >>>>>> # file: dir >>>>>> # owner: root >>>>>> # group: root >>>>>> user::rwx >>>>>> group::r-x >>>>>> other::r-x >>>>>> >>>>>> # setfacl -m g:gluster:rw file >>>>>> # getfacl file >>>>>> # file: file >>>>>> # owner: root >>>>>> # group: root >>>>>> user::rw- >>>>>> group::r-- >>>>>> group:gluster:rw- >>>>>> mask::rw- >>>>>> other::r-- >>>>>> >>>>>> setfacl -m g:gluster:r-x dir >>>>>> getfacl dir >>>>>> # file: dir >>>>>> # owner: root >>>>>> # group: root >>>>>> user::rwx >>>>>> group::r-x >>>>>> group:gluster:r-x >>>>>> mask::r-x >>>>>> other::r-x >>>>>> >>>>>> >>>>>> So can u share the following information from the server. >>>>>> 1.) gluster vol info >>>>>> 2.) nfs.log (nfs-server log) >>>>>> 3.) brick logs >>>>>> >>>>>> and also can u try the same on fuse mount(gluster native mount). >>>>>> >>>>>> -- >>>>>> Jiffin >>>>>> >>>>>>>>>> J. >>>>>>>>>> On 29 Jul 2015, at 15:16, Jiffin Thottan <jthottan@xxxxxxxxxx >>>>>>>>>> <mailto:jthottan@xxxxxxxxxx> >>>>>>>>>> <mailto:jthottan@xxxxxxxxxx>> wrote: >>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Jüri Palis" <jyri.palis@xxxxxxxxx >>>>>>>>>>> <mailto:jyri.palis@xxxxxxxxx> >>>>>>>>>>> <mailto:jyri.palis@xxxxxxxxx>> >>>>>>>>>>> To:gluster-users@xxxxxxxxxxx >>>>>>>>>>> <mailto:gluster-users@xxxxxxxxxxx><mailto:gluster-users@xxxxxxxxxxx> >>>>>>>>>>> Sent: Wednesday, July 29, 2015 4:19:20 PM >>>>>>>>>>> Subject: GlusterFS 3.7.2 and ACL >>>>>>>>>>> >>>>>>>>>>> Hi >>>>>>>>>>> >>>>>>>>>>> Setup: >>>>>>>>>>> GFS 3.7.2, NFS is used for host access >>>>>>>>>>> >>>>>>>>>>> Problem: >>>>>>>>>>> POSIX ACL work correctly when ACLs are applied to files but do >>>>>>>>>>> not work when ACLs are applied to directories on GFS volumes. >>>>>>>>>>> >>>>>>>>>>> How can I debug this issue more deeply? >>>>>>>>>>> >>>>>>>>>>> Can you please explain the issue with more details, i.e what >>>>>>>>>>> exactly not working properly , is it setting acl or any >>>>>>>>>>> functionality issue, in which client? >>>>>>>>>>> __ >>>>>>>>>>> Jiffin >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Jyri >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>> Gluster-users@xxxxxxxxxxx >>>>>>>>>>> <mailto:Gluster-users@xxxxxxxxxxx><mailto:Gluster-users@xxxxxxxxxxx> >>>>>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>> _______________________________________________ >>>>>>>>>> Gluster-users mailing list >>>>>>>>>> Gluster-users@xxxxxxxxxxx >>>>>>>>>> <mailto:Gluster-users@xxxxxxxxxxx><mailto:Gluster-users@xxxxxxxxxxx> >>>>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>>>>>> _______________________________________________ >>>>>>>>> Gluster-users mailing list >>>>>>>>> Gluster-users@xxxxxxxxxxx >>>>>>>>> <mailto:Gluster-users@xxxxxxxxxxx><mailto:Gluster-users@xxxxxxxxxxx> >>>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>>>>> _______________________________________________ >>>>>>>> Gluster-users mailing list >>>>>>>> Gluster-users@xxxxxxxxxxx >>>>>>>> <mailto:Gluster-users@xxxxxxxxxxx><mailto:Gluster-users@xxxxxxxxxxx> >>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users@xxxxxxxxxxx >>>>>>> <mailto:Gluster-users@xxxxxxxxxxx><mailto:Gluster-users@xxxxxxxxxxx> >>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>>> >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users@xxxxxxxxxxx >>>>>> <mailto:Gluster-users@xxxxxxxxxxx><mailto:Gluster-users@xxxxxxxxxxx> >>>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users@xxxxxxxxxxx <mailto:Gluster-users@xxxxxxxxxxx> >>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users@xxxxxxxxxxx >> http://www.gluster.org/mailman/listinfo/gluster-users >> _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
