Re: nfs acces denied

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hello Carlos,

 

I created a new disk formatted xfs with an inode size of 512.

I created a new gluster volume and migrated the data. Again, when I mount it as a gluster volume, everything works fine.

On nfs (mounted on the same server that’s running the gluster volume) I get the following weirdness:

 

[root@lpr-nas01 /]# mount -t nfs -o mountproto=tcp,vers=3 lpr-nas01:/caviar_data11 /media

[root@lpr-nas01 /]# ll /media/*/

 

….

/media/2012/201206:

ls: /media/2012/201206/20120621: No such file or directory

total 0

drwxrwsr-x 3 960 1003 15 Jun 15  2011 20120621

….

drwxrwsr-x 2 nfsnobody 1003 4096 Jun  6  2011 38

drwxrwsr-x 2 nfsnobody 1003 4096 Jun  6  2011 39

drwxrwsr-x 2 nfsnobody 1003  138 Jun  6  2011 40

drwxrwsr-x 2 nfsnobody 1003 4096 Jun  6  2011 42

drwxrwsr-x 2 nfsnobody 1003  171 Jun  6  2011 43

drwxrwsr-x 2 nfsnobody 1003 4096 Jun  6  2011 45

drwxrwsr-x 2 nfsnobody 1003 4096 Jun  6  2011 46

drwxrwsr-x 2 nfsnobody 1003 4096 Jun  6  2011 47

drwxrwsr-x 2 nfsnobody 1003  369 Jun  6  2011 48

…

ls: cannot access /media/2011/201106/20110606/81: Invalid argument

ls: cannot access /media/2011/201106/20110606/55: Invalid argument

ls: cannot access /media/2011/201106/20110606/30: Invalid argument

ls: cannot access /media/2011/201106/20110606/90: Invalid argument

total 12

d????????? ? ?   ?        ?            ? 00

d????????? ? ?   ?        ?            ? 01

d????????? ? ?   ?        ?            ? 02

d????????? ? ?   ?        ?            ? 03

d????????? ? ?   ?        ?            ? 04

d????????? ? ?   ?        ?            ? 05

d????????? ? ?   ?        ?            ? 06

d????????? ? ?   ?        ?            ? 07

d????????? ? ?   ?        ?            ? 08

d????????? ? ?   ?        ?            ? 09

d????????? ? ?   ?        ?            ? 10

….

 

 

In the nfs log, I get the following errors:

…

[2014-04-03 08:50:13.624517] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:a1acf77c-2b81-4b5f-a113-521c6ab8fd23>: Invalid argument

[2014-04-03 08:50:13.624547] E [acl3.c:334:acl3_getacl_resume] 0-nfs-ACL: Unable to resolve FH: (192.168.151.21:954) caviar_data11 : a1acf77c-2b81-4b5f-a113-521c6ab8fd23

[2014-04-03 08:50:13.624562] E [acl3.c:342:acl3_getacl_resume] 0-nfs-ACL: unable to open_and_resume

[2014-04-03 08:50:13.624914] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:980a29f7-29f4-4c88-9896-2b1a549370e2>: Invalid argument

[2014-04-03 08:50:13.624960] E [acl3.c:334:acl3_getacl_resume] 0-nfs-ACL: Unable to resolve FH: (192.168.151.21:954) caviar_data11 : 980a29f7-29f4-4c88-9896-2b1a549370e2

[2014-04-03 08:50:13.624970] E [acl3.c:342:acl3_getacl_resume] 0-nfs-ACL: unable to open_and_resume

[2014-04-03 08:50:13.625290] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:b67993d2-c647-4493-aa78-64033614dc33>: Invalid argument

[2014-04-03 08:50:13.625322] E [nfs3.c:755:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.151.21:954) caviar_data11 : b67993d2-c647-4493-aa78-64033614dc33

[2014-04-03 08:50:13.625335] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: c838d715, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)

[2014-04-03 08:50:13.626268] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201108/20110802. holes=1 overlaps=0

[2014-04-03 08:50:13.627701] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201108/20110803. holes=1 overlaps=0

[2014-04-03 08:50:13.628839] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:bb587d95-ffa3-42f3-9b27-b1cf0c5c05cb>: Invalid argument

….

[2014-04-03 08:50:13.706106] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 6639d715, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)

[2014-04-03 08:50:13.706585] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:897bf28d-41a3-4cc2-a120-8374656be858>: Invalid argument

[2014-04-03 08:50:13.706626] E [nfs3.c:755:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.151.21:954) caviar_data11 : 897bf28d-41a3-4cc2-a120-8374656be858

[2014-04-03 08:50:13.706647] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 6739d715, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)

[2014-04-03 08:50:13.707113] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:ded9c3fe-171b-4104-bc0e-e64c0d3e18e2>: Invalid argument

[2014-04-03 08:50:13.707158] E [nfs3.c:755:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.151.21:954) caviar_data11 : ded9c3fe-171b-4104-bc0e-e64c0d3e18e2

[2014-04-03 08:50:13.707174] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 6839d715, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)

[2014-04-03 08:50:13.707597] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:c5ebeb07-ee2e-4161-9e65-65d65f614628>: Invalid argument

[2014-04-03 08:50:13.707644] E [nfs3.c:755:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.151.21:954) caviar_data11 : c5ebeb07-ee2e-4161-9e65-65d65f614628

[2014-04-03 08:50:13.707707] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 6939d715, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)

[2014-04-03 08:50:13.708357] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:da363aa4-6d43-47c0-90fe-edead3689064>: Invalid argument

[2014-04-03 08:50:13.708407] E [nfs3.c:755:nfs3_getattr_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.151.21:954) caviar_data11 : da363aa4-6d43-47c0-90fe-edead3689064

[2014-04-03 08:50:13.708432] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 6a39d715, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)

[2014-04-03 08:50:13.708796] E [dht-helper.c:429:dht_subvol_get_hashed] (-->/usr/lib64/glusterfs/3.4.2/xlator/debug/io-stats.so(io_stats_lookup+0x157) [0x7f4465bf52e7] (-->/usr/lib64/libglusterfs.so.0(default_lookup+0x6d) [0x3dfe01c03d] (-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup+0xa7e) [0x7f4466037f2e]))) 0-caviar_data11-dht: invalid argument: loc->parent

[2014-04-03 08:50:13.709066] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-caviar_data11-client-0: remote operation failed: Invalid argument. Path: <gfid:00000000-0000-0000-0000-000000000000> (00000000-0000-0000-0000-000000000000)

[2014-04-03 08:50:13.709128] E [acl3.c:334:acl3_getacl_resume] 0-nfs-ACL: Unable to resolve FH: (192.168.151.21:954) caviar_data11 : 00000000-0000-0000-0000-000000000000

[2014-04-03 08:50:13.709159] E [acl3.c:342:acl3_getacl_resume] 0-nfs-ACL: unable to open_and_resume

[2014-04-03 08:50:13.709741] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:db0eec80-e122-4751-9695-d903a6e6f29e>: Invalid argument

….

 

 

So….. any ideas?

 

 

From: Carlos Capriotti [mailto:capriotti.carlos@xxxxxxxxx]
Sent: maandag 31 maart 2014 18:03
To: VAN CAUSBROECK Wannes
Cc: gluster-users@xxxxxxxxxxx
Subject: Re: nfs acces denied

 

maybe it would be nice to see your volume info for affected volumes.

 

Also, on the server side, what happens if you mount the share using glusterfs instead of nfs ?

 

any change the native nfs server is running on your server ?

 

Are there any auto-heal processes running ?

 

There are a few name resolution messages on your logs, that seem to refer to the nodes themselves. Any DNS conflicts ? Maybe add the names of servers to the hosts file ? 

 

You MS client seems to be having issues with user/group translation. It seems to create files with gid 1003. (I could be wrong).

 

Again, is SElinux/ACLs/iptables disabled ?

 

All is very inconclusive os far.

 

On Mon, Mar 31, 2014 at 5:26 PM, VAN CAUSBROECK Wannes <Wannes.VANCAUSBROECK@xxxxxxxxxxxxxx> wrote:

Well, with 'client' i do actually mean the server itself.

i've tried forcing linux and windows to nfs V3 and tcp, and on windows i played around with the uid and gid, but the result is always the same

 


On 31 Mar 2014, at 17:22, "Carlos Capriotti" <capriotti.carlos@xxxxxxxxx> wrote:

Well, saying your client-side is "linux" does not help much. Distro, flavor, etc helps a lot, but I'll take a wild guess here.

 

First, force your NFS mount (client) to use nfs version 3. 

 

The same for Microsoft. (It is fair to say I have no idea if the MS client supports v4 or not).

 

Additionally, check that firewalls are disabled on both sides, just for testing. The same goes for SElinux.

 

Windows and ACL, and user mapping is something that might be in your way too. There is a Technet document that describes how to handle this mapping if I am not wrong.

Just for testing, mount your nfs share you your own server, using localhost:/nfs_share and see how it goes.

 

It is a good start.

 

Kr,

 

Carlos

 

On Mon, Mar 31, 2014 at 3:58 PM, VAN CAUSBROECK Wannes <Wannes.VANCAUSBROECK@xxxxxxxxxxxxxx> wrote:

Hello all,

 

I’ve already tried to post this, but i’m unsure it arrived to the mailing list.

 

I have some issues regarding my nfs mounts. My setup is as follows:

Rhel 6.4, gluster 3.4.2-1 running on a vm (4 cores, 8GB ram) attached to a san. I have one disk on which are all the bricks (formatted ext4 in 64 bit mode) of 25TB.

On the gluster side of things, everything works without issues. The trouble starts when I mount a volume as an nfs mount.

Lots of volumes work without issues, but others behave strangely. The volumes that act weird generally contain many files (can be accidental?).

The volumes in question mount without issues, but when I try to go into any subdirectory sometimes it works, sometimes I get errors.

 

On windows with nfs client: access denied

 

In nfslog:

[2014-03-31 13:57:58.771241] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in <gfid:c8d94120-6851-46ea-9f28-c629a44b1015>. holes=1 overlaps=0

[2014-03-31 13:57:58.771348] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:c8d94120-6851-46ea-9f28-c629a44b1015>: Invalid argument

[2014-03-31 13:57:58.771380] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : c8d94120-6851-46ea-9f28-c629a44b1015

[2014-03-31 13:57:58.771819] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28530, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)

[2014-03-31 13:57:58.798967] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in <gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b>. holes=1 overlaps=0

[2014-03-31 13:57:58.799039] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b>: Invalid argument

[2014-03-31 13:57:58.799056] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : 14972193-1039-4d7a-aed5-0d7e7eccf57b

[2014-03-31 13:57:58.799088] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28531, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)

….

 

 

On linux:

[root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/

ls: /media/2011/201105/20110530/37: No such file or directory

total 332

…

drwxrwsr-x 2 nfsnobody 1003 4096 Jun  6  2011 32

drwxrwsr-x 2 nfsnobody 1003 4096 Jun  6  2011 34

drwxrwsr-x 2 nfsnobody 1003 4096 Jun  6  2011 35

drwxrwsr-x 2 nfsnobody 1003 4096 Jun  6  2011 36

drwxrwsr-x 2 nfsnobody 1003 4096 Jun  6  2011 37

…

 

[root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/37

ls: /media/2011/201105/20110530/37/NN.0000073824357.00001.tif: No such file or directory

ls: /media/2011/201105/20110530/37/NN.0000073824357.00003.tif: No such file or directory

total 54

-rwxrwxr-x 0 nfsnobody 1003  9340 Jun  6  2011 NN.0000073824357.00001.tif

-rwxrwxr-x 1 nfsnobody 1003 35312 Jun  6  2011 NN.0000073824357.00002.tif

-rwxrwxr-x 0 nfsnobody 1003  9340 Jun  6  2011 NN.0000073824357.00003.tif

 

 

I see in the nfslog:

…

[2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. holes=1 overlaps=0

[2014-03-31 12:44:18.958078] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/30. holes=1 overlaps=0

[2014-03-31 12:44:18.959980] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/90. holes=1 overlaps=0

[2014-03-31 12:44:18.961094] E [dht-helper.c:429:dht_subvol_get_hashed] (-->/usr/lib64/glusterfs/3.4.2/xlator/debug/io-stats.so(io_stats_lookup+0x157) [0x7fd6a61282e7] (-->/usr/lib64/libglusterfs.so.0(default_lookup+0x6d) [0x3dfe01c03d] (-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup+0xa7e) [0x7fd6a656af2e]))) 0-caviar_data11-dht: invalid argument: loc->parent

[2014-03-31 12:44:18.961283] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-caviar_data11-client-0: remote operation failed: Invalid argument. Path: <gfid:00000000-0000-0000-0000-000000000000> (00000000-0000-0000-0000-000000000000)

[2014-03-31 12:44:18.961319] E [acl3.c:334:acl3_getacl_resume] 0-nfs-ACL: Unable to resolve FH: (192.168.151.21:740) caviar_data11 : 00000000-0000-0000-0000-000000000000

[2014-03-31 12:44:18.961338] E [acl3.c:342:acl3_getacl_resume] 0-nfs-ACL: unable to open_and_resume

…

 

The weirdest thing is it changes from time to time which files and directories work and which don’t

Any ideas?

 

Thanks!


_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users

 

 

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux