Re: Cannot rename files with root squashing and r-x folder group permissions

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Raghavendra,
I'm don't quite understand the issue. Yes, a rebalance was was performed in December last year when I added a brick. However, we have only just come across this (reproducible) problem upon upgrading to 3.5. 

Is there anything I can do to try and correct the issue? Perhaps turn off root squashing while running "gluster volume rebalance VOLNAME fix-layout start"?

Cheers,
Dave


On 17 July 2014 17:31, Raghavendra Bhat <rabhat@xxxxxxxxxx> wrote:
On Wednesday 16 July 2014 10:18 AM, David Raffelt wrote:
Hi Raghavendra,
No
Thanks 
Dave



As per the cmd_log_history file (a hidden file present in the log directory which stores the CLI commands executed on that peer), rebalance seems to be running (or was run).

[2013-12-17 03:08:59.081232]  : volume rebalance data start : SUCCESS
[2013-12-17 03:09:14.631826]  : volume rebalance data status : SUCCESS
[2013-12-17 03:09:22.761097]  : volume rebalance data status : SUCCESS
[2013-12-17 03:09:27.748014]  : volume rebalance data status : SUCCESS
[2013-12-17 03:09:28.839242]  : volume rebalance data status : SUCCESS
[2013-12-17 03:10:39.982747]  : volume rebalance data status : SUCCESS
[2013-12-17 03:14:30.919676]  : volume rebalance data status : SUCCESS
[2013-12-17 03:14:33.772300]  : volume rebalance data status : SUCCESS
[2013-12-17 03:29:14.467954]  : volume rebalance data status : SUCCESS
[2013-12-17 03:29:43.303852]  : volume rebalance data status : SUCCESS
[2013-12-17 03:30:04.309054]  : volume rebalance data status : SUCCESS
[2013-12-17 04:35:45.631119]  : volume rebalance data status : SUCCESS


 I think this is what has happened. As part of rebalance layout might have changed for some directories and distribute tries to repair it by doing a self-heal when a lookup is performed on the directory. Distribute performs self-heal as root. But when the requests from that client comes to brick process, the requests from root are changed by default to nfsnobody (uid: 65534) and that uid does not have permissions to do some modifications (in this case self-heal) on the directory which brick thinks is owned by root. So self-heal does not happen properly and because of that some operations performed (in this case rename of a file within that directory)
fails.

Dave,
Please let me know if I have missed anything. This is my observation based on the log files.

CCing Raghavendra G who might be able to clarify whether this is what happened.

Regards,
Raghavendra Bhat


On 16 July 2014 14:47, Raghavendra Bhat <rabhat@xxxxxxxxxx> wrote:
On Tuesday 15 July 2014 01:57 PM, David Raffelt wrote:
Hi Raghavendra,
Thanks for looking into this. Attached are the log files from the 3 peers. The glusterfs server is running on "Beauty".  All 3 peers mount the native gluster client on /home. Each peer has a direct connection to each other, addressable via the /etc/hosts file. 

Note that I do not see any new output in the log when this error occurs.  Also note that I tried to replicate this issue on Ubuntu 14.04 with a single brick and could not replicate it. 

Below is some more output that might help.
Thanks!
Dave



dave@beauty:~$ glusterfs --version
glusterfs 3.5git built on Jun 30 2014 15:58:19
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.


dave@beauty:~$ uname -r
3.15.4-1-ARCH


dave@beauty:~$ sudo gluster volume info 
Volume Name: data
Type: Distribute
Volume ID: 1d5948c7-9b7a-40ca-8aa7-85c74bcef3bc
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: beauty:/export/beauty
Brick2: beast:/export/beast
Brick3: benji:/export/benji
Options Reconfigured:
performance.cache-size: 32MB
performance.write-behind-window-size: 1MB
auth.allow: 172.30.25.173,172.30.25.158,172.30.25.234,172.30.26.76,172.30.26.77,192.168.0.1,192.168.1.1,192.168.1.2,192.168.2.2,192.168.3.2,192.168.4.1,192.168.4.2,192.168.5.1,192.168.5.2
nfs.disable: off
diagnostics.brick-log-level: ERROR
diagnostics.client-log-level: ERROR
server.root-squash: enable




Hi Dave,

Was rebalance running when you did above operations?


Regards,
Raghavendra Bhat

















On 15 July 2014 15:29, Raghavendra Bhat <rabhat@xxxxxxxxxx> wrote:
On Monday 14 July 2014 09:10 PM, Pranith Kumar Karampuri wrote:
CCed Raghavendra Bhat who may know about the issue

Pranith
On 07/14/2014 08:01 PM, Joe Julian wrote:
https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS

Please file a bug report.

On July 14, 2014 12:38:11 AM PDT, David Raffelt <d.raffelt@xxxxxxxxxxxx> wrote:
Hi All,
After a recent update to gluster 3.5 we are having some issues renaming files when root squashing is enabled and the folder group permissions are not set to write. 

For example if I create a folder with the following permissions
$ mkdir test
$ chmod g-w test
$ ls -l 
drwxr-xr-x  2 dave dave  22 Jul 14 17:16 test

When I create a file within this folder, and try to rename it I get a file permissions error. 

$ cd test
$ touch asdf
$ mv asdf asdf2
mv: cannot move ‘asdf’ to ‘asdf2’: Permission denied

A strace on the mv command reveals the rename system call fails with:
rename("asdf", "asdf2") = -1 EACCES (Permission denied)

However I can copy the file and delete the old one fine. 

If I either disable gluster root squashing, or change the test_dir folder group permission to write then I can rename the file without any problems.

System details are:
Arch linux
System umask is set to 002
Distributed volume, 3 peers, 1 brick per peer. 

Any help is much appreciated!
Dave



Hi Dave,

Can you please provide the brick and client log files? Which client you were using? fuse or nfs?

Regards,
Raghavendra Bhat



--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users



_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users


_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users



--
David Raffelt (PhD)
Postdoctoral Fellow

The Florey Institute of Neuroscience and Mental Health
Melbourne Brain Centre - Austin Campus
245 Burgundy Street
Heidelberg Vic 3084


_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users



--
David Raffelt (PhD)
Postdoctoral Fellow

The Florey Institute of Neuroscience and Mental Health
Melbourne Brain Centre - Austin Campus
245 Burgundy Street
Heidelberg Vic 3084


_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users



--
David Raffelt (PhD)
Post Doctoral Fellow

The Florey Institute of Neuroscience and Mental Health
Melbourne Brain Centre - Austin Campus
245 Burgundy Street
Heidelberg Vic 3084
www.florey.edu.au
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux