Hi Raghavendra,
I'm don't quite understand the issue. Yes, a rebalance was was performed in December last year when I added a brick. However, we have only just come across this (reproducible) problem upon upgrading to 3.5.
Is there anything I can do to try and correct the issue? Perhaps turn off root squashing while running "gluster volume rebalance VOLNAME fix-layout start"?
Cheers,
Dave
On 17 July 2014 17:31, Raghavendra Bhat <rabhat@xxxxxxxxxx> wrote:
On Wednesday 16 July 2014 10:18 AM, David Raffelt wrote:
Hi Raghavendra,NoThanksDave
As per the cmd_log_history file (a hidden file present in the log directory which stores the CLI commands executed on that peer), rebalance seems to be running (or was run).
[2013-12-17 03:08:59.081232] : volume rebalance data start : SUCCESS
[2013-12-17 03:09:14.631826] : volume rebalance data status : SUCCESS
[2013-12-17 03:09:22.761097] : volume rebalance data status : SUCCESS
[2013-12-17 03:09:27.748014] : volume rebalance data status : SUCCESS
[2013-12-17 03:09:28.839242] : volume rebalance data status : SUCCESS
[2013-12-17 03:10:39.982747] : volume rebalance data status : SUCCESS
[2013-12-17 03:14:30.919676] : volume rebalance data status : SUCCESS
[2013-12-17 03:14:33.772300] : volume rebalance data status : SUCCESS
[2013-12-17 03:29:14.467954] : volume rebalance data status : SUCCESS
[2013-12-17 03:29:43.303852] : volume rebalance data status : SUCCESS
[2013-12-17 03:30:04.309054] : volume rebalance data status : SUCCESS
[2013-12-17 04:35:45.631119] : volume rebalance data status : SUCCESS
I think this is what has happened. As part of rebalance layout might have changed for some directories and distribute tries to repair it by doing a self-heal when a lookup is performed on the directory. Distribute performs self-heal as root. But when the requests from that client comes to brick process, the requests from root are changed by default to nfsnobody (uid: 65534) and that uid does not have permissions to do some modifications (in this case self-heal) on the directory which brick thinks is owned by root. So self-heal does not happen properly and because of that some operations performed (in this case rename of a file within that directory)
fails.
Dave,
Please let me know if I have missed anything. This is my observation based on the log files.
CCing Raghavendra G who might be able to clarify whether this is what happened.
Regards,
Raghavendra Bhat
On 16 July 2014 14:47, Raghavendra Bhat <rabhat@xxxxxxxxxx> wrote:
Hi Dave,On Tuesday 15 July 2014 01:57 PM, David Raffelt wrote:
Hi Raghavendra,Thanks for looking into this. Attached are the log files from the 3 peers. The glusterfs server is running on "Beauty". All 3 peers mount the native gluster client on /home. Each peer has a direct connection to each other, addressable via the /etc/hosts file.
Note that I do not see any new output in the log when this error occurs. Also note that I tried to replicate this issue on Ubuntu 14.04 with a single brick and could not replicate it.
Below is some more output that might help.Thanks!Dave
dave@beauty:~$ glusterfs --versionglusterfs 3.5git built on Jun 30 2014 15:58:19Repository revision: git://git.gluster.com/glusterfs.gitCopyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>GlusterFS comes with ABSOLUTELY NO WARRANTY.It is licensed to you under your choice of the GNU LesserGeneral Public License, version 3 or any later version (LGPLv3or later), or the GNU General Public License, version 2 (GPLv2),in all cases as published by the Free Software Foundation.
dave@beauty:~$ uname -r3.15.4-1-ARCH
dave@beauty:~$ sudo gluster volume infoVolume Name: dataType: DistributeVolume ID: 1d5948c7-9b7a-40ca-8aa7-85c74bcef3bcStatus: StartedNumber of Bricks: 3Transport-type: tcpBricks:Brick1: beauty:/export/beautyBrick2: beast:/export/beastBrick3: benji:/export/benjiOptions Reconfigured:performance.cache-size: 32MBperformance.write-behind-window-size: 1MBauth.allow: 172.30.25.173,172.30.25.158,172.30.25.234,172.30.26.76,172.30.26.77,192.168.0.1,192.168.1.1,192.168.1.2,192.168.2.2,192.168.3.2,192.168.4.1,192.168.4.2,192.168.5.1,192.168.5.2nfs.disable: offdiagnostics.brick-log-level: ERRORdiagnostics.client-log-level: ERRORserver.root-squash: enable
Was rebalance running when you did above operations?
Regards,
Raghavendra Bhat
On 15 July 2014 15:29, Raghavendra Bhat <rabhat@xxxxxxxxxx> wrote:
Hi Dave,On Monday 14 July 2014 09:10 PM, Pranith Kumar Karampuri wrote:
CCed Raghavendra Bhat who may know about the issue
Pranith
On 07/14/2014 08:01 PM, Joe Julian wrote:
https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
Please file a bug report.
On July 14, 2014 12:38:11 AM PDT, David Raffelt <d.raffelt@xxxxxxxxxxxx> wrote:Hi All,After a recent update to gluster 3.5 we are having some issues renaming files when root squashing is enabled and the folder group permissions are not set to write.
For example if I create a folder with the following permissions$ mkdir test$ chmod g-w test$ ls -ldrwxr-xr-x 2 dave dave 22 Jul 14 17:16 test
When I create a file within this folder, and try to rename it I get a file permissions error.
$ cd test$ touch asdf$ mv asdf asdf2mv: cannot move ‘asdf’ to ‘asdf2’: Permission denied
A strace on the mv command reveals the rename system call fails with:rename("asdf", "asdf2") = -1 EACCES (Permission denied)
However I can copy the file and delete the old one fine.
If I either disable gluster root squashing, or change the test_dir folder group permission to write then I can rename the file without any problems.
System details are:
Arch linuxSystem umask is set to 002
Distributed volume, 3 peers, 1 brick per peer.
Any help is much appreciated!Dave
Can you please provide the brick and client log files? Which client you were using? fuse or nfs?
Regards,
Raghavendra Bhat
Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
--
David Raffelt (PhD)Postdoctoral Fellow
The Florey Institute of Neuroscience and Mental HealthMelbourne Brain Centre - Austin Campus245 Burgundy StreetHeidelberg Vic 3084Ph: +61 3 9035 7024
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
--
David Raffelt (PhD)Postdoctoral Fellow
The Florey Institute of Neuroscience and Mental HealthMelbourne Brain Centre - Austin Campus245 Burgundy StreetHeidelberg Vic 3084Ph: +61 3 9035 7024
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
David Raffelt (PhD)
Postdoctoral Fellow
The Florey Institute of Neuroscience and Mental Health
Melbourne Brain Centre - Austin Campus
245 Burgundy Street
Heidelberg Vic 3084
Ph: +61 3 9035 7024
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
