Hi,
We run a fairly large two server replica with 5 2TB bricks on each. We
recently added two more bricks each and started a rebalance.
We immediately started getting client errors and halted the rebalance.
Since then we've been getting more and more errors.
Symptoms:
Client:
gets Permission denied errors when accessing a file. If root, or
occasionally another user accesses the file, it works, then works for the
original user on that client.
logs contain lots of:
[2014-10-30 03:14:08.912942] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-storage-client-3: remote operation failed: Permission denied. Path: /wms_pyramid/r_00000525/12285/r_00000525/8/8.qix (00000000-0000-0000-0000-000000000000)
errors.
Server:
link files and source files (that have caused problems above) seem to have different ownership:
$ ls -l brick*b/vol/raster/r_000002ff/000008ba.tif
---------T 2 otm otm 0 Oct 30 17:03 brick0b/vol/raster/r_000002ff/000008ba.tif
-rw-rw-r-- 2 dandelion dandelion 15867977 Mar 22 2013 brick1b/vol/raster/r_000002ff/000008ba.tif
$ sudo getfattr -e hex -m- -d brick*b/vol/raster/r_000002ff/000008ba.tif
# file: brick0b/vol/raster/r_000002ff/000008ba.tif
trusted.afr.storage-client-0=0x000000000000000100000000
trusted.afr.storage-client-1=0x000000000000000100000000
trusted.gfid=0x54b3b2ae42504754a193505a933c30b7
trusted.glusterfs.dht.linkto=0x73746f726167652d7265706c69636174652d3100
# file: brick1b/vol/raster/r_000002ff/000008ba.tif
trusted.afr.storage-client-2=0x000000000000000000000000
trusted.afr.storage-client-3=0x000000000000000000000000
trusted.gfid=0x54b3b2ae42504754a193505a933c30b7
logs contain lots of:
[2014-10-29 19:43:35.838942] I [server-rpc-fops.c:575:server_mknod_cbk] 0-storage-server: 15505: MKNOD /media/settings/branding/mfe-logo-white-2-1.png (1776f2b8-8857-4801-8c58-266eafcd7a87/mfe-logo-white-2-1.png) ==> (Permission denied)
and
[2014-10-30 04:28:58.707352] E [marker.c:2080:marker_setattr_cbk] 0-storage-marker: Operation not permitted occurred during setattr of <nul>
[2014-10-30 04:28:58.707407] I [server-rpc-fops.c:1778:server_setattr_cbk] 0-storage-server: 1210198: SETATTR /raster/r_0000081a/0000005e.jp2.aux.xml (0df0682b-0791-4327-bb5d-72ed916349fd) ==> (Operation not permitted)
This is happening to old files that haven't been changed since long before the rebalance, and is still happening, even though I believe the rebalance has been stopped.
We have restarted gluster-server on both servers. volume heal shows no current split-brain or heal-failed.
It does seem, to me, to match:
Except that is supposedly fixed before 3.4.5.
Help!?
- Colin
-- _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
