gluster 3.4.5: lots of permission problems after add-brick/rebalance

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

We run a fairly large two server replica with 5 2TB bricks on each. We
recently added two more bricks each and started a rebalance.

We immediately started getting client errors and halted the rebalance.

Since then we've been getting more and more errors.

Symptoms:

Client: 

  gets Permission denied errors when accessing a file. If root, or
  occasionally another user accesses the file, it works, then works for the
  original user on that client.
 
  logs contain lots of:

     [2014-10-30 03:14:08.912942] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-storage-client-3: remote operation failed: Permission denied. Path: /wms_pyramid/r_00000525/12285/r_00000525/8/8.qix (00000000-0000-0000-0000-000000000000)


   errors.


Server:

   link files and source files (that have caused problems above) seem to have different ownership:

   $ ls -l brick*b/vol/raster/r_000002ff/000008ba.tif
    ---------T 2 otm       otm              0 Oct 30 17:03 brick0b/vol/raster/r_000002ff/000008ba.tif
    -rw-rw-r-- 2 dandelion dandelion 15867977 Mar 22  2013 brick1b/vol/raster/r_000002ff/000008ba.tif


  $ sudo getfattr -e hex -m- -d brick*b/vol/raster/r_000002ff/000008ba.tif
  # file: brick0b/vol/raster/r_000002ff/000008ba.tif
  trusted.afr.storage-client-0=0x000000000000000100000000
  trusted.afr.storage-client-1=0x000000000000000100000000
  trusted.gfid=0x54b3b2ae42504754a193505a933c30b7
  trusted.glusterfs.dht.linkto=0x73746f726167652d7265706c69636174652d3100

  # file: brick1b/vol/raster/r_000002ff/000008ba.tif
  trusted.afr.storage-client-2=0x000000000000000000000000
  trusted.afr.storage-client-3=0x000000000000000000000000
  trusted.gfid=0x54b3b2ae42504754a193505a933c30b7


  logs contain lots of:

   [2014-10-29 19:43:35.838942] I [server-rpc-fops.c:575:server_mknod_cbk] 0-storage-server: 15505: MKNOD /media/settings/branding/mfe-logo-white-2-1.png (1776f2b8-8857-4801-8c58-266eafcd7a87/mfe-logo-white-2-1.png) ==> (Permission denied)

  and

   [2014-10-30 04:28:58.707352] E [marker.c:2080:marker_setattr_cbk] 0-storage-marker: Operation not permitted occurred during setattr of <nul>
[2014-10-30 04:28:58.707407] I [server-rpc-fops.c:1778:server_setattr_cbk] 0-storage-server: 1210198: SETATTR /raster/r_0000081a/0000005e.jp2.aux.xml (0df0682b-0791-4327-bb5d-72ed916349fd) ==> (Operation not permitted)


  This is happening to old files that haven't been changed since long before the rebalance, and is still happening, even though I believe the rebalance has been stopped.

We have restarted gluster-server on both servers.   volume heal  shows no current split-brain or heal-failed.


 It does seem, to me, to match:
     https://bugzilla.redhat.com/show_bug.cgi?id=884597

 Except that is supposedly fixed before 3.4.5.


Help!?


- Colin
-- 
--
Colin Coghill
DevOps Engineer
Koordinates
colin.coghill@xxxxxxxxxxxxxxx
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux