Remote operation failed: Stale NFS file handle

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



btw... My bug entry to change that glib and kernel error string 
(https://bugzilla.redhat.com/show_bug.cgi?id=832694) have been accepted 
and patched. Just waiting for them to make it downstream.

On 10/15/2013 2:54 PM, Justin Dossey wrote:
> I've seen these errors too on GlusterFS 3.3.1 nodes with 
> glusterfs-fuse mounts.  It's particularly strange because we're not 
> using NFS to mount the volumes.
>
>
> On Tue, Oct 15, 2013 at 1:44 PM, Neil Van Lysel <van-lyse at cs.wisc.edu 
> <mailto:van-lyse at cs.wisc.edu>> wrote:
>
>     Hello!
>
>     Many of our Gluster client nodes are seeing a lot of these errors
>     in their log files:
>
>     [2013-10-15 06:48:59.467263] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-6:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (3cfbebf4-40e4-4300-aa6e-bd43b4310b94)
>     [2013-10-15 06:48:59.467331] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-7:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (3cfbebf4-40e4-4300-aa6e-bd43b4310b94)
>     [2013-10-15 06:48:59.470554] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-0:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (d662e7db-7864-4b18-b587-bdc5e8756076)
>     [2013-10-15 06:48:59.470624] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-1:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (d662e7db-7864-4b18-b587-bdc5e8756076)
>     [2013-10-15 06:49:04.537548] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-3:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (a4ea32e0-25f8-440d-b258-23430490624d)
>     [2013-10-15 06:49:04.537651] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-2:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (a4ea32e0-25f8-440d-b258-23430490624d)
>     [2013-10-15 06:49:14.380551] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-0:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (669a2d6b-2998-48b2-8f3f-93d5f65cdd87)
>     [2013-10-15 06:49:14.380663] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-1:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (669a2d6b-2998-48b2-8f3f-93d5f65cdd87)
>     [2013-10-15 06:49:14.386390] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-4:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (016aafa9-35ac-4f6f-90bd-b4ac5d435ad0)
>     [2013-10-15 06:49:14.386471] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-5:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (016aafa9-35ac-4f6f-90bd-b4ac5d435ad0)
>     [2013-10-15 18:28:10.630357] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-2:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (5d6153cc-64b3-4151-85cd-2646c33c6918)
>     [2013-10-15 18:28:10.630425] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-3:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (5d6153cc-64b3-4151-85cd-2646c33c6918)
>     [2013-10-15 18:28:10.636301] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-4:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (2f64b9fe-02a0-408b-9edb-0c5e5bf0ed0e)
>     [2013-10-15 18:28:10.636377] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-5:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (2f64b9fe-02a0-408b-9edb-0c5e5bf0ed0e)
>     [2013-10-15 18:28:10.638574] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-5:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (990de721-1fc9-461d-8412-8c17c23ebbbd)
>     [2013-10-15 18:28:10.638647] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-4:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (990de721-1fc9-461d-8412-8c17c23ebbbd)
>     [2013-10-15 18:28:10.645043] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-7:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (0d8d3c5a-d26e-4c15-a8d5-987a4033a6d0)
>     [2013-10-15 18:28:10.645157] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-6:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (0d8d3c5a-d26e-4c15-a8d5-987a4033a6d0)
>     [2013-10-15 18:28:10.648126] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-6:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (c1c84d57-f54d-4dc1-a5df-9be563da78fb)
>     [2013-10-15 18:28:10.648276] W
>     [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-7:
>     remote operation failed: Stale NFS file handle. Path: /path
>     (c1c84d57-f54d-4dc1-a5df-9be563da78fb)
>
>
>     How can I resolve these errors?
>
>
>     *gluster --version:
>     glusterfs 3.4.0 built on Jul 25 2013 04:12:27
>
>
>     *gluster volume info:
>     Volume Name: scratch
>     Type: Distributed-Replicate
>     Volume ID: 198b9d77-96e6-4c7f-9f0c-3618cbcaa940
>     Status: Started
>     Number of Bricks: 4 x 2 = 8
>     Transport-type: tcp
>     Bricks:
>     Brick1: 10.129.40.21:/data/glusterfs/brick1/scratch
>     Brick2: 10.129.40.22:/data/glusterfs/brick1/scratch
>     Brick3: 10.129.40.23:/data/glusterfs/brick1/scratch
>     Brick4: 10.129.40.24:/data/glusterfs/brick1/scratch
>     Brick5: 10.129.40.21:/data/glusterfs/brick2/scratch
>     Brick6: 10.129.40.22:/data/glusterfs/brick2/scratch
>     Brick7: 10.129.40.23:/data/glusterfs/brick2/scratch
>     Brick8: 10.129.40.24:/data/glusterfs/brick2/scratch
>     Options Reconfigured:
>     features.quota: off
>
>     Volume Name: home
>     Type: Distributed-Replicate
>     Volume ID: 0d8ebafc-471e-4b16-a4a9-787ce8616225
>     Status: Started
>     Number of Bricks: 4 x 2 = 8
>     Transport-type: tcp
>     Bricks:
>     Brick1: 10.129.40.21:/data/glusterfs/brick1/home
>     Brick2: 10.129.40.22:/data/glusterfs/brick1/home
>     Brick3: 10.129.40.23:/data/glusterfs/brick1/home
>     Brick4: 10.129.40.24:/data/glusterfs/brick1/home
>     Brick5: 10.129.40.21:/data/glusterfs/brick2/home
>     Brick6: 10.129.40.22:/data/glusterfs/brick2/home
>     Brick7: 10.129.40.23:/data/glusterfs/brick2/home
>     Brick8: 10.129.40.24:/data/glusterfs/brick2/home
>     Options Reconfigured:
>     features.quota: off
>
>
>     *gluster volume status:
>     Status of volume: scratch
>     Gluster process                                         Port
>     Online  Pid
>     ------------------------------------------------------------------------------
>     Brick 10.129.40.21:/data/glusterfs/brick1/scratch 49154 Y       7536
>     Brick 10.129.40.22:/data/glusterfs/brick1/scratch 49154 Y       27976
>     Brick 10.129.40.23:/data/glusterfs/brick1/scratch 49154 Y       7436
>     Brick 10.129.40.24:/data/glusterfs/brick1/scratch 49154 Y       19773
>     Brick 10.129.40.21:/data/glusterfs/brick2/scratch 49155 Y       7543
>     Brick 10.129.40.22:/data/glusterfs/brick2/scratch 49155 Y       27982
>     Brick 10.129.40.23:/data/glusterfs/brick2/scratch 49155 Y       7442
>     Brick 10.129.40.24:/data/glusterfs/brick2/scratch 49155 Y       19778
>     NFS Server on localhost                                 2049 Y    
>       7564
>     Self-heal Daemon on localhost                           N/A Y    
>       7569
>     NFS Server on 10.129.40.24                              2049 Y    
>       19788
>     Self-heal Daemon on 10.129.40.24                        N/A Y    
>       19792
>     NFS Server on 10.129.40.23                              2049 Y    
>       7464
>     Self-heal Daemon on 10.129.40.23                        N/A Y    
>       7468
>     NFS Server on 10.129.40.22                              2049 Y    
>       28004
>     Self-heal Daemon on 10.129.40.22                        N/A Y    
>       28008
>
>     There are no active volume tasks
>     Status of volume: home
>     Gluster process                                         Port
>     Online  Pid
>     ------------------------------------------------------------------------------
>     Brick 10.129.40.21:/data/glusterfs/brick1/home  49152 Y       7549
>     Brick 10.129.40.22:/data/glusterfs/brick1/home  49152 Y       27989
>     Brick 10.129.40.23:/data/glusterfs/brick1/home  49152 Y       7449
>     Brick 10.129.40.24:/data/glusterfs/brick1/home  49152 Y       19760
>     Brick 10.129.40.21:/data/glusterfs/brick2/home  49153 Y       7554
>     Brick 10.129.40.22:/data/glusterfs/brick2/home  49153 Y       27994
>     Brick 10.129.40.23:/data/glusterfs/brick2/home  49153 Y       7454
>     Brick 10.129.40.24:/data/glusterfs/brick2/home  49153 Y       19766
>     NFS Server on localhost                                 2049 Y    
>       7564
>     Self-heal Daemon on localhost                           N/A Y    
>       7569
>     NFS Server on 10.129.40.24                              2049 Y    
>       19788
>     Self-heal Daemon on 10.129.40.24                        N/A Y    
>       19792
>     NFS Server on 10.129.40.22                              2049 Y    
>       28004
>     Self-heal Daemon on 10.129.40.22                        N/A Y    
>       28008
>     NFS Server on 10.129.40.23                              2049 Y    
>       7464
>     Self-heal Daemon on 10.129.40.23                        N/A Y    
>       7468
>
>     There are no active volume tasks
>
>
>     *The gluster volumes are mounted using the glusterfs-fuse package
>     (glusterfs-fuse-3.4.0-3.el6.x86_64) on the clients like so:
>     /sbin/mount.glusterfs 10.129.40.21:home /home
>     /sbin/mount.glusterfs 10.129.40.21:scratch /scratch
>
>
>     *Gluster packages on Gluster servers:
>     glusterfs-server-3.4.0-3.el6.x86_64
>     glusterfs-libs-3.4.0-8.el6.x86_64
>     glusterfs-3.4.0-3.el6.x86_64
>     glusterfs-geo-replication-3.4.0-3.el6.x86_64
>     glusterfs-fuse-3.4.0-3.el6.x86_64
>     glusterfs-rdma-3.4.0-3.el6.x86_64
>
>
>     *Gluster packages on clients:
>     glusterfs-fuse-3.4.0-3.el6.x86_64
>     glusterfs-3.4.0-3.el6.x86_64
>
>
>     All clients and servers are running the same OS and kernel:
>
>     *uname -a:
>     Linux <hostname> 2.6.32-358.6.1.el6.x86_64 #1 SMP Tue Apr 23
>     16:15:13 CDT 2013 x86_64 x86_64 x86_64 GNU/Linux
>
>     *cat /etc/redhat-release :
>     Scientific Linux release 6.3 (Carbon)
>
>
>     Thanks for your help,
>
>     Neil Van Lysel
>     UNIX Systems Administrator
>     Center for High Throughput Computing
>     University of Wisconsin - Madison
>
>
>
>     _______________________________________________
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>     http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> -- 
> Justin Dossey
> CTO, PodOmatic
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131015/477d0c1d/attachment.html>


[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux