Everything seems to be working great, but the logs are filled with these warnings. We can access the files that are reporting a stale nfs handle fine. What would these warnings be a symptom of? Warnings worry me so I don't want to ignore these errors even if it is log spam. On 10/15/2013 06:19 PM, Joe Julian wrote: > Note the " W ". Errors are " E ". If something's not working, however, > please elaborate. Those warnings may be a symptom, or just might be > log spam. > > On 10/15/2013 2:54 PM, Justin Dossey wrote: >> I've seen these errors too on GlusterFS 3.3.1 nodes with >> glusterfs-fuse mounts. It's particularly strange because we're not >> using NFS to mount the volumes. >> >> >> On Tue, Oct 15, 2013 at 1:44 PM, Neil Van Lysel <van-lyse at cs.wisc.edu >> <mailto:van-lyse at cs.wisc.edu>> wrote: >> >> Hello! >> >> Many of our Gluster client nodes are seeing a lot of these errors >> in their log files: >> >> [2013-10-15 06:48:59.467263] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-6: >> remote operation failed: Stale NFS file handle. Path: /path >> (3cfbebf4-40e4-4300-aa6e-bd43b4310b94) >> [2013-10-15 06:48:59.467331] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-7: >> remote operation failed: Stale NFS file handle. Path: /path >> (3cfbebf4-40e4-4300-aa6e-bd43b4310b94) >> [2013-10-15 06:48:59.470554] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-0: >> remote operation failed: Stale NFS file handle. Path: /path >> (d662e7db-7864-4b18-b587-bdc5e8756076) >> [2013-10-15 06:48:59.470624] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-1: >> remote operation failed: Stale NFS file handle. Path: /path >> (d662e7db-7864-4b18-b587-bdc5e8756076) >> [2013-10-15 06:49:04.537548] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-3: >> remote operation failed: Stale NFS file handle. Path: /path >> (a4ea32e0-25f8-440d-b258-23430490624d) >> [2013-10-15 06:49:04.537651] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-2: >> remote operation failed: Stale NFS file handle. Path: /path >> (a4ea32e0-25f8-440d-b258-23430490624d) >> [2013-10-15 06:49:14.380551] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-0: >> remote operation failed: Stale NFS file handle. Path: /path >> (669a2d6b-2998-48b2-8f3f-93d5f65cdd87) >> [2013-10-15 06:49:14.380663] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-1: >> remote operation failed: Stale NFS file handle. Path: /path >> (669a2d6b-2998-48b2-8f3f-93d5f65cdd87) >> [2013-10-15 06:49:14.386390] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-4: >> remote operation failed: Stale NFS file handle. Path: /path >> (016aafa9-35ac-4f6f-90bd-b4ac5d435ad0) >> [2013-10-15 06:49:14.386471] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-5: >> remote operation failed: Stale NFS file handle. Path: /path >> (016aafa9-35ac-4f6f-90bd-b4ac5d435ad0) >> [2013-10-15 18:28:10.630357] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-2: >> remote operation failed: Stale NFS file handle. Path: /path >> (5d6153cc-64b3-4151-85cd-2646c33c6918) >> [2013-10-15 18:28:10.630425] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-3: >> remote operation failed: Stale NFS file handle. Path: /path >> (5d6153cc-64b3-4151-85cd-2646c33c6918) >> [2013-10-15 18:28:10.636301] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-4: >> remote operation failed: Stale NFS file handle. Path: /path >> (2f64b9fe-02a0-408b-9edb-0c5e5bf0ed0e) >> [2013-10-15 18:28:10.636377] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-5: >> remote operation failed: Stale NFS file handle. Path: /path >> (2f64b9fe-02a0-408b-9edb-0c5e5bf0ed0e) >> [2013-10-15 18:28:10.638574] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-5: >> remote operation failed: Stale NFS file handle. Path: /path >> (990de721-1fc9-461d-8412-8c17c23ebbbd) >> [2013-10-15 18:28:10.638647] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-4: >> remote operation failed: Stale NFS file handle. Path: /path >> (990de721-1fc9-461d-8412-8c17c23ebbbd) >> [2013-10-15 18:28:10.645043] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-7: >> remote operation failed: Stale NFS file handle. Path: /path >> (0d8d3c5a-d26e-4c15-a8d5-987a4033a6d0) >> [2013-10-15 18:28:10.645157] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-6: >> remote operation failed: Stale NFS file handle. Path: /path >> (0d8d3c5a-d26e-4c15-a8d5-987a4033a6d0) >> [2013-10-15 18:28:10.648126] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-6: >> remote operation failed: Stale NFS file handle. Path: /path >> (c1c84d57-f54d-4dc1-a5df-9be563da78fb) >> [2013-10-15 18:28:10.648276] W >> [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-7: >> remote operation failed: Stale NFS file handle. Path: /path >> (c1c84d57-f54d-4dc1-a5df-9be563da78fb) >> >> >> How can I resolve these errors? >> >> >> *gluster --version: >> glusterfs 3.4.0 built on Jul 25 2013 04:12:27 >> >> >> *gluster volume info: >> Volume Name: scratch >> Type: Distributed-Replicate >> Volume ID: 198b9d77-96e6-4c7f-9f0c-3618cbcaa940 >> Status: Started >> Number of Bricks: 4 x 2 = 8 >> Transport-type: tcp >> Bricks: >> Brick1: 10.129.40.21:/data/glusterfs/brick1/scratch >> Brick2: 10.129.40.22:/data/glusterfs/brick1/scratch >> Brick3: 10.129.40.23:/data/glusterfs/brick1/scratch >> Brick4: 10.129.40.24:/data/glusterfs/brick1/scratch >> Brick5: 10.129.40.21:/data/glusterfs/brick2/scratch >> Brick6: 10.129.40.22:/data/glusterfs/brick2/scratch >> Brick7: 10.129.40.23:/data/glusterfs/brick2/scratch >> Brick8: 10.129.40.24:/data/glusterfs/brick2/scratch >> Options Reconfigured: >> features.quota: off >> >> Volume Name: home >> Type: Distributed-Replicate >> Volume ID: 0d8ebafc-471e-4b16-a4a9-787ce8616225 >> Status: Started >> Number of Bricks: 4 x 2 = 8 >> Transport-type: tcp >> Bricks: >> Brick1: 10.129.40.21:/data/glusterfs/brick1/home >> Brick2: 10.129.40.22:/data/glusterfs/brick1/home >> Brick3: 10.129.40.23:/data/glusterfs/brick1/home >> Brick4: 10.129.40.24:/data/glusterfs/brick1/home >> Brick5: 10.129.40.21:/data/glusterfs/brick2/home >> Brick6: 10.129.40.22:/data/glusterfs/brick2/home >> Brick7: 10.129.40.23:/data/glusterfs/brick2/home >> Brick8: 10.129.40.24:/data/glusterfs/brick2/home >> Options Reconfigured: >> features.quota: off >> >> >> *gluster volume status: >> Status of volume: scratch >> Gluster process Port Online Pid >> ------------------------------------------------------------------------------ >> Brick 10.129.40.21:/data/glusterfs/brick1/scratch 49154 Y 7536 >> Brick 10.129.40.22:/data/glusterfs/brick1/scratch 49154 Y 27976 >> Brick 10.129.40.23:/data/glusterfs/brick1/scratch 49154 Y 7436 >> Brick 10.129.40.24:/data/glusterfs/brick1/scratch 49154 Y 19773 >> Brick 10.129.40.21:/data/glusterfs/brick2/scratch 49155 Y 7543 >> Brick 10.129.40.22:/data/glusterfs/brick2/scratch 49155 Y 27982 >> Brick 10.129.40.23:/data/glusterfs/brick2/scratch 49155 Y 7442 >> Brick 10.129.40.24:/data/glusterfs/brick2/scratch 49155 Y 19778 >> NFS Server on localhost 2049 Y 7564 >> Self-heal Daemon on localhost N/A Y 7569 >> NFS Server on 10.129.40.24 2049 Y 19788 >> Self-heal Daemon on 10.129.40.24 N/A Y 19792 >> NFS Server on 10.129.40.23 2049 Y 7464 >> Self-heal Daemon on 10.129.40.23 N/A Y 7468 >> NFS Server on 10.129.40.22 2049 Y 28004 >> Self-heal Daemon on 10.129.40.22 N/A Y 28008 >> >> There are no active volume tasks >> Status of volume: home >> Gluster process Port Online Pid >> ------------------------------------------------------------------------------ >> Brick 10.129.40.21:/data/glusterfs/brick1/home 49152 Y 7549 >> Brick 10.129.40.22:/data/glusterfs/brick1/home 49152 Y 27989 >> Brick 10.129.40.23:/data/glusterfs/brick1/home 49152 Y 7449 >> Brick 10.129.40.24:/data/glusterfs/brick1/home 49152 Y 19760 >> Brick 10.129.40.21:/data/glusterfs/brick2/home 49153 Y 7554 >> Brick 10.129.40.22:/data/glusterfs/brick2/home 49153 Y 27994 >> Brick 10.129.40.23:/data/glusterfs/brick2/home 49153 Y 7454 >> Brick 10.129.40.24:/data/glusterfs/brick2/home 49153 Y 19766 >> NFS Server on localhost 2049 Y 7564 >> Self-heal Daemon on localhost N/A Y 7569 >> NFS Server on 10.129.40.24 2049 Y 19788 >> Self-heal Daemon on 10.129.40.24 N/A Y 19792 >> NFS Server on 10.129.40.22 2049 Y 28004 >> Self-heal Daemon on 10.129.40.22 N/A Y 28008 >> NFS Server on 10.129.40.23 2049 Y 7464 >> Self-heal Daemon on 10.129.40.23 N/A Y 7468 >> >> There are no active volume tasks >> >> >> *The gluster volumes are mounted using the glusterfs-fuse package >> (glusterfs-fuse-3.4.0-3.el6.x86_64) on the clients like so: >> /sbin/mount.glusterfs 10.129.40.21:home /home >> /sbin/mount.glusterfs 10.129.40.21:scratch /scratch >> >> >> *Gluster packages on Gluster servers: >> glusterfs-server-3.4.0-3.el6.x86_64 >> glusterfs-libs-3.4.0-8.el6.x86_64 >> glusterfs-3.4.0-3.el6.x86_64 >> glusterfs-geo-replication-3.4.0-3.el6.x86_64 >> glusterfs-fuse-3.4.0-3.el6.x86_64 >> glusterfs-rdma-3.4.0-3.el6.x86_64 >> >> >> *Gluster packages on clients: >> glusterfs-fuse-3.4.0-3.el6.x86_64 >> glusterfs-3.4.0-3.el6.x86_64 >> >> >> All clients and servers are running the same OS and kernel: >> >> *uname -a: >> Linux <hostname> 2.6.32-358.6.1.el6.x86_64 #1 SMP Tue Apr 23 >> 16:15:13 CDT 2013 x86_64 x86_64 x86_64 GNU/Linux >> >> *cat /etc/redhat-release : >> Scientific Linux release 6.3 (Carbon) >> >> >> Thanks for your help, >> >> Neil Van Lysel >> UNIX Systems Administrator >> Center for High Throughput Computing >> University of Wisconsin - Madison >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >> http://supercolony.gluster.org/mailman/listinfo/gluster-users >> >> >> >> >> -- >> Justin Dossey >> CTO, PodOmatic >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://supercolony.gluster.org/mailman/listinfo/gluster-users > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131016/c031350d/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3528 bytes Desc: S/MIME Cryptographic Signature URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131016/c031350d/attachment.p7s>
