Hi, On April 1st, I started seeing the following errors in my nfs.log file. I didn't notice until today, when most of my VM's under Citrix XenServer made their drives read-only. The VM disk images are NFS mounted via gluster nfs. I didn't notice any issues until the remount as read-only today. Three of my VM's failed to boot after this, just sitting at initramfs. All VM's that re-mounted read-only needs an fsck. I'm running Gluster 3.2.5 under Ubuntu 10.04.4 (64 bit), and everything is happening via NFS. Gerald NFS.LOG: Given volfile: +------------------------------------------------------------------------------+ 1: volume gl-raid1-client-0 2: type protocol/client 3: option remote-host 192.168.20.1 4: option remote-subvolume /export/gluster/RAID-1 5: option transport-type tcp 6: end-volume 7: 8: volume gl-raid1-replicate-0 9: type cluster/replicate 10: subvolumes gl-raid1-client-0 11: end-volume 12: 13: volume gl-raid1-write-behind 14: type performance/write-behind 15: option cache-size 4mb 16: subvolumes gl-raid1-replicate-0 17: end-volume 18: 19: volume gl-raid1-read-ahead 20: type performance/read-ahead 21: subvolumes gl-raid1-write-behind 22: end-volume 23: 24: volume gl-raid1-io-cache 25: type performance/io-cache 26: option cache-size 268435456 27: subvolumes gl-raid1-read-ahead 28: end-volume 29: 30: volume gl-raid1-quick-read 31: type performance/quick-read 32: option cache-size 268435456 33: subvolumes gl-raid1-io-cache 34: end-volume 35: 36: volume gl-raid1 37: type debug/io-stats 38: option latency-measurement off 39: option count-fop-hits off 40: subvolumes gl-raid1-quick-read 41: end-volume 42: 43: volume gl-raid6-client-0 44: type protocol/client 45: option remote-host 192.168.10.1 46: option remote-subvolume /export/gluster/RAID-6 47: option transport-type tcp 48: end-volume 49: 50: volume gl-raid6-replicate-0 51: type cluster/replicate 52: subvolumes gl-raid6-client-0 53: end-volume 54: 55: volume gl-raid6-write-behind 56: type performance/write-behind 57: option cache-size 4mb 58: subvolumes gl-raid6-replicate-0 59: end-volume 60: 61: volume gl-raid6-read-ahead 62: type performance/read-ahead 63: subvolumes gl-raid6-write-behind 64: end-volume 65: 66: volume gl-raid6-io-cache 67: type performance/io-cache 68: option cache-size 268435456 69: subvolumes gl-raid6-read-ahead 70: end-volume 71: 72: volume gl-raid6-quick-read 73: type performance/quick-read 74: option cache-size 268435456 75: subvolumes gl-raid6-io-cache 76: end-volume 77: 78: volume gl-raid6 79: type debug/io-stats 80: option latency-measurement off 81: option count-fop-hits off 82: subvolumes gl-raid6-quick-read 83: end-volume 84: 85: volume nfs-server 86: type nfs/server 87: option nfs.dynamic-volumes on 88: option rpc-auth.addr.gl-raid6.allow * 89: option nfs3.gl-raid6.volume-id 149df6f8-ffbf-44cc-90f0-ee894d6626ba 90: option rpc-auth.addr.gl-raid1.allow * 91: option nfs3.gl-raid1.volume-id f8c1b8c4-8511-40dc-a4eb-f338522b9662 92: option nfs.port 2049 93: option nfs3.gl-raid6.trusted-sync on 94: option nfs3.gl-raid1.trusted-sync on 95: subvolumes gl-raid6 gl-raid1 96: end-volume +------------------------------------------------------------------------------+ [2012-03-12 22:28:53.706616] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-gl-raid6-client-0: changing port to 24009 (from 0) [2012-03-12 22:28:53.706650] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-gl-raid1-client-0: changing port to 24010 (from 0) [2012-03-12 22:28:57.706828] I [client-handshake.c:1090:select_server_supported_programs] 0-gl-raid6-client-0: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2012-03-12 22:28:57.706993] I [client-handshake.c:1090:select_server_supported_programs] 0-gl-raid1-client-0: Using Program GlusterFS 3.2.5, Num (1298437), Version (310) [2012-03-12 22:28:57.708254] I [client-handshake.c:913:client_setvolume_cbk] 0-gl-raid6-client-0: Connected to 192.168.10.1:24009, attached to remote volume '/export/gluster/R [2012-03-12 22:28:57.708296] I [afr-common.c:3141:afr_notify] 0-gl-raid6-replicate-0: Subvolume 'gl-raid6-client-0' came back up; going online. [2012-03-12 22:28:57.708447] I [client-handshake.c:913:client_setvolume_cbk] 0-gl-raid1-client-0: Connected to 192.168.20.1:24010, attached to remote volume '/export/gluster/R [2012-03-12 22:28:57.708464] I [afr-common.c:3141:afr_notify] 0-gl-raid1-replicate-0: Subvolume 'gl-raid1-client-0' came back up; going online. [2012-03-12 22:28:57.708568] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-gl-raid6-replicate-0: added root inode [2012-03-12 22:28:57.708635] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-gl-raid1-replicate-0: added root inode [2012-04-01 13:34:05.870006] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message [2012-04-01 13:34:07.715548] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed [2012-04-01 13:34:07.715720] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message [2012-04-01 13:34:07.715742] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed [2012-04-01 13:34:07.715806] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message [2012-04-01 13:34:07.715825] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed [2012-04-01 13:34:07.715886] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message [2012-04-01 13:34:07.715904] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed [2012-04-01 13:34:07.715963] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message [2012-04-01 13:34:07.715981] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed [2012-04-01 13:34:07.716040] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message [2012-04-01 13:34:07.716057] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed [2012-04-01 13:34:07.716117] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message [2012-04-01 13:34:07.716135] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed [2012-04-01 13:34:19.146413] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message [2012-04-01 13:34:19.146463] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed [2012-04-01 13:34:19.146775] E [event.c:703:event_select_on_epoll] 0-epoll: index not found for fd=-1 (idx_hint=0) [2012-04-01 13:34:42.297040] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message [2012-04-01 13:34:42.297081] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed [2012-04-01 13:34:42.297172] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message [2012-04-01 13:34:42.297188] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed [2012-04-01 13:34:42.297246] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message [2012-04-01 13:34:42.297261] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed [2012-04-01 13:34:42.297310] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message [2012-04-01 13:34:42.297325] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed [2012-04-01 13:34:42.297373] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message [2012-04-01 13:34:42.297387] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed [2012-04-01 13:34:42.297438] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message [2012-04-01 13:34:42.297451] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed [2012-04-01 13:34:42.297499] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message
