+glusterd folks. Pranith On 11/25/2014 02:41 AM, Scott Merrill wrote:
After upgrading to Gluster 3.6.1, I'm seeing a lot more (stale?) locks between my replicated servers. This prevents me from executing commands on either server. From one server: root@gluster1:PRODUCTION:~> gluster volume status epa Locking failed on gluster2.domain.local. Please check log file for details. root@gluster1:PRODUCTION:~> gluster volume status store Locking failed on gluster2.domain.local. Please check log file for details. From the server that reports the lock: root@gluster2:PRODUCTION:~> gluster volume status epa Another transaction is in progress. Please try again after sometime. root@gluster2:PRODUCTION:~> gluster volume status store Another transaction is in progress. Please try again after sometime. I dutifully consulted the logs, as instructed, first from one server: root@gluster1:PRODUCTION:~> tail /var/log/glusterfs/etc-glusterfs-glusterd.vol.log [2014-11-24 21:06:34.402067] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed (Invalid argument) [2014-11-24 21:06:37.402400] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed (Invalid argument) [2014-11-24 21:06:40.402722] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed (Invalid argument) [2014-11-24 21:06:43.403091] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed (Invalid argument) [2014-11-24 21:06:46.403416] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed (Invalid argument) [2014-11-24 21:06:49.403757] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed (Invalid argument) [2014-11-24 21:06:52.404099] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed (Invalid argument) [2014-11-24 21:06:55.404440] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed (Invalid argument) [2014-11-24 21:06:58.404770] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed (Invalid argument) The message "I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has disconnected from glusterd." repeated 39 times between [2014-11-24 21:05:01.391554] and [2014-11-24 21:06:58.405175] and then from the other: root@gluster2:PRODUCTION:~> tail /var/log/glusterfs/etc-glusterfs-glusterd.vol.log [2014-11-24 21:07:23.532144] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed (Invalid argument) [2014-11-24 21:07:26.532426] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed (Invalid argument) [2014-11-24 21:07:29.532721] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed (Invalid argument) [2014-11-24 21:07:32.533014] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed (Invalid argument) [2014-11-24 21:07:35.533368] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed (Invalid argument) [2014-11-24 21:07:38.533669] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed (Invalid argument) [2014-11-24 21:07:41.533998] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed (Invalid argument) [2014-11-24 21:07:44.534330] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed (Invalid argument) [2014-11-24 21:07:47.534668] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed (Invalid argument) [2014-11-24 21:07:50.534984] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed (Invalid argument) (The NFS message from the first server is interesting, since all of my volumes explicitly declare "nfs.disable: true".) I can release the lock by `service glusterd restart`, but that seems sub-optimal. It's terribly manual; and eventually some other lock will stick, reproducing this situation. How can I diagnose what is locking, and what can I do to remedy this? Thanks! Scott _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
