Thanks all for commenting. It turned out to be an easier problem. This is an interactive login node with a lot of users logging in. So, I put in lots of limits to prevent abuse of cpu time, number of processes, open file limits, etc. Account provisioning process mistakenly puts root into a group that has limitations. Unfortunately, the limits enforced by /etc/security/limits.conf is not logged anywhere (so, the debugging process got a lot longer). I found out the limit when I decided to probe into /proc/<PID>/limits To ensure the proper limits, I put the following in mount.glusterfs: ulimit -S -s 10240 ulimit -n 65536 ulimit -t unlimited Thanks all for commenting. Robin On 12/17/12 10:59 AM, Robin, Robin wrote: > Yup. I rsync over to a Gluster mounted (and not the actual brick) via > client over the network. > > Robin > > > From: "Washer, Bryan" <bwasher at netsuite.com <mailto:bwasher at netsuite.com>> > Date: Mon, 17 Dec 2012 10:47:38 -0500 > To: Joe Julian <joe at julianfamily.org <mailto:joe at julianfamily.org>>, > "gluster-users at gluster.org <mailto:gluster-users at gluster.org>" > <gluster-users at gluster.org <mailto:gluster-users at gluster.org>> > Subject: Re: Transport endpoint > > > > Just to make sure we don't miss the obvious?when you say sync'd over to > the mount point?you mean where you did a glusterfs mount and not eh actual > location of the brick on one of the mirrors in your replica. > > Once you setup the volume and start it.you should NEVER write or delete > directly on the backend brick unless you really know what you are doing. > > Bryan > > -----Original Message----- > From: Joe Julian <joe at julianfamily.org <mailto:joe at julianfamily.org>> > Date: Monday, December 17, 2012 9:29 AM > To: "gluster-users at gluster.org <mailto:gluster-users at gluster.org>" > <gluster-users at gluster.org <mailto:gluster-users at gluster.org>> > Subject: Re: Transport endpoint > >>On 12/17/2012 06:56 AM, Robin, Robin wrote: >>> Hi, >>> >>> I've got Gluster error: Transport endpoint not connected. >>> >>> It came up twice after trying to rsync 2 TB filesystem over; it >>> reached about 1.8 TB and got the error. >>> >>> Logs on the server side (on reverse time order): >>> [2012-12-15 00:53:24.747934] I >>> [server-helpers.c:629:server_connection_destroy] >>> 0-RedhawkShared-server: destroyed connection of >>> >>>mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared- >>>client-0-0 >>> [2012-12-15 00:53:24.743459] I [server-helpers.c:474:do_fd_cleanup] >>> 0-RedhawkShared-server: fd cleanup on >>> >>>/mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQ >>>swL >>> [2012-12-15 00:53:24.743430] I >>> [server-helpers.c:330:do_lock_table_cleanup] 0-RedhawkShared-server: >>> finodelk released on >>> >>>/mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQ >>>swL >>> [2012-12-15 00:53:24.743400] I >>> [server-helpers.c:741:server_connection_put] 0-RedhawkShared-server: >>> Shutting down connection >>> >>>mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared- >>>client-0-0 >>> [2012-12-15 00:53:24.743368] I [server.c:685:server_rpc_notify] >>> 0-RedhawkShared-server: disconnecting connectionfrom >>> >>>mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared- >>>client-0-0 >>> [2012-12-15 00:53:24.740055] W [socket.c:195:__socket_rwv] >>> 0-tcp.RedhawkShared-server: readv failed (Connection reset by peer) >>> >>> I can't find relevant logs on the client side. >>> >>> From the logs, can we judge for sure that this is a network reset >>> problem ? >>> >>When you say, "I can't find relevant logs on the client side," do you >>mean that you can't find the log, or that there's nothing in there from >>around the same timestamp? The client log will be in /var/log/glusterfs >>and will be named based on the mountpoint. >>_______________________________________________ >>Gluster-users mailing list >>Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >>http://supercolony.gluster.org/mailman/listinfo/gluster-users >> > > > NOTICE: This email and any attachments may contain confidential and > proprietary information of NetSuite Inc. and is for the sole use of the > intended recipient for the stated purpose. Any improper use or > distribution is prohibited. If you are not the intended recipient, > please notify the sender; do not review, copy or distribute; and > promptly delete or destroy all transmitted information. Please note that > all communications and information transmitted through this email system > may be monitored by NetSuite or its agents and that all incoming email > is automatically scanned by a third party spam and filtering service.
