To be more precious the hang is clearly seen when there is some IO(write) to the mount point. Even rm -rf takes time to clear the files. Below, time command showing the delay. Typically it should take less then a second, but glusterfs take more than 5seconds just to list 32x 2GB files. [root@client-host ~]# time ls -l /mnt/gluster/ total 34575680 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.0.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.1.0 -rw-r--r--. 1 root root 2147454976 Aug 10 12:23 rand.10.0 -rw-r--r--. 1 root root 2147463168 Aug 10 12:23 rand.11.0 -rw-r--r--. 1 root root 2147467264 Aug 10 12:23 rand.12.0 -rw-r--r--. 1 root root 2147475456 Aug 10 12:23 rand.13.0 -rw-r--r--. 1 root root 2147479552 Aug 10 12:23 rand.14.0 -rw-r--r--. 1 root root 2147479552 Aug 10 12:23 rand.15.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.16.0 -rw-r--r--. 1 root root 2147479552 Aug 10 12:23 rand.17.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.18.0 -rw-r--r--. 1 root root 2147467264 Aug 10 12:23 rand.19.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.2.0 -rw-r--r--. 1 root root 2147475456 Aug 10 12:23 rand.20.0 -rw-r--r--. 1 root root 2147479552 Aug 10 12:23 rand.21.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.22.0 -rw-r--r--. 1 root root 2147459072 Aug 10 12:23 rand.23.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.24.0 -rw-r--r--. 1 root root 2147471360 Aug 10 12:23 rand.25.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.26.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.27.0 -rw-r--r--. 1 root root 2147479552 Aug 10 12:23 rand.28.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.29.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.3.0 -rw-r--r--. 1 root root 2147442688 Aug 10 12:23 rand.30.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.31.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.4.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.5.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.6.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.7.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.8.0 -rw-r--r--. 1 root root 2147483648 Aug 10 12:23 rand.9.0 real 0m7.478s user 0m0.001s sys 0m0.005s [root@client-host ~]# -- Deepak -----Original Message----- From: gluster-users-bounces@xxxxxxxxxxx [mailto:gluster-users-bounces@xxxxxxxxxxx] On Behalf Of Deepak Naidu Sent: Wednesday, August 10, 2016 2:18 PM To: Vijay Bellur Cc: gluster-users@xxxxxxxxxxx Subject: Re: Linux (ls -l) command pauses/slow on GlusterFS mounts * PGP Signed: 08/10/2016 at 02:18:22 PM, Decrypted I did strace & its waiting on IO. -- Deepak -----Original Message----- From: Vijay Bellur [mailto:vbellur@xxxxxxxxxx] Sent: Wednesday, August 10, 2016 2:17 PM To: Deepak Naidu Cc: gluster-users@xxxxxxxxxxx Subject: Re: Linux (ls -l) command pauses/slow on GlusterFS mounts On 08/10/2016 05:12 PM, Deepak Naidu wrote: > Before we can try physical we wanted POC on VM. > > Just a note the VMs are decently powerful 18cpus, 10gig NIC, 45GB Ram 1TB SSD drives. This is per node spec. > > I don't see the ls -l command hanging when I try to list the files from the gluster-node VMs itself So the question. The reason I alluded to a physical setup was to remove the variables that can affect performance in a virtual setup. The behavior is not usual for the scale of deployment that you mention. You could use strace in conjunction with gluster volume profile to determine where the latency is stemming from. Regards, Vijay > > -- > Deepak > >> On Aug 10, 2016, at 2:01 PM, Vijay Bellur <vbellur@xxxxxxxxxx> wrote: >> >>> On 08/10/2016 04:54 PM, Deepak Naidu wrote: >>> Anyone who has seen the issue in their env ? >> >> >>> -- >>> Deepak >>> >>> -----Original Message----- >>> From: gluster-users-bounces@xxxxxxxxxxx >>> [mailto:gluster-users-bounces@xxxxxxxxxxx] On Behalf Of Deepak Naidu >>> Sent: Tuesday, August 09, 2016 9:14 PM >>> To: gluster-users@xxxxxxxxxxx >>> Subject: Linux (ls -l) command pauses/slow on >>> GlusterFS mounts >>> >>> Greetings, >>> >>> I have 3node GlusterFS on VM for POC each node has 2x bricks of 200GB. Regardless of what type of volume I create when listing files under directory using ls command the GlusterFS mount hangs pauses for few seconds. This is same if there're 2-5 19gb file each or 2gb file each. There are less than 10 files under the GlusterFS mount. >>> >>> I am using NFS-Ganesha for NFS server with GlusterFS and the Linux client is mounted using GlusterFS fuse mount with direct-io enabled. >>> >>> GlusterFS version 3.8(latest) >>> >>> >>> Any insight is appreciated. >> >> This does not seem usual for the deployment that you describe. Can you try on a physical setup to see if the same behavior is observed? >> >> -Vijay >> >> > ---------------------------------------------------------------------- > ------------- This email message is for the sole use of the intended > recipient(s) and may contain confidential information. Any > unauthorized review, use, disclosure or distribution is prohibited. > If you are not the intended recipient, please contact the sender by > reply email and destroy all copies of the original message. > ---------------------------------------------------------------------- > ------------- > _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users * Deepak Naidu <dnaidu@xxxxxxxxxx> * 0x15098040 _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
