On 01/28/2015 09:24 PM, Paul E Stallworth wrote:
We are running version 3.6.1 on Ubuntu 12.04.5 LTS. The file system for the volume is ext4 and is mounted on clients as glusterfs.
I verified apache wasn't running, and there shouldn't be any other workload on the web server and ran another test using the following:
"dd if=/dev/zero of=/web/ddtest bs=8k count=250000 conv=fdatasync" with the profiler running.
The dd operation took 38 seconds and the two big results from the profiler for this particular brick were:
42.79 202.15 us 27.00 us 1008658.00 us 27649 WRITE
56.14 7332621.00 us 7332621.00 us 7332621.00 us 1 FSYNC
I repeated the test and the results for this brick are similar:
29.60 176.56 us 27.00 us 1008658.00 us 55273 WRITE
62.41 1714773.67 us 10098.00 us 10996032.00 us 12 FSYNC
Could you do the same test on the brick directly and see how much time
it is taking using the following command:
strace -Tc dd if=/dev/zero of=<path-on-brick> bs=8k count=250000
conv=fdatasync
If the times are comparable, there is nothing wrong, otherwise there is
something that needs to be looked into.
Pranith
Thanks,
Paul
Paul Stallworth
Housing Information Technology
University of Colorado Boulder
Boulder, Colorado 80309
T 303.735.6623
M 303.735.6623
-----Original Message-----
From: Pranith Kumar Karampuri [mailto:pkarampu@xxxxxxxxxx]
Sent: Wednesday, January 28, 2015 2:16 AM
To: Paul E Stallworth; gluster-users@xxxxxxxxxxx
Subject: Re: High latency on FSYNC, INODELK, FINODELK
On 01/27/2015 11:29 PM, Paul E Stallworth wrote:
Hello,
I ran a gluster volume profile while doing a dd to read from /dev/zero and write to a file and noticed unusually high latency on the FSYNC, INODELK, and FINODELK operations. This latency seems to correspond with very slow page loads of our website. I'm looking for any help in finding where this latency might be coming from and how we might mitigate it.
FSYNC shouldn't happen in the workload you specified. Are you sure this is the only workload on the volume when you executed 'dd'? What is the version of gluster you are using?
Pranith
Here are the values from the profile
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 110601 FORGET
0.00 0.00 us 0.00 us 0.00 us 104872 RELEASE
0.00 0.00 us 0.00 us 0.00 us 269678 RELEASEDIR
0.00 50.00 us 45.00 us 55.00 us 2 OPEN
0.00 73.50 us 32.00 us 115.00 us 2 FLUSH
0.00 47.00 us 33.00 us 57.00 us 4 ENTRYLK
0.00 226.00 us 226.00 us 226.00 us 1 CREATE
0.00 83.75 us 42.00 us 136.00 us 4 STATFS
0.00 72.80 us 23.00 us 175.00 us 5 READDIR
0.00 54.90 us 15.00 us 97.00 us 10 GETXATTR
0.00 79.19 us 26.00 us 193.00 us 32 FXATTROP
0.00 45.36 us 14.00 us 271.00 us 206 STAT
0.01 66.26 us 2.00 us 586.00 us 352 OPENDIR
0.09 252262.00 us 252262.00 us 252262.00 us 1 UNLINK
1.02 139.54 us 29.00 us 10777.00 us 20263 WRITE
7.98 1843032.83 us 22261.00 us 16023044.00 us 12 FSYNC
8.19 1620932.64 us 27.00 us 17171453.00 us 14 INODELK
9.03 94.52 us 22.00 us 9533.00 us 264526 LOOKUP
73.67 563743.05 us 14.00 us 17173239.00 us 362 FINODELK
Thanks,
Paul
Paul Stallworth
Housing Information Technology
University of Colorado Boulder
Boulder, Colorado 80309
T 303.735.6623
M 303.735.6623
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
