I haven't heard anything from anyone re this post yet so I have a bit more info to add. The last time I posted about extremely high process utilization for glusterfs and glusterfsd, I was asked to see if I could capture the output of a profile. I did so and here is the output of 2 captures (they appear very similar). Perhaps this will assist. Note that previously I was mounting with the fuse client. This time I tried NFS.
Profile Output 1:
Brick: 192.168.10.92:/export/brick1
-----------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 133 6 37
Block Size: 8b+ 16b+ 32b+
No. of Reads: 28 93 31
No. of Writes: 4576 6494 22228
Block Size: 64b+ 128b+ 256b+
No. of Reads: 217 216 254
No. of Writes: 37627 41711 20519
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 484 978 684
No. of Writes: 39094 42589 203747
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1073 1800 2838
No. of Writes: 220154 229725 215083
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 4224 14941 57812
No. of Writes: 60938 61967 793400
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 4 4 2
No. of Writes: 5 0 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1835064 FORGET
0.00 0.00 us 0.00 us 0.00 us 2639321 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1439759 RELEASEDIR
0.00 153.50 us 142.00 us 165.00 us 2 XATTROP
0.00 102.67 us 97.00 us 110.00 us 3 SETATTR
0.00 93.20 us 53.00 us 129.00 us 5 GETXATTR
0.00 55.07 us 33.00 us 100.00 us 15 INODELK
0.00 46.61 us 35.00 us 69.00 us 18 STATFS
0.01 48.40 us 31.00 us 64.00 us 25 LK
0.01 50.72 us 32.00 us 88.00 us 47 FINODELK
0.01 35.49 us 17.00 us 68.00 us 86 FLUSH
0.02 50.59 us 27.00 us 76.00 us 68 ENTRYLK
0.03 75.72 us 47.00 us 112.00 us 74 OPEN
0.06 58.79 us 32.00 us 100.00 us 222 OPENDIR
0.08 1535.92 us 129.00 us 16583.00 us 12 UNLINK
0.11 24814.00 us 24814.00 us 24814.00 us 1 TRUNCATE
0.12 34.25 us 11.00 us 1950.00 us 784 ACCESS
0.13 603.50 us 90.00 us 21829.00 us 46 FXATTROP
0.16 7225.40 us 63.00 us 14563.00 us 5 READLINK
0.17 437.90 us 54.00 us 19042.00 us 83 READ
0.36 85.92 us 56.00 us 313.00 us 912 FSTAT
0.42 59.86 us 22.00 us 1443.00 us 1533 STAT
0.45 4318.65 us 70.00 us 56839.00 us 23 WRITE
0.84 8405.18 us 161.00 us 37688.00 us 22 CREATE
8.33 2005.77 us 361.00 us 2796.00 us 912 READDIRP
88.68 921.10 us 34.00 us 159393.00 us 21137 LOOKUP
Duration: 338151 seconds
Data Read: 9290348715 bytes
Data Written: 122127653133 bytes
Interval 1 Stats:
Block Size: 8b+ 32b+ 64b+
No. of Reads: 2 0 1
No. of Writes: 0 1 4
Block Size: 128b+ 256b+ 512b+
No. of Reads: 1 2 5
No. of Writes: 3 0 0
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 7 13 11
No. of Writes: 6 0 0
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 10 2 9
No. of Writes: 0 0 0
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 1 1 1
No. of Writes: 0 0 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 7 FORGET
0.00 0.00 us 0.00 us 0.00 us 45 RELEASE
0.00 0.00 us 0.00 us 0.00 us 177 RELEASEDIR
0.00 109.00 us 109.00 us 109.00 us 1 GETXATTR
0.00 110.00 us 110.00 us 110.00 us 1 SETATTR
0.00 52.78 us 33.00 us 81.00 us 9 INODELK
0.00 48.92 us 36.00 us 69.00 us 12 STATFS
0.01 146.80 us 129.00 us 171.00 us 5 UNLINK
0.01 47.89 us 31.00 us 63.00 us 18 LK
0.01 34.25 us 17.00 us 61.00 us 40 FLUSH
0.01 49.89 us 35.00 us 64.00 us 28 FINODELK
0.02 49.12 us 27.00 us 76.00 us 42 ENTRYLK
0.02 78.14 us 47.00 us 99.00 us 29 OPEN
0.08 58.01 us 34.00 us 100.00 us 177 OPENDIR
0.13 241.08 us 54.00 us 10214.00 us 66 READ
0.18 34.76 us 11.00 us 1950.00 us 649 ACCESS
0.21 906.82 us 94.00 us 21829.00 us 28 FXATTROP
0.29 9014.75 us 63.00 us 14563.00 us 4 READLINK
0.32 85.78 us 56.00 us 313.00 us 455 FSTAT
0.59 60.01 us 22.00 us 1443.00 us 1215 STAT
0.67 5906.21 us 70.00 us 56839.00 us 14 WRITE
1.02 7854.81 us 161.00 us 27303.00 us 16 CREATE
7.42 2008.99 us 361.00 us 2796.00 us 455 READDIRP
89.00 1052.36 us 37.00 us 132924.00 us 10418 LOOKUP
Duration: 18 seconds
Data Read: 1285736 bytes
Data Written: 8465 bytes
Brick: 192.168.10.90:/export/brick1
-----------------------------------
Cumulative Stats:
Block Size: 2b+ 8b+ 16b+
No. of Reads: 0 6 32
No. of Writes: 1 1928 2604
Block Size: 32b+ 64b+ 128b+
No. of Reads: 21 112 169
No. of Writes: 6983 12066 26008
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 85 299 809
No. of Writes: 743 18584 7209
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 790 1418 2884
No. of Writes: 1650 10647 16901
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 4588 5856 32428
No. of Writes: 2974 5758 3387
Block Size: 131072b+
No. of Reads: 455199
No. of Writes: 4601
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 604780 FORGET
0.00 0.00 us 0.00 us 0.00 us 1410794 RELEASE
0.00 0.00 us 0.00 us 0.00 us 989700 RELEASEDIR
0.00 45.50 us 29.00 us 62.00 us 2 READDIR
0.00 58.00 us 44.00 us 77.00 us 4 GETXATTR
0.00 92.00 us 86.00 us 101.00 us 3 SETATTR
0.00 147.50 us 136.00 us 159.00 us 2 XATTROP
0.00 55.89 us 41.00 us 84.00 us 9 FSTAT
0.00 54.22 us 33.00 us 76.00 us 18 STATFS
0.00 46.04 us 30.00 us 60.00 us 25 LK
0.01 172.17 us 115.00 us 222.00 us 12 UNLINK
0.01 38.75 us 15.00 us 150.00 us 56 FLUSH
0.01 54.46 us 30.00 us 101.00 us 46 FINODELK
0.01 52.78 us 27.00 us 91.00 us 68 ENTRYLK
0.01 93.74 us 49.00 us 352.00 us 43 OPEN
0.01 204.50 us 150.00 us 511.00 us 22 CREATE
0.03 58.25 us 3.00 us 94.00 us 183 OPENDIR
0.03 247.33 us 73.00 us 5551.00 us 45 FXATTROP
0.11 35786.00 us 35786.00 us 35786.00 us 1 TRUNCATE
0.15 669.42 us 41.00 us 23631.00 us 76 READLINK
0.17 3501.75 us 60.00 us 27637.00 us 16 READ
0.34 5093.86 us 60.00 us 69136.00 us 22 WRITE
4.56 100248.87 us 25.00 us 1502989.00 us 15 INODELK
9.64 1930.07 us 30.00 us 3404.00 us 1646 READDIRP
84.89 1367.04 us 34.00 us 828308.00 us 20463 LOOKUP
Duration: 76033 seconds
Data Read: 63112532714 bytes
Data Written: 1489661828 bytes
Interval 1 Stats:
Block Size: 32b+ 64b+ 128b+
No. of Reads: 0 0 0
No. of Writes: 1 4 3
Block Size: 1024b+ 65536b+ 131072b+
No. of Reads: 0 2 1
No. of Writes: 6 0 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 9 FORGET
0.00 0.00 us 0.00 us 0.00 us 45 RELEASE
0.00 0.00 us 0.00 us 0.00 us 177 RELEASEDIR
0.00 86.00 us 86.00 us 86.00 us 1 SETATTR
0.00 55.89 us 41.00 us 84.00 us 9 FSTAT
0.00 53.50 us 40.00 us 76.00 us 12 STATFS
0.00 160.00 us 130.00 us 184.00 us 5 UNLINK
0.00 46.83 us 30.00 us 60.00 us 18 LK
0.01 51.11 us 30.00 us 88.00 us 28 FINODELK
0.01 37.88 us 15.00 us 150.00 us 40 FLUSH
0.01 52.14 us 27.00 us 83.00 us 42 ENTRYLK
0.01 101.34 us 52.00 us 352.00 us 29 OPEN
0.02 209.44 us 150.00 us 511.00 us 16 CREATE
0.04 319.57 us 73.00 us 5551.00 us 28 FXATTROP
0.05 58.58 us 29.00 us 94.00 us 177 OPENDIR
0.23 669.42 us 41.00 us 23631.00 us 76 READLINK
0.26 3728.27 us 60.00 us 27637.00 us 15 READ
0.37 5742.93 us 60.00 us 69136.00 us 14 WRITE
6.88 167043.89 us 25.00 us 1502989.00 us 9 INODELK
10.95 1908.32 us 30.00 us 3404.00 us 1254 READDIRP
81.16 1701.62 us 34.00 us 828308.00 us 10421 LOOKUP
Duration: 18 seconds
Data Read: 310621 bytes
Data Written: 8465 bytes
Brick: 192.168.10.91:/export/brick1
-----------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 3 3 6
Block Size: 8b+ 16b+ 32b+
No. of Reads: 13 173 58
No. of Writes: 3370 5495 13458
Block Size: 64b+ 128b+ 256b+
No. of Reads: 194 280 500
No. of Writes: 33703 34265 7020
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1302 1888 3596
No. of Writes: 26022 17086 12495
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 5031 13204 21838
No. of Writes: 21889 32294 18289
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 73578 190508 2710663
No. of Writes: 17017 13980 144272
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 817553 FORGET
0.00 0.00 us 0.00 us 0.00 us 1663792 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1438610 RELEASEDIR
0.00 85.00 us 85.00 us 85.00 us 1 READDIR
0.00 58.50 us 56.00 us 60.00 us 4 FSTAT
0.00 34.86 us 24.00 us 46.00 us 7 STAT
0.00 89.67 us 81.00 us 100.00 us 3 SETATTR
0.00 136.50 us 108.00 us 165.00 us 2 XATTROP
0.00 63.20 us 35.00 us 87.00 us 5 GETXATTR
0.00 54.73 us 39.00 us 66.00 us 15 INODELK
0.01 53.39 us 33.00 us 88.00 us 18 STATFS
0.01 58.00 us 32.00 us 259.00 us 25 LK
0.01 208.50 us 112.00 us 691.00 us 12 UNLINK
0.02 60.76 us 40.00 us 125.00 us 46 FINODELK
0.02 176.63 us 74.00 us 302.00 us 19 READ
0.02 52.51 us 23.00 us 97.00 us 68 ENTRYLK
0.02 48.00 us 18.00 us 579.00 us 81 FLUSH
0.03 206.95 us 163.00 us 497.00 us 22 CREATE
0.04 85.16 us 51.00 us 168.00 us 70 OPEN
0.04 145.02 us 92.00 us 344.00 us 45 FXATTROP
0.07 59.91 us 3.00 us 145.00 us 184 OPENDIR
0.13 1024.41 us 61.00 us 12251.00 us 22 WRITE
0.14 444.38 us 44.00 us 11184.00 us 52 READLINK
0.27 45104.00 us 45104.00 us 45104.00 us 1 TRUNCATE
20.33 1895.33 us 27.00 us 6083.00 us 1819 READDIRP
78.84 645.59 us 35.00 us 209082.00 us 20706 LOOKUP
Duration: 269212 seconds
Data Read: 376538217873 bytes
Data Written: 21968843126 bytes
Interval 1 Stats:
Block Size: 32b+ 64b+ 128b+
No. of Reads: 0 0 0
No. of Writes: 1 4 3
Block Size: 1024b+ 32768b+ 131072b+
No. of Reads: 0 1 10
No. of Writes: 6 0 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8 FORGET
0.00 0.00 us 0.00 us 0.00 us 45 RELEASE
0.00 0.00 us 0.00 us 0.00 us 178 RELEASEDIR
0.00 81.00 us 81.00 us 81.00 us 1 SETATTR
0.00 85.00 us 85.00 us 85.00 us 1 READDIR
0.00 63.33 us 35.00 us 87.00 us 3 GETXATTR
0.00 58.50 us 56.00 us 60.00 us 4 FSTAT
0.00 34.86 us 24.00 us 46.00 us 7 STAT
0.00 54.00 us 47.00 us 66.00 us 9 INODELK
0.01 51.25 us 33.00 us 78.00 us 12 STATFS
0.01 156.20 us 112.00 us 202.00 us 5 UNLINK
0.01 61.33 us 34.00 us 259.00 us 18 LK
0.02 61.64 us 40.00 us 125.00 us 28 FINODELK
0.02 54.65 us 21.00 us 579.00 us 40 FLUSH
0.02 54.12 us 28.00 us 97.00 us 42 ENTRYLK
0.02 161.93 us 74.00 us 302.00 us 15 READ
0.02 83.97 us 68.00 us 107.00 us 29 OPEN
0.03 214.88 us 163.00 us 497.00 us 16 CREATE
0.04 144.21 us 92.00 us 344.00 us 28 FXATTROP
0.10 60.01 us 3.00 us 145.00 us 178 OPENDIR
0.10 316.62 us 44.00 us 8512.00 us 34 READLINK
0.15 1121.14 us 61.00 us 12251.00 us 14 WRITE
32.97 1895.33 us 27.00 us 6083.00 us 1819 READDIRP
66.46 667.09 us 35.00 us 67882.00 us 10418 LOOKUP
Duration: 18 seconds
Data Read: 1352163 bytes
Data Written: 8465 bytes
Profile 2:
Brick: 192.168.10.92:/export/brick1
-----------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 133 6 37
Block Size: 8b+ 16b+ 32b+
No. of Reads: 28 93 31
No. of Writes: 4577 6495 22231
Block Size: 64b+ 128b+ 256b+
No. of Reads: 219 218 254
No. of Writes: 37641 41723 20519
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 484 981 696
No. of Writes: 39094 42613 203748
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1079 1802 2840
No. of Writes: 220154 229725 215083
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 4227 14944 57816
No. of Writes: 60938 61967 793400
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 6 5 2
No. of Writes: 5 0 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1835089 FORGET
0.00 0.00 us 0.00 us 0.00 us 2639439 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1440418 RELEASEDIR
0.00 53.50 us 43.00 us 64.00 us 2 READDIR
0.00 153.50 us 142.00 us 165.00 us 2 XATTROP
0.00 113.00 us 97.00 us 144.00 us 4 SETATTR
0.00 114.60 us 49.00 us 339.00 us 10 GETXATTR
0.00 55.00 us 33.00 us 100.00 us 24 INODELK
0.00 48.85 us 25.00 us 81.00 us 71 STATFS
0.01 49.47 us 31.00 us 76.00 us 92 LK
0.01 36.07 us 16.00 us 68.00 us 180 FLUSH
0.01 52.45 us 32.00 us 88.00 us 159 FINODELK
0.02 79.16 us 47.00 us 112.00 us 137 OPEN
0.02 51.06 us 26.00 us 95.00 us 216 ENTRYLK
0.03 24814.00 us 24814.00 us 24814.00 us 1 TRUNCATE
0.06 271.36 us 86.00 us 21829.00 us 158 FXATTROP
0.07 59.02 us 2.00 us 100.00 us 881 OPENDIR
0.07 5910.00 us 56.00 us 16873.00 us 9 READLINK
0.11 33.81 us 11.00 us 1950.00 us 2348 ACCESS
0.17 4181.53 us 129.00 us 41541.00 us 30 UNLINK
0.31 2788.77 us 59.00 us 56839.00 us 79 WRITE
0.37 2096.99 us 54.00 us 61234.00 us 125 READ
0.38 84.58 us 53.00 us 313.00 us 3191 FSTAT
0.50 61.21 us 20.00 us 14904.00 us 5873 STAT
0.76 7023.69 us 145.00 us 102353.00 us 78 CREATE
8.90 2002.80 us 361.00 us 3545.00 us 3191 READDIRP
88.19 1112.81 us 34.00 us 307904.00 us 56876 LOOKUP
Duration: 338222 seconds
Data Read: 9292898491 bytes
Data Written: 122127690284 bytes
Interval 4 Stats:
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 2
No. of Writes: 1 3 12
Block Size: 128b+ 1024b+ 2048b+
No. of Reads: 2 3 10
No. of Writes: 8 16 0
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 4 2 1
No. of Writes: 0 0 0
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 2 3 1
No. of Writes: 0 0 0
Block Size: 262144b+
No. of Reads: 2
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 21 FORGET
0.00 0.00 us 0.00 us 0.00 us 76 RELEASE
0.00 0.00 us 0.00 us 0.00 us 407 RELEASEDIR
0.00 144.00 us 144.00 us 144.00 us 1 SETATTR
0.00 63.33 us 56.00 us 69.00 us 3 READLINK
0.00 226.00 us 113.00 us 339.00 us 2 GETXATTR
0.00 54.89 us 39.00 us 70.00 us 9 INODELK
0.01 48.03 us 25.00 us 72.00 us 37 STATFS
0.01 36.75 us 16.00 us 65.00 us 59 FLUSH
0.01 50.44 us 33.00 us 76.00 us 48 LK
0.01 83.73 us 55.00 us 111.00 us 37 OPEN
0.01 53.79 us 32.00 us 80.00 us 80 FINODELK
0.02 50.55 us 26.00 us 83.00 us 104 ENTRYLK
0.03 135.21 us 86.00 us 228.00 us 80 FXATTROP
0.07 58.75 us 35.00 us 98.00 us 407 OPENDIR
0.10 1044.22 us 55.00 us 30268.00 us 32 READ
0.14 33.74 us 13.00 us 1044.00 us 1384 ACCESS
0.19 5076.42 us 141.00 us 41541.00 us 12 UNLINK
0.36 2860.70 us 59.00 us 29520.00 us 40 WRITE
0.36 84.16 us 53.00 us 182.00 us 1369 FSTAT
0.65 5225.18 us 145.00 us 28783.00 us 40 CREATE
0.73 61.98 us 20.00 us 14904.00 us 3771 STAT
8.53 2009.18 us 518.00 us 2532.00 us 1369 READDIRP
88.77 1156.14 us 36.00 us 232288.00 us 24747 LOOKUP
Duration: 48 seconds
Data Read: 1424095 bytes
Data Written: 22682 bytes
Brick: 192.168.10.91:/export/brick1
-----------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 3 3 6
Block Size: 8b+ 16b+ 32b+
No. of Reads: 13 173 58
No. of Writes: 3371 5496 13461
Block Size: 64b+ 128b+ 256b+
No. of Reads: 194 280 500
No. of Writes: 33717 34277 7020
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1302 1888 3596
No. of Writes: 26022 17110 12496
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 5031 13204 21838
No. of Writes: 21889 32294 18289
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 73579 190511 2710676
No. of Writes: 17017 13980 144272
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 817592 FORGET
0.00 0.00 us 0.00 us 0.00 us 1663910 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1439269 RELEASEDIR
0.00 61.67 us 39.00 us 85.00 us 3 READDIR
0.00 136.50 us 108.00 us 165.00 us 2 XATTROP
0.00 91.00 us 81.00 us 100.00 us 4 SETATTR
0.00 58.86 us 35.00 us 87.00 us 7 GETXATTR
0.00 60.57 us 55.00 us 69.00 us 7 FSTAT
0.00 35.53 us 24.00 us 53.00 us 15 STAT
0.00 57.12 us 39.00 us 72.00 us 24 INODELK
0.01 51.25 us 33.00 us 90.00 us 71 STATFS
0.01 52.09 us 32.00 us 259.00 us 92 LK
0.01 175.87 us 112.00 us 691.00 us 30 UNLINK
0.01 41.84 us 16.00 us 579.00 us 175 FLUSH
0.02 56.23 us 25.00 us 125.00 us 158 FINODELK
0.02 51.59 us 23.00 us 97.00 us 216 ENTRYLK
0.02 85.10 us 51.00 us 168.00 us 133 OPEN
0.03 200.55 us 150.00 us 554.00 us 78 CREATE
0.06 163.03 us 38.00 us 11184.00 us 204 READLINK
0.07 1022.60 us 67.00 us 24446.00 us 40 READ
0.08 45104.00 us 45104.00 us 45104.00 us 1 TRUNCATE
0.09 59.52 us 3.00 us 145.00 us 843 OPENDIR
0.13 936.73 us 56.00 us 12251.00 us 78 WRITE
0.19 687.17 us 88.00 us 85925.00 us 157 FXATTROP
22.66 1881.29 us 23.00 us 6083.00 us 6698 READDIRP
76.59 754.63 us 33.00 us 209082.00 us 56449 LOOKUP
Duration: 269283 seconds
Data Read: 376540270509 bytes
Data Written: 21968880277 bytes
Interval 4 Stats:
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 1 3 12
Block Size: 128b+ 1024b+ 32768b+
No. of Reads: 0 0 1
No. of Writes: 8 16 0
Block Size: 65536b+ 131072b+
No. of Reads: 2 9
No. of Writes: 0 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 36 FORGET
0.00 0.00 us 0.00 us 0.00 us 76 RELEASE
0.00 0.00 us 0.00 us 0.00 us 408 RELEASEDIR
0.00 95.00 us 95.00 us 95.00 us 1 SETATTR
0.00 48.00 us 39.00 us 57.00 us 2 GETXATTR
0.00 50.00 us 39.00 us 61.00 us 2 READDIR
0.00 35.50 us 27.00 us 53.00 us 4 STAT
0.00 63.33 us 55.00 us 69.00 us 3 FSTAT
0.00 61.11 us 50.00 us 72.00 us 9 INODELK
0.01 50.41 us 34.00 us 90.00 us 37 STATFS
0.01 160.08 us 125.00 us 213.00 us 12 UNLINK
0.01 37.19 us 18.00 us 71.00 us 59 FLUSH
0.01 48.46 us 32.00 us 78.00 us 48 LK
0.01 84.92 us 57.00 us 118.00 us 37 OPEN
0.02 54.89 us 25.00 us 89.00 us 80 FINODELK
0.02 50.57 us 27.00 us 74.00 us 104 ENTRYLK
0.03 66.88 us 38.00 us 100.00 us 113 READLINK
0.03 196.90 us 150.00 us 554.00 us 40 CREATE
0.04 139.06 us 88.00 us 233.00 us 80 FXATTROP
0.10 58.95 us 8.00 us 93.00 us 408 OPENDIR
0.11 1675.56 us 67.00 us 24446.00 us 16 READ
0.17 1066.62 us 62.00 us 8506.00 us 40 WRITE
23.14 1889.02 us 23.00 us 3775.00 us 3073 READDIRP
76.29 773.40 us 33.00 us 135928.00 us 24746 LOOKUP
Duration: 48 seconds
Data Read: 1446611 bytes
Data Written: 22682 bytes
Brick: 192.168.10.90:/export/brick1
-----------------------------------
Cumulative Stats:
Block Size: 2b+ 8b+ 16b+
No. of Reads: 0 6 32
No. of Writes: 1 1929 2605
Block Size: 32b+ 64b+ 128b+
No. of Reads: 21 112 169
No. of Writes: 6986 12080 26020
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 85 299 809
No. of Writes: 743 18584 7233
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 790 1419 2885
No. of Writes: 1651 10647 16901
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 4588 5858 32434
No. of Writes: 2974 5758 3387
Block Size: 131072b+
No. of Reads: 455218
No. of Writes: 4601
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 604813 FORGET
0.00 0.00 us 0.00 us 0.00 us 1410912 RELEASE
0.00 0.00 us 0.00 us 0.00 us 990360 RELEASEDIR
0.00 44.33 us 16.00 us 62.00 us 6 READDIR
0.00 147.50 us 136.00 us 159.00 us 2 XATTROP
0.00 100.25 us 86.00 us 125.00 us 4 SETATTR
0.00 60.11 us 39.00 us 92.00 us 9 GETXATTR
0.00 60.19 us 41.00 us 84.00 us 16 FSTAT
0.00 45.50 us 17.00 us 182.00 us 38 STAT
0.00 51.37 us 31.00 us 81.00 us 71 STATFS
0.00 49.05 us 30.00 us 79.00 us 92 LK
0.00 168.87 us 115.00 us 287.00 us 30 UNLINK
0.01 38.74 us 15.00 us 150.00 us 150 FLUSH
0.01 54.76 us 30.00 us 236.00 us 158 FINODELK
0.01 87.17 us 49.00 us 352.00 us 106 OPEN
0.01 52.56 us 27.00 us 91.00 us 216 ENTRYLK
0.02 174.25 us 73.00 us 5551.00 us 157 FXATTROP
0.03 35786.00 us 35786.00 us 35786.00 us 1 TRUNCATE
0.05 62.12 us 3.00 us 1628.00 us 843 OPENDIR
0.10 685.58 us 41.00 us 51432.00 us 157 READLINK
0.36 6483.54 us 59.00 us 47225.00 us 63 READ
0.58 8304.77 us 60.00 us 185503.00 us 78 WRITE
0.67 9646.06 us 150.00 us 686579.00 us 78 CREATE
1.82 85338.88 us 25.00 us 1502989.00 us 24 INODELK
14.62 1920.48 us 23.00 us 9822.00 us 8567 READDIRP
81.71 1635.58 us 30.00 us 828308.00 us 56208 LOOKUP
Duration: 76104 seconds
Data Read: 63115678019 bytes
Data Written: 1489698979 bytes
Interval 4 Stats:
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 1 3 12
Block Size: 128b+ 1024b+ 4096b+
No. of Reads: 0 0 1
No. of Writes: 8 16 0
Block Size: 8192b+ 32768b+ 65536b+
No. of Reads: 1 1 6
No. of Writes: 0 0 0
Block Size: 131072b+
No. of Reads: 10
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 29 FORGET
0.00 0.00 us 0.00 us 0.00 us 76 RELEASE
0.00 0.00 us 0.00 us 0.00 us 409 RELEASEDIR
0.00 125.00 us 125.00 us 125.00 us 1 SETATTR
0.00 43.75 us 16.00 us 60.00 us 4 READDIR
0.00 61.80 us 39.00 us 92.00 us 5 GETXATTR
0.00 65.71 us 44.00 us 76.00 us 7 FSTAT
0.00 48.50 us 24.00 us 182.00 us 28 STAT
0.00 50.08 us 31.00 us 70.00 us 37 STATFS
0.00 165.25 us 119.00 us 238.00 us 12 UNLINK
0.00 39.66 us 19.00 us 62.00 us 59 FLUSH
0.00 51.12 us 33.00 us 79.00 us 48 LK
0.01 82.78 us 64.00 us 104.00 us 37 OPEN
0.01 56.45 us 30.00 us 236.00 us 80 FINODELK
0.01 52.17 us 28.00 us 84.00 us 104 ENTRYLK
0.02 154.12 us 88.00 us 1577.00 us 80 FXATTROP
0.05 65.00 us 3.00 us 1628.00 us 409 OPENDIR
0.09 909.64 us 43.00 us 51432.00 us 61 READLINK
0.60 9748.22 us 59.00 us 47225.00 us 36 READ
0.86 12688.17 us 60.00 us 185503.00 us 40 WRITE
0.92 60488.89 us 50.00 us 543898.00 us 9 INODELK
1.26 18623.40 us 161.00 us 686579.00 us 40 CREATE
19.51 1951.60 us 25.00 us 9822.00 us 5889 READDIRP
76.64 1823.96 us 30.00 us 318395.00 us 24747 LOOKUP
Duration: 48 seconds
Data Read: 1931486 bytes
Data Written: 22682 bytes
Again, any assistance on this issue would be much appreciated.
Thank you.
On Sat, Dec 20, 2014 at 11:58 PM, Kyle Harris <kyle.harris98@xxxxxxxxx> wrote:
Hello All,I have been experiencing difficulty with high process utilization on the glusterfs and glusterfsd processes. I thought I had it tracked down to Apache looking for files on the cluster drive that don't exist. I have been working to clean that up and now I have discovered another problem.When I reboot one of the nodes in my 3-node replicated cluster, as soon as it comes back up I see a bunch of entries such as follows:[2014-12-21 05:33:49.873989] I [dict.c:370:dict_get] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7fe3bd0d2420] (--> /usr/lib64/libglusterfs.so.0(dict_get+0x4f)[0x7fe3bd0cbbdf] (--> /usr/lib64/glusterfs/3.6.1/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x1e1)[0x7fe3aad3ada1] (--> /usr/lib64/glusterfs/3.6.1/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x113)[0x7fe3aaf524e3] (--> /usr/lib64/glusterfs/3.6.1/xlator/performance/md-cache.so(mdc_lookup+0x300)[0x7fe3ab167310] ))))) 0-dict: !this || key=system.posix_acl_accessWhile this is occurring, the process utilization for glusterfs and glusterfsd go sky high. Eventually they calm down on their own but the servers are virtually unusable until they do. Note this is on Centos 6.6 with Gluster 3.6.1 and kernel 2.6.32 using a fuse mount with each of the 3 servers having the client mounted for hosting web sites via Apache. There are many small files and some are php. I have not done any tuning to the default gluster parameters.Can someone tell me why this occurs after one of the nodes is rebooted and perhaps lend some guidance on what can be done to resolve this issue?Thank you.--Kyle
Kyle A. Harris
Kyle@xxxxxxxxxxxxxxxxx
615-364-6752
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
