On 11/09/2013 09:36 PM, Emmanuel Dreyfus wrote:
Hi
On 3.4.1, I had a crash server side because of system limit (heap size). I
restarted, and now the fs is extremely slow.
Here are profile info. Any hint of where is the issue?
With lookups taking bulk of the time, was there any self heal in progress?
Did the situation recover after a while or has it been the same?
-Vijay
silo# gluster volume profile gfs341 info
Brick: silo:/export/wd2a
------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 119 89 207
Block Size: 8b+ 16b+ 32b+
No. of Reads: 5 5 6
No. of Writes: 2685 20559 26884
Block Size: 64b+ 128b+ 256b+
No. of Reads: 183 440 333
No. of Writes: 16666 22414 26828
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 162 87 86
No. of Writes: 47158 39862 46764
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 454 454 740
No. of Writes: 22016 14543 9947
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1897 5631 209278
No. of Writes: 12471 3710 197
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 279478 FORGET
0.00 0.00 us 0.00 us 0.00 us 267232 RELEASE
0.00 0.00 us 0.00 us 0.00 us 28625 RELEASEDIR
0.00 47.67 us 28.00 us 61.00 us 3 STAT
0.00 106.33 us 92.00 us 133.00 us 3 TRUNCATE
0.00 62.33 us 53.00 us 68.00 us 6 WRITE
0.00 97.50 us 65.00 us 124.00 us 4 FSTAT
0.00 81.78 us 46.00 us 109.00 us 9 FTRUNCATE
0.00 101.38 us 27.00 us 451.00 us 8 STATFS
0.00 242.00 us 158.00 us 321.00 us 5 READDIR
0.00 1501.00 us 1501.00 us 1501.00 us 1 SETXATTR
0.00 531.67 us 49.00 us 1483.00 us 3 OPEN
0.03 3865.83 us 1687.00 us 11607.00 us 6 FSYNC
0.06 16270.00 us 6346.00 us 36005.00 us 3 RENAME
0.18 5054.03 us 2822.00 us 22519.00 us 30 FXATTROP
0.23 39790.00 us 21818.00 us 89369.00 us 5 CREATE
0.25 6793.66 us 19.00 us 215339.00 us 32 FINODELK
0.35 2506.14 us 114.00 us 168464.00 us 120 READ
0.43 61041.83 us 3273.00 us 239458.00 us 6 XATTROP
3.07 17162.43 us 51.00 us 431101.00 us 153 SETATTR
8.95 45045.94 us 30.00 us 2065080.00 us 170 GETXATTR
10.82 31385.63 us 17.00 us 2941755.00 us 295 ENTRYLK
16.73 14315.53 us 17.00 us 2502697.00 us 1000 INODELK
58.89 14729.58 us 86.00 us 3603064.00 us 3420 LOOKUP
Duration: 561493 seconds
Data Read: 28085570022 bytes
Data Written: 1503297110 bytes
Interval 3 Stats:
Block Size: 512b+ 131072b+
No. of Reads: 0 40
No. of Writes: 2 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 FORGET
0.00 0.00 us 0.00 us 0.00 us 3 RELEASE
0.00 61.00 us 61.00 us 61.00 us 1 STAT
0.00 63.00 us 63.00 us 63.00 us 1 OPEN
0.00 94.00 us 94.00 us 94.00 us 1 TRUNCATE
0.00 62.50 us 61.00 us 64.00 us 2 WRITE
0.00 68.00 us 44.00 us 92.00 us 2 STATFS
0.00 86.33 us 46.00 us 109.00 us 3 FTRUNCATE
0.00 30.80 us 23.00 us 41.00 us 10 FINODELK
0.00 184.00 us 158.00 us 210.00 us 2 READDIR
0.00 97.50 us 65.00 us 124.00 us 4 FSTAT
0.03 2557.00 us 1691.00 us 3423.00 us 2 FSYNC
0.03 6346.00 us 6346.00 us 6346.00 us 1 RENAME
0.03 166.03 us 121.00 us 278.00 us 40 READ
0.09 8913.00 us 3273.00 us 14553.00 us 2 XATTROP
0.24 4524.00 us 2895.00 us 10112.00 us 10 FXATTROP
0.60 57210.50 us 25052.00 us 89369.00 us 2 CREATE
2.86 11905.35 us 57.00 us 190800.00 us 46 SETATTR
6.60 19771.62 us 396.00 us 608832.00 us 64 GETXATTR
9.39 13431.40 us 19.00 us 971690.00 us 134 ENTRYLK
11.96 6496.44 us 17.00 us 950230.00 us 353 INODELK
68.17 14177.33 us 94.00 us 1380707.00 us 922 LOOKUP
Duration: 28 seconds
Data Read: 5242880 bytes
Data Written: 1800 bytes
Brick: debacle:/export/wd1a
---------------------------
Cumulative Stats:
Block Size: 1b+ 4b+ 8b+
No. of Reads: 0 3 54
No. of Writes: 1 0 4
Block Size: 16b+ 32b+ 64b+
No. of Reads: 1125 1277 837
No. of Writes: 6 7 18
Block Size: 128b+ 256b+ 512b+
No. of Reads: 944 1177 1707
No. of Writes: 31 79 121
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 1436 3085 1828
No. of Writes: 281 596 317
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 983 3788 461
No. of Writes: 289 504 4091
Block Size: 65536b+ 131072b+
No. of Reads: 44771 50582
No. of Writes: 2242 159
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 288178 FORGET
0.00 0.00 us 0.00 us 0.00 us 276611 RELEASE
0.00 0.00 us 0.00 us 0.00 us 29662 RELEASEDIR
0.00 65.50 us 30.00 us 101.00 us 2 STAT
0.00 65.00 us 38.00 us 115.00 us 3 OPEN
0.00 47.62 us 24.00 us 106.00 us 8 STATFS
0.00 53.88 us 32.00 us 72.00 us 8 FTRUNCATE
0.00 26.56 us 10.00 us 84.00 us 18 FINODELK
0.00 24.23 us 8.00 us 63.00 us 22 ENTRYLK
0.00 212.00 us 189.00 us 229.00 us 5 FSYNC
0.01 2144.00 us 2144.00 us 2144.00 us 1 READDIR
0.01 84.46 us 73.00 us 121.00 us 37 READ
0.03 215.22 us 44.00 us 6190.00 us 37 FSTAT
0.09 7455.00 us 2237.00 us 13052.00 us 3 RENAME
0.23 19000.33 us 62.00 us 56797.00 us 3 TRUNCATE
0.47 19886.67 us 1217.00 us 75636.00 us 6 XATTROP
0.50 833.23 us 40.00 us 66927.00 us 151 SETATTR
0.77 4586.33 us 27.00 us 115632.00 us 42 GETXATTR
0.91 14307.44 us 1107.00 us 107253.00 us 16 FXATTROP
2.06 103694.20 us 14683.00 us 447120.00 us 5 CREATE
12.50 2043.20 us 9.00 us 1030204.00 us 1540 INODELK
82.41 3470.41 us 76.00 us 998615.00 us 5976 LOOKUP
Duration: 561508 seconds
Data Read: 11439890349 bytes
Data Written: 393336122 bytes
Interval 3 Stats:
Block Size: 65536b+
No. of Reads: 11
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 FORGET
0.00 0.00 us 0.00 us 0.00 us 3 RELEASE
0.00 30.00 us 30.00 us 30.00 us 1 STAT
0.00 37.50 us 26.00 us 49.00 us 2 STATFS
0.00 115.00 us 115.00 us 115.00 us 1 OPEN
0.01 17.12 us 12.00 us 23.00 us 8 ENTRYLK
0.01 142.00 us 142.00 us 142.00 us 1 TRUNCATE
0.01 56.67 us 39.00 us 69.00 us 3 FTRUNCATE
0.01 32.43 us 13.00 us 84.00 us 7 FINODELK
0.02 209.00 us 189.00 us 229.00 us 2 FSYNC
0.02 48.73 us 48.00 us 51.00 us 11 FSTAT
0.04 84.09 us 76.00 us 117.00 us 11 READ
0.56 13052.00 us 13052.00 us 13052.00 us 1 RENAME
1.59 18707.50 us 14683.00 us 22732.00 us 2 CREATE
2.14 172.60 us 12.00 us 23403.00 us 291 INODELK
3.28 38566.50 us 1497.00 us 75636.00 us 2 XATTROP
4.19 16414.83 us 1176.00 us 85980.00 us 6 FXATTROP
5.02 2619.51 us 45.00 us 66927.00 us 45 SETATTR
83.10 1549.53 us 80.00 us 171269.00 us 1260 LOOKUP
Duration: 27 seconds
Data Read: 1146101 bytes
Data Written: 0 bytes
Brick: hangar:/export/wd3a
--------------------------
Cumulative Stats:
Block Size: 1b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 3 58
Block Size: 16b+ 32b+ 64b+
No. of Reads: 1 0 0
No. of Writes: 1130 1284 855
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 0 0
No. of Writes: 975 1256 1827
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 3 3 32
No. of Writes: 1716 3677 2110
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 25 166 141
No. of Writes: 1248 1042 4118
Block Size: 65536b+ 131072b+
No. of Reads: 904 132762
No. of Writes: 2293 341
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 289743 FORGET
0.00 0.00 us 0.00 us 0.00 us 276611 RELEASE
0.00 0.00 us 0.00 us 0.00 us 29662 RELEASEDIR
0.00 127.67 us 103.00 us 150.00 us 3 TRUNCATE
0.00 80.00 us 42.00 us 127.00 us 8 FTRUNCATE
0.00 276.00 us 78.00 us 800.00 us 4 FSTAT
0.00 550.00 us 69.00 us 1512.00 us 3 OPEN
0.00 174.13 us 121.00 us 258.00 us 15 READDIR
0.00 155.94 us 21.00 us 1508.00 us 18 FINODELK
0.03 15207.67 us 6407.00 us 32591.00 us 3 RENAME
0.03 9230.33 us 2991.00 us 26321.00 us 6 XATTROP
0.07 1779.54 us 150.00 us 33052.00 us 67 READ
0.29 65312.50 us 38.00 us 226590.00 us 8 STATFS
0.30 106863.80 us 25134.00 us 403476.00 us 5 CREATE
0.69 77226.81 us 2748.00 us 773357.00 us 16 FXATTROP
0.71 255734.60 us 334.00 us 1032546.00 us 5 FSYNC
1.52 18027.92 us 51.00 us 863671.00 us 151 SETATTR
3.85 12032.38 us 40.00 us 2023922.00 us 574 GETXATTR
11.50 21288.11 us 15.00 us 4791081.00 us 969 ENTRYLK
23.47 16990.93 us 18.00 us 3870756.00 us 2479 INODELK
57.55 22760.43 us 94.00 us 4753189.00 us 4537 LOOKUP
Duration: 561502 seconds
Data Read: 17497120878 bytes
Data Written: 470443381 bytes
Interval 3 Stats:
Block Size: 131072b+
No. of Reads: 24
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 FORGET
0.00 0.00 us 0.00 us 0.00 us 3 RELEASE
0.00 69.00 us 69.00 us 69.00 us 1 OPEN
0.00 51.00 us 38.00 us 64.00 us 2 STATFS
0.00 150.00 us 150.00 us 150.00 us 1 TRUNCATE
0.00 88.33 us 42.00 us 127.00 us 3 FTRUNCATE
0.00 200.33 us 143.00 us 254.00 us 3 READDIR
0.00 276.00 us 78.00 us 800.00 us 4 FSTAT
0.00 346.14 us 24.00 us 1508.00 us 7 FINODELK
0.01 6407.00 us 6407.00 us 6407.00 us 1 RENAME
0.03 9475.00 us 5680.00 us 13270.00 us 2 XATTROP
0.11 2800.33 us 154.00 us 19781.00 us 24 READ
0.41 122646.00 us 610.00 us 244682.00 us 2 FSYNC
0.67 66579.33 us 2976.00 us 368832.00 us 6 FXATTROP
0.72 214305.00 us 25134.00 us 403476.00 us 2 CREATE
1.57 20415.13 us 53.00 us 863671.00 us 46 SETATTR
7.13 43065.32 us 380.00 us 2023922.00 us 99 GETXATTR
17.08 57053.73 us 21.00 us 4791081.00 us 179 ENTRYLK
19.27 24833.95 us 18.00 us 1721894.00 us 464 INODELK
52.99 31373.74 us 99.00 us 4753189.00 us 1010 LOOKUP
Duration: 27 seconds
Data Read: 3145728 bytes
Data Written: 0 bytes
Brick: hangar:/export/wd1a
--------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 1 1
No. of Writes: 119 89 206
Block Size: 8b+ 16b+ 32b+
No. of Reads: 18 456 638
No. of Writes: 2674 20109 26254
Block Size: 64b+ 128b+ 256b+
No. of Reads: 653 1114 1047
No. of Writes: 16225 21757 26094
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 815 445 1022
No. of Writes: 46483 39502 45837
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1188 2113 808
No. of Writes: 21375 14211 9791
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 528 6478 153851
No. of Writes: 12407 3688 153
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 278772 FORGET
0.00 0.00 us 0.00 us 0.00 us 267233 RELEASE
0.00 0.00 us 0.00 us 0.00 us 28625 RELEASEDIR
0.00 66.60 us 59.00 us 79.00 us 5 WRITE
0.00 1908.00 us 1908.00 us 1908.00 us 1 READDIR
0.01 1413.71 us 35.00 us 9513.00 us 7 STATFS
0.01 4227.67 us 63.00 us 12551.00 us 3 OPEN
0.01 4651.33 us 75.00 us 13710.00 us 3 TRUNCATE
0.01 416.73 us 14.00 us 8592.00 us 37 FSTAT
0.02 19443.00 us 19443.00 us 19443.00 us 1 SETXATTR
0.02 4094.20 us 1650.00 us 9259.00 us 5 FSYNC
0.03 4631.62 us 42.00 us 36533.00 us 8 FTRUNCATE
0.05 1433.56 us 35.00 us 48945.00 us 36 READLINK
0.13 24155.33 us 5373.00 us 83746.00 us 6 XATTROP
0.14 5630.43 us 21.00 us 74418.00 us 28 FINODELK
0.15 32752.40 us 26251.00 us 40382.00 us 5 CREATE
0.19 69616.33 us 5391.00 us 188134.00 us 3 RENAME
0.19 2929.67 us 151.00 us 66289.00 us 72 READ
0.28 11943.19 us 2855.00 us 95080.00 us 26 FXATTROP
0.95 7093.66 us 53.00 us 418369.00 us 151 SETATTR
1.50 76999.59 us 17.00 us 1692792.00 us 22 ENTRYLK
3.39 88812.86 us 32.00 us 1730394.00 us 43 GETXATTR
23.83 37513.31 us 17.00 us 4552377.00 us 716 INODELK
69.10 20612.44 us 88.00 us 5304073.00 us 3778 LOOKUP
Duration: 561502 seconds
Data Read: 20818299002 bytes
Data Written: 1477574380 bytes
Interval 3 Stats:
Block Size: 512b+ 131072b+
No. of Reads: 0 24
No. of Writes: 2 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 FORGET
0.00 0.00 us 0.00 us 0.00 us 3 RELEASE
0.00 69.00 us 69.00 us 69.00 us 1 OPEN
0.00 62.00 us 39.00 us 85.00 us 2 STATFS
0.00 64.00 us 59.00 us 69.00 us 2 WRITE
0.00 169.00 us 169.00 us 169.00 us 1 TRUNCATE
0.00 76.00 us 52.00 us 100.00 us 3 FTRUNCATE
0.00 139.11 us 39.00 us 509.00 us 9 READLINK
0.01 2932.50 us 1751.00 us 4114.00 us 2 FSYNC
0.03 6897.50 us 5373.00 us 8422.00 us 2 XATTROP
0.04 1268.91 us 14.00 us 8592.00 us 11 FSTAT
0.14 5535.10 us 2905.00 us 13880.00 us 10 FXATTROP
0.15 29488.00 us 26251.00 us 32725.00 us 2 CREATE
0.25 4117.08 us 151.00 us 38975.00 us 24 READ
0.29 11564.40 us 22.00 us 74418.00 us 10 FINODELK
0.47 188134.00 us 188134.00 us 188134.00 us 1 RENAME
2.40 20793.83 us 53.00 us 418369.00 us 46 SETATTR
4.25 211640.62 us 17.00 us 1692792.00 us 8 ENTRYLK
4.41 439321.25 us 640.00 us 1730394.00 us 4 GETXATTR
24.88 43685.58 us 20.00 us 2885874.00 us 227 INODELK
62.67 22440.90 us 108.00 us 2617203.00 us 1113 LOOKUP
Duration: 27 seconds
Data Read: 3145728 bytes
Data Written: 1800 bytes
