Anand Avati <anand.avati@xxxxxxxxx> wrote:
> Can you get the 'gluster volume profile <name> info' output for each run?
I had to restart servers in order to get profiling enabled. And then the
problem vanished, at least in the slow-forever state.
There are still performances issues, but it is not exactly the same
problem. For instance an initial ls -l took a long time to complete,
but replaying it got decent performances afterwards. Here is an
output for that below.
I am now waiting to drop into the always slow mode.
Brick: hotstuff:/export/wd1a
----------------------------
Cumulative Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
1.80 36.00 us 36.00 us 36.00 us 1 STATFS
3.71 74.00 us 74.00 us 74.00 us 1 GETXATTR
6.31 126.00 us 126.00 us 126.00 us 1 OPEN
18.08 120.33 us 107.00 us 132.00 us 3 OPENDIR
70.11 127.27 us 52.00 us 219.00 us 11 LOOKUP
Duration: 132 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Interval 0 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
1.80 36.00 us 36.00 us 36.00 us 1 STATFS
3.71 74.00 us 74.00 us 74.00 us 1 GETXATTR
6.31 126.00 us 126.00 us 126.00 us 1 OPEN
18.08 120.33 us 107.00 us 132.00 us 3 OPENDIR
70.11 127.27 us 52.00 us 219.00 us 11 LOOKUP
Duration: 132 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: silo:/export/wd3a
------------------------
Cumulative Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 RELEASEDIR
0.00 41.00 us 41.00 us 41.00 us 1 GETXATTR
0.01 38.00 us 24.00 us 52.00 us 2 ENTRYLK
0.04 318.00 us 304.00 us 332.00 us 2 OPEN
0.07 253.25 us 40.00 us 383.00 us 4 OPENDIR
0.80 5755.50 us 78.00 us 11433.00 us 2 READDIRP
2.31 33373.00 us 33373.00 us 33373.00 us 1 FXATTROP
14.07 101845.00 us 58861.00 us 144829.00 us 2 XATTROP
82.71 85528.07 us 85.00 us 248598.00 us 14 LOOKUP
Duration: 117 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Interval 0 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 RELEASEDIR
0.00 41.00 us 41.00 us 41.00 us 1 GETXATTR
0.01 38.00 us 24.00 us 52.00 us 2 ENTRYLK
0.04 318.00 us 304.00 us 332.00 us 2 OPEN
0.07 253.25 us 40.00 us 383.00 us 4 OPENDIR
0.80 5755.50 us 78.00 us 11433.00 us 2 READDIRP
2.31 33373.00 us 33373.00 us 33373.00 us 1 FXATTROP
14.07 101845.00 us 58861.00 us 144829.00 us 2 XATTROP
82.71 85528.07 us 85.00 us 248598.00 us 14 LOOKUP
Duration: 117 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: hangar:/export/wd1a
--------------------------
Cumulative Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 48.00 us 48.00 us 48.00 us 1 GETXATTR
0.05 249.67 us 185.00 us 330.00 us 3 OPENDIR
1.03 14307.00 us 14307.00 us 14307.00 us 1 OPEN
1.60 7394.00 us 37.00 us 21957.00 us 3 READDIRP
97.31 122412.18 us 147.00 us 446739.00 us 11 LOOKUP
Duration: 126 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Interval 0 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 48.00 us 48.00 us 48.00 us 1 GETXATTR
0.05 249.67 us 185.00 us 330.00 us 3 OPENDIR
1.03 14307.00 us 14307.00 us 14307.00 us 1 OPEN
1.60 7394.00 us 37.00 us 21957.00 us 3 READDIRP
97.31 122412.18 us 147.00 us 446739.00 us 11 LOOKUP
Duration: 126 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: hangar:/export/wd3a
--------------------------
Cumulative Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 RELEASEDIR
0.00 46.00 us 46.00 us 46.00 us 1 STATFS
0.00 52.00 us 52.00 us 52.00 us 1 GETXATTR
0.00 30.00 us 29.00 us 31.00 us 2 ENTRYLK
0.00 219.00 us 46.00 us 321.00 us 4 OPENDIR
0.05 16486.00 us 93.00 us 32879.00 us 2 OPEN
0.12 19635.75 us 26.00 us 45547.00 us 4 READDIRP
0.82 556046.00 us 556046.00 us 556046.00 us 1 MKNOD
1.12 756390.00 us 756390.00 us 756390.00 us 1 MKDIR
1.26 65604.69 us 69.00 us 221650.00 us 13 LOOKUP
1.57 266513.25 us 51.00 us 532977.00 us 4 SETATTR
95.07 64470146.00 us 64470146.00 us 64470146.00 us 1 FXATTROP
Duration: 127 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Interval 0 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 RELEASEDIR
0.00 46.00 us 46.00 us 46.00 us 1 STATFS
0.00 52.00 us 52.00 us 52.00 us 1 GETXATTR
0.00 30.00 us 29.00 us 31.00 us 2 ENTRYLK
0.00 219.00 us 46.00 us 321.00 us 4 OPENDIR
0.05 16486.00 us 93.00 us 32879.00 us 2 OPEN
0.12 19635.75 us 26.00 us 45547.00 us 4 READDIRP
0.82 556046.00 us 556046.00 us 556046.00 us 1 MKNOD
1.12 756390.00 us 756390.00 us 756390.00 us 1 MKDIR
1.26 65604.69 us 69.00 us 221650.00 us 13 LOOKUP
1.57 266513.25 us 51.00 us 532977.00 us 4 SETATTR
95.07 64470146.00 us 64470146.00 us 64470146.00 us 1 FXATTR
--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@xxxxxxxxxx
