All the volume commands are taking several minutes to complete. Here's the profiler's output:
Node3's brick is the one that was replaced. It's replica is node6
Brick: node1:/bricks/brickname
---------------------------------------------------
Cumulative Stats:
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 13 2
No. of Writes: 60 0 141
Block Size: 64b+ 128b+ 256b+
No. of Reads: 3 26 17
No. of Writes: 325 87 738
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 99 114 222
No. of Writes: 877 343 128
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 110 5 2
No. of Writes: 29401 78829 1448
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 19 33 34679
No. of Writes: 6233 22903 41202
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 1 513
No. of Writes: 1 0 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 126138 FORGET
0.00 0.00 us 0.00 us 0.00 us 141671 RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718 RELEASEDIR
0.02 43.00 us 21.00 us 80.00 us 17 STAT
0.02 51.62 us 27.00 us 131.00 us 21 STATFS
0.09 42.78 us 11.00 us 1640.00 us 95 FLUSH
0.21 50.01 us 23.00 us 567.00 us 189 ENTRYLK
0.21 50.01 us 19.00 us 291.00 us 190 FINODELK
0.56 8578.33 us 53.00 us 25625.00 us 3 GETXATTR
0.62 148.89 us 71.00 us 2761.00 us 190 XATTROP
0.70 168.11 us 83.00 us 1019.00 us 190 FXATTROP
0.91 219.84 us 47.00 us 13732.00 us 190 SETATTR
1.21 71.74 us 16.00 us 11516.00 us 775 INODELK
1.47 354.88 us 56.00 us 22669.00 us 190 REMOVEXATTR
2.60 1254.69 us 122.00 us 12514.00 us 95 WRITE
3.25 194.10 us 51.00 us 48823.00 us 770 LOOKUP
88.15 43068.81 us 265.00 us 418819.00 us 94 CREATE
Duration: 644070 seconds
Data Read: 5089537361 bytes
Data Written: 9083513756 bytes
Interval 0 Stats:
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 13 2
No. of Writes: 60 0 141
Block Size: 64b+ 128b+ 256b+
No. of Reads: 3 26 17
No. of Writes: 325 87 738
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 99 114 222
No. of Writes: 877 343 128
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 110 5 2
No. of Writes: 29401 78829 1448
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 19 33 34679
No. of Writes: 6233 22903 41202
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 1 513
No. of Writes: 1 0 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 126138 FORGET
0.00 0.00 us 0.00 us 0.00 us 141671 RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718 RELEASEDIR
0.02 43.00 us 21.00 us 80.00 us 17 STAT
0.02 51.62 us 27.00 us 131.00 us 21 STATFS
0.09 42.78 us 11.00 us 1640.00 us 95 FLUSH
0.21 50.01 us 23.00 us 567.00 us 189 ENTRYLK
0.21 50.01 us 19.00 us 291.00 us 190 FINODELK
0.56 8578.33 us 53.00 us 25625.00 us 3 GETXATTR
0.62 148.89 us 71.00 us 2761.00 us 190 XATTROP
0.70 168.11 us 83.00 us 1019.00 us 190 FXATTROP
0.91 219.84 us 47.00 us 13732.00 us 190 SETATTR
1.21 71.74 us 16.00 us 11516.00 us 775 INODELK
1.47 354.88 us 56.00 us 22669.00 us 190 REMOVEXATTR
2.60 1254.69 us 122.00 us 12514.00 us 95 WRITE
3.25 194.10 us 51.00 us 48823.00 us 770 LOOKUP
88.15 43068.81 us 265.00 us 418819.00 us 94 CREATE
Duration: 644070 seconds
Data Read: 5089537361 bytes
Data Written: 9083513756 bytes
Brick: node2:/bricks/brickname
-----------------------------------------------
Cumulative Stats:
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 1 23
No. of Writes: 60 0 141
Block Size: 64b+ 128b+ 256b+
No. of Reads: 45 47 363
No. of Writes: 325 87 738
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 515 37 42
No. of Writes: 877 343 128
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 17 0 1
No. of Writes: 29401 78829 1448
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 15 39 67031
No. of Writes: 6233 22903 41202
Block Size: 262144b+ 1048576b+
No. of Reads: 1 105
No. of Writes: 1 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 126136 FORGET
0.00 0.00 us 0.00 us 0.00 us 141671 RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718 RELEASEDIR
0.02 74.00 us 74.00 us 74.00 us 1 STAT
0.15 196.00 us 141.00 us 288.00 us 3 GETXATTR
0.59 105.23 us 44.00 us 146.00 us 22 STATFS
2.05 83.88 us 11.00 us 137.00 us 96 FLUSH
4.79 98.61 us 20.00 us 146.00 us 191 ENTRYLK
5.03 102.93 us 22.00 us 158.00 us 192 FINODELK
5.52 226.07 us 136.00 us 295.00 us 96 WRITE
6.31 261.08 us 150.00 us 345.00 us 95 CREATE
6.72 137.53 us 50.00 us 214.00 us 192 SETATTR
7.68 157.14 us 75.00 us 237.00 us 192 REMOVEXATTR
8.13 166.49 us 81.00 us 282.00 us 192 XATTROP
8.26 169.01 us 76.00 us 275.00 us 192 FXATTROP
17.22 86.46 us 16.00 us 216.00 us 783 INODELK
27.54 138.09 us 43.00 us 266.00 us 784 LOOKUP
Duration: 644071 seconds
Data Read: 8902589511 bytes
Data Written: 9083513756 bytes
Interval 0 Stats:
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 1 23
No. of Writes: 60 0 141
Block Size: 64b+ 128b+ 256b+
No. of Reads: 45 47 363
No. of Writes: 325 87 738
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 515 37 42
No. of Writes: 877 343 128
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 17 0 1
No. of Writes: 29401 78829 1448
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 15 39 67031
No. of Writes: 6233 22903 41202
Block Size: 262144b+ 1048576b+
No. of Reads: 1 105
No. of Writes: 1 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 126136 FORGET
0.00 0.00 us 0.00 us 0.00 us 141671 RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718 RELEASEDIR
0.02 74.00 us 74.00 us 74.00 us 1 STAT
0.15 196.00 us 141.00 us 288.00 us 3 GETXATTR
0.59 105.23 us 44.00 us 146.00 us 22 STATFS
2.05 83.88 us 11.00 us 137.00 us 96 FLUSH
4.79 98.61 us 20.00 us 146.00 us 191 ENTRYLK
5.03 102.93 us 22.00 us 158.00 us 192 FINODELK
5.52 226.07 us 136.00 us 295.00 us 96 WRITE
6.31 261.08 us 150.00 us 345.00 us 95 CREATE
6.72 137.53 us 50.00 us 214.00 us 192 SETATTR
7.68 157.14 us 75.00 us 237.00 us 192 REMOVEXATTR
8.13 166.49 us 81.00 us 282.00 us 192 XATTROP
8.26 169.01 us 76.00 us 275.00 us 192 FXATTROP
17.22 86.46 us 16.00 us 216.00 us 783 INODELK
27.54 138.09 us 43.00 us 266.00 us 784 LOOKUP
Duration: 644071 seconds
Data Read: 8902589511 bytes
Data Written: 9083513756 bytes
Brick: node3(sink):/bricks/brickname
------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 11 26 125
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 829 2341 9599
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 12674 9229 27346
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 2 10 0
No. of Writes: 23414 28727 18372
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1 0 0
No. of Writes: 48347 92134 9675
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 2 11 50
No. of Writes: 11717 24948 1022216
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 13805186 FORGET
0.00 0.00 us 0.00 us 0.00 us 17674891 RELEASE
0.00 0.00 us 0.00 us 0.00 us 218068 RELEASEDIR
0.00 24.00 us 24.00 us 24.00 us 1 OPENDIR
0.00 13.06 us 8.00 us 36.00 us 16 STAT
0.00 19.22 us 9.00 us 46.00 us 18 STATFS
0.00 45.54 us 22.00 us 82.00 us 13 SETXATTR
0.00 120.93 us 77.00 us 156.00 us 14 XATTROP
0.00 11.01 us 7.00 us 68.00 us 156 ENTRYLK
0.01 283.15 us 246.00 us 504.00 us 59 READDIR
0.02 899.19 us 39.00 us 17518.00 us 26 SETATTR
0.02 2004.85 us 38.00 us 10406.00 us 13 WRITE
0.02 2022.77 us 24.00 us 21677.00 us 13 REMOVEXATTR
0.03 2965.85 us 34.00 us 37695.00 us 13 FTRUNCATE
0.04 3691.62 us 31.00 us 18386.00 us 13 FLUSH
0.31 2105.65 us 23.00 us 57417.00 us 177 OPEN
0.43 2603.12 us 57.00 us 73929.00 us 202 FXATTROP
0.46 3030.94 us 7.00 us 87892.00 us 186 FSTAT
1.07 33.17 us 18.00 us 17545.00 us 39491 GETXATTR
1.31 123033.46 us 75610.00 us 269227.00 us 13 FSYNC
46.14 704.24 us 6.00 us 268597.00 us 79866 INODELK
50.13 699.93 us 20.00 us 267607.00 us 87307 LOOKUP
Duration: 112674 seconds
Data Read: 7441454 bytes
Data Written: 138577629032 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 11 26 125
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 829 2341 9599
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 12674 9229 27346
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 2 10 0
No. of Writes: 23414 28727 18372
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1 0 0
No. of Writes: 48347 92134 9675
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 2 11 50
No. of Writes: 11717 24948 1022216
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 13805186 FORGET
0.00 0.00 us 0.00 us 0.00 us 17674862 RELEASE
0.00 0.00 us 0.00 us 0.00 us 218068 RELEASEDIR
0.00 24.00 us 24.00 us 24.00 us 1 OPENDIR
0.00 13.06 us 8.00 us 36.00 us 16 STAT
0.00 19.22 us 9.00 us 46.00 us 18 STATFS
0.00 45.54 us 22.00 us 82.00 us 13 SETXATTR
0.00 120.93 us 77.00 us 156.00 us 14 XATTROP
0.00 11.01 us 7.00 us 68.00 us 156 ENTRYLK
0.01 283.15 us 246.00 us 504.00 us 59 READDIR
0.02 899.19 us 39.00 us 17518.00 us 26 SETATTR
0.02 2004.85 us 38.00 us 10406.00 us 13 WRITE
0.02 2022.77 us 24.00 us 21677.00 us 13 REMOVEXATTR
0.03 2965.85 us 34.00 us 37695.00 us 13 FTRUNCATE
0.04 3691.62 us 31.00 us 18386.00 us 13 FLUSH
0.31 2105.65 us 23.00 us 57417.00 us 177 OPEN
0.43 2603.12 us 57.00 us 73929.00 us 202 FXATTROP
0.46 3030.94 us 7.00 us 87892.00 us 186 FSTAT
1.07 33.17 us 18.00 us 17545.00 us 39491 GETXATTR
1.31 123033.46 us 75610.00 us 269227.00 us 13 FSYNC
46.14 704.24 us 6.00 us 268597.00 us 79866 INODELK
50.13 699.93 us 20.00 us 267607.00 us 87307 LOOKUP
Duration: 112674 seconds
Data Read: 7441454 bytes
Data Written: 138577629032 bytes
Brick: node4:/bricks/brickname
-----------------------------------------------
Cumulative Stats:
Block Size: 8b+ 32b+ 64b+
No. of Reads: 0 9 24
No. of Writes: 62 128 335
Block Size: 128b+ 256b+ 512b+
No. of Reads: 21 177 257
No. of Writes: 186 779 885
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 30 14 7
No. of Writes: 286 101 29410
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 0 9 0
No. of Writes: 79662 1379 6187
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 29 3924 0
No. of Writes: 22467 32424 1
Block Size: 1048576b+
No. of Reads: 0
No. of Writes: 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 126295 FORGET
0.00 0.00 us 0.00 us 0.00 us 141875 RELEASE
0.00 0.00 us 0.00 us 0.00 us 117220 RELEASEDIR
0.12 119.50 us 102.00 us 147.00 us 4 GETXATTR
0.19 68.18 us 42.00 us 109.00 us 11 STAT
0.44 92.16 us 19.00 us 141.00 us 19 STATFS
1.20 68.03 us 11.00 us 120.00 us 71 FLUSH
2.93 83.02 us 18.00 us 136.00 us 142 ENTRYLK
3.15 89.18 us 16.00 us 160.00 us 142 FINODELK
3.40 192.82 us 76.00 us 271.00 us 71 WRITE
4.04 114.43 us 35.00 us 204.00 us 142 SETATTR
4.87 138.05 us 49.00 us 222.00 us 142 REMOVEXATTR
5.63 159.52 us 56.00 us 262.00 us 142 FXATTROP
10.68 73.81 us 11.00 us 202.00 us 582 INODELK
19.90 1127.35 us 116.00 us 27717.00 us 71 CREATE
21.67 613.82 us 46.00 us 65260.00 us 142 XATTROP
21.80 130.68 us 35.00 us 241.00 us 671 LOOKUP
Duration: 458509 seconds
Data Read: 517180943 bytes
Data Written: 7895152670 bytes
Interval 0 Stats:
Block Size: 8b+ 32b+ 64b+
No. of Reads: 0 9 24
No. of Writes: 62 128 335
Block Size: 128b+ 256b+ 512b+
No. of Reads: 21 177 257
No. of Writes: 186 779 885
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 30 14 7
No. of Writes: 286 101 29410
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 0 9 0
No. of Writes: 79662 1379 6187
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 29 3924 0
No. of Writes: 22467 32424 1
Block Size: 1048576b+
No. of Reads: 0
No. of Writes: 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 126295 FORGET
0.00 0.00 us 0.00 us 0.00 us 141875 RELEASE
0.00 0.00 us 0.00 us 0.00 us 117220 RELEASEDIR
0.12 119.50 us 102.00 us 147.00 us 4 GETXATTR
0.19 68.18 us 42.00 us 109.00 us 11 STAT
0.44 92.16 us 19.00 us 141.00 us 19 STATFS
1.20 68.03 us 11.00 us 120.00 us 71 FLUSH
2.93 83.02 us 18.00 us 136.00 us 142 ENTRYLK
3.15 89.18 us 16.00 us 160.00 us 142 FINODELK
3.40 192.82 us 76.00 us 271.00 us 71 WRITE
4.04 114.43 us 35.00 us 204.00 us 142 SETATTR
4.87 138.05 us 49.00 us 222.00 us 142 REMOVEXATTR
5.63 159.52 us 56.00 us 262.00 us 142 FXATTROP
10.68 73.81 us 11.00 us 202.00 us 582 INODELK
19.90 1127.35 us 116.00 us 27717.00 us 71 CREATE
21.67 613.82 us 46.00 us 65260.00 us 142 XATTROP
21.80 130.68 us 35.00 us 241.00 us 671 LOOKUP
Duration: 458509 seconds
Data Read: 517180943 bytes
Data Written: 7895152670 bytes
Brick: node5:/bricks/brickname
------------------------------------------------
Cumulative Stats:
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 14 9
No. of Writes: 62 0 128
Block Size: 64b+ 128b+ 256b+
No. of Reads: 23 56 225
No. of Writes: 335 186 779
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 357 106 233
No. of Writes: 885 286 102
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 128 11 15
No. of Writes: 29410 79662 1379
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 16 34 28965
No. of Writes: 6191 22467 32424
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 5 3 984
No. of Writes: 1 0 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 126301 FORGET
0.00 0.00 us 0.00 us 0.00 us 141880 RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718 RELEASEDIR
0.01 51.75 us 41.00 us 69.00 us 4 GETXATTR
0.02 59.50 us 35.00 us 108.00 us 4 STAT
0.05 44.06 us 28.00 us 118.00 us 16 STATFS
0.10 24.78 us 15.00 us 99.00 us 59 FLUSH
0.34 41.01 us 24.00 us 107.00 us 118 ENTRYLK
0.37 44.43 us 25.00 us 156.00 us 118 FINODELK
0.76 90.88 us 70.00 us 183.00 us 118 SETATTR
0.81 193.88 us 162.00 us 283.00 us 59 WRITE
0.98 116.89 us 85.00 us 212.00 us 118 REMOVEXATTR
1.10 131.18 us 86.00 us 219.00 us 118 FXATTROP
1.52 44.02 us 24.00 us 1004.00 us 487 INODELK
3.35 86.29 us 52.00 us 183.00 us 549 LOOKUP
4.21 504.03 us 83.00 us 43099.00 us 118 XATTROP
86.39 20696.02 us 207.00 us 69802.00 us 59 CREATE
Duration: 644071 seconds
Data Read: 4837930222 bytes
Data Written: 7895351133 bytes
Interval 0 Stats:
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 14 9
No. of Writes: 62 0 128
Block Size: 64b+ 128b+ 256b+
No. of Reads: 23 56 225
No. of Writes: 335 186 779
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 357 106 233
No. of Writes: 885 286 102
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 128 11 15
No. of Writes: 29410 79662 1379
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 16 34 28965
No. of Writes: 6191 22467 32424
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 5 3 984
No. of Writes: 1 0 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 126301 FORGET
0.00 0.00 us 0.00 us 0.00 us 141880 RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718 RELEASEDIR
0.01 51.75 us 41.00 us 69.00 us 4 GETXATTR
0.02 59.50 us 35.00 us 108.00 us 4 STAT
0.05 44.06 us 28.00 us 118.00 us 16 STATFS
0.10 24.78 us 15.00 us 99.00 us 59 FLUSH
0.34 41.01 us 24.00 us 107.00 us 118 ENTRYLK
0.37 44.43 us 25.00 us 156.00 us 118 FINODELK
0.76 90.88 us 70.00 us 183.00 us 118 SETATTR
0.81 193.88 us 162.00 us 283.00 us 59 WRITE
0.98 116.89 us 85.00 us 212.00 us 118 REMOVEXATTR
1.10 131.18 us 86.00 us 219.00 us 118 FXATTROP
1.52 44.02 us 24.00 us 1004.00 us 487 INODELK
3.35 86.29 us 52.00 us 183.00 us 549 LOOKUP
4.21 504.03 us 83.00 us 43099.00 us 118 XATTROP
86.39 20696.02 us 207.00 us 69802.00 us 59 CREATE
Duration: 644071 seconds
Data Read: 4837930222 bytes
Data Written: 7895351133 bytes
Brick: node6(source):/bricks/brickname
--------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 7 18 89
No. of Writes: 4 8 37
Block Size: 8b+ 16b+ 32b+
No. of Reads: 727 2325 9459
No. of Writes: 108 54 188
Block Size: 64b+ 128b+ 256b+
No. of Reads: 12419 9313 27616
No. of Writes: 360 85 772
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 23708 28691 18594
No. of Writes: 847 313 138
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 19484 12596 8458
No. of Writes: 29185 79632 1431
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 5695 5755 1062899
No. of Writes: 6168 19435 32017
Block Size: 262144b+ 1048576b+
No. of Reads: 0 0
No. of Writes: 1 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 13806534 FORGET
0.00 0.00 us 0.00 us 0.00 us 17813646 RELEASE
0.00 0.00 us 0.00 us 0.00 us 223324 RELEASEDIR
0.00 560.00 us 560.00 us 560.00 us 1 ENTRYLK
0.00 3901.00 us 3901.00 us 3901.00 us 1 SETXATTR
0.00 4010.00 us 4010.00 us 4010.00 us 1 REMOVEXATTR
0.01 62446.08 us 8.00 us 365433.00 us 13 FLUSH
0.01 93887.77 us 52.00 us 588566.00 us 13 SETATTR
0.03 10772.83 us 28.00 us 1121761.00 us 253 GETXATTR
0.04 3190096.00 us 3190096.00 us 3190096.00 us 1 READDIR
0.09 558307.69 us 179931.00 us 3188951.00 us 13 READ
0.11 616756.00 us 74.00 us 7307745.00 us 14 XATTROP
0.12 4754785.50 us 48.00 us 9509523.00 us 2 OPENDIR
0.15 1799185.00 us 2310.00 us 5023537.00 us 7 STATFS
0.16 68757.98 us 10.00 us 872148.00 us 189 FSTAT
0.31 143533.93 us 42.00 us 7002195.00 us 174 OPEN
0.40 160262.95 us 661.00 us 2825083.00 us 202 READDIRP
1.55 624450.87 us 31.00 us 7397432.00 us 203 FXATTROP
22.43 212161.62 us 12.00 us 7397413.00 us 8639 INODELK
74.60 541421.09 us 63.00 us 14463033.00 us 11261 LOOKUP
Duration: 644071 seconds
Data Read: 140706386722 bytes
Data Written: 7549422894 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 7 18 89
No. of Writes: 4 8 37
Block Size: 8b+ 16b+ 32b+
No. of Reads: 727 2325 9459
No. of Writes: 108 54 188
Block Size: 64b+ 128b+ 256b+
No. of Reads: 12419 9313 27616
No. of Writes: 360 85 772
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 23708 28691 18594
No. of Writes: 847 313 138
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 19484 12596 8458
No. of Writes: 29185 79632 1431
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 5695 5755 1062899
No. of Writes: 6168 19435 32017
Block Size: 262144b+ 1048576b+
No. of Reads: 0 0
No. of Writes: 1 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 13806534 FORGET
0.00 0.00 us 0.00 us 0.00 us 17813657 RELEASE
0.00 0.00 us 0.00 us 0.00 us 223324 RELEASEDIR
0.00 560.00 us 560.00 us 560.00 us 1 ENTRYLK
0.00 3901.00 us 3901.00 us 3901.00 us 1 SETXATTR
0.00 4010.00 us 4010.00 us 4010.00 us 1 REMOVEXATTR
0.01 62446.08 us 8.00 us 365433.00 us 13 FLUSH
0.01 93887.77 us 52.00 us 588566.00 us 13 SETATTR
0.03 10772.83 us 28.00 us 1121761.00 us 253 GETXATTR
0.04 3190096.00 us 3190096.00 us 3190096.00 us 1 READDIR
0.09 558307.69 us 179931.00 us 3188951.00 us 13 READ
0.11 616756.00 us 74.00 us 7307745.00 us 14 XATTROP
0.12 4754785.50 us 48.00 us 9509523.00 us 2 OPENDIR
0.15 1799185.00 us 2310.00 us 5023537.00 us 7 STATFS
0.16 68757.98 us 10.00 us 872148.00 us 189 FSTAT
0.31 143533.93 us 42.00 us 7002195.00 us 174 OPEN
0.40 160262.95 us 661.00 us 2825083.00 us 202 READDIRP
1.55 624450.87 us 31.00 us 7397432.00 us 203 FXATTROP
22.43 212161.62 us 12.00 us 7397413.00 us 8639 INODELK
74.60 541421.09 us 63.00 us 14463033.00 us 11261 LOOKUP
Duration: 644071 seconds
Data Read: 140706386722 bytes
Data Written: 7549422894 bytes
On Fri, Aug 7, 2015 at 12:17 AM, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:
On 08/07/2015 12:11 PM, Prasun Gera wrote:
No, no noticeable difference. Still very high, possibly higher than before.
I was guessing that the cpu usage could be because of the diff algorithm which computes checksums (which is a cpu intensive task). That doesn't seem to be the case. Could you do a volume profile and see the FOPS that are happening on the bricks and share the result?
1.gluster volume profile <volname> start
2. gluster volume profile <volname> info
3. wait 10-15 seconds
4.gluster volume profile <volname> info
The system has come down to a crawl. It's difficult to even ssh or run any commands on the terminal. Do you make anything of the logs ? The brick log is just a giant alternating stream of those two lines I mentioned earlier.
On Thu, Aug 6, 2015 at 10:10 PM, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:
On 08/07/2015 01:33 AM, Prasun Gera wrote:
I replaced the brick in a node in my 3x2 dist+repl volume (RHS 3). I'm seeing that the heal process, which should essentially be a dump from the working replica to the newly added one is taking exceptionally long. It has moved ~100 G over a day on a 1Gigabit network. The CPU usage on both the nodes of the replica has been pretty high.
Does setting `cluster.data-self-heal-algorithm` to full make a difference in the cpu usage?
I also think that nagios is making it worse. The heal is slow enough as it is, and nagios keeps triggering heal info, which I think never completes. I also see my logs filling up These are some of the log contents which I got by running tail on them:
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
