The volume is almost completely idle now and the CPU for the brick
process has returned to normal. I've included the profile and I think it
shows the latency for the bad brick (data12) is unusually high, probably
indicating the filesystem is at fault after all??
On Sun, 2014-06-01 at 01:01 -0400, Pranith Kumar Karampuri wrote:
> Franco,
> Could you do the following to get more information:
>
> "gluster volume profile <volname> start"
>
> Wait for some time, this will start gathering what operations are coming to all the bricks"
> Now execute "gluster volume profile <volname> info" > /file/you/should/reply/to/this/mail/with
>
> Then execute:
> gluster volume profile <volname> stop
>
> Lets see if this throws any light on the problem at hand
>
> Pranith
> ----- Original Message -----
> > From: "Franco Broi" <franco.broi@xxxxxxxxxx>
> > To: gluster-users@xxxxxxxxxxx
> > Sent: Sunday, June 1, 2014 9:02:48 AM
> > Subject: glusterfsd process spinning
> >
> > Hi
> >
> > I've been suffering from continual problems with my gluster filesystem
> > slowing down due to what I thought was congestion on a single brick
> > being caused by a problem with the underlying filesystem running slow
> > but I've just noticed that the glusterfsd process for that particular
> > brick is running at 100%+, even when the filesystem is almost idle.
> >
> > I've done a couple of straces of the brick and another on the same
> > server, does the high number of futex errors give any clues as to what
> > might be wrong?
> >
> > % time seconds usecs/call calls errors syscall
> > ------ ----------- ----------- --------- --------- ----------------
> > 45.58 0.027554 0 191665 20772 futex
> > 28.26 0.017084 0 137133 readv
> > 26.04 0.015743 0 66259 epoll_wait
> > 0.13 0.000077 3 23 writev
> > 0.00 0.000000 0 1 epoll_ctl
> > ------ ----------- ----------- --------- --------- ----------------
> > 100.00 0.060458 395081 20772 total
> >
> > % time seconds usecs/call calls errors syscall
> > ------ ----------- ----------- --------- --------- ----------------
> > 99.25 0.334020 133 2516 epoll_wait
> > 0.40 0.001347 0 4090 26 futex
> > 0.35 0.001192 0 5064 readv
> > 0.00 0.000000 0 20 writev
> > ------ ----------- ----------- --------- --------- ----------------
> > 100.00 0.336559 11690 26 total
> >
> >
> >
> > Cheers,
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users@xxxxxxxxxxx
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> >
Brick: nas3-10g:/data10/gvol
----------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 151 240 313821945
Block Size: 8b+ 16b+ 32b+
No. of Reads: 634 19 15
No. of Writes: 11210623 45720500 24490029
Block Size: 64b+ 128b+ 256b+
No. of Reads: 22 17006 950
No. of Writes: 992498404 1024176715 1055850217
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 954 3561 2564
No. of Writes: 597572813 225090885 227894690
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 5826398 2161421 1101178
No. of Writes: 398642340 174916651 47685643
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1749653 53735599 1098632674
No. of Writes: 10906391 11251050 23780900
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 99
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 108512 FORGET
0.00 0.00 us 0.00 us 0.00 us 26578056 RELEASE
0.00 0.00 us 0.00 us 0.00 us 422494 RELEASEDIR
0.00 51.00 us 51.00 us 51.00 us 1 ACCESS
0.00 61.00 us 61.00 us 61.00 us 1 OPEN
0.28 37.33 us 15.00 us 127.00 us 322 STAT
0.47 40.70 us 13.00 us 94.00 us 492 FLUSH
1.90 81542.00 us 81542.00 us 81542.00 us 1 CREATE
2.18 46.26 us 18.00 us 119.00 us 2024 STATFS
7.17 94.84 us 37.00 us 250.00 us 3251 WRITE
16.39 329.89 us 38.00 us 131946.00 us 2136 LOOKUP
71.62 1083.06 us 53.00 us 130116.00 us 2844 READ
Duration: 1644260 seconds
Data Read: 147664953639565 bytes
Data Written: 12749788231214 bytes
Interval 2 Stats:
Block Size: 131072b+
No. of Reads: 490
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8 RELEASE
0.82 41.71 us 19.00 us 65.00 us 96 FLUSH
1.27 33.68 us 15.00 us 127.00 us 184 STAT
1.76 44.53 us 19.00 us 119.00 us 193 STATFS
6.74 101.27 us 52.00 us 217.00 us 325 LOOKUP
89.42 891.45 us 53.00 us 123191.00 us 490 READ
Duration: 96 seconds
Data Read: 64225280 bytes
Data Written: 0 bytes
Brick: nas3-10g:/data12/gvol
----------------------------
Cumulative Stats:
Block Size: 1b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 7 436253984 4926880
Block Size: 16b+ 32b+ 64b+
No. of Reads: 1 0 1
No. of Writes: 1606681 5947 2506
Block Size: 128b+ 256b+ 512b+
No. of Reads: 720 6 14
No. of Writes: 7649781 30681943 4496119
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 73 52 9825
No. of Writes: 2777352 2097176 56527057
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 484366 11934 114306
No. of Writes: 20436877 9192065 2441283
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 186673 144740142 0
No. of Writes: 4296733 3896350 16
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 783 FORGET
0.00 0.00 us 0.00 us 0.00 us 554946 RELEASE
0.00 0.00 us 0.00 us 0.00 us 46014 RELEASEDIR
0.00 33.19 us 10.00 us 91.00 us 16 ACCESS
0.02 46.22 us 15.00 us 120.00 us 322 STAT
0.08 45.78 us 12.00 us 108.00 us 1102 FLUSH
0.18 56.27 us 21.00 us 292.00 us 2024 STATFS
1.29 354.23 us 43.00 us 113589.00 us 2294 LOOKUP
14.40 100.54 us 33.00 us 576.00 us 89962 WRITE
84.02 4839.60 us 48.00 us 705200.00 us 10905 READ
Duration: 273953 seconds
Data Read: 18992872093553 bytes
Data Written: 1969722263215 bytes
Interval 2 Stats:
Block Size: 131072b+
No. of Reads: 714
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 18 RELEASE
0.03 31.00 us 10.00 us 91.00 us 12 ACCESS
0.55 39.12 us 15.00 us 113.00 us 184 STAT
0.67 44.83 us 21.00 us 114.00 us 193 STATFS
0.67 40.13 us 14.00 us 75.00 us 216 FLUSH
2.66 96.76 us 46.00 us 266.00 us 357 LOOKUP
95.43 1738.16 us 52.00 us 230583.00 us 714 READ
Duration: 96 seconds
Data Read: 93585408 bytes
Data Written: 0 bytes
Brick: nas3-10g:/data9/gvol
---------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 5 0
No. of Writes: 226 38 382035
Block Size: 8b+ 16b+ 32b+
No. of Reads: 503 1 26
No. of Writes: 8121 44024302 29347327
Block Size: 64b+ 128b+ 256b+
No. of Reads: 23 14499 696
No. of Writes: 1097991782 1082664231 908683224
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 562 1538 1359
No. of Writes: 490505979 198697761 283101385
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 3475245 717394 516582
No. of Writes: 434127250 182773820 47503801
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1285964 72630385 1093265270
No. of Writes: 10739639 11453329 34706166
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 103
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 100526 FORGET
0.00 0.00 us 0.00 us 0.00 us 26961620 RELEASE
0.00 0.00 us 0.00 us 0.00 us 422499 RELEASEDIR
0.01 33.27 us 11.00 us 46.00 us 11 ACCESS
0.41 50.28 us 15.00 us 109.00 us 322 STAT
0.53 42.90 us 12.00 us 99.00 us 480 FLUSH
2.73 52.71 us 20.00 us 134.00 us 2024 STATFS
20.75 380.28 us 38.00 us 121585.00 us 2135 LOOKUP
75.57 1068.90 us 50.00 us 243959.00 us 2766 READ
Duration: 1644260 seconds
Data Read: 148144416725315 bytes
Data Written: 14520329792942 bytes
Interval 2 Stats:
Block Size: 131072b+
No. of Reads: 511
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8 RELEASE
0.07 29.88 us 11.00 us 44.00 us 8 ACCESS
1.17 40.70 us 18.00 us 68.00 us 96 FLUSH
2.54 46.32 us 15.00 us 109.00 us 184 STAT
2.88 49.96 us 22.00 us 121.00 us 193 STATFS
10.61 109.34 us 48.00 us 262.00 us 325 LOOKUP
82.73 542.40 us 61.00 us 60856.00 us 511 READ
Duration: 96 seconds
Data Read: 66977792 bytes
Data Written: 0 bytes
Brick: nas3-10g:/data11/gvol
----------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 166 18 432965
Block Size: 8b+ 16b+ 32b+
No. of Reads: 415 10 22
No. of Writes: 8114 44848404 49704831
Block Size: 64b+ 128b+ 256b+
No. of Reads: 46 493741 1325
No. of Writes: 1013779056 788776921 791567418
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 756 1828 1810
No. of Writes: 421897696 193233666 241937475
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 2232414 1165468 346570
No. of Writes: 425717230 167837566 45478150
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1527982 37900085 925191366
No. of Writes: 10730607 8915356 21685151
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 110
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 82647 FORGET
0.00 0.00 us 0.00 us 0.00 us 21556792 RELEASE
0.00 0.00 us 0.00 us 0.00 us 422493 RELEASEDIR
0.00 23.83 us 10.00 us 43.00 us 12 ACCESS
0.17 39.05 us 14.00 us 102.00 us 322 STAT
0.55 43.08 us 13.00 us 110.00 us 960 FLUSH
1.21 44.83 us 18.00 us 126.00 us 2024 STATFS
1.64 47.36 us 16.00 us 115.00 us 2603 FSTAT
8.64 303.41 us 39.00 us 73264.00 us 2134 LOOKUP
8.91 79.36 us 35.00 us 182.00 us 8414 WRITE
78.88 2104.41 us 49.00 us 701544.00 us 2810 READ
Duration: 1644260 seconds
Data Read: 123844112929707 bytes
Data Written: 11977554070196 bytes
Interval 2 Stats:
Block Size: 16b+ 512b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 590 84 1166
Block Size: 4096b+ 131072b+
No. of Reads: 0 446
No. of Writes: 6 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 16 RELEASE
0.04 20.89 us 10.00 us 43.00 us 9 ACCESS
1.27 33.66 us 14.00 us 91.00 us 184 STAT
1.63 41.56 us 17.00 us 63.00 us 192 FLUSH
1.64 41.70 us 20.00 us 110.00 us 193 STATFS
5.51 46.10 us 18.00 us 74.00 us 585 FSTAT
6.73 101.37 us 43.00 us 330.00 us 325 LOOKUP
29.23 77.50 us 37.00 us 137.00 us 1846 WRITE
53.96 592.26 us 49.00 us 114792.00 us 446 READ
Duration: 96 seconds
Data Read: 58458112 bytes
Data Written: 3675120 bytes
Brick: nas2-10g:/data6/gvol
---------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 174 10 70244
Block Size: 8b+ 16b+ 32b+
No. of Reads: 50 0 5
No. of Writes: 5201 6014165 8506259
Block Size: 64b+ 128b+ 256b+
No. of Reads: 3 11745 651
No. of Writes: 1434815362 702223997 503658696
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 121 476 342
No. of Writes: 147148463 83746530 79063284
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 4253441 1519343 562779
No. of Writes: 189219348 68756300 19102239
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 458648 17234860 659534440
No. of Writes: 3670738 2242352 6425894
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 77
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 50581 FORGET
0.00 0.00 us 0.00 us 0.00 us 14025563 RELEASE
0.00 0.00 us 0.00 us 0.00 us 422483 RELEASEDIR
0.01 46.20 us 36.00 us 75.00 us 5 ACCESS
0.64 46.98 us 13.00 us 120.00 us 322 STAT
0.88 43.21 us 11.00 us 99.00 us 480 FLUSH
4.14 48.52 us 16.00 us 130.00 us 2024 STATFS
44.41 495.23 us 32.00 us 61768.00 us 2126 LOOKUP
49.92 529.94 us 49.00 us 146637.00 us 2233 READ
Duration: 1642269 seconds
Data Read: 87641300019913 bytes
Data Written: 4639013858093 bytes
Interval 2 Stats:
Block Size: 131072b+
No. of Reads: 365
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8 RELEASE
0.09 50.33 us 36.00 us 75.00 us 3 ACCESS
2.20 38.66 us 11.00 us 61.00 us 96 FLUSH
4.81 44.11 us 13.00 us 102.00 us 184 STAT
5.10 44.63 us 17.00 us 89.00 us 193 STATFS
17.30 89.82 us 36.00 us 189.00 us 325 LOOKUP
70.50 325.92 us 49.00 us 30989.00 us 365 READ
Duration: 96 seconds
Data Read: 47841280 bytes
Data Written: 0 bytes
Brick: nas2-10g:/data7/gvol
---------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 127 7 74738
Block Size: 8b+ 16b+ 32b+
No. of Reads: 32 0 0
No. of Writes: 7917 6326619 22661985
Block Size: 64b+ 128b+ 256b+
No. of Reads: 2 12193 695
No. of Writes: 21919447 121465603 507089230
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 101 1312 491
No. of Writes: 98551947 99287896 69556748
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 364897 1287122 30135
No. of Writes: 162759130 42628532 11803784
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 208091 15161747 695588501
No. of Writes: 3882169 2502859 4392089
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 97
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 51107 FORGET
0.00 0.00 us 0.00 us 0.00 us 11669143 RELEASE
0.00 0.00 us 0.00 us 0.00 us 422488 RELEASEDIR
0.00 65.00 us 65.00 us 65.00 us 1 OPEN
0.01 49.00 us 14.00 us 91.00 us 16 ACCESS
0.28 45.42 us 13.00 us 109.00 us 322 STAT
0.44 23310.00 us 23310.00 us 23310.00 us 1 CREATE
0.75 40.94 us 12.00 us 101.00 us 972 FLUSH
1.93 50.64 us 16.00 us 153.00 us 2024 STATFS
18.59 463.21 us 34.00 us 153463.00 us 2129 LOOKUP
20.48 78.82 us 30.00 us 209.00 us 13782 WRITE
57.52 935.01 us 44.00 us 107746.00 us 3263 READ
Duration: 1642269 seconds
Data Read: 92189487962936 bytes
Data Written: 3589594156624 bytes
Interval 2 Stats:
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 0 0 0
No. of Writes: 6887 4986 140
Block Size: 131072b+
No. of Reads: 511
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 16 RELEASE
0.03 55.75 us 14.00 us 91.00 us 8 ACCESS
0.48 38.26 us 17.00 us 77.00 us 192 FLUSH
0.52 43.19 us 13.00 us 109.00 us 184 STAT
0.60 47.59 us 16.00 us 115.00 us 193 STATFS
2.08 98.00 us 34.00 us 265.00 us 325 LOOKUP
35.34 1059.94 us 44.00 us 107746.00 us 511 READ
60.95 77.75 us 30.00 us 193.00 us 12013 WRITE
Duration: 96 seconds
Data Read: 66977792 bytes
Data Written: 6497280 bytes
Brick: nas2-10g:/data5/gvol
---------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 157 63 94918
Block Size: 8b+ 16b+ 32b+
No. of Reads: 69 1 1
No. of Writes: 2032 6946372 15138774
Block Size: 64b+ 128b+ 256b+
No. of Reads: 4 10350 616
No. of Writes: 673853459 415954645 462803157
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 84 436 237
No. of Writes: 91307202 75997986 43601841
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1384551 1503179 315362
No. of Writes: 152102275 42970573 11506162
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 409211 11698640 656491132
No. of Writes: 3414508 1959650 6498862
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 129
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 48867 FORGET
0.00 0.00 us 0.00 us 0.00 us 10202393 RELEASE
0.00 0.00 us 0.00 us 0.00 us 422484 RELEASEDIR
0.03 45.00 us 23.00 us 94.00 us 35 ACCESS
0.25 46.58 us 12.00 us 154.00 us 322 STAT
0.51 41.26 us 14.00 us 101.00 us 731 FLUSH
1.69 49.61 us 16.00 us 125.00 us 2024 STATFS
12.21 654.70 us 47.00 us 132377.00 us 1105 READ
12.92 360.17 us 33.00 us 46417.00 us 2126 LOOKUP
72.39 92.78 us 28.00 us 233.00 us 46230 WRITE
Duration: 1642269 seconds
Data Read: 86858503076859 bytes
Data Written: 3670162546832 bytes
Interval 2 Stats:
Block Size: 131072b+
No. of Reads: 88
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 13 RELEASE
1.06 40.75 us 23.00 us 71.00 us 16 ACCESS
10.05 39.83 us 14.00 us 89.00 us 155 FLUSH
12.16 40.58 us 12.00 us 87.00 us 184 STAT
12.66 88.34 us 54.00 us 240.00 us 88 READ
14.45 45.96 us 16.00 us 79.00 us 193 STATFS
49.61 93.72 us 35.00 us 165.00 us 325 LOOKUP
Duration: 96 seconds
Data Read: 11534336 bytes
Data Written: 0 bytes
Brick: nas2-10g:/data8/gvol
---------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 2
No. of Writes: 141 8 65118
Block Size: 8b+ 16b+ 32b+
No. of Reads: 48 0 4
No. of Writes: 4802 6760117 28814469
Block Size: 64b+ 128b+ 256b+
No. of Reads: 7 10507 625
No. of Writes: 23235333 124052110 477018875
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 72 454 269
No. of Writes: 100740601 77797389 63701386
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 4799964 2133846 448385
No. of Writes: 155757464 45770857 10418699
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1093962 26191376 656875259
No. of Writes: 2756571 1784738 5124758
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 88
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 49393 FORGET
0.00 0.00 us 0.00 us 0.00 us 12888692 RELEASE
0.00 0.00 us 0.00 us 0.00 us 422479 RELEASEDIR
0.00 62.00 us 62.00 us 62.00 us 1 OPEN
0.02 47.64 us 11.00 us 93.00 us 11 ACCESS
0.45 13366.00 us 13366.00 us 13366.00 us 1 CREATE
0.50 45.46 us 12.00 us 112.00 us 322 STAT
0.75 50.07 us 23.00 us 112.00 us 440 READLINK
0.87 42.01 us 13.00 us 98.00 us 612 FLUSH
3.38 49.33 us 17.00 us 127.00 us 2024 STATFS
5.53 90.01 us 29.00 us 177.00 us 1815 WRITE
31.95 418.34 us 33.00 us 101223.00 us 2258 LOOKUP
56.57 678.00 us 49.00 us 311465.00 us 2467 READ
Duration: 1642269 seconds
Data Read: 87911734109028 bytes
Data Written: 3450048479478 bytes
Interval 2 Stats:
Block Size: 256b+ 131072b+
No. of Reads: 0 369
No. of Writes: 14 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 11 RELEASE
0.09 42.88 us 11.00 us 74.00 us 8 ACCESS
0.31 83.50 us 68.00 us 128.00 us 14 WRITE
1.15 54.65 us 23.00 us 112.00 us 80 READLINK
1.72 49.68 us 14.00 us 98.00 us 131 FLUSH
2.04 41.97 us 12.00 us 107.00 us 184 STAT
2.31 45.49 us 19.00 us 111.00 us 193 STATFS
8.36 92.14 us 35.00 us 383.00 us 344 LOOKUP
84.03 863.67 us 50.00 us 142787.00 us 369 READ
Duration: 96 seconds
Data Read: 48365568 bytes
Data Written: 5264 bytes
Brick: nas4-10g:/data13/gvol
----------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 124 11 60682
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 4
No. of Writes: 793 10025717 10374704
Block Size: 64b+ 128b+ 256b+
No. of Reads: 2 1553 20
No. of Writes: 1884073 8729975 90222445
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 68 183 96
No. of Writes: 14709941 11216938 8118854
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 16557 13443 14207
No. of Writes: 95258335 28252430 8459854
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 52993 471640 204527123
No. of Writes: 2759266 1514093 7579155
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 77
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 4661 FORGET
0.00 0.00 us 0.00 us 0.00 us 1439364 RELEASE
0.00 0.00 us 0.00 us 0.00 us 99546 RELEASEDIR
0.00 33.80 us 13.00 us 45.00 us 5 ACCESS
0.01 38.71 us 11.00 us 106.00 us 322 STAT
0.03 38.96 us 6.00 us 118.00 us 960 FLUSH
0.08 51.00 us 17.00 us 124.00 us 2024 STATFS
0.49 516.72 us 46.00 us 150624.00 us 1182 READ
0.64 378.49 us 35.00 us 78068.00 us 2126 LOOKUP
15.63 58.73 us 9.00 us 1207.00 us 333723 FSTAT
83.12 102.33 us 29.00 us 1739.00 us 1018542 WRITE
Duration: 424409 seconds
Data Read: 26841972052360 bytes
Data Written: 2563585335420 bytes
Interval 2 Stats:
Block Size: 16b+ 32b+ 4096b+
No. of Reads: 0 0 0
No. of Writes: 76341 150580 8664
Block Size: 8192b+ 16384b+ 131072b+
No. of Reads: 0 0 112
No. of Writes: 4284 16 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 16 RELEASE
0.00 28.67 us 13.00 us 41.00 us 3 ACCESS
0.02 32.59 us 11.00 us 106.00 us 184 STAT
0.03 37.77 us 6.00 us 103.00 us 192 FLUSH
0.03 48.23 us 19.00 us 121.00 us 193 STATFS
0.12 106.78 us 41.00 us 245.00 us 325 LOOKUP
0.19 471.36 us 49.00 us 20113.00 us 112 READ
14.49 54.34 us 9.00 us 1207.00 us 75290 FSTAT
85.12 100.23 us 29.00 us 1739.00 us 239885 WRITE
Duration: 96 seconds
Data Read: 14680064 bytes
Data Written: 148440000 bytes
Brick: nas4-10g:/data15/gvol
----------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 1 0
No. of Writes: 107 0 68152
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 1
No. of Writes: 3884 8300782 8516312
Block Size: 64b+ 128b+ 256b+
No. of Reads: 2 232782 7
No. of Writes: 3461081 15911651 91527571
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 60 338 74
No. of Writes: 16154130 12732671 5311766
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 33878 247197 20326
No. of Writes: 95504061 29337720 8187030
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 177564 619264 247831629
No. of Writes: 2473961 1419888 2575526
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 68
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 5376 FORGET
0.00 0.00 us 0.00 us 0.00 us 1266475 RELEASE
0.00 0.00 us 0.00 us 0.00 us 109404 RELEASEDIR
0.00 93.00 us 93.00 us 93.00 us 1 ACCESS
0.20 34.01 us 13.00 us 102.00 us 322 STAT
0.58 38.66 us 12.00 us 114.00 us 840 FLUSH
1.59 43.99 us 20.00 us 151.00 us 2024 STATFS
16.50 433.87 us 43.00 us 77186.00 us 2127 LOOKUP
81.13 1187.53 us 53.00 us 355507.00 us 3822 READ
Duration: 431368 seconds
Data Read: 32534712233462 bytes
Data Written: 1913767903640 bytes
Interval 2 Stats:
Block Size: 131072b+
No. of Reads: 631
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 14 RELEASE
0.66 29.80 us 13.00 us 85.00 us 184 STAT
0.82 40.67 us 13.00 us 73.00 us 168 FLUSH
0.98 42.52 us 20.00 us 113.00 us 193 STATFS
3.98 102.29 us 46.00 us 270.00 us 326 LOOKUP
93.56 1241.13 us 60.00 us 156647.00 us 631 READ
Duration: 96 seconds
Data Read: 82706432 bytes
Data Written: 0 bytes
Brick: nas4-10g:/data14/gvol
----------------------------
Cumulative Stats:
Block Size: 1b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 150 56945 3144
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 5 1
No. of Writes: 6945350 13373779 6572652
Block Size: 128b+ 256b+ 512b+
No. of Reads: 2600 7 84
No. of Writes: 15657875 96604095 27470300
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 280 93 39962
No. of Writes: 13158968 18008018 104566143
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 247581 61525 424768
No. of Writes: 32013956 8099478 2463916
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 1762271 275293932 0
No. of Writes: 2375462 14621188 104
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 9289 FORGET
0.00 0.00 us 0.00 us 0.00 us 1567656 RELEASE
0.00 0.00 us 0.00 us 0.00 us 137113 RELEASEDIR
0.00 32.83 us 17.00 us 44.00 us 6 ACCESS
0.23 36.76 us 14.00 us 118.00 us 322 STAT
0.63 38.06 us 13.00 us 97.00 us 840 FLUSH
1.88 46.82 us 19.00 us 179.00 us 2024 STATFS
13.09 101.45 us 64.00 us 252.00 us 6510 WRITE
15.69 372.21 us 34.00 us 69938.00 us 2126 LOOKUP
68.47 1150.92 us 54.00 us 340203.00 us 3001 READ
Duration: 516081 seconds
Data Read: 36220486986088 bytes
Data Written: 3706236391485 bytes
Interval 2 Stats:
Block Size: 131072b+
No. of Reads: 482
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 14 RELEASE
0.01 41.00 us 38.00 us 44.00 us 3 ACCESS
0.41 33.18 us 14.00 us 118.00 us 184 STAT
0.44 38.89 us 14.00 us 90.00 us 168 FLUSH
0.61 46.86 us 22.00 us 112.00 us 193 STATFS
2.33 106.87 us 49.00 us 297.00 us 325 LOOKUP
96.20 2972.94 us 62.00 us 340203.00 us 482 READ
Duration: 96 seconds
Data Read: 63176704 bytes
Data Written: 0 bytes
Brick: nas4-10g:/data16/gvol
----------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 257 42 343818
Block Size: 8b+ 16b+ 32b+
No. of Reads: 461 0 10
No. of Writes: 39602 41604782 45688731
Block Size: 64b+ 128b+ 256b+
No. of Reads: 24 20447 744
No. of Writes: 1562272399 2661869208 1282744508
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 831 2591 1943
No. of Writes: 582413527 197225820 316982056
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 4723753 3196854 673805
No. of Writes: 612708974 257698531 64463657
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1375642 46137369 1214376777
No. of Writes: 13992804 14030587 31263025
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 202
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 104780 FORGET
0.00 0.00 us 0.00 us 0.00 us 23962953 RELEASE
0.00 0.00 us 0.00 us 0.00 us 545444 RELEASEDIR
0.09 40.62 us 13.00 us 87.00 us 131 FLUSH
0.19 35.21 us 13.00 us 108.00 us 322 STAT
1.50 43.27 us 21.00 us 126.00 us 2024 STATFS
7.61 906.02 us 51.00 us 85240.00 us 491 READ
9.41 258.78 us 43.00 us 28299.00 us 2126 LOOKUP
81.20 85.58 us 33.00 us 207.00 us 55476 WRITE
Duration: 2083859 seconds
Data Read: 162319927356315 bytes
Data Written: 17503378837925 bytes
Interval 2 Stats:
Block Size: 131072b+
No. of Reads: 37
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 2 RELEASE
1.84 38.12 us 13.00 us 59.00 us 24 FLUSH
7.84 105.49 us 75.00 us 137.00 us 37 READ
11.19 30.28 us 13.00 us 78.00 us 184 STAT
15.74 40.61 us 21.00 us 102.00 us 193 STATFS
63.38 97.08 us 49.00 us 278.00 us 325 LOOKUP
Duration: 96 seconds
Data Read: 4849664 bytes
Data Written: 0 bytes
Brick: nas1-10g:/data3/gvol
---------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 12866 24859 323807
Block Size: 8b+ 16b+ 32b+
No. of Reads: 412 11 41
No. of Writes: 10645 41105912 25134078
Block Size: 64b+ 128b+ 256b+
No. of Reads: 48 27542 975
No. of Writes: 1292875339 843578811 744024598
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1037 2743 2651
No. of Writes: 514219067 157387945 221165334
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 7095518 1659221 682413
No. of Writes: 377770795 149646392 26457687
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 756642 21515090 893474661
No. of Writes: 6883983 7293778 23116967
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 73
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 78022 FORGET
0.00 0.00 us 0.00 us 0.00 us 21609000 RELEASE
0.00 0.00 us 0.00 us 0.00 us 422486 RELEASEDIR
0.05 36.47 us 14.00 us 96.00 us 17 ACCESS
0.98 38.82 us 14.00 us 100.00 us 322 STAT
7.31 46.00 us 20.00 us 174.00 us 2024 STATFS
91.66 546.81 us 32.00 us 53829.00 us 2134 LOOKUP
Duration: 1643022 seconds
Data Read: 118613759650761 bytes
Data Written: 10929293256154 bytes
Interval 2 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.49 28.00 us 14.00 us 65.00 us 8 ACCESS
13.26 32.66 us 14.00 us 60.00 us 184 STAT
17.70 41.56 us 21.00 us 111.00 us 193 STATFS
68.55 95.59 us 44.00 us 285.00 us 325 LOOKUP
Duration: 95 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: nas1-10g:/data4/gvol
---------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 1316 1165 102009
Block Size: 8b+ 16b+ 32b+
No. of Reads: 86 0 2
No. of Writes: 8839 9266893 15627653
Block Size: 64b+ 128b+ 256b+
No. of Reads: 4 1896 522
No. of Writes: 1398213448 920823375 693751586
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 135 797 1065
No. of Writes: 250611823 77787294 131519640
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 3491363 1095468 1207276
No. of Writes: 263505479 90307664 20141698
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1399405 8171475 306788386
No. of Writes: 5206176 3607247 12105216
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 196
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 51459 FORGET
0.00 0.00 us 0.00 us 0.00 us 20885594 RELEASE
0.00 0.00 us 0.00 us 0.00 us 422485 RELEASEDIR
0.00 63.00 us 63.00 us 63.00 us 1 OPEN
0.00 40.38 us 9.00 us 88.00 us 8 ACCESS
0.02 51.71 us 16.00 us 108.00 us 34 FLUSH
0.12 10179.00 us 10179.00 us 10179.00 us 1 CREATE
0.17 43.90 us 14.00 us 116.00 us 322 STAT
1.30 53.90 us 19.00 us 128.00 us 2024 STATFS
13.36 522.98 us 39.00 us 89328.00 us 2136 LOOKUP
85.02 108.00 us 31.00 us 1321.00 us 65823 WRITE
Duration: 1643022 seconds
Data Read: 40856739676704 bytes
Data Written: 6647121567894 bytes
Interval 2 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.42 32.00 us 9.00 us 62.00 us 6 ACCESS
15.15 37.99 us 14.00 us 97.00 us 184 STAT
18.03 43.12 us 22.00 us 105.00 us 193 STATFS
66.41 94.30 us 42.00 us 437.00 us 325 LOOKUP
Duration: 95 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: nas1-10g:/data2/gvol
---------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 3 0
No. of Writes: 152 86 228980
Block Size: 8b+ 16b+ 32b+
No. of Reads: 305 1 0
No. of Writes: 8038 40193578 18301406
Block Size: 64b+ 128b+ 256b+
No. of Reads: 4 8174 755
No. of Writes: 1179643287 773912641 723775462
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 530 825 1563
No. of Writes: 345033982 98562726 189979361
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 4004519 1662938 580793
No. of Writes: 350504951 149569626 37434932
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 968566 22065542 628847152
No. of Writes: 7361059 5337716 29462826
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 95
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 61432 FORGET
0.00 0.00 us 0.00 us 0.00 us 17730243 RELEASE
0.00 0.00 us 0.00 us 0.00 us 422491 RELEASEDIR
0.00 65.00 us 65.00 us 65.00 us 1 OPEN
0.02 44.74 us 16.00 us 76.00 us 23 FLUSH
0.03 43.65 us 11.00 us 91.00 us 52 ACCESS
0.21 44.87 us 13.00 us 105.00 us 322 STAT
0.39 27014.00 us 27014.00 us 27014.00 us 1 CREATE
1.49 50.56 us 18.00 us 151.00 us 2024 STATFS
17.10 548.98 us 40.00 us 62271.00 us 2136 LOOKUP
80.76 94.12 us 32.00 us 225.00 us 58844 WRITE
Duration: 1643022 seconds
Data Read: 83957540265576 bytes
Data Written: 11073257699744 bytes
Interval 2 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 RELEASE
0.65 44.82 us 16.00 us 66.00 us 11 FLUSH
1.33 37.33 us 11.00 us 90.00 us 27 ACCESS
10.24 42.33 us 13.00 us 98.00 us 184 STAT
11.90 46.90 us 18.00 us 108.00 us 193 STATFS
75.88 177.52 us 42.00 us 26117.00 us 325 LOOKUP
Duration: 95 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: nas1-10g:/data1/gvol
---------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 151 13 81196
Block Size: 8b+ 16b+ 32b+
No. of Reads: 14 2 0
No. of Writes: 1603 7608086 7955758
Block Size: 64b+ 128b+ 256b+
No. of Reads: 3 4314 557
No. of Writes: 738224969 430264864 261225993
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 65 75 186
No. of Writes: 98925790 37066531 45695936
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 4902061 1406500 639922
No. of Writes: 143325821 43522270 8861235
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 778372 6662218 311945542
No. of Writes: 2914316 1746266 6558560
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 76
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 41056 FORGET
0.00 0.00 us 0.00 us 0.00 us 4546126 RELEASE
0.00 0.00 us 0.00 us 0.00 us 422484 RELEASEDIR
0.00 69.00 us 69.00 us 69.00 us 1 OPEN
0.01 43.44 us 36.00 us 67.00 us 16 ACCESS
0.02 38.74 us 12.00 us 101.00 us 23 FLUSH
0.29 51.97 us 15.00 us 161.00 us 322 STAT
0.56 32456.00 us 32456.00 us 32456.00 us 1 CREATE
2.24 63.82 us 19.00 us 140.00 us 2024 STATFS
18.89 509.61 us 42.00 us 102367.00 us 2139 LOOKUP
77.99 105.72 us 32.00 us 1151.00 us 42561 WRITE
Duration: 1643022 seconds
Data Read: 41404034760227 bytes
Data Written: 3321036917134 bytes
Interval 2 Stats:
Block Size: 4096b+ 8192b+
No. of Reads: 0 0
No. of Writes: 8 3
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 RELEASE
0.49 39.43 us 36.00 us 42.00 us 7 ACCESS
0.51 26.09 us 12.00 us 101.00 us 11 FLUSH
1.54 79.36 us 38.00 us 163.00 us 11 WRITE
15.33 47.07 us 15.00 us 111.00 us 184 STAT
18.71 54.77 us 21.00 us 140.00 us 193 STATFS
63.43 110.27 us 42.00 us 270.00 us 325 LOOKUP
Duration: 95 seconds
Data Read: 0 bytes
Data Written: 114408 bytes
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
