| Dear all, I just noticed on my main volume of my HPC cluster my IO operations become impressively poor.. Doing some file operations above a linux kernel sources compressed file, the untar operation can take more than 1/2 hours for this file (roughly 80MB and 52 000 files inside) as you read below: ####################################################### ################ UNTAR time consumed ################ ####################################################### real 32m42.967s user 0m11.783s sys 0m15.050s ####################################################### ################# DU time consumed ################## ####################################################### 557M linux-4.1-rc6 real 0m25.060s user 0m0.068s sys 0m0.344s ####################################################### ################# FIND time consumed ################ ####################################################### 52663 real 0m25.687s user 0m0.084s sys 0m0.387s ####################################################### ################# GREP time consumed ################ ####################################################### 7952 real 2m15.890s user 0m0.887s sys 0m2.777s ####################################################### ################# TAR time consumed ################# ####################################################### real 1m5.551s user 0m26.536s sys 0m2.609s ####################################################### ################# RM time consumed ################## ####################################################### real 2m51.485s user 0m0.167s sys 0m1.663s For information, this volume is a distributed replicated one and is composed by 4 servers with 2 bricks each. Each bricks is a 12-drives RAID6 vdisk with nice native performances (around 1.2GBs). In comparison, when I use DD to generate a 100GB file on the same volume, my write throughput is around 1GB (client side) and 500MBs (server side) because of replication: Client side: [root@node056 ~]# ifstat -i ib0 ib0 KB/s in KB/s out 3251.45 1.09e+06 3139.80 1.05e+06 3185.29 1.06e+06 3293.84 1.09e+06 ... Server side: [root@lucifer ~]# ifstat -i ib0 ib0 KB/s in KB/s out 561818.1 1746.42 560020.3 1737.92 526337.1 1648.20 513972.7 1613.69 ... DD command: [root@node056 ~]# dd if=/dev/zero of=/home/root/test.dd bs=1M count=100000 100000+0 enregistrements lus 100000+0 enregistrements écrits 104857600000 octets (105 GB) copiés, 202,99 s, 517 MB/s So this issue doesn’t seem coming from the network (which is Infiniband technology in this case) You can find in attachments a set of files: - mybench.sh: the bench script - benches.txt: output of my "bench" - profile.txt: gluster volume profile during the "bench" - vol_status.txt: gluster volume status - vol_info.txt: gluster volume info Can someone help me to fix it (it’s very critical because this volume is on a HPC cluster in production). Thanks by advance, Geoffrey -----------------------------------------------
Geoffrey Letessier Responsable informatique & ingénieur système CNRS - UPR 9080 - Laboratoire de Biochimie Théorique Institut de Biologie Physico-Chimique 13, rue Pierre et Marie Curie - 75005 Paris Tel: 01 58 41 50 93 - eMail: geoffrey.letessier@xxxxxxx |
####################################################### ################ UNTAR time consumed ################ ####################################################### real 32m42.967s user 0m11.783s sys 0m15.050s ####################################################### ################# DU time consumed ################## ####################################################### 557M linux-4.1-rc6 real 0m25.060s user 0m0.068s sys 0m0.344s ####################################################### ################# FIND time consumed ################ ####################################################### 52663 real 0m25.687s user 0m0.084s sys 0m0.387s ####################################################### ################# GREP time consumed ################ ####################################################### 7952 real 2m15.890s user 0m0.887s sys 0m2.777s ####################################################### ################# TAR time consumed ################# ####################################################### real 1m5.551s user 0m26.536s sys 0m2.609s ####################################################### ################# RM time consumed ################## ####################################################### real 2m51.485s user 0m0.167s sys 0m1.663s
Attachment:
mybench.sh
Description: Binary data
Brick: ib-storage1:/export/brick_home/brick1
--------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 4 5
No. of Writes: 2503 3565 677590
Block Size: 8b+ 16b+ 32b+
No. of Reads: 6 267 95
No. of Writes: 6520399 7921488 102967
Block Size: 64b+ 128b+ 256b+
No. of Reads: 162 300 551
No. of Writes: 1413431 2067947 145362
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 857 1609 161526
No. of Writes: 296020 2032864 1894595
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 17889 11758 9624
No. of Writes: 19938454 337311 13717970
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 18532 522995 25990763
No. of Writes: 2326904 1936216 10774752
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 367
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 638901 FORGET
0.00 0.00 us 0.00 us 0.00 us 21813916 RELEASE
0.00 0.00 us 0.00 us 0.00 us 348145 RELEASEDIR
0.00 96.00 us 96.00 us 96.00 us 1 LINK
0.00 25.80 us 17.00 us 80.00 us 368 READLINK
0.01 16410.00 us 211.00 us 32676.00 us 7 SYMLINK
0.03 47.17 us 9.00 us 18825.00 us 6472 READDIR
0.03 41.90 us 12.00 us 204.00 us 7704 STAT
0.04 52.28 us 14.00 us 218.00 us 7696 FSTAT
0.04 132.28 us 79.00 us 32726.00 us 3207 RMDIR
0.05 83.08 us 33.00 us 1094.00 us 6428 SETXATTR
0.05 110.99 us 65.00 us 645.00 us 5336 XATTROP
0.06 111.66 us 19.00 us 9216.00 us 5498 GETXATTR
0.10 43.91 us 17.00 us 459.00 us 26195 OPEN
0.12 23.09 us 7.00 us 1858.00 us 58971 FLUSH
0.13 40.34 us 18.00 us 9435.00 us 36761 OPENDIR
0.18 110.57 us 33.00 us 233287.00 us 18116 FTRUNCATE
0.32 34.78 us 7.00 us 1098.00 us 100440 ENTRYLK
0.47 55.02 us 23.00 us 13738.00 us 93929 SETATTR
0.53 472.87 us 81.00 us 1060168.00 us 12466 UNLINK
0.58 43.62 us 9.00 us 185279.00 us 147550 FINODELK
0.78 2349.35 us 566.00 us 302825.00 us 3674 STATFS
0.87 38.93 us 10.00 us 129683.00 us 248111 INODELK
0.93 88.29 us 35.00 us 41846.00 us 117204 WRITE
2.44 1676.53 us 63.00 us 1096272.00 us 16125 FSYNC
4.04 734.44 us 23.00 us 341615.00 us 61003 READDIRP
4.63 327.10 us 51.00 us 1572422.00 us 156987 FXATTROP
8.02 13845.49 us 514.00 us 878375.00 us 6414 MKDIR
11.76 1432.18 us 30.00 us 1424172.00 us 90993 READ
28.45 12656.31 us 126.00 us 815661.00 us 24905 CREATE
35.33 1015.71 us 29.00 us 1993181.00 us 385368 LOOKUP
Duration: 3332887 seconds
Data Read: 3446122000339 bytes
Data Written: 2112162576108 bytes
Interval 6 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 24 4 147
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 290704 257822 343
Block Size: 64b+ 128b+ 256b+
No. of Reads: 4 0 6
No. of Writes: 47480 89041 5689
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 7 3 10
No. of Writes: 66843 7420 169036
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 80 516 442
No. of Writes: 1392575 14641 5378
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1219 48198 832387
No. of Writes: 4822 100530 123303
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 3
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 35039 FORGET
0.00 0.00 us 0.00 us 0.00 us 771685 RELEASE
0.00 0.00 us 0.00 us 0.00 us 47426 RELEASEDIR
0.00 25.80 us 17.00 us 80.00 us 368 READLINK
0.01 20149.33 us 11630.00 us 32676.00 us 3 SYMLINK
0.01 115.32 us 70.00 us 514.00 us 938 XATTROP
0.01 122.94 us 19.00 us 9216.00 us 1008 GETXATTR
0.03 42.81 us 12.00 us 176.00 us 5226 STAT
0.03 85.58 us 35.00 us 329.00 us 3215 SETXATTR
0.03 44.87 us 9.00 us 18825.00 us 6436 READDIR
0.04 54.01 us 16.00 us 208.00 us 5871 FSTAT
0.05 132.28 us 79.00 us 32726.00 us 3207 RMDIR
0.08 44.50 us 17.00 us 299.00 us 16713 OPEN
0.09 23.27 us 7.00 us 1858.00 us 34260 FLUSH
0.12 96.24 us 33.00 us 41868.00 us 11537 FTRUNCATE
0.17 40.30 us 18.00 us 9435.00 us 36688 OPENDIR
0.26 33.39 us 7.00 us 1098.00 us 69102 ENTRYLK
0.30 57.20 us 25.00 us 13738.00 us 46997 SETATTR
0.44 45.52 us 9.00 us 185279.00 us 86466 FINODELK
0.56 41.04 us 11.00 us 129683.00 us 120644 INODELK
0.62 2523.96 us 566.00 us 302825.00 us 2193 STATFS
0.66 473.00 us 139.00 us 1060168.00 us 12461 UNLINK
0.71 89.83 us 35.00 us 41846.00 us 70103 WRITE
2.72 2342.45 us 135.00 us 1096272.00 us 10311 FSYNC
4.27 417.12 us 52.00 us 1572422.00 us 90907 FXATTROP
4.96 723.50 us 23.00 us 341615.00 us 60915 READDIRP
5.94 16456.63 us 530.00 us 464098.00 us 3207 MKDIR
14.64 1465.20 us 32.00 us 1424172.00 us 88799 READ
21.11 15057.75 us 130.00 us 670879.00 us 12458 CREATE
42.13 1449.10 us 30.00 us 1993181.00 us 258368 LOOKUP
Duration: 130678 seconds
Data Read: 112634411874 bytes
Data Written: 37104014555 bytes
Brick: ib-storage1:/export/brick_home/brick2
--------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 1 26 1
No. of Writes: 162 210 998509
Block Size: 8b+ 16b+ 32b+
No. of Reads: 49 42 95
No. of Writes: 7255477 7480105 115606
Block Size: 64b+ 128b+ 256b+
No. of Reads: 157 301 450
No. of Writes: 1656535 1178920 111314
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 874 1588 1991
No. of Writes: 419785 1999041 1597559
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 17690 14133 9716
No. of Writes: 14779519 237867 4943912
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 18535 531296 26209755
No. of Writes: 2014514 2436442 7612781
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 414
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 582649 FORGET
0.00 0.00 us 0.00 us 0.00 us 23283301 RELEASE
0.00 0.00 us 0.00 us 0.00 us 337651 RELEASEDIR
0.00 247.00 us 214.00 us 280.00 us 2 RENAME
0.00 40.93 us 21.00 us 61.00 us 15 READLINK
0.00 8590.33 us 650.00 us 24313.00 us 3 MKNOD
0.00 11961.00 us 6810.00 us 17024.00 us 3 SYMLINK
0.02 41.69 us 12.00 us 592.00 us 5122 STAT
0.02 48.73 us 14.00 us 212.00 us 4533 FSTAT
0.03 50.30 us 9.00 us 48239.00 us 6471 READDIR
0.03 119.86 us 82.00 us 1540.00 us 3207 RMDIR
0.05 82.39 us 36.00 us 247.00 us 6428 SETXATTR
0.06 109.19 us 65.00 us 929.00 us 6883 XATTROP
0.07 109.58 us 19.00 us 9203.00 us 7038 GETXATTR
0.07 41.92 us 17.00 us 290.00 us 19584 OPEN
0.10 22.54 us 6.00 us 1318.00 us 53065 FLUSH
0.13 39.83 us 16.00 us 15248.00 us 36748 OPENDIR
0.26 203.59 us 33.00 us 1563067.00 us 14798 FTRUNCATE
0.29 33.64 us 7.00 us 663.00 us 100016 ENTRYLK
0.33 312.28 us 133.00 us 957757.00 us 12380 UNLINK
0.47 58.11 us 23.00 us 156892.00 us 93610 SETATTR
0.56 52.35 us 9.00 us 284549.00 us 125238 FINODELK
0.71 91.53 us 36.00 us 28059.00 us 90354 WRITE
0.83 39.13 us 11.00 us 266939.00 us 248932 INODELK
1.30 4149.23 us 909.00 us 302884.00 us 3672 STATFS
1.91 164.64 us 49.00 us 486411.00 us 135360 FXATTROP
2.17 2777.19 us 133.00 us 1231294.00 us 9138 FSYNC
2.86 549.03 us 23.00 us 871225.00 us 60914 READDIRP
8.24 15024.78 us 503.00 us 1024291.00 us 6414 MKDIR
11.96 1201.78 us 27.00 us 1521246.00 us 116302 READ
30.08 14183.64 us 126.00 us 1366919.00 us 24790 CREATE
37.45 1107.39 us 29.00 us 2418700.00 us 395335 LOOKUP
Duration: 3332887 seconds
Data Read: 3474783830508 bytes
Data Written: 1531183844503 bytes
Interval 6 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 13 1 7
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 201791 103365 362
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 2
No. of Writes: 65495 32729 6118
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1 2 9
No. of Writes: 84277 8572 92260
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 67 527 472
No. of Writes: 674402 8759 10442
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1050 52273 1218750
No. of Writes: 2821 201335 1096401
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 5
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 35031 FORGET
0.00 0.00 us 0.00 us 0.00 us 602698 RELEASE
0.00 0.00 us 0.00 us 0.00 us 46990 RELEASEDIR
0.00 40.93 us 21.00 us 61.00 us 15 READLINK
0.00 17024.00 us 17024.00 us 17024.00 us 1 SYMLINK
0.01 115.41 us 65.00 us 929.00 us 959 XATTROP
0.01 41.81 us 13.00 us 592.00 us 2852 STAT
0.01 120.97 us 21.00 us 9203.00 us 1024 GETXATTR
0.02 48.86 us 16.00 us 202.00 us 3168 FSTAT
0.03 85.42 us 37.00 us 247.00 us 3215 SETXATTR
0.03 47.23 us 9.00 us 48239.00 us 6435 READDIR
0.04 119.86 us 82.00 us 1540.00 us 3207 RMDIR
0.06 41.53 us 17.00 us 290.00 us 13056 OPEN
0.07 22.41 us 6.00 us 597.00 us 31146 FLUSH
0.16 39.78 us 16.00 us 15248.00 us 36680 OPENDIR
0.24 32.15 us 7.00 us 410.00 us 68752 ENTRYLK
0.27 252.42 us 33.00 us 1563067.00 us 10011 FTRUNCATE
0.31 61.96 us 26.00 us 156892.00 us 46758 SETATTR
0.41 312.31 us 133.00 us 957757.00 us 12377 UNLINK
0.44 54.73 us 10.00 us 268631.00 us 74842 FINODELK
0.50 38.86 us 11.00 us 22112.00 us 119367 INODELK
0.56 93.20 us 36.00 us 28059.00 us 56038 WRITE
1.02 4339.86 us 909.00 us 302884.00 us 2191 STATFS
1.46 172.96 us 54.00 us 486411.00 us 78882 FXATTROP
1.73 2577.60 us 142.00 us 1231294.00 us 6262 FSYNC
3.35 514.16 us 23.00 us 5523.00 us 60826 READDIRP
6.14 17879.08 us 503.00 us 954711.00 us 3207 MKDIR
14.96 1210.34 us 27.00 us 1521246.00 us 115423 READ
23.07 17411.99 us 131.00 us 1253896.00 us 12377 CREATE
45.11 1600.87 us 29.00 us 2418700.00 us 263192 LOOKUP
Duration: 130678 seconds
Data Read: 163529971099 bytes
Data Written: 165033472212 bytes
Brick: ib-storage4:/export/brick_home/brick2
--------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 417 21 21
No. of Writes: 2799 44458 1920989
Block Size: 8b+ 16b+ 32b+
No. of Reads: 155 17212 383
No. of Writes: 120150069 123441450 1284806
Block Size: 64b+ 128b+ 256b+
No. of Reads: 887 1181 2361
No. of Writes: 3385737 3966309 675767
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 5231 7376 13678
No. of Writes: 3515157 2689689 2380036
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 41771 64205 115832
No. of Writes: 342914077 4318646 12575888
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 12130828 1088246 100747675
No. of Writes: 11367062 7674095 36049629
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 1256
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 2373725 FORGET
0.00 0.00 us 0.00 us 0.00 us 285379523 RELEASE
0.00 0.00 us 0.00 us 0.00 us 998266 RELEASEDIR
0.00 111.00 us 111.00 us 111.00 us 1 LINK
0.00 1469.50 us 459.00 us 2480.00 us 2 RENAME
0.00 48.11 us 21.00 us 118.00 us 362 READLINK
0.01 16737.14 us 178.00 us 38401.00 us 7 SYMLINK
0.03 68.16 us 30.00 us 1232.00 us 7480 REMOVEXATTR
0.05 118.38 us 68.00 us 1609.00 us 7480 XATTROP
0.06 22.60 us 7.00 us 691.00 us 46115 FLUSH
0.06 56.21 us 19.00 us 191.00 us 19803 OPEN
0.06 82.52 us 34.00 us 351.00 us 13917 SETXATTR
0.07 168.68 us 17.00 us 3043.00 us 7668 GETXATTR
0.07 370.83 us 16.00 us 237921.00 us 3759 FSTAT
0.08 225.61 us 11.00 us 372254.00 us 6473 READDIR
0.10 402.45 us 13.00 us 951568.00 us 4487 STAT
0.10 52.57 us 21.00 us 65823.00 us 36751 OPENDIR
0.14 805.40 us 97.00 us 1157607.00 us 3208 RMDIR
0.21 39.82 us 9.00 us 599.00 us 99982 ENTRYLK
0.22 35.51 us 11.00 us 10079.00 us 114585 FINODELK
0.33 60.18 us 28.00 us 64586.00 us 100986 SETATTR
0.53 44.28 us 11.00 us 326232.00 us 222108 INODELK
0.63 127.57 us 39.00 us 445997.00 us 92621 WRITE
1.53 3168.53 us 38.00 us 4933571.00 us 8977 FTRUNCATE
1.85 2779.44 us 154.00 us 5294733.00 us 12393 UNLINK
1.85 9334.79 us 22.00 us 10582833.00 us 3697 STATFS
2.33 1420.33 us 25.00 us 2699561.00 us 30539 READDIRP
3.60 546.24 us 58.00 us 2198782.00 us 122938 FXATTROP
6.13 7225.74 us 26.00 us 2715877.00 us 15817 READ
6.30 18012.53 us 150.00 us 3784104.00 us 6519 FSYNC
7.64 22184.56 us 628.00 us 996693.00 us 6414 MKDIR
28.04 21101.37 us 156.00 us 1056408.00 us 24756 CREATE
37.98 1613.62 us 22.00 us 2171864.00 us 438497 LOOKUP
Duration: 8245643 seconds
Data Read: 13781191303643 bytes
Data Written: 8899094872577 bytes
Interval 8 Stats:
Block Size: 1b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 9 17 61796
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 2
No. of Writes: 197721 497 47678
Block Size: 128b+ 256b+ 512b+
No. of Reads: 1 2 3
No. of Writes: 85079 8377 16024
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 5 3 75
No. of Writes: 13918 84114 570032
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 457 2478 249
No. of Writes: 5621 2445 1931
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 45784 858666 0
No. of Writes: 175805 227232 11
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 35025 FORGET
0.00 0.00 us 0.00 us 0.00 us 519793 RELEASE
0.00 0.00 us 0.00 us 0.00 us 46990 RELEASEDIR
0.00 48.11 us 21.00 us 118.00 us 362 READLINK
0.00 11342.67 us 178.00 us 33644.00 us 3 SYMLINK
0.00 66.33 us 31.00 us 361.00 us 1010 REMOVEXATTR
0.01 118.80 us 68.00 us 1609.00 us 1010 XATTROP
0.01 164.75 us 17.00 us 1027.00 us 1077 GETXATTR
0.02 84.53 us 35.00 us 351.00 us 4225 SETXATTR
0.04 53.48 us 21.00 us 191.00 us 10655 OPEN
0.04 22.30 us 7.00 us 691.00 us 26203 FLUSH
0.09 539.23 us 16.00 us 237921.00 us 2413 FSTAT
0.10 223.58 us 11.00 us 372254.00 us 6435 READDIR
0.12 635.83 us 14.00 us 951568.00 us 2712 STAT
0.13 52.55 us 21.00 us 65823.00 us 36680 OPENDIR
0.15 35.49 us 11.00 us 2418.00 us 63844 FINODELK
0.17 37.40 us 9.00 us 599.00 us 68774 ENTRYLK
0.18 805.59 us 97.00 us 1157607.00 us 3207 RMDIR
0.20 60.07 us 29.00 us 64586.00 us 47779 SETATTR
0.32 44.53 us 12.00 us 272234.00 us 106021 INODELK
0.33 113.37 us 40.00 us 410447.00 us 43123 WRITE
1.41 9444.66 us 883.00 us 10582833.00 us 2191 STATFS
1.67 4275.34 us 38.00 us 4933571.00 us 5735 FTRUNCATE
2.34 2781.47 us 158.00 us 5294733.00 us 12383 UNLINK
2.71 585.66 us 58.00 us 2198782.00 us 68014 FXATTROP
2.86 1384.26 us 25.00 us 2699561.00 us 30408 READDIRP
5.45 20084.19 us 210.00 us 3784104.00 us 3995 FSYNC
5.99 27503.67 us 643.00 us 996693.00 us 3207 MKDIR
7.72 12154.58 us 40.00 us 2715877.00 us 9344 READ
21.29 25301.02 us 159.00 us 977262.00 us 12380 CREATE
46.64 2453.60 us 22.00 us 2171864.00 us 279710 LOOKUP
Duration: 130678 seconds
Data Read: 115885000924 bytes
Data Written: 51705278415 bytes
Brick: ib-storage4:/export/brick_home/brick1
--------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 14 59 24
No. of Writes: 3102 143118 1913649
Block Size: 8b+ 16b+ 32b+
No. of Reads: 93 363 322
No. of Writes: 146972063 137996676 955372
Block Size: 64b+ 128b+ 256b+
No. of Reads: 602 1657 1914
No. of Writes: 3607769 3526969 593478
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 4941 10475 15626
No. of Writes: 3170256 3519492 3487318
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 42721 2865963 122751
No. of Writes: 421096284 6505251 16204384
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 241098 1246959 125695304
No. of Writes: 24321299 17112293 36754149
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 1137
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 2395042 FORGET
0.00 0.00 us 0.00 us 0.00 us 355901765 RELEASE
0.00 0.00 us 0.00 us 0.00 us 999338 RELEASEDIR
0.00 146.00 us 146.00 us 146.00 us 1 LINK
0.00 1843.67 us 817.00 us 3374.00 us 3 RENAME
0.00 10584.00 us 10584.00 us 10584.00 us 1 MKNOD
0.00 37.25 us 18.00 us 114.00 us 365 READLINK
0.00 21181.00 us 203.00 us 50727.00 us 3 SYMLINK
0.03 68.35 us 30.00 us 579.00 us 5992 REMOVEXATTR
0.03 76.14 us 15.00 us 134626.00 us 5533 STAT
0.04 95.69 us 14.00 us 38936.00 us 5679 FSTAT
0.05 203.33 us 96.00 us 142249.00 us 3208 RMDIR
0.05 107.27 us 11.00 us 138966.00 us 6473 READDIR
0.05 117.88 us 66.00 us 1731.00 us 5992 XATTROP
0.06 56.93 us 12.00 us 6374.00 us 13698 LK
0.07 83.66 us 36.00 us 9810.00 us 12429 SETXATTR
0.07 171.51 us 21.00 us 2559.00 us 6179 GETXATTR
0.12 56.28 us 17.00 us 15160.00 us 29042 OPEN
0.12 25.36 us 6.00 us 8211.00 us 69485 FLUSH
0.14 55.61 us 20.00 us 18606.00 us 36751 OPENDIR
0.28 40.26 us 9.00 us 2822.00 us 100078 ENTRYLK
0.37 36.25 us 11.00 us 876.00 us 144480 FINODELK
0.41 59.11 us 27.00 us 45322.00 us 99394 SETATTR
0.74 47.66 us 11.00 us 893771.00 us 218811 INODELK
0.90 3434.93 us 23.00 us 302647.00 us 3697 STATFS
0.96 118.00 us 36.00 us 584830.00 us 115233 WRITE
1.44 670.48 us 26.00 us 492717.00 us 30374 READDIRP
1.91 2174.28 us 155.00 us 3897490.00 us 12478 UNLINK
2.24 1640.66 us 33.00 us 3419939.00 us 19311 FTRUNCATE
4.25 394.50 us 54.00 us 1598213.00 us 152708 FXATTROP
5.08 6835.66 us 157.00 us 1509652.00 us 10528 FSYNC
8.49 18772.67 us 676.00 us 1475838.00 us 6414 MKDIR
9.15 901.13 us 19.00 us 641961.00 us 143997 READ
29.92 17156.74 us 155.00 us 1576799.00 us 24720 CREATE
33.02 1111.79 us 34.00 us 1251645.00 us 421029 LOOKUP
Duration: 8245643 seconds
Data Read: 16618489558419 bytes
Data Written: 11059829249488 bytes
Interval 8 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 11 3 16
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 174531 170339 480
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 4 0
No. of Writes: 73389 43453 7791
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 30 3 31
No. of Writes: 20833 13945 191335
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 73 404 2978
No. of Writes: 1399938 10827 5718
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 178 87941 1359963
No. of Writes: 4554 272285 280883
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 10
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 34961 FORGET
0.00 0.00 us 0.00 us 0.00 us 651615 RELEASE
0.00 0.00 us 0.00 us 0.00 us 46990 RELEASEDIR
0.00 6408.00 us 203.00 us 12613.00 us 2 SYMLINK
0.00 37.25 us 18.00 us 114.00 us 365 READLINK
0.01 66.95 us 30.00 us 266.00 us 995 REMOVEXATTR
0.01 120.86 us 66.00 us 1731.00 us 995 XATTROP
0.01 52.25 us 17.00 us 212.00 us 2969 STAT
0.02 169.76 us 21.00 us 1494.00 us 1062 GETXATTR
0.03 87.01 us 36.00 us 9810.00 us 4210 SETXATTR
0.04 114.75 us 14.00 us 38936.00 us 3685 FSTAT
0.06 203.36 us 96.00 us 142249.00 us 3207 RMDIR
0.06 104.05 us 11.00 us 138966.00 us 6435 READDIR
0.07 54.16 us 23.00 us 15160.00 us 14075 OPEN
0.07 22.60 us 8.00 us 1020.00 us 34107 FLUSH
0.18 55.58 us 20.00 us 18606.00 us 36680 OPENDIR
0.23 37.95 us 9.00 us 2822.00 us 69138 ENTRYLK
0.25 58.53 us 27.00 us 45322.00 us 48045 SETATTR
0.25 35.42 us 11.00 us 876.00 us 80318 FINODELK
0.50 52.27 us 12.00 us 893771.00 us 106580 INODELK
0.74 3764.05 us 678.00 us 302647.00 us 2191 STATFS
0.77 127.19 us 38.00 us 584830.00 us 67703 WRITE
1.69 622.66 us 26.00 us 492717.00 us 30272 READDIRP
2.43 2174.91 us 159.00 us 3897490.00 us 12474 UNLINK
2.52 2691.21 us 38.00 us 3419939.00 us 10441 FTRUNCATE
4.03 532.52 us 54.00 us 1598213.00 us 84534 FXATTROP
4.52 7344.64 us 157.00 us 1509652.00 us 6877 FSYNC
6.68 23269.92 us 688.00 us 1475838.00 us 3207 MKDIR
11.58 934.12 us 38.00 us 641961.00 us 138480 READ
23.12 20702.08 us 155.00 us 1576799.00 us 12472 CREATE
40.12 1681.59 us 34.00 us 1251645.00 us 266403 LOOKUP
Duration: 130678 seconds
Data Read: 184440024688 bytes
Data Written: 73336009406 bytes
Brick: ib-storage2:/export/brick_home/brick2
--------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 3 8 6
No. of Writes: 162 210 998462
Block Size: 8b+ 16b+ 32b+
No. of Reads: 31 152 112
No. of Writes: 7255468 7480098 115554
Block Size: 64b+ 128b+ 256b+
No. of Reads: 231 528 732
No. of Writes: 1656480 1178874 111295
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 3765 2602 5057
No. of Writes: 419643 1997143 1594991
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 24673 30770 45075
No. of Writes: 14775885 237814 4559647
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 148825 884156 34437754
No. of Writes: 2006298 2436275 7594413
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 411
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 574011 FORGET
0.00 0.00 us 0.00 us 0.00 us 23285514 RELEASE
0.00 0.00 us 0.00 us 0.00 us 331847 RELEASEDIR
0.00 328.00 us 314.00 us 342.00 us 2 RENAME
0.00 70.92 us 37.00 us 139.00 us 12 READLINK
0.00 7881.67 us 1128.00 us 12950.00 us 3 MKNOD
0.01 29591.33 us 5819.00 us 56502.00 us 3 SYMLINK
0.02 63.03 us 18.00 us 950.00 us 3647 STAT
0.02 103.18 us 14.00 us 23715.00 us 3447 FSTAT
0.04 80.13 us 15.00 us 35916.00 us 6473 READDIR
0.04 188.79 us 119.00 us 13576.00 us 3207 RMDIR
0.04 89.58 us 35.00 us 1233.00 us 6877 REMOVEXATTR
0.07 158.64 us 82.00 us 1369.00 us 6877 XATTROP
0.09 65.28 us 23.00 us 2597.00 us 19598 OPEN
0.10 108.32 us 39.00 us 511.00 us 13305 SETXATTR
0.10 219.41 us 25.00 us 6959.00 us 7048 GETXATTR
0.12 33.32 us 9.00 us 988.00 us 53065 FLUSH
0.18 71.60 us 24.00 us 26355.00 us 36748 OPENDIR
0.21 211.82 us 46.00 us 244514.00 us 14798 FTRUNCATE
0.36 432.13 us 194.00 us 1563312.00 us 12380 UNLINK
0.37 54.59 us 11.00 us 685.00 us 100016 ENTRYLK
0.45 48.95 us 14.00 us 88451.00 us 137017 FINODELK
0.55 80.90 us 36.00 us 131149.00 us 100487 SETATTR
0.86 140.44 us 49.00 us 27274.00 us 90354 WRITE
0.90 53.56 us 15.00 us 31422.00 us 249040 INODELK
1.74 2804.45 us 178.00 us 1583694.00 us 9138 FSYNC
1.85 194.26 us 72.00 us 88526.00 us 140362 FXATTROP
2.09 43401.94 us 46.00 us 1108683.00 us 711 READDIRP
2.30 9230.89 us 1090.00 us 10582852.00 us 3672 STATFS
8.01 18426.27 us 941.00 us 2077151.00 us 6414 MKDIR
10.14 1747.37 us 51.00 us 1636611.00 us 85635 READ
28.67 17069.63 us 165.00 us 1729993.00 us 24790 CREATE
40.68 1451.32 us 37.00 us 2361218.00 us 413672 LOOKUP
Duration: 3287662 seconds
Data Read: 4593814739723 bytes
Data Written: 1522157031477 bytes
Interval 6 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 13 1 7
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 201791 103365 362
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 2 0
No. of Writes: 65495 32729 6118
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 23 2 1
No. of Writes: 84277 8572 92260
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 90 438 302
No. of Writes: 674402 8759 10442
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 291 85944 716849
No. of Writes: 2821 201335 1096401
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 5
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 35116 FORGET
0.00 0.00 us 0.00 us 0.00 us 603134 RELEASE
0.00 0.00 us 0.00 us 0.00 us 46990 RELEASEDIR
0.00 70.92 us 37.00 us 139.00 us 12 READLINK
0.00 56502.00 us 56502.00 us 56502.00 us 1 SYMLINK
0.01 91.61 us 42.00 us 457.00 us 959 REMOVEXATTR
0.01 169.26 us 96.00 us 1369.00 us 959 XATTROP
0.02 63.31 us 18.00 us 950.00 us 3564 STAT
0.02 247.53 us 30.00 us 6370.00 us 1032 GETXATTR
0.02 109.52 us 14.00 us 23715.00 us 2499 FSTAT
0.04 113.40 us 47.00 us 511.00 us 4174 SETXATTR
0.04 80.19 us 15.00 us 35916.00 us 6435 READDIR
0.05 188.79 us 119.00 us 13576.00 us 3207 RMDIR
0.07 64.72 us 23.00 us 2597.00 us 13064 OPEN
0.08 33.44 us 9.00 us 961.00 us 31146 FLUSH
0.17 212.83 us 46.00 us 244514.00 us 10011 FTRUNCATE
0.21 71.58 us 24.00 us 26355.00 us 36680 OPENDIR
0.30 53.07 us 11.00 us 685.00 us 68752 ENTRYLK
0.31 79.83 us 36.00 us 43346.00 us 47717 SETATTR
0.32 49.23 us 14.00 us 88451.00 us 80557 FINODELK
0.44 432.18 us 194.00 us 1563312.00 us 12377 UNLINK
0.53 54.24 us 15.00 us 31422.00 us 119434 INODELK
0.65 142.49 us 49.00 us 27274.00 us 56038 WRITE
1.29 194.44 us 73.00 us 88526.00 us 81590 FXATTROP
1.64 9201.79 us 1090.00 us 10582852.00 us 2191 STATFS
1.84 3595.43 us 178.00 us 1583694.00 us 6262 FSYNC
2.49 49818.49 us 75.00 us 1108683.00 us 612 READDIRP
6.91 26413.22 us 941.00 us 2077151.00 us 3207 MKDIR
12.20 1758.74 us 51.00 us 1636611.00 us 85043 READ
24.48 24253.61 us 192.00 us 1729993.00 us 12377 CREATE
45.85 1997.26 us 59.00 us 2361218.00 us 281557 LOOKUP
Duration: 130678 seconds
Data Read: 99960233724 bytes
Data Written: 165033472212 bytes
Brick: ib-storage2:/export/brick_home/brick1
--------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 4 6 3
No. of Writes: 2503 3565 677582
Block Size: 8b+ 16b+ 32b+
No. of Reads: 24 300 98
No. of Writes: 6520394 7921488 102885
Block Size: 64b+ 128b+ 256b+
No. of Reads: 281 449 821
No. of Writes: 1413392 2067916 145349
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 4018 2387 297908
No. of Writes: 295881 2031027 1892018
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 24837 30359 45248
No. of Writes: 19934955 337179 13338686
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 154966 822618 40237418
No. of Writes: 2317370 1936040 10737467
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 362
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 630341 FORGET
0.00 0.00 us 0.00 us 0.00 us 21814593 RELEASE
0.00 0.00 us 0.00 us 0.00 us 342341 RELEASEDIR
0.00 126.00 us 126.00 us 126.00 us 1 LINK
0.00 51.48 us 18.00 us 241.00 us 1287 STAT
0.01 21034.00 us 248.00 us 48661.00 us 7 SYMLINK
0.04 3683.78 us 46.00 us 99904.00 us 134 READDIRP
0.04 93.05 us 37.00 us 405.00 us 5331 REMOVEXATTR
0.04 87.73 us 16.00 us 44298.00 us 6473 READDIR
0.04 191.38 us 112.00 us 7419.00 us 3207 RMDIR
0.06 150.18 us 17.00 us 77306.00 us 5540 FSTAT
0.06 164.26 us 77.00 us 1495.00 us 5331 XATTROP
0.09 223.02 us 33.00 us 7473.00 us 5508 GETXATTR
0.09 111.19 us 43.00 us 649.00 us 11759 SETXATTR
0.13 67.38 us 23.00 us 3477.00 us 26207 OPEN
0.15 35.78 us 8.00 us 4286.00 us 58969 FLUSH
0.19 71.94 us 24.00 us 17725.00 us 36761 OPENDIR
0.40 55.64 us 11.00 us 8442.00 us 100440 ENTRYLK
0.41 451.15 us 168.00 us 799700.00 us 12466 UNLINK
0.56 429.19 us 44.00 us 1333102.00 us 18115 FTRUNCATE
0.56 49.73 us 13.00 us 1242.00 us 156945 FINODELK
0.57 79.78 us 34.00 us 34626.00 us 99260 SETATTR
0.96 53.88 us 15.00 us 28786.00 us 248213 INODELK
1.15 136.13 us 52.00 us 108559.00 us 117201 WRITE
2.25 194.82 us 72.00 us 161079.00 us 160001 FXATTROP
2.37 8955.90 us 665.00 us 10582939.00 us 3674 STATFS
6.67 14437.77 us 923.00 us 860778.00 us 6414 MKDIR
8.61 7413.79 us 124.00 us 3159235.00 us 16124 FSYNC
10.03 1565.37 us 39.00 us 696532.00 us 88922 READ
23.48 13084.14 us 184.00 us 3548041.00 us 24905 CREATE
41.02 1446.27 us 38.00 us 3968026.00 us 393626 LOOKUP
Duration: 3287662 seconds
Data Read: 5351151149122 bytes
Data Written: 2100690985553 bytes
Interval 6 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 24 4 147
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 290704 257822 343
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 4 24
No. of Writes: 47480 89041 5689
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 5 3 122
No. of Writes: 66843 7420 169036
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 67 438 298
No. of Writes: 1392575 14641 5378
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 322 75334 801470
No. of Writes: 4822 100530 123303
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 3
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 35118 FORGET
0.00 0.00 us 0.00 us 0.00 us 772121 RELEASE
0.00 0.00 us 0.00 us 0.00 us 47426 RELEASEDIR
0.00 531.38 us 52.00 us 1507.00 us 24 READDIRP
0.01 51.56 us 18.00 us 241.00 us 1195 STAT
0.01 94.37 us 45.00 us 405.00 us 938 REMOVEXATTR
0.01 32141.67 us 19610.00 us 48661.00 us 3 SYMLINK
0.01 180.00 us 99.00 us 1495.00 us 938 XATTROP
0.02 256.49 us 35.00 us 7473.00 us 1015 GETXATTR
0.04 115.42 us 49.00 us 649.00 us 4153 SETXATTR
0.05 86.37 us 16.00 us 44298.00 us 6435 READDIR
0.05 191.38 us 112.00 us 7419.00 us 3207 RMDIR
0.06 193.03 us 17.00 us 77306.00 us 3532 FSTAT
0.10 67.72 us 25.00 us 810.00 us 16721 OPEN
0.11 36.30 us 8.00 us 2116.00 us 34260 FLUSH
0.23 71.90 us 24.00 us 17725.00 us 36688 OPENDIR
0.33 54.34 us 11.00 us 8442.00 us 69102 ENTRYLK
0.34 79.89 us 34.00 us 34626.00 us 47935 SETATTR
0.40 50.24 us 14.00 us 876.00 us 91170 FINODELK
0.49 451.25 us 168.00 us 799700.00 us 12461 UNLINK
0.55 541.77 us 44.00 us 1333102.00 us 11537 FTRUNCATE
0.57 54.27 us 15.00 us 20314.00 us 120665 INODELK
0.84 135.99 us 52.00 us 20007.00 us 70103 WRITE
1.60 197.66 us 72.00 us 161079.00 us 92485 FXATTROP
1.72 8928.45 us 665.00 us 10582939.00 us 2193 STATFS
5.17 18384.45 us 951.00 us 860778.00 us 3207 MKDIR
10.04 11103.24 us 187.00 us 3159235.00 us 10311 FSYNC
12.20 1577.97 us 39.00 us 696532.00 us 88156 READ
18.43 16873.19 us 193.00 us 3548041.00 us 12458 CREATE
46.62 1993.57 us 54.00 us 3968026.00 us 266654 LOOKUP
Duration: 130678 seconds
Data Read: 110320985300 bytes
Data Written: 37104014555 bytes
Brick: ib-storage3:/export/brick_home/brick1
--------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 278 19 25
No. of Writes: 3102 143118 1913649
Block Size: 8b+ 16b+ 32b+
No. of Reads: 80 6032 286
No. of Writes: 146971892 137996567 955372
Block Size: 64b+ 128b+ 256b+
No. of Reads: 588 1410 2207
No. of Writes: 3607770 3526971 593478
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 5785 12100 14835
No. of Writes: 3170256 3519473 3487316
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 42840 7783521 109914
No. of Writes: 421096135 6505124 16204424
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 239345 1333729 130336390
No. of Writes: 24321304 17112308 36754137
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 1137
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 2394796 FORGET
0.00 0.00 us 0.00 us 0.00 us 355899920 RELEASE
0.00 0.00 us 0.00 us 0.00 us 999334 RELEASEDIR
0.00 188.00 us 188.00 us 188.00 us 1 LINK
0.00 1908.33 us 874.00 us 3563.00 us 3 RENAME
0.00 37467.00 us 37467.00 us 37467.00 us 1 MKNOD
0.00 14059.33 us 222.00 us 24467.00 us 3 SYMLINK
0.03 53.88 us 12.00 us 25596.00 us 6473 READDIR
0.03 67.33 us 15.00 us 14007.00 us 5914 FSTAT
0.04 86.94 us 34.00 us 234.00 us 6437 SETXATTR
0.06 123.88 us 23.00 us 1022.00 us 6178 GETXATTR
0.06 130.48 us 71.00 us 513.00 us 5990 XATTROP
0.08 177.54 us 13.00 us 375360.00 us 5709 STAT
0.08 334.94 us 95.00 us 618297.00 us 3208 RMDIR
0.12 55.30 us 14.00 us 2287.00 us 29015 OPEN
0.13 25.14 us 6.00 us 1710.00 us 69454 FLUSH
0.14 139.47 us 13.00 us 18248.00 us 13658 LK
0.16 56.22 us 19.00 us 9100.00 us 36752 OPENDIR
0.31 40.78 us 8.00 us 571.00 us 100078 ENTRYLK
0.39 36.27 us 10.00 us 2852.00 us 144224 FINODELK
0.41 57.80 us 27.00 us 14441.00 us 93402 SETATTR
0.79 47.91 us 12.00 us 955514.00 us 218883 INODELK
0.92 105.89 us 39.00 us 178789.00 us 115232 WRITE
1.14 1210.05 us 162.00 us 2372502.00 us 12478 UNLINK
1.51 5443.69 us 20.00 us 8027637.00 us 3697 STATFS
2.14 920.62 us 27.00 us 303564.00 us 30951 READDIRP
2.94 256.06 us 57.00 us 1014635.00 us 152672 FXATTROP
3.14 2159.56 us 33.00 us 1969773.00 us 19302 FTRUNCATE
8.14 16868.37 us 651.00 us 1587347.00 us 6414 MKDIR
8.17 10306.76 us 145.00 us 2630139.00 us 10528 FSYNC
10.37 1359.83 us 19.00 us 1810293.00 us 101304 READ
28.56 902.70 us 29.00 us 1302788.00 us 420456 LOOKUP
30.14 16201.16 us 156.00 us 1816457.00 us 24720 CREATE
Duration: 8245643 seconds
Data Read: 17274440006231 bytes
Data Written: 11059827627342 bytes
Interval 8 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 11 3 16
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 174531 170339 480
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 2
No. of Writes: 73389 43453 7791
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 24 6 134
No. of Writes: 20833 13945 191335
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 60 483 508
No. of Writes: 1399938 10827 5718
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 187 91224 2359724
No. of Writes: 4554 272285 280883
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 10
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 34970 FORGET
0.00 0.00 us 0.00 us 0.00 us 651615 RELEASE
0.00 0.00 us 0.00 us 0.00 us 46990 RELEASEDIR
0.00 12344.50 us 222.00 us 24467.00 us 2 SYMLINK
0.01 130.94 us 74.00 us 466.00 us 995 XATTROP
0.01 125.63 us 28.00 us 999.00 us 1062 GETXATTR
0.03 70.81 us 17.00 us 14007.00 us 3920 FSTAT
0.03 86.66 us 39.00 us 234.00 us 3215 SETXATTR
0.03 51.48 us 12.00 us 25596.00 us 6435 READDIR
0.07 53.33 us 19.00 us 2287.00 us 14075 OPEN
0.08 22.46 us 7.00 us 919.00 us 34107 FLUSH
0.09 315.35 us 13.00 us 375360.00 us 2782 STAT
0.11 335.01 us 95.00 us 618297.00 us 3207 RMDIR
0.20 56.19 us 19.00 us 9100.00 us 36680 OPENDIR
0.26 38.71 us 8.00 us 571.00 us 69138 ENTRYLK
0.27 58.00 us 27.00 us 14441.00 us 47050 SETATTR
0.28 35.56 us 10.00 us 2852.00 us 80074 FINODELK
0.41 38.56 us 12.00 us 5965.00 us 106597 INODELK
0.71 105.27 us 40.00 us 47018.00 us 67703 WRITE
0.77 3551.20 us 819.00 us 302771.00 us 2191 STATFS
1.49 1210.38 us 162.00 us 2372502.00 us 12474 UNLINK
2.05 244.56 us 59.00 us 1014635.00 us 84516 FXATTROP
2.76 904.05 us 27.00 us 303564.00 us 30887 READDIRP
3.67 3555.15 us 40.00 us 1969773.00 us 10441 FTRUNCATE
6.16 19414.95 us 659.00 us 1587347.00 us 3207 MKDIR
9.25 13593.16 us 151.00 us 2630139.00 us 6877 FSYNC
13.58 1426.05 us 36.00 us 1810293.00 us 96239 READ
22.27 18037.26 us 156.00 us 1816457.00 us 12472 CREATE
35.39 1344.44 us 29.00 us 1302788.00 us 265921 LOOKUP
Duration: 130678 seconds
Data Read: 315654875415 bytes
Data Written: 73336009406 bytes
Brick: ib-storage3:/export/brick_home/brick2
--------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 274 11 31
No. of Writes: 2799 44458 1920989
Block Size: 8b+ 16b+ 32b+
No. of Reads: 338 389 538
No. of Writes: 120149698 123441417 1284806
Block Size: 64b+ 128b+ 256b+
No. of Reads: 597 1162 2157
No. of Writes: 3385739 3966312 675768
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 5046 8420 13634
No. of Writes: 3515158 2689726 2380039
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 41888 63782 102251
No. of Writes: 342913738 4318286 12575829
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 7970182 966549 105807027
No. of Writes: 11367054 7674071 36049609
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 1256
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 2374161 FORGET
0.00 0.00 us 0.00 us 0.00 us 285378948 RELEASE
0.00 0.00 us 0.00 us 0.00 us 998265 RELEASEDIR
0.00 116.00 us 116.00 us 116.00 us 1 LINK
0.00 1426.00 us 532.00 us 2320.00 us 2 RENAME
0.00 2123.29 us 184.00 us 8690.00 us 7 SYMLINK
0.00 43.69 us 21.00 us 109.00 us 365 READLINK
0.02 52.39 us 16.00 us 270.00 us 3657 STAT
0.03 50.68 us 11.00 us 19034.00 us 6473 READDIR
0.03 81.07 us 18.00 us 25598.00 us 4302 FSTAT
0.05 144.31 us 98.00 us 818.00 us 3208 RMDIR
0.05 85.72 us 36.00 us 228.00 us 6437 SETXATTR
0.09 124.79 us 22.00 us 918.00 us 7665 GETXATTR
0.10 129.62 us 72.00 us 406.00 us 7477 XATTROP
0.10 22.89 us 8.00 us 1333.00 us 46116 FLUSH
0.11 56.61 us 22.00 us 764.00 us 19791 OPEN
0.19 51.17 us 21.00 us 9579.00 us 36752 OPENDIR
0.41 41.90 us 9.00 us 1097.00 us 99982 ENTRYLK
0.41 36.76 us 11.00 us 1136.00 us 114122 FINODELK
0.50 412.58 us 113.00 us 587678.00 us 12393 UNLINK
0.53 593.55 us 41.00 us 488559.00 us 8976 FTRUNCATE
0.53 57.84 us 27.00 us 18244.00 us 93506 SETATTR
1.05 114.82 us 38.00 us 20336.00 us 92620 WRITE
1.22 55.48 us 12.00 us 1269670.00 us 222190 INODELK
2.78 904.43 us 25.00 us 191013.00 us 31121 READDIRP
3.31 273.00 us 60.00 us 771031.00 us 122843 FXATTROP
3.33 9119.50 us 22.00 us 10582919.00 us 3697 STATFS
3.58 5565.15 us 160.00 us 2523803.00 us 6518 FSYNC
8.21 12973.44 us 644.00 us 1111351.00 us 6414 MKDIR
9.84 961.03 us 24.00 us 372996.00 us 103820 READ
27.55 11281.70 us 156.00 us 1231731.00 us 24756 CREATE
35.98 834.65 us 29.00 us 841807.00 us 437074 LOOKUP
Duration: 8245643 seconds
Data Read: 14271820772976 bytes
Data Written: 8899080749496 bytes
Interval 8 Stats:
Block Size: 1b+ 4b+ 8b+
No. of Reads: 0 0 1
No. of Writes: 9 17 61796
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 197721 497 47678
Block Size: 128b+ 256b+ 512b+
No. of Reads: 2 25 1
No. of Writes: 85080 8377 16024
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 4 11 47
No. of Writes: 13918 84114 570032
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 486 327 330
No. of Writes: 5621 2445 1931
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 45298 683840 0
No. of Writes: 175805 227232 11
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 35044 FORGET
0.00 0.00 us 0.00 us 0.00 us 519794 RELEASE
0.00 0.00 us 0.00 us 0.00 us 46990 RELEASEDIR
0.00 3030.00 us 184.00 us 8690.00 us 3 SYMLINK
0.00 43.69 us 21.00 us 109.00 us 365 READLINK
0.02 127.22 us 72.00 us 406.00 us 1010 XATTROP
0.02 123.31 us 22.00 us 918.00 us 1077 GETXATTR
0.02 52.98 us 17.00 us 270.00 us 3039 STAT
0.03 87.69 us 18.00 us 25598.00 us 2962 FSTAT
0.04 85.54 us 40.00 us 228.00 us 3215 SETXATTR
0.04 48.65 us 11.00 us 19034.00 us 6435 READDIR
0.06 144.29 us 98.00 us 818.00 us 3207 RMDIR
0.07 54.20 us 22.00 us 764.00 us 10658 OPEN
0.08 22.71 us 8.00 us 1226.00 us 26206 FLUSH
0.24 51.14 us 21.00 us 9579.00 us 36680 OPENDIR
0.30 36.62 us 11.00 us 634.00 us 63664 FINODELK
0.35 39.08 us 9.00 us 1097.00 us 68774 ENTRYLK
0.35 57.78 us 27.00 us 18244.00 us 46769 SETATTR
0.45 603.52 us 41.00 us 488559.00 us 5735 FTRUNCATE
0.54 39.61 us 12.00 us 46305.00 us 106064 INODELK
0.57 102.49 us 39.00 us 20336.00 us 43126 WRITE
0.66 412.69 us 161.00 us 587678.00 us 12383 UNLINK
2.21 4278.90 us 173.00 us 1120606.00 us 3995 FSYNC
2.22 252.31 us 60.00 us 771031.00 us 67986 FXATTROP
2.60 9176.09 us 769.00 us 10582919.00 us 2191 STATFS
3.60 895.74 us 25.00 us 191013.00 us 31066 READDIRP
6.37 15343.35 us 648.00 us 661938.00 us 3207 MKDIR
12.69 1109.54 us 30.00 us 372996.00 us 88343 READ
21.82 13615.09 us 159.00 us 1231731.00 us 12380 CREATE
44.62 1238.03 us 29.00 us 841807.00 us 278388 LOOKUP
Duration: 130678 seconds
Data Read: 92861175346 bytes
Data Written: 51705278558 bytes
Volume Name: vol_home Type: Distributed-Replicate Volume ID: f6ebcfc1-b735-4a0e-b1d7-47ed2d2e7af6 Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp,rdma Bricks: Brick1: ib-storage1:/export/brick_home/brick1 Brick2: ib-storage2:/export/brick_home/brick1 Brick3: ib-storage3:/export/brick_home/brick1 Brick4: ib-storage4:/export/brick_home/brick1 Brick5: ib-storage1:/export/brick_home/brick2 Brick6: ib-storage2:/export/brick_home/brick2 Brick7: ib-storage3:/export/brick_home/brick2 Brick8: ib-storage4:/export/brick_home/brick2 Options Reconfigured: performance.cache-max-file-size: 64MB performance.read-ahead: on performance.write-behind: on features.quota-deem-statfs: on performance.stat-prefetch: on performance.flush-behind: on features.default-soft-limit: 90% features.quota: on diagnostics.brick-log-level: CRITICAL auth.allow: localhost,127.0.0.1,10.* nfs.disable: on performance.cache-size: 1GB performance.write-behind-window-size: 4MB performance.quick-read: on performance.io-cache: on performance.io-thread-count: 64 nfs.enable-ino32: off
Status of volume: vol_home Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick ib-storage1:/export/brick_home/brick1 49170 Y 9324 Brick ib-storage2:/export/brick_home/brick1 49154 Y 4814 Brick ib-storage3:/export/brick_home/brick1 49152 Y 4602 Brick ib-storage4:/export/brick_home/brick1 49152 Y 4574 Brick ib-storage1:/export/brick_home/brick2 49172 Y 9330 Brick ib-storage2:/export/brick_home/brick2 49156 Y 4815 Brick ib-storage3:/export/brick_home/brick2 49154 Y 4614 Brick ib-storage4:/export/brick_home/brick2 49154 Y 4586 Self-heal Daemon on localhost N/A Y 9430 Quota Daemon on localhost N/A Y 9437 Self-heal Daemon on ib-storage2 N/A Y 4890 Quota Daemon on ib-storage2 N/A Y 4910 Self-heal Daemon on ib-storage3 N/A Y 4656 Quota Daemon on ib-storage3 N/A Y 4639 Self-heal Daemon on ib-storage4 N/A Y 4633 Quota Daemon on ib-storage4 N/A Y 4616 Task Status of Volume vol_home ------------------------------------------------------------------------------ There are no active volume tasks
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
