Hi all,
I'm running a 3 node gluster cluster in AWS (m4.xlarge, 4 x st1 500G disks) and just starting out.
glusterfs 3.7.11 built on Apr 18 2016 12:57:30 --servers
glusterfs 3.7.13 built on Jul 8 2016 15:46:50 -- fuse clients
Volume Name: marketplace_nfs
Type: Distributed-Replicate
Volume ID: 261f1e1e-49c1-4e93-b7a5-082590fd606f
Status: Started
Number of Bricks: 6 x 2 = 12
Transport-type: tcp
Bricks:
Brick1: 10.90.3.4:/data/data1/marketplace_nfs
Brick2: 10.90.4.82:/data/data1/marketplace_nfs
Brick3: 10.90.5.26:/data/data1/marketplace_nfs
Brick4: 10.90.3.4:/data/data0/marketplace_nfs
Brick5: 10.90.4.82:/data/data0/marketplace_nfs
Brick6: 10.90.5.26:/data/data0/marketplace_nfs
Brick7: 10.90.3.4:/data/data3/marketplace_nfs
Brick8: 10.90.4.82:/data/data3/marketplace_nfs
Brick9: 10.90.5.26:/data/data3/marketplace_nfs
Brick10: 10.90.3.4:/data/data2/marketplace_nfs
Brick11: 10.90.4.82:/data/data2/marketplace_nfs
Brick12: 10.90.5.26:/data/data2/marketplace_nfs
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
network.ping-timeout: 15
cluster.quorum-type: none
performance.readdir-ahead: on
The idea is that at least 2 nodes have a copy of the data in the volume.
I mount this on the client
10.90.3.4:/marketplace_nfs on /data type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
(10.90.3.4:/marketplace_nfs /data glusterfs backupvolfile-server=nfs.market.customer.envato.internal 0 0)
When I execute on a fuse mounted client:
$ time find ./* -maxdepth 1 -type d |wc -l
6219
real 10m42.753s
user 0m0.730s
sys 0m0.467s
Locally on one of the gluster nodes (over a larger number of files).
time find /data/data*/marketplace_nfs/ -maxdepth 1 -type d |wc -l
24908
real 0m0.202s
user 0m0.058s
sys 0m0.126s
I am not sure of the performance I can reasonably expect. The question I have is is this reasonable and if not, how can I improve performance? Should I reduce the number of bricks per node? Should I increase the replica count? Is there a fundamental problem in my configuration?
Any help appreciated,
Den
further information:
Each gluster node has similar top:
top - 03:19:57 up 7 days, 22:19, 1 user, load average: 12.84, 9.13, 6.56
Tasks: 142 total, 2 running, 140 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.4 us, 0.3 sy, 0.0 ni, 99.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16431672 total, 239224 free, 644316 used, 15548132 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 15365572 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3782 root 20 0 2002532 83264 7736 S 206.2 0.5 101:34.25 /usr/sbin/glusterfsd -s 10.90.3.4 --volfile-id marketplace_nfs.10.90.3.4.data-data1-marketplace_nfs -p /var/lib/glusterd/vols/marketplace_nfs/run/10.90.3.4-data-data1-marketplace_nfs.pid -S /var/run/gluster/aef8f955e6f50664936da6eee0158803.socket --brick-name /data/data1/marketplace_nfs -l /va+
3820 root 20 0 2068064 83124 8120 S 175.0 0.5 99:59.40 /usr/sbin/glusterfsd -s 10.90.3.4 --volfile-id marketplace_nfs.10.90.3.4.data-data3-marketplace_nfs -p /var/lib/glusterd/vols/marketplace_nfs/run/10.90.3.4-data-data3-marketplace_nfs.pid -S /var/run/gluster/a4ab633dc313d2e6fd8e53a37c77c30b.socket --brick-name /data/data3/marketplace_nfs -l /va+
3801 root 20 0 1335868 72172 8128 S 6.2 0.4 9:12.61 /usr/sbin/glusterfsd -s 10.90.3.4 --volfile-id marketplace_nfs.10.90.3.4.data-data0-marketplace_nfs -p /var/lib/glusterd/vols/marketplace_nfs/run/10.90.3.4-data-data0-marketplace_nfs.pid -S /var/run/gluster/ff9b43a6f3ea1b1ec0c04b9aee936dc8.socket --brick-name /data/data0/marketplace_nfs -l /va+
1 root 20 0 128556 8576 5736 S 0.0 0.1 0:06.74 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
2 root 20 0 0 0 0 S 0.0 0.0 0:00.10 [kthreadd]
Each glusterfs server is showing this behaviour, massive network sends and little disk usage:
---total-cpu-usage---- --dsk/xvdg----dsk/xvdh----dsk/xvdi----dsk/xvdj- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ: read writ: read writ: read writ| recv send| in out | int csw
24 74 2 0 0 0| 0 0 : 0 1024B: 0 0 : 0 0 | 129k 14M| 0 0 |6533 6308
25 74 1 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 160k 14M| 0 0 |6973 6669
24 74 2 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 161k 15M| 0 0 |7300 7562
25 74 2 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 149k 14M| 0 0 |6642 6528
31 68 1 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 124k 14M| 0 0 |6683 6343
24 64 12 0 0 0| 0 0 : 0 0 : 0 0 : 0 1024B| 142k 13M| 0 0 |6800 6939
25 73 1 0 0 1| 0 0 : 0 0 : 0 0 : 0 0 | 154k 14M| 0 0 |7054 6484
26 70 4 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 132k 14M| 0 0 |6803 6539
25 69 5 0 0 1| 0 0 : 0 0 : 0 0 : 0 0 | 160k 14M| 0 0 |7323 9557
24 70 5 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 143k 14M| 0 0 |6769 6935
25 73 2 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 137k 14M| 0 0 |6976 6257
29 70 0 0 0 0| 0 1024B: 0 0 : 0 0 : 0 0 | 171k 14M| 0 0 |6836 5853
27 70 3 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 179k 14M| 0 0 |7081 7359
22 62 16 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 133k 13M| 0 0 |6499 6478
15 36 48 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 91k 8798k| 0 0 |4686 3832
15 44 41 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 132k 10M| 0 0 |5708 5834
19 53 28 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 140k 12M| 0 0 |6471 5599
27 70 3 0 0 1| 0 0 : 0 0 : 0 0 : 0 0 | 151k 13M| 0 0 |7134 7197
24 74 2 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 159k 14M| 0 0 |7682 8206
28 69 3 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 134k 13M| 0 0 |7194 8798
29 70 1 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 72k 14M| 0 0 |6365 6426
15 44 42 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 61k 11M| 0 0 |4929 6895
0 0 100 0 0 0| 0 0 : 0 0 : 0 1024B: 0 0 | 25k 15k| 0 0 |1121 1259
0 0 100 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |4268B 3186B| 0 0 | 458 562
The profile info is:
$ cat /tmp/dataout1
Brick: 10.90.3.4:/data/data2/marketplace_nfs
--------------------------------------------
Cumulative Stats:
Block Size: 1b+ 64b+ 128b+
No. of Reads: 0 0 0
No. of Writes: 15 1 2
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 0 0 0
No. of Writes: 4 5 20
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 50 67 114
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 0 1 0
No. of Writes: 273 391 831
Block Size: 131072b+
No. of Reads: 14301
No. of Writes: 397404
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8687 FORGET
0.00 0.00 us 0.00 us 0.00 us 1800 RELEASE
0.00 0.00 us 0.00 us 0.00 us 199314 RELEASEDIR
0.00 30.00 us 26.00 us 34.00 us 3 STATFS
0.01 121.00 us 94.00 us 155.00 us 9 UNLINK
0.01 31.45 us 14.00 us 51.00 us 44 READDIR
0.03 106.95 us 13.00 us 618.00 us 60 GETXATTR
0.08 32.99 us 16.00 us 1321.00 us 470 STAT
3.68 40.79 us 19.00 us 588.00 us 16548 INODELK
5.16 114.49 us 60.00 us 8674.00 us 8274 RMDIR
7.20 39.89 us 8.00 us 1114.00 us 33114 ENTRYLK
17.10 49.37 us 1.00 us 9456.00 us 63574 OPENDIR
31.39 115.54 us 15.00 us 4769.00 us 49862 LOOKUP
35.33 123.04 us 23.00 us 7915.00 us 52693 READDIRP
Duration: 684652 seconds
Data Read: 1874518431 bytes
Data Written: 52198234474 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 926 RELEASEDIR
0.06 23.40 us 23.00 us 24.00 us 5 STAT
23.14 45.95 us 29.00 us 3089.00 us 929 OPENDIR
34.89 115.37 us 25.00 us 301.00 us 558 READDIRP
41.91 116.09 us 75.00 us 1106.00 us 666 LOOKUP
Duration: 58 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.3.4:/data/data0/marketplace_nfs
--------------------------------------------
Cumulative Stats:
Block Size: 1b+ 64b+ 128b+
No. of Reads: 0 0 0
No. of Writes: 19 1 1
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 0 0 0
No. of Writes: 3 8 15
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 55 71 92
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 0 1 1
No. of Writes: 274 413 838
Block Size: 131072b+
No. of Reads: 3
No. of Writes: 246923
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8688 FORGET
0.00 0.00 us 0.00 us 0.00 us 1792 RELEASE
0.00 0.00 us 0.00 us 0.00 us 199385 RELEASEDIR
0.00 35.67 us 32.00 us 38.00 us 3 STATFS
0.01 31.00 us 17.00 us 56.00 us 44 READDIR
0.01 178.92 us 94.00 us 632.00 us 13 UNLINK
0.03 119.72 us 13.00 us 597.00 us 60 GETXATTR
0.08 37.79 us 18.00 us 1467.00 us 470 STAT
3.29 42.40 us 19.00 us 624.00 us 16548 INODELK
5.06 130.39 us 58.00 us 12961.00 us 8274 RMDIR
7.53 48.42 us 9.00 us 2416.00 us 33122 ENTRYLK
19.89 66.66 us 1.00 us 8525.00 us 63574 OPENDIR
31.02 125.13 us 24.00 us 6020.00 us 52812 READDIRP
33.08 140.87 us 16.00 us 8952.00 us 50033 LOOKUP
Duration: 684652 seconds
Data Read: 558591 bytes
Data Written: 32474329315 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 926 RELEASEDIR
0.06 27.60 us 21.00 us 37.00 us 5 STAT
30.85 76.03 us 29.00 us 4056.00 us 929 OPENDIR
31.14 130.61 us 29.00 us 3018.00 us 546 READDIRP
37.95 130.47 us 52.00 us 1154.00 us 666 LOOKUP
Duration: 58 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.3.4:/data/data1/marketplace_nfs
--------------------------------------------
Cumulative Stats:
Block Size: 1b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 11 3 2
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 5 6 47
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 62 100 263
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 3 168
No. of Writes: 412 838 334487
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8657 FORGET
0.00 0.00 us 0.00 us 0.00 us 1779 RELEASE
0.00 0.00 us 0.00 us 0.00 us 199385 RELEASEDIR
0.00 54.33 us 42.00 us 66.00 us 3 STATFS
0.00 29.43 us 15.00 us 44.00 us 44 READDIR
0.00 5174.50 us 81.00 us 33257.00 us 8 UNLINK
0.01 44.76 us 19.00 us 26107.00 us 16548 INODELK
0.01 131.59 us 59.00 us 10776.00 us 8274 RMDIR
0.01 48.54 us 9.00 us 28337.00 us 33112 ENTRYLK
0.04 4452.26 us 18.00 us 132302.00 us 1263 STAT
0.24 548915.85 us 13.00 us 5224650.00 us 60 GETXATTR
0.93 2414.17 us 16.00 us 203291.00 us 51699 LOOKUP
1.48 3129.09 us 1.00 us 228575.00 us 63574 OPENDIR
97.28 68862.50 us 17.00 us 460340.00 us 189998 READDIRP
Duration: 684653 seconds
Data Read: 22303103 bytes
Data Written: 43952561838 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 926 RELEASEDIR
0.02 5927.05 us 25.00 us 75030.00 us 20 STAT
0.73 5543.27 us 78.00 us 141158.00 us 668 LOOKUP
1.29 7060.32 us 27.00 us 144956.00 us 930 OPENDIR
97.96 105543.88 us 25.00 us 384946.00 us 4722 READDIRP
Duration: 58 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.3.4:/data/data3/marketplace_nfs
--------------------------------------------
Cumulative Stats:
Block Size: 1b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 8 1 3
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 7 21 44
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 62 104 241
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 2 1 6
No. of Writes: 464 791 285704
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8659 FORGET
0.00 0.00 us 0.00 us 0.00 us 1762 RELEASE
0.00 0.00 us 0.00 us 0.00 us 199385 RELEASEDIR
0.00 38.33 us 30.00 us 52.00 us 3 STATFS
0.00 31.25 us 15.00 us 111.00 us 44 READDIR
0.00 1703.00 us 97.00 us 15797.00 us 14 UNLINK
0.00 42.25 us 19.00 us 13149.00 us 16548 INODELK
0.01 121.61 us 61.00 us 4492.00 us 8274 RMDIR
0.01 40.81 us 9.00 us 5622.00 us 33124 ENTRYLK
0.05 123021.10 us 13.00 us 2265444.00 us 60 GETXATTR
0.12 13702.60 us 17.00 us 296082.00 us 1263 STAT
1.16 3452.46 us 15.00 us 272058.00 us 50064 LOOKUP
1.45 3379.65 us 1.00 us 325140.00 us 63574 OPENDIR
97.20 75905.73 us 21.00 us 533385.00 us 190235 READDIRP
Duration: 684652 seconds
Data Read: 974227 bytes
Data Written: 37555843577 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 926 RELEASEDIR
0.08 20941.70 us 20.00 us 179031.00 us 20 STAT
0.98 5891.36 us 27.00 us 180270.00 us 929 OPENDIR
1.15 9582.95 us 81.00 us 194474.00 us 666 LOOKUP
97.79 113708.24 us 22.00 us 424985.00 us 4784 READDIRP
Duration: 58 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.4.82:/data/data1/marketplace_nfs
---------------------------------------------
Cumulative Stats:
Block Size: 1b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 11 3 2
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 5 6 47
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 62 100 263
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 2 43
No. of Writes: 412 838 334487
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8662 FORGET
0.00 0.00 us 0.00 us 0.00 us 1779 RELEASE
0.00 0.00 us 0.00 us 0.00 us 199383 RELEASEDIR
0.00 56.33 us 35.00 us 67.00 us 3 STATFS
0.01 155.88 us 112.00 us 235.00 us 8 UNLINK
0.01 30.86 us 16.00 us 61.00 us 44 READDIR
0.03 120.55 us 13.00 us 343.00 us 60 GETXATTR
0.10 45.39 us 22.00 us 2903.00 us 470 STAT
3.27 41.38 us 20.00 us 1282.00 us 16548 INODELK
4.04 102.23 us 60.00 us 6847.00 us 8274 RMDIR
5.91 37.39 us 9.00 us 1601.00 us 33112 ENTRYLK
21.07 69.38 us 37.00 us 9144.00 us 63574 OPENDIR
30.59 121.72 us 36.00 us 7740.00 us 52616 READDIRP
34.97 141.63 us 87.00 us 5463.00 us 51699 LOOKUP
Duration: 684653 seconds
Data Read: 5784405 bytes
Data Written: 43952561838 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 926 RELEASEDIR
0.04 22.60 us 22.00 us 43.00 us 5 STAT
27.35 131.09 us 36.00 us 2312.00 us 554 READDIRP
31.73 90.78 us 37.00 us 9144.00 us 928 OPENDIR
40.88 162.48 us 87.00 us 4127.00 us 668 LOOKUP
Duration: 57 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.4.82:/data/data0/marketplace_nfs
---------------------------------------------
Cumulative Stats:
Block Size: 1b+ 64b+ 128b+
No. of Reads: 0 0 0
No. of Writes: 11 1 1
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 0 0 0
No. of Writes: 7 10 15
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 2
No. of Writes: 46 63 120
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 0 0 0
No. of Writes: 248 438 866
Block Size: 131072b+
No. of Reads: 171
No. of Writes: 350568
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8702 FORGET
0.00 0.00 us 0.00 us 0.00 us 1862 RELEASE
0.00 0.00 us 0.00 us 0.00 us 199383 RELEASEDIR
0.00 33.45 us 15.00 us 66.00 us 44 READDIR
0.00 1804.00 us 43.00 us 5262.00 us 3 STATFS
0.00 678.89 us 90.00 us 5115.00 us 9 UNLINK
0.00 465.45 us 13.00 us 11085.00 us 60 GETXATTR
0.01 45.88 us 19.00 us 11892.00 us 16548 INODELK
0.01 105.35 us 63.00 us 2869.00 us 8274 RMDIR
0.02 38.91 us 9.00 us 18766.00 us 33114 ENTRYLK
0.10 6311.11 us 30.00 us 230926.00 us 1264 STAT
0.96 1596.82 us 101.00 us 304352.00 us 50059 LOOKUP
1.11 1456.50 us 30.00 us 309713.00 us 63574 OPENDIR
97.81 43329.13 us 27.00 us 1075779.00 us 188746 READDIRP
Duration: 684653 seconds
Data Read: 22434084 bytes
Data Written: 46064215951 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 926 RELEASEDIR
0.03 2744.90 us 30.00 us 49925.00 us 20 STAT
0.66 2076.87 us 101.00 us 149921.00 us 666 LOOKUP
0.83 1897.86 us 30.00 us 204514.00 us 928 OPENDIR
98.48 44524.85 us 27.00 us 1075779.00 us 4669 READDIRP
Duration: 57 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.4.82:/data/data3/marketplace_nfs
---------------------------------------------
Cumulative Stats:
Block Size: 1b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 8 1 3
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 7 21 44
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 1 0
No. of Writes: 62 104 241
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 0 2099
No. of Writes: 464 791 285704
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8665 FORGET
0.00 0.00 us 0.00 us 0.00 us 1762 RELEASE
0.00 0.00 us 0.00 us 0.00 us 199383 RELEASEDIR
0.00 28.00 us 24.00 us 31.00 us 3 STATFS
0.01 32.00 us 16.00 us 61.00 us 44 READDIR
0.01 110.00 us 84.00 us 138.00 us 14 UNLINK
0.04 117.45 us 13.00 us 636.00 us 60 GETXATTR
0.09 37.84 us 58.00 us 2402.00 us 470 STAT
3.58 40.91 us 19.00 us 3205.00 us 16548 INODELK
4.53 103.60 us 59.00 us 4561.00 us 8274 RMDIR
6.06 34.60 us 9.00 us 1404.00 us 33124 ENTRYLK
18.68 55.57 us 33.00 us 5689.00 us 63575 OPENDIR
32.75 117.30 us 30.00 us 6893.00 us 52807 READDIRP
34.25 129.40 us 103.00 us 5439.00 us 50064 LOOKUP
Duration: 684653 seconds
Data Read: 275135247 bytes
Data Written: 37555843577 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 926 RELEASEDIR
0.06 27.80 us 58.00 us 58.00 us 5 STAT
28.17 70.89 us 33.00 us 4767.00 us 928 OPENDIR
30.52 126.15 us 30.00 us 3003.00 us 565 READDIRP
41.25 144.62 us 103.00 us 2822.00 us 666 LOOKUP
Duration: 57 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.4.82:/data/data2/marketplace_nfs
---------------------------------------------
Cumulative Stats:
Block Size: 1b+ 16b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 12 1 4
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 0 0
No. of Writes: 1 3 7
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 0 0 0
No. of Writes: 14 43 68
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 0 1 0
No. of Writes: 114 263 419
Block Size: 65536b+ 131072b+
No. of Reads: 2 352
No. of Writes: 824 322197
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8705 FORGET
0.00 0.00 us 0.00 us 0.00 us 1807 RELEASE
0.00 0.00 us 0.00 us 0.00 us 199308 RELEASEDIR
0.00 29.33 us 22.00 us 36.00 us 3 STATFS
0.00 68.50 us 47.00 us 90.00 us 2 READLINK
0.00 28.57 us 14.00 us 46.00 us 44 READDIR
0.00 344.00 us 10.00 us 4695.00 us 60 GETXATTR
0.00 1442.17 us 93.00 us 8467.00 us 18 UNLINK
0.01 100.33 us 60.00 us 8658.00 us 8274 RMDIR
0.01 70.57 us 20.00 us 485119.00 us 16548 INODELK
0.03 86.06 us 9.00 us 889356.00 us 33132 ENTRYLK
0.12 9973.16 us 24.00 us 403909.00 us 1264 STAT
1.06 2120.37 us 89.00 us 411175.00 us 50520 LOOKUP
1.20 1900.07 us 30.00 us 374073.00 us 63575 OPENDIR
97.57 52219.09 us 28.00 us 1051061.00 us 188703 READDIRP
Duration: 684653 seconds
Data Read: 46382302 bytes
Data Written: 42340686672 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 926 RELEASEDIR
0.06 6935.55 us 24.00 us 135183.00 us 20 STAT
0.37 1290.03 us 89.00 us 168901.00 us 668 LOOKUP
0.60 1503.80 us 30.00 us 185597.00 us 929 OPENDIR
98.97 49757.74 us 28.00 us 1051061.00 us 4625 READDIRP
Duration: 57 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.5.26:/data/data0/marketplace_nfs
---------------------------------------------
Cumulative Stats:
Block Size: 1b+ 64b+ 128b+
No. of Reads: 0 0 0
No. of Writes: 11 1 1
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 0 0 0
No. of Writes: 7 10 15
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 1 0
No. of Writes: 46 63 120
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 2 0 1
No. of Writes: 248 438 866
Block Size: 131072b+
No. of Reads: 905
No. of Writes: 350568
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8710 FORGET
0.00 0.00 us 0.00 us 0.00 us 1862 RELEASE
0.00 0.00 us 0.00 us 0.00 us 199383 RELEASEDIR
0.00 109.67 us 32.00 us 238.00 us 3 STATFS
0.01 37.95 us 20.00 us 82.00 us 44 READDIR
0.02 407.44 us 95.00 us 2742.00 us 9 UNLINK
0.03 114.35 us 17.00 us 417.00 us 60 GETXATTR
0.10 45.22 us 18.00 us 1975.00 us 470 STAT
3.23 39.57 us 17.00 us 512.00 us 16548 INODELK
4.09 100.09 us 61.00 us 5656.00 us 8274 RMDIR
6.11 37.33 us 9.00 us 1217.00 us 33114 ENTRYLK
20.72 65.99 us 1.00 us 6995.00 us 63574 OPENDIR
31.05 119.10 us 24.00 us 7180.00 us 52780 READDIRP
34.64 140.09 us 22.00 us 7989.00 us 50059 LOOKUP
Duration: 684616 seconds
Data Read: 118761609 bytes
Data Written: 46064215951 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 926 RELEASEDIR
0.45 229.20 us 22.00 us 1051.00 us 5 STAT
29.27 133.60 us 25.00 us 5005.00 us 558 READDIRP
29.65 81.38 us 30.00 us 3405.00 us 928 OPENDIR
40.63 155.39 us 67.00 us 2723.00 us 666 LOOKUP
Duration: 57 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.5.26:/data/data2/marketplace_nfs
---------------------------------------------
Cumulative Stats:
Block Size: 1b+ 16b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 12 1 4
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 0 0
No. of Writes: 1 3 7
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 0 0 1
No. of Writes: 14 43 68
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 0 0 1
No. of Writes: 114 263 419
Block Size: 65536b+ 131072b+
No. of Reads: 1 10072
No. of Writes: 824 322197
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8714 FORGET
0.00 0.00 us 0.00 us 0.00 us 1807 RELEASE
0.00 0.00 us 0.00 us 0.00 us 199308 RELEASEDIR
0.00 44.33 us 25.00 us 73.00 us 3 STATFS
0.01 35.36 us 17.00 us 64.00 us 44 READDIR
0.07 204.87 us 14.00 us 3290.00 us 60 GETXATTR
0.07 706.28 us 98.00 us 10710.00 us 18 UNLINK
0.09 34.82 us 18.00 us 2003.00 us 470 STAT
2.80 53.16 us 24.00 us 5770.00 us 9488 READLINK
3.50 38.14 us 19.00 us 2399.00 us 16548 INODELK
4.40 95.87 us 54.00 us 10825.00 us 8274 RMDIR
5.82 31.65 us 8.00 us 2535.00 us 33132 ENTRYLK
17.28 49.00 us 1.00 us 4676.00 us 63574 OPENDIR
32.61 116.38 us 21.00 us 8548.00 us 50519 LOOKUP
33.36 114.35 us 25.00 us 8772.00 us 52588 READDIRP
Duration: 684615 seconds
Data Read: 1320327387 bytes
Data Written: 42340686672 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 926 RELEASEDIR
0.05 45.50 us 45.00 us 46.00 us 2 READLINK
0.07 24.20 us 21.00 us 28.00 us 5 STAT
21.85 43.76 us 28.00 us 829.00 us 928 OPENDIR
35.71 121.53 us 25.00 us 2901.00 us 546 READDIRP
42.33 117.76 us 55.00 us 296.00 us 668 LOOKUP
Duration: 57 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.5.26:/data/data1/marketplace_nfs
---------------------------------------------
Cumulative Stats:
Block Size: 1b+ 64b+ 128b+
No. of Reads: 0 0 0
No. of Writes: 19 1 1
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 0 0 0
No. of Writes: 3 8 15
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 55 71 92
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 2 0 4
No. of Writes: 274 413 838
Block Size: 131072b+
No. of Reads: 15212
No. of Writes: 246923
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8683 FORGET
0.00 0.00 us 0.00 us 0.00 us 1792 RELEASE
0.00 0.00 us 0.00 us 0.00 us 199383 RELEASEDIR
0.00 73.67 us 24.00 us 152.00 us 3 STATFS
0.00 28.59 us 14.00 us 55.00 us 44 READDIR
0.00 179.22 us 11.00 us 2137.00 us 60 GETXATTR
0.00 1016.15 us 93.00 us 8833.00 us 13 UNLINK
0.01 43.64 us 20.00 us 8567.00 us 16548 INODELK
0.01 105.54 us 60.00 us 18316.00 us 8274 RMDIR
0.02 37.47 us 10.00 us 2952.00 us 33122 ENTRYLK
0.07 3915.24 us 17.00 us 210047.00 us 1263 STAT
0.76 1109.99 us 19.00 us 206017.00 us 50033 LOOKUP
0.99 1141.17 us 1.00 us 240750.00 us 63574 OPENDIR
98.15 40529.31 us 23.00 us 1009517.00 us 177968 READDIRP
Duration: 684616 seconds
Data Read: 1994272237 bytes
Data Written: 32474329315 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 926 RELEASEDIR
0.18 16199.70 us 21.00 us 169000.00 us 20 STAT
0.45 1231.90 us 57.00 us 80090.00 us 666 LOOKUP
0.51 990.99 us 28.00 us 117563.00 us 928 OPENDIR
98.86 42375.20 us 27.00 us 425098.00 us 4223 READDIRP
Duration: 57 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.5.26:/data/data3/marketplace_nfs
---------------------------------------------
Cumulative Stats:
Block Size: 1b+ 64b+ 128b+
No. of Reads: 0 0 0
No. of Writes: 15 1 2
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 0 0 0
No. of Writes: 4 5 20
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 0
No. of Writes: 50 67 114
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 2 1 1
No. of Writes: 273 391 831
Block Size: 131072b+
No. of Reads: 14324
No. of Writes: 397404
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8669 FORGET
0.00 0.00 us 0.00 us 0.00 us 1800 RELEASE
0.00 0.00 us 0.00 us 0.00 us 199389 RELEASEDIR
0.00 47.00 us 30.00 us 79.00 us 3 STATFS
0.00 171.32 us 15.00 us 6207.00 us 44 READDIR
0.00 463.12 us 13.00 us 7289.00 us 60 GETXATTR
0.00 3234.22 us 99.00 us 24983.00 us 9 UNLINK
0.01 45.34 us 19.00 us 21238.00 us 16548 INODELK
0.01 104.78 us 54.00 us 15752.00 us 8274 RMDIR
0.01 38.32 us 9.00 us 22041.00 us 33114 ENTRYLK
0.14 11662.03 us 15.00 us 313625.00 us 1263 STAT
1.07 2314.18 us 29.00 us 288612.00 us 49863 LOOKUP
1.30 2209.79 us 1.00 us 363556.00 us 63573 OPENDIR
97.46 55158.65 us 25.00 us 602484.00 us 190328 READDIRP
Duration: 684616 seconds
Data Read: 1877692288 bytes
Data Written: 52198234474 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 926 RELEASEDIR
0.16 18884.20 us 21.00 us 172352.00 us 20 STAT
0.61 1582.79 us 27.00 us 232040.00 us 927 OPENDIR
0.75 2693.21 us 58.00 us 212646.00 us 666 LOOKUP
98.49 50363.67 us 26.00 us 438706.00 us 4708 READDIRP
Duration: 57 seconds
Data Read: 0 bytes
Data Written: 0 bytes
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
