Am Di., 5. Nov. 2019 um 12:06 Uhr schrieb RAFI KC <rkavunga@xxxxxxxxxx>:
On 11/4/19 8:46 PM, David Spisla wrote:
Dear Gluster Community,
I also have a issue concerning performance. The last days I updated our test cluster from GlusterFS v5.5 to v7.0 . The setup in general:
2 HP DL380 Servers with 10Gbit NICs, 1 Distribute-Replica 2 Volume with 2 Replica Pairs. Client is SMB Samba (access via vfs_glusterfs) . I did several tests to ensure that Samba don't causes the fall.The setup ist completely the same except the Gluster Version
Here are my results:64KiB 1MiB 10MiB (Filesize)3,49 47,41 300,50 (Values in MiB/s with GlusterFS v5.5)
0,16 2,61 76,63 (Values in MiB/s with GlusterFS v7.0)
Can you please share the profile information [1] for both versions? Also it would be really helpful if you can mention the io patterns that used for this tests.
[1] : https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/
Hello Rafi,
thank you for your help.
* First more information about the io patterns: As a client we use a DL360 Windws Server 2017 machine with 10Gbit NIC connected to the storage machines. The share will be mounted via SMB and the tests writes with fio. We use this job files (see attachment). Each job file will be executed separetely and there is a sleep about 60s between each test run to calm down the system before starting a new test.
* Attached below you find the profile output from the tests with v5.5 (ctime enabled), v7.0 (ctime enabled).
* Beside of the tests with Samba I did also some fio tests directly on the FUSE Mounts (locally on one of the storage nodes). The results show that there is only a small decrease of performance between v5.5 and v7.0
(All values in MiB/s)
64KiB 1MiB 10MiB
50,09 679,96 1023,02 (v5.5)
47,00 656,46 977,60 (v7.0)
50,09 679,96 1023,02 (v5.5)
47,00 656,46 977,60 (v7.0)
It seems to be that the combination of samba + gluster7.0 has a lot of problems, or not?
We use this volume options (GlusterFS 7.0):
Volume Name: archive1
Type: Distributed-Replicate
Volume ID: 44c17844-0bd4-4ca2-98d8-a1474add790c
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
Brick2: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
Brick3: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
Brick4: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
user.smb: disable
features.read-only: off
features.worm: off
features.worm-file-level: on
features.retention-mode: enterprise
features.default-retention-period: 120
network.ping-timeout: 10
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.nl-cache: on
performance.nl-cache-timeout: 600
client.event-threads: 32
server.event-threads: 32
cluster.lookup-optimize: on
performance.stat-prefetch: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
performance.cache-samba-metadata: on
performance.cache-ima-xattrs: on
performance.io-thread-count: 64
cluster.use-compound-fops: on
performance.cache-size: 512MB
performance.cache-refresh-timeout: 10
performance.read-ahead: off
performance.write-behind-window-size: 4MB
performance.write-behind: on
storage.build-pgfid: on
features.ctime: on
cluster.quorum-type: fixed
cluster.quorum-count: 1
features.bitrot: on
features.scrub: Active
features.scrub-freq: daily
For GlusterFS 5.5 its nearly the same except the fact that there were 2 options to enable ctime feature.
Ctime stores additional metadata information as an extended attributes which sometimes exceeds the default inode size. In such scenarios the additional xattrs won't fit into the default size. This will result in additional blocks to be used to store xattrs in the inide, which will effect the latency. This is purely based on the i/o operations and the total xattrs size stored in the inode.
Is it possible for you to repeat the test by disabling ctime or increasing the inode size to a higher value say 1024KB?
I will do so but for today I could not finish tests with ctime disabled (or higher inode value) because it takes a lot of time with v7.0 due to the low performance and I will perform it tomorrow. As soon as possible I give you the results.
By the way: You really mean inode size on xfs layer 1024KB? Or do you mean 1024Bytes? We use per default 512Bytes, because this is the recommended size until now . But it seems to be that there is a need for a new recommendation when using ctime feature as a default. I can not image that this is the real cause for the low performance because in v5.5 we also use ctime feature with inode size 512Bytes.
Regards
David
Our optimization for Samba looks like this (for every version):
[global]
workgroup = SAMBA
netbios name = CLUSTER
kernel share modes = no
aio read size = 1
aio write size = 1
kernel oplocks = no
max open files = 100000
nt acl support = no
security = user
server min protocol = SMB2
store dos attributes = no
strict locking = no
full_audit:failure = pwrite_send pwrite_recv pwrite offload_write_send offload_write_recv create_file open unlink connect disconnect rename chown fchown lchown chmod fchmod mkdir rmdir ntimes ftruncate fallocate
full_audit:success = pwrite_send pwrite_recv pwrite offload_write_send offload_write_recv create_file open unlink connect disconnect rename chown fchown lchown chmod fchmod mkdir rmdir ntimes ftruncate fallocate
full_audit:facility = local5
durable handles = yes
posix locking = no
log level = 2
max log size = 100000
debug pid = yes
What can be the cause for this rapid falling of the performance for small files? Are some of our vol options not recommended anymore?
There were some patches concerning performance for small files in v6.0 und v7.0 :
#1670031: performance regression seen with smallfile workload tests
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/118564314 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/118564314 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
Gluster5.5 (Ctime enabled)
==========================
fs-dl380-c1-n1:~ # gluster vo profile repo1 info
Brick: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 13722 125700 15237
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 3236 746 30
Block Size: 4096b+ 16384b+ 32768b+
No. of Reads: 0 0 0
No. of Writes: 2 9051 9050
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 7 163 0
No. of Writes: 3034 214018 99
Block Size: 524288b+ 1048576b+ 2097152b+
No. of Reads: 0 0 0
No. of Writes: 431 4898 1732
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 19937 FORGET
0.00 0.00 us 0.00 us 0.00 us 28745 RELEASE
0.00 0.00 us 0.00 us 0.00 us 27033 RELEASEDIR
0.00 372.52 us 372.52 us 372.52 us 1 XATTROP
0.00 155.97 us 148.40 us 163.74 us 3 SETXATTR
0.00 355.85 us 319.36 us 392.34 us 3 MKDIR
0.00 35.10 us 14.17 us 58.11 us 39 IPC
0.00 2092.77 us 759.52 us 2785.95 us 3 RMDIR
0.00 57.69 us 13.35 us 514.30 us 135 READDIR
0.01 894.26 us 34.03 us 56491.66 us 182 READ
0.01 374.15 us 22.21 us 19577.39 us 827 STATFS
0.03 69.78 us 21.13 us 352.94 us 7778 FSTAT
0.03 282.54 us 17.73 us 23688.04 us 2268 FSETXATTR
0.04 60.61 us 14.01 us 10210.17 us 13348 FINODELK
0.04 355.90 us 31.44 us 56158.91 us 2325 OPEN
0.05 306.29 us 109.78 us 674.82 us 3665 SETATTR
0.10 276.25 us 21.09 us 1058896.03 us 7498 INODELK
0.16 507.07 us 181.63 us 5291.74 us 6607 CREATE
0.18 297.21 us 90.56 us 11128.32 us 13316 FXATTROP
0.24 773.47 us 179.92 us 27742.67 us 6606 UNLINK
0.33 265.41 us 12.03 us 62361.64 us 26444 ENTRYLK
0.88 234.65 us 18.24 us 15480.79 us 80510 LOOKUP
0.93 750.11 us 12.26 us 14390.35 us 26600 GETXATTR
1.04 1127.56 us 11.10 us 227020.48 us 19784 FLUSH
1.09 881.86 us 1.22 us 33956.54 us 26502 OPENDIR
4.70 595.48 us 68.68 us 230191.32 us 168825 WRITE
90.14 33977.47 us 84.81 us 129099.85 us 56766 READDIRP
0.00 0.00 us 0.00 us 0.00 us 1958138 UPCALL
0.00 0.00 us 0.00 us 0.00 us 1958010 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 2261 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 128 CI_FORGET
Duration: 6513 seconds
Data Read: 21823488 bytes
Data Written: 38141710505 bytes
Interval 1 Stats:
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 13722 125700 15237
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 3236 746 30
Block Size: 4096b+ 65536b+ 131072b+
No. of Reads: 0 7 163
No. of Writes: 2 3034 2
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 0
No. of Writes: 55 431 4898
Block Size: 2097152b+
No. of Reads: 0
No. of Writes: 1732
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 126 FORGET
0.00 0.00 us 0.00 us 0.00 us 8927 RELEASE
0.00 0.00 us 0.00 us 0.00 us 26500 RELEASEDIR
0.00 372.52 us 372.52 us 372.52 us 1 XATTROP
0.00 155.97 us 148.40 us 163.74 us 3 SETXATTR
0.00 355.85 us 319.36 us 392.34 us 3 MKDIR
0.00 36.18 us 14.17 us 58.11 us 34 IPC
0.00 2092.77 us 759.52 us 2785.95 us 3 RMDIR
0.00 57.02 us 13.35 us 514.30 us 127 READDIR
0.01 894.26 us 34.03 us 56491.66 us 182 READ
0.01 380.67 us 22.21 us 19577.39 us 806 STATFS
0.03 69.78 us 21.13 us 352.94 us 7778 FSTAT
0.03 282.54 us 17.73 us 23688.04 us 2268 FSETXATTR
0.04 60.61 us 14.01 us 10210.17 us 13348 FINODELK
0.04 355.90 us 31.44 us 56158.91 us 2325 OPEN
0.05 306.29 us 109.78 us 674.82 us 3665 SETATTR
0.10 276.25 us 21.09 us 1058896.03 us 7498 INODELK
0.16 507.07 us 181.63 us 5291.74 us 6607 CREATE
0.18 297.21 us 90.56 us 11128.32 us 13316 FXATTROP
0.24 773.47 us 179.92 us 27742.67 us 6606 UNLINK
0.33 265.41 us 12.03 us 62361.64 us 26444 ENTRYLK
0.88 234.65 us 18.24 us 15480.79 us 80496 LOOKUP
0.93 750.43 us 12.26 us 14390.35 us 26587 GETXATTR
1.04 1127.56 us 11.10 us 227020.48 us 19784 FLUSH
1.09 882.00 us 1.22 us 33956.54 us 26498 OPENDIR
4.70 595.47 us 68.68 us 230191.32 us 168825 WRITE
90.14 33978.50 us 84.81 us 129099.85 us 56766 READDIRP
0.00 0.00 us 0.00 us 0.00 us 1958138 UPCALL
0.00 0.00 us 0.00 us 0.00 us 1958010 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 2261 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 128 CI_FORGET
Duration: 789 seconds
Data Read: 21823488 bytes
Data Written: 9485570217 bytes
Brick: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 0 0 0
No. of Writes: 8952 8950 2966
Block Size: 131072b+ 262144b+ 524288b+
No. of Reads: 0 0 0
No. of Writes: 217821 88 484
Block Size: 1048576b+ 2097152b+
No. of Reads: 0 0
No. of Writes: 4982 1846
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 20194 FORGET
0.00 0.00 us 0.00 us 0.00 us 26431 RELEASE
0.00 0.00 us 0.00 us 0.00 us 26795 RELEASEDIR
0.00 164.43 us 161.90 us 167.62 us 3 SETXATTR
0.00 30.57 us 17.79 us 44.45 us 21 IPC
0.00 390.62 us 383.31 us 397.93 us 2 XATTROP
0.00 56.79 us 26.70 us 152.48 us 23 FSETXATTR
0.00 447.69 us 400.71 us 510.30 us 3 MKDIR
0.00 129.81 us 55.26 us 289.15 us 28 OPEN
0.00 75.71 us 14.13 us 816.87 us 75 READDIR
0.00 2188.47 us 796.44 us 2993.06 us 3 RMDIR
0.01 160.12 us 21.94 us 6000.03 us 813 STATFS
0.06 70.99 us 24.27 us 29788.04 us 7349 INODELK
0.07 73.26 us 24.65 us 677.95 us 7784 FSTAT
0.09 60.50 us 17.42 us 521.01 us 13194 FINODELK
0.13 312.52 us 117.48 us 565.10 us 3652 SETATTR
0.39 508.69 us 179.00 us 1052.82 us 6594 CREATE
0.44 292.42 us 90.47 us 4546.87 us 13194 FXATTROP
0.77 337.08 us 12.36 us 56472.78 us 19800 FLUSH
0.81 1067.61 us 218.87 us 50116.71 us 6594 UNLINK
1.08 356.53 us 1.54 us 22107.25 us 26472 OPENDIR
1.12 368.18 us 13.26 us 68738.68 us 26406 ENTRYLK
1.19 1003.64 us 166.98 us 6085.26 us 10330 WRITE
2.29 753.23 us 12.07 us 5793.53 us 26507 GETXATTR
2.44 253.59 us 16.41 us 13225.37 us 83900 LOOKUP
89.11 12640.00 us 20.25 us 145809.04 us 61411 READDIRP
0.00 0.00 us 0.00 us 0.00 us 425 UPCALL
0.00 0.00 us 0.00 us 0.00 us 35 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 27 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 390 CI_FORGET
Duration: 6513 seconds
Data Read: 0 bytes
Data Written: 38958530560 bytes
Interval 1 Stats:
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 0 0 0
No. of Writes: 2966 3 49
Block Size: 524288b+ 1048576b+ 2097152b+
No. of Reads: 0 0 0
No. of Writes: 484 4982 1846
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 387 FORGET
0.00 0.00 us 0.00 us 0.00 us 6622 RELEASE
0.00 0.00 us 0.00 us 0.00 us 26472 RELEASEDIR
0.00 164.43 us 161.90 us 167.62 us 3 SETXATTR
0.00 30.40 us 17.79 us 44.45 us 19 IPC
0.00 390.62 us 383.31 us 397.93 us 2 XATTROP
0.00 56.79 us 26.70 us 152.48 us 23 FSETXATTR
0.00 447.69 us 400.71 us 510.30 us 3 MKDIR
0.00 129.81 us 55.26 us 289.15 us 28 OPEN
0.00 77.61 us 14.13 us 816.87 us 71 READDIR
0.00 2188.47 us 796.44 us 2993.06 us 3 RMDIR
0.01 161.55 us 21.94 us 6000.03 us 792 STATFS
0.06 70.99 us 24.27 us 29788.04 us 7349 INODELK
0.07 73.26 us 24.65 us 677.95 us 7784 FSTAT
0.09 60.50 us 17.42 us 521.01 us 13194 FINODELK
0.13 312.52 us 117.48 us 565.10 us 3652 SETATTR
0.39 508.69 us 179.00 us 1052.82 us 6594 CREATE
0.44 292.42 us 90.47 us 4546.87 us 13194 FXATTROP
0.77 337.08 us 12.36 us 56472.78 us 19800 FLUSH
0.81 1067.61 us 218.87 us 50116.71 us 6594 UNLINK
1.08 356.55 us 1.54 us 22107.25 us 26470 OPENDIR
1.12 368.18 us 13.26 us 68738.68 us 26406 ENTRYLK
1.19 1003.64 us 166.98 us 6085.26 us 10330 WRITE
2.29 753.43 us 12.07 us 5793.53 us 26499 GETXATTR
2.44 253.59 us 16.41 us 13225.37 us 83890 LOOKUP
89.11 12640.71 us 20.25 us 145809.04 us 61411 READDIRP
0.00 0.00 us 0.00 us 0.00 us 425 UPCALL
0.00 0.00 us 0.00 us 0.00 us 35 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 27 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 390 CI_FORGET
Duration: 789 seconds
Data Read: 0 bytes
Data Written: 9811918848 bytes
Brick: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 0 0 0
No. of Writes: 8952 8950 2966
Block Size: 131072b+ 262144b+ 524288b+
No. of Reads: 24 0 0
No. of Writes: 217821 88 484
Block Size: 1048576b+ 2097152b+
No. of Reads: 0 0
No. of Writes: 4982 1846
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 26404 FORGET
0.00 0.00 us 0.00 us 0.00 us 26426 RELEASE
0.00 0.00 us 0.00 us 0.00 us 27037 RELEASEDIR
0.00 242.57 us 242.57 us 242.57 us 1 READDIRP
0.00 270.43 us 224.72 us 316.13 us 2 REMOVEXATTR
0.00 376.30 us 357.25 us 395.35 us 2 XATTROP
0.00 411.18 us 363.84 us 462.52 us 3 MKDIR
0.00 36.88 us 25.54 us 54.42 us 36 IPC
0.00 176.85 us 32.37 us 492.36 us 15 FSETXATTR
0.01 206.12 us 108.73 us 412.08 us 21 OPEN
0.01 197.53 us 85.44 us 451.06 us 27 READ
0.01 1995.17 us 760.64 us 2693.43 us 3 RMDIR
0.01 1366.94 us 127.39 us 3611.57 us 5 SETXATTR
0.02 88.63 us 25.71 us 863.15 us 135 READDIR
0.13 116.18 us 34.05 us 2167.66 us 828 STATFS
0.78 78.60 us 26.05 us 52960.99 us 7352 INODELK
1.19 67.15 us 20.02 us 2608.91 us 13194 FINODELK
1.58 59.32 us 18.09 us 2981.27 us 19800 FLUSH
1.64 334.42 us 118.08 us 855.90 us 3654 SETATTR
2.57 72.45 us 19.16 us 732.36 us 26406 ENTRYLK
3.84 108.04 us 2.38 us 677.07 us 26502 OPENDIR
4.56 515.76 us 191.84 us 883.80 us 6594 CREATE
5.32 300.53 us 96.16 us 6531.99 us 13194 FXATTROP
7.63 862.82 us 248.89 us 4093.19 us 6594 UNLINK
14.59 1052.78 us 173.50 us 3653.35 us 10330 WRITE
26.87 753.92 us 21.93 us 5703.54 us 26567 GETXATTR
29.25 276.32 us 32.87 us 3104.49 us 78907 LOOKUP
0.00 0.00 us 0.00 us 0.00 us 6646 UPCALL
0.00 0.00 us 0.00 us 0.00 us 30 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 17 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 6616 CI_FORGET
Duration: 6596 seconds
Data Read: 3145728 bytes
Data Written: 38958530560 bytes
Interval 1 Stats:
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 0 24 0
No. of Writes: 2966 3 49
Block Size: 524288b+ 1048576b+ 2097152b+
No. of Reads: 0 0 0
No. of Writes: 484 4982 1846
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6597 FORGET
0.00 0.00 us 0.00 us 0.00 us 6615 RELEASE
0.00 0.00 us 0.00 us 0.00 us 26500 RELEASEDIR
0.00 242.57 us 242.57 us 242.57 us 1 READDIRP
0.00 270.43 us 224.72 us 316.13 us 2 REMOVEXATTR
0.00 376.30 us 357.25 us 395.35 us 2 XATTROP
0.00 411.18 us 363.84 us 462.52 us 3 MKDIR
0.00 37.24 us 25.54 us 54.42 us 34 IPC
0.00 176.85 us 32.37 us 492.36 us 15 FSETXATTR
0.01 206.12 us 108.73 us 412.08 us 21 OPEN
0.01 197.53 us 85.44 us 451.06 us 27 READ
0.01 1995.17 us 760.64 us 2693.43 us 3 RMDIR
0.01 1366.94 us 127.39 us 3611.57 us 5 SETXATTR
0.02 88.84 us 25.71 us 863.15 us 127 READDIR
0.13 116.49 us 34.05 us 2167.66 us 806 STATFS
0.78 78.60 us 26.05 us 52960.99 us 7352 INODELK
1.19 67.15 us 20.02 us 2608.91 us 13194 FINODELK
1.58 59.32 us 18.09 us 2981.27 us 19800 FLUSH
1.64 334.42 us 118.08 us 855.90 us 3654 SETATTR
2.57 72.45 us 19.16 us 732.36 us 26406 ENTRYLK
3.84 108.06 us 2.38 us 677.07 us 26498 OPENDIR
4.56 515.76 us 191.84 us 883.80 us 6594 CREATE
5.32 300.53 us 96.16 us 6531.99 us 13194 FXATTROP
7.63 862.82 us 248.89 us 4093.19 us 6594 UNLINK
14.59 1052.78 us 173.50 us 3653.35 us 10330 WRITE
26.87 754.21 us 21.93 us 5703.54 us 26555 GETXATTR
29.25 276.33 us 32.87 us 3104.49 us 78893 LOOKUP
0.00 0.00 us 0.00 us 0.00 us 6646 UPCALL
0.00 0.00 us 0.00 us 0.00 us 30 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 17 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 6616 CI_FORGET
Duration: 789 seconds
Data Read: 3145728 bytes
Data Written: 9811918848 bytes
Brick: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 13722 125700 15237
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 3236 746 30
Block Size: 4096b+ 16384b+ 32768b+
No. of Reads: 0 0 0
No. of Writes: 2 9051 9050
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 7 139 0
No. of Writes: 3034 214018 99
Block Size: 524288b+ 1048576b+ 2097152b+
No. of Reads: 0 0 0
No. of Writes: 431 4898 1732
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 26420 FORGET
0.00 0.00 us 0.00 us 0.00 us 28806 RELEASE
0.00 0.00 us 0.00 us 0.00 us 27035 RELEASEDIR
0.00 373.31 us 373.31 us 373.31 us 1 READDIRP
0.00 410.11 us 410.11 us 410.11 us 1 XATTROP
0.00 172.89 us 140.25 us 229.22 us 3 SETXATTR
0.00 348.41 us 319.63 us 404.97 us 3 MKDIR
0.00 40.45 us 31.64 us 58.41 us 36 IPC
0.00 2020.28 us 690.06 us 2754.73 us 3 RMDIR
0.01 74.65 us 18.21 us 868.57 us 135 READDIR
0.02 170.34 us 80.49 us 293.60 us 155 READ
0.07 107.76 us 30.67 us 5006.30 us 827 STATFS
0.14 77.87 us 27.17 us 17142.15 us 2329 FSETXATTR
0.25 136.56 us 42.42 us 1019.27 us 2386 OPEN
0.36 61.10 us 16.56 us 3013.27 us 7498 INODELK
0.60 57.67 us 18.69 us 337.84 us 13355 FINODELK
0.78 50.74 us 13.71 us 556.16 us 19784 FLUSH
0.90 315.86 us 109.57 us 1634.45 us 3665 SETATTR
1.26 61.32 us 17.83 us 759.14 us 26444 ENTRYLK
2.03 98.28 us 2.27 us 4248.68 us 26502 OPENDIR
2.58 500.80 us 174.65 us 1841.95 us 6607 CREATE
3.16 304.42 us 93.33 us 46617.14 us 13316 FXATTROP
4.43 859.31 us 203.35 us 3433.33 us 6606 UNLINK
13.58 230.56 us 25.82 us 1441.09 us 75517 LOOKUP
15.45 745.15 us 20.75 us 6030.94 us 26595 GETXATTR
54.36 412.94 us 74.85 us 5222.73 us 168825 WRITE
0.00 0.00 us 0.00 us 0.00 us 1966855 UPCALL
0.00 0.00 us 0.00 us 0.00 us 1957919 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 2325 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 8936 CI_FORGET
Duration: 6596 seconds
Data Read: 18677760 bytes
Data Written: 38141710505 bytes
Interval 1 Stats:
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 13722 125700 15237
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 3236 746 30
Block Size: 4096b+ 65536b+ 131072b+
No. of Reads: 0 7 139
No. of Writes: 2 3034 2
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 0
No. of Writes: 55 431 4898
Block Size: 2097152b+
No. of Reads: 0
No. of Writes: 1732
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6609 FORGET
0.00 0.00 us 0.00 us 0.00 us 8988 RELEASE
0.00 0.00 us 0.00 us 0.00 us 26500 RELEASEDIR
0.00 373.31 us 373.31 us 373.31 us 1 READDIRP
0.00 410.11 us 410.11 us 410.11 us 1 XATTROP
0.00 172.89 us 140.25 us 229.22 us 3 SETXATTR
0.00 348.41 us 319.63 us 404.97 us 3 MKDIR
0.00 40.85 us 31.64 us 58.41 us 34 IPC
0.00 2020.28 us 690.06 us 2754.73 us 3 RMDIR
0.01 74.66 us 18.21 us 868.57 us 127 READDIR
0.02 170.34 us 80.49 us 293.60 us 155 READ
0.07 107.62 us 30.67 us 5006.30 us 806 STATFS
0.14 77.87 us 27.17 us 17142.15 us 2329 FSETXATTR
0.25 136.56 us 42.42 us 1019.27 us 2386 OPEN
0.36 61.10 us 16.56 us 3013.27 us 7498 INODELK
0.60 57.67 us 18.69 us 337.84 us 13355 FINODELK
0.78 50.74 us 13.71 us 556.16 us 19784 FLUSH
0.90 315.86 us 109.57 us 1634.45 us 3665 SETATTR
1.26 61.32 us 17.83 us 759.14 us 26444 ENTRYLK
2.03 98.29 us 2.27 us 4248.68 us 26498 OPENDIR
2.58 500.80 us 174.65 us 1841.95 us 6607 CREATE
3.16 304.42 us 93.33 us 46617.14 us 13316 FXATTROP
4.43 859.31 us 203.35 us 3433.33 us 6606 UNLINK
13.58 230.56 us 25.82 us 1441.09 us 75503 LOOKUP
15.45 745.45 us 20.75 us 6030.94 us 26582 GETXATTR
54.37 412.94 us 74.85 us 5222.73 us 168825 WRITE
0.00 0.00 us 0.00 us 0.00 us 1966855 UPCALL
0.00 0.00 us 0.00 us 0.00 us 1957919 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 2325 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 8936 CI_FORGET
Duration: 789 seconds
Data Read: 18677760 bytes
Data Written: 9485570217 bytes
Gluster7.0 (Ctime enabled)
==========================
fs-dl380-c1-n1:~ # gluster vo profile repo1 info
Brick: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 23977 143222 10220
Block Size: 512b+ 1024b+ 8192b+
No. of Reads: 0 0 1
No. of Writes: 2067 51 0
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 8 171
No. of Writes: 26 2954 0
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 0
No. of Writes: 50 480 5002
Block Size: 2097152b+
No. of Reads: 0
No. of Writes: 1841
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6590 FORGET
0.00 0.00 us 0.00 us 0.00 us 7432 RELEASE
0.00 0.00 us 0.00 us 0.00 us 53292 RELEASEDIR
0.00 129.79 us 114.92 us 140.65 us 3 SETXATTR
0.00 386.13 us 331.50 us 440.75 us 2 READDIRP
0.00 292.58 us 249.60 us 360.18 us 3 XATTROP
0.00 576.10 us 446.41 us 775.50 us 3 MKDIR
0.00 1857.67 us 820.07 us 2405.74 us 3 RMDIR
0.00 33.78 us 14.96 us 120.77 us 216 IPC
0.02 73.47 us 11.54 us 315.66 us 412 FSETXATTR
0.02 169.11 us 43.46 us 405.47 us 188 READ
0.04 69.85 us 9.31 us 2344.93 us 906 READDIR
0.05 79.66 us 7.34 us 267.55 us 1151 GETXATTR
0.06 123.14 us 28.68 us 517.29 us 846 OPEN
0.32 85.70 us 25.91 us 399.66 us 6594 FGETXATTR
0.48 107.57 us 25.52 us 14326.25 us 7828 STATFS
0.49 43.59 us 11.85 us 2791.32 us 19801 FLUSH
0.50 62.56 us 13.76 us 1445.20 us 14181 FINODELK
0.67 177.44 us 56.71 us 687.67 us 6594 FSTAT
0.92 61.48 us 11.24 us 2399.86 us 26408 ENTRYLK
1.31 315.70 us 96.90 us 915.84 us 7305 SETATTR
1.97 525.68 us 173.64 us 3103.01 us 6595 CREATE
2.20 277.03 us 68.90 us 13473.77 us 13942 FXATTROP
2.70 214.72 us 9.56 us 2757.47 us 22114 LOOKUP
3.01 802.37 us 182.07 us 5259.34 us 6594 UNLINK
3.08 101.50 us 0.65 us 4039.32 us 53271 OPENDIR
36.60 4043.67 us 15.93 us 994013.08 us 15909 INODELK
45.55 421.68 us 64.99 us 9380.82 us 189890 WRITE
0.00 0.00 us 0.00 us 0.00 us 3677977 UPCALL
0.00 0.00 us 0.00 us 0.00 us 3671345 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 36 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 6632 CI_FORGET
Duration: 5698 seconds
Data Read: 22977840 bytes
Data Written: 9841829137 bytes
Interval 0 Stats:
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 23977 143222 10220
Block Size: 512b+ 1024b+ 8192b+
No. of Reads: 0 0 1
No. of Writes: 2067 51 0
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 8 171
No. of Writes: 26 2954 0
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 0
No. of Writes: 50 480 5002
Block Size: 2097152b+
No. of Reads: 0
No. of Writes: 1841
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6590 FORGET
0.00 0.00 us 0.00 us 0.00 us 7432 RELEASE
0.00 0.00 us 0.00 us 0.00 us 53292 RELEASEDIR
0.00 129.79 us 114.92 us 140.65 us 3 SETXATTR
0.00 386.13 us 331.50 us 440.75 us 2 READDIRP
0.00 292.58 us 249.60 us 360.18 us 3 XATTROP
0.00 576.10 us 446.41 us 775.50 us 3 MKDIR
0.00 1857.67 us 820.07 us 2405.74 us 3 RMDIR
0.00 33.78 us 14.96 us 120.77 us 216 IPC
0.02 73.47 us 11.54 us 315.66 us 412 FSETXATTR
0.02 169.11 us 43.46 us 405.47 us 188 READ
0.04 69.85 us 9.31 us 2344.93 us 906 READDIR
0.05 79.66 us 7.34 us 267.55 us 1151 GETXATTR
0.06 123.14 us 28.68 us 517.29 us 846 OPEN
0.32 85.70 us 25.91 us 399.66 us 6594 FGETXATTR
0.48 107.70 us 25.52 us 14326.25 us 7828 STATFS
0.49 43.59 us 11.85 us 2791.32 us 19801 FLUSH
0.50 62.56 us 13.76 us 1445.20 us 14181 FINODELK
0.67 177.44 us 56.71 us 687.67 us 6594 FSTAT
0.92 61.48 us 11.24 us 2399.86 us 26408 ENTRYLK
1.31 315.70 us 96.90 us 915.84 us 7305 SETATTR
1.97 525.68 us 173.64 us 3103.01 us 6595 CREATE
2.20 277.03 us 68.90 us 13473.77 us 13942 FXATTROP
2.70 214.72 us 9.56 us 2757.47 us 22114 LOOKUP
3.01 802.37 us 182.07 us 5259.34 us 6594 UNLINK
3.08 101.50 us 0.65 us 4039.32 us 53271 OPENDIR
36.60 4043.67 us 15.93 us 994013.08 us 15909 INODELK
45.55 421.68 us 64.99 us 9380.82 us 189890 WRITE
0.00 0.00 us 0.00 us 0.00 us 3677977 UPCALL
0.00 0.00 us 0.00 us 0.00 us 3671345 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 36 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 6632 CI_FORGET
Duration: 5698 seconds
Data Read: 22977840 bytes
Data Written: 9841829137 bytes
Brick: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
Block Size: 4096b+ 32768b+ 65536b+
No. of Reads: 0 0 0
No. of Writes: 1 24 3023
Block Size: 131072b+ 262144b+ 524288b+
No. of Reads: 320 0 0
No. of Writes: 0 41 436
Block Size: 1048576b+ 2097152b+
No. of Reads: 0 0
No. of Writes: 4856 1768
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6605 FORGET
0.00 0.00 us 0.00 us 0.00 us 7075 RELEASE
0.00 0.00 us 0.00 us 0.00 us 53292 RELEASEDIR
0.00 138.84 us 112.92 us 160.96 us 3 SETXATTR
0.00 399.64 us 344.42 us 454.87 us 2 READDIRP
0.00 571.87 us 398.94 us 844.47 us 3 MKDIR
0.01 2045.86 us 777.17 us 2709.66 us 3 RMDIR
0.01 35.10 us 12.94 us 93.54 us 216 IPC
0.06 72.26 us 12.14 us 334.52 us 432 FSETXATTR
0.10 113.49 us 26.00 us 484.88 us 473 OPEN
0.10 174.90 us 87.44 us 453.04 us 324 READ
0.11 65.07 us 11.03 us 4413.29 us 907 READDIR
0.11 67.32 us 13.89 us 1223.39 us 918 GETXATTR
1.06 87.26 us 26.59 us 194.28 us 6606 FGETXATTR
1.34 93.17 us 20.71 us 8258.97 us 7828 STATFS
1.37 56.19 us 14.38 us 588.75 us 13240 FINODELK
1.74 47.75 us 9.44 us 1288.06 us 19775 FLUSH
2.27 186.16 us 58.73 us 1957.66 us 6606 FSTAT
2.90 59.56 us 15.95 us 2550.84 us 26442 ENTRYLK
4.20 310.72 us 92.26 us 920.97 us 7338 SETATTR
6.31 517.93 us 194.94 us 3140.66 us 6606 CREATE
6.84 279.99 us 71.20 us 2480.15 us 13240 FXATTROP
8.44 214.97 us 15.99 us 3525.82 us 21284 LOOKUP
9.83 807.18 us 187.73 us 3510.05 us 6606 UNLINK
9.91 100.86 us 1.15 us 2942.26 us 53271 OPENDIR
19.45 1039.19 us 126.76 us 5976.65 us 10149 WRITE
23.82 873.49 us 13.82 us 996890.14 us 14792 INODELK
0.00 0.00 us 0.00 us 0.00 us 6707 UPCALL
0.00 0.00 us 0.00 us 0.00 us 82 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 21 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 6625 CI_FORGET
Duration: 5698 seconds
Data Read: 41943040 bytes
Data Written: 9455665152 bytes
Interval 0 Stats:
Block Size: 4096b+ 32768b+ 65536b+
No. of Reads: 0 0 0
No. of Writes: 1 24 3023
Block Size: 131072b+ 262144b+ 524288b+
No. of Reads: 320 0 0
No. of Writes: 0 41 436
Block Size: 1048576b+ 2097152b+
No. of Reads: 0 0
No. of Writes: 4856 1768
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6605 FORGET
0.00 0.00 us 0.00 us 0.00 us 7075 RELEASE
0.00 0.00 us 0.00 us 0.00 us 53292 RELEASEDIR
0.00 138.84 us 112.92 us 160.96 us 3 SETXATTR
0.00 399.64 us 344.42 us 454.87 us 2 READDIRP
0.00 571.87 us 398.94 us 844.47 us 3 MKDIR
0.01 2045.86 us 777.17 us 2709.66 us 3 RMDIR
0.01 35.10 us 12.94 us 93.54 us 216 IPC
0.06 72.26 us 12.14 us 334.52 us 432 FSETXATTR
0.10 113.49 us 26.00 us 484.88 us 473 OPEN
0.10 174.90 us 87.44 us 453.04 us 324 READ
0.11 65.07 us 11.03 us 4413.29 us 907 READDIR
0.11 67.32 us 13.89 us 1223.39 us 918 GETXATTR
1.06 87.26 us 26.59 us 194.28 us 6606 FGETXATTR
1.34 93.17 us 20.71 us 8258.97 us 7828 STATFS
1.37 56.19 us 14.38 us 588.75 us 13240 FINODELK
1.74 47.75 us 9.44 us 1288.06 us 19775 FLUSH
2.27 186.16 us 58.73 us 1957.66 us 6606 FSTAT
2.90 59.56 us 15.95 us 2550.84 us 26442 ENTRYLK
4.20 310.72 us 92.26 us 920.97 us 7338 SETATTR
6.31 517.93 us 194.94 us 3140.66 us 6606 CREATE
6.84 279.99 us 71.20 us 2480.15 us 13240 FXATTROP
8.44 214.97 us 15.99 us 3525.82 us 21284 LOOKUP
9.83 807.18 us 187.73 us 3510.05 us 6606 UNLINK
9.91 100.86 us 1.15 us 2942.26 us 53271 OPENDIR
19.45 1039.20 us 126.76 us 5976.65 us 10149 WRITE
23.82 873.49 us 13.82 us 996890.14 us 14792 INODELK
0.00 0.00 us 0.00 us 0.00 us 6707 UPCALL
0.00 0.00 us 0.00 us 0.00 us 82 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 21 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 6625 CI_FORGET
Duration: 5698 seconds
Data Read: 41943040 bytes
Data Written: 9455665152 bytes
Brick: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 23977 143222 10220
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 1
No. of Writes: 2067 51 0
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 1 170
No. of Writes: 26 2954 0
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 0
No. of Writes: 50 480 5002
Block Size: 2097152b+
No. of Reads: 0
No. of Writes: 1841
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 944 FORGET
0.00 0.00 us 0.00 us 0.00 us 7430 RELEASE
0.00 0.00 us 0.00 us 0.00 us 53292 RELEASEDIR
0.00 134.44 us 75.05 us 193.89 us 3 SETXATTR
0.00 268.64 us 259.90 us 278.55 us 3 XATTROP
0.00 439.49 us 203.80 us 682.65 us 3 MKDIR
0.00 1853.60 us 823.20 us 2393.82 us 3 RMDIR
0.00 34.05 us 13.77 us 101.66 us 215 IPC
0.00 50.12 us 19.28 us 294.14 us 410 FSETXATTR
0.00 176.78 us 47.52 us 268.32 us 173 READ
0.00 67.16 us 9.59 us 4569.92 us 906 READDIR
0.00 135.83 us 38.87 us 18143.17 us 837 OPEN
0.02 76.76 us 23.00 us 192.51 us 6594 FGETXATTR
0.04 65.77 us 10.26 us 19823.69 us 14227 FINODELK
0.04 68.49 us 11.99 us 18024.04 us 15909 INODELK
0.05 155.32 us 16.76 us 21625.26 us 7828 STATFS
0.08 142.52 us 37.26 us 4159.63 us 14358 FSTAT
0.08 296.31 us 85.27 us 2190.78 us 7305 SETATTR
0.13 509.05 us 179.26 us 6280.47 us 6595 CREATE
0.15 274.40 us 79.84 us 13303.23 us 13942 FXATTROP
0.18 233.24 us 8.82 us 53288.69 us 19801 FLUSH
0.19 738.29 us 191.82 us 26804.67 us 6594 UNLINK
0.44 416.08 us 11.09 us 31352.67 us 27762 LOOKUP
0.55 551.61 us 10.11 us 46117.06 us 26408 ENTRYLK
0.74 367.23 us 0.64 us 34196.75 us 53271 OPENDIR
2.73 372.74 us 8.60 us 61916.08 us 192531 GETXATTR
4.06 560.96 us 62.01 us 62168.04 us 189890 WRITE
90.53 13985.76 us 15.94 us 60188.98 us 170026 READDIRP
0.00 0.00 us 0.00 us 0.00 us 3672463 UPCALL
0.00 0.00 us 0.00 us 0.00 us 3671509 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 37 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 954 CI_FORGET
Duration: 5691 seconds
Data Read: 22379229 bytes
Data Written: 9841829137 bytes
Interval 0 Stats:
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 23977 143222 10220
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 1
No. of Writes: 2067 51 0
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 1 170
No. of Writes: 26 2954 0
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 0
No. of Writes: 50 480 5002
Block Size: 2097152b+
No. of Reads: 0
No. of Writes: 1841
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 944 FORGET
0.00 0.00 us 0.00 us 0.00 us 7430 RELEASE
0.00 0.00 us 0.00 us 0.00 us 53292 RELEASEDIR
0.00 134.44 us 75.05 us 193.89 us 3 SETXATTR
0.00 268.64 us 259.90 us 278.55 us 3 XATTROP
0.00 439.49 us 203.80 us 682.65 us 3 MKDIR
0.00 1853.60 us 823.20 us 2393.82 us 3 RMDIR
0.00 34.05 us 13.77 us 101.66 us 215 IPC
0.00 50.12 us 19.28 us 294.14 us 410 FSETXATTR
0.00 176.78 us 47.52 us 268.32 us 173 READ
0.00 67.16 us 9.59 us 4569.92 us 906 READDIR
0.00 135.83 us 38.87 us 18143.17 us 837 OPEN
0.02 76.76 us 23.00 us 192.51 us 6594 FGETXATTR
0.04 65.77 us 10.26 us 19823.69 us 14227 FINODELK
0.04 68.49 us 11.99 us 18024.04 us 15909 INODELK
0.05 155.32 us 16.76 us 21625.26 us 7828 STATFS
0.08 142.52 us 37.26 us 4159.63 us 14358 FSTAT
0.08 296.31 us 85.27 us 2190.78 us 7305 SETATTR
0.13 509.05 us 179.26 us 6280.47 us 6595 CREATE
0.15 274.40 us 79.84 us 13303.23 us 13942 FXATTROP
0.18 233.24 us 8.82 us 53288.69 us 19801 FLUSH
0.19 738.29 us 191.82 us 26804.67 us 6594 UNLINK
0.44 416.08 us 11.09 us 31352.67 us 27762 LOOKUP
0.55 551.61 us 10.11 us 46117.06 us 26408 ENTRYLK
0.74 367.23 us 0.64 us 34196.75 us 53271 OPENDIR
2.73 372.74 us 8.60 us 61916.08 us 192531 GETXATTR
4.06 560.96 us 62.01 us 62168.04 us 189890 WRITE
90.53 13985.90 us 15.94 us 60188.98 us 170026 READDIRP
0.00 0.00 us 0.00 us 0.00 us 3672463 UPCALL
0.00 0.00 us 0.00 us 0.00 us 3671509 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 37 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 954 CI_FORGET
Duration: 5691 seconds
Data Read: 22379229 bytes
Data Written: 9841829137 bytes
Brick: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
Block Size: 4096b+ 32768b+ 65536b+
No. of Reads: 0 0 7
No. of Writes: 1 24 3023
Block Size: 131072b+ 262144b+ 524288b+
No. of Reads: 320 0 0
No. of Writes: 0 41 436
Block Size: 1048576b+ 2097152b+
No. of Reads: 0 0
No. of Writes: 4856 1768
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 249 FORGET
0.00 0.00 us 0.00 us 0.00 us 7069 RELEASE
0.00 0.00 us 0.00 us 0.00 us 53292 RELEASEDIR
0.00 163.77 us 143.80 us 187.63 us 3 SETXATTR
0.00 539.28 us 199.35 us 909.49 us 3 MKDIR
0.00 1861.11 us 734.86 us 2433.60 us 3 RMDIR
0.00 34.80 us 14.08 us 120.65 us 215 IPC
0.00 47.22 us 16.75 us 265.51 us 426 FSETXATTR
0.00 105.54 us 34.69 us 438.87 us 474 OPEN
0.00 58.10 us 9.99 us 2784.86 us 906 READDIR
0.00 193.84 us 78.00 us 1070.84 us 338 READ
0.01 78.11 us 27.12 us 267.42 us 6606 FGETXATTR
0.02 62.29 us 15.49 us 2094.30 us 13240 FINODELK
0.03 71.48 us 13.41 us 2627.65 us 14792 INODELK
0.03 146.99 us 13.72 us 18406.23 us 7828 STATFS
0.06 144.89 us 37.17 us 1728.42 us 14262 FSTAT
0.06 294.14 us 95.08 us 1298.49 us 7338 SETATTR
0.09 501.65 us 171.12 us 3673.36 us 6606 CREATE
0.10 277.18 us 83.58 us 2255.78 us 13240 FXATTROP
0.11 618.23 us 182.22 us 31669.04 us 6606 UNLINK
0.19 247.20 us 12.45 us 56440.26 us 27624 LOOKUP
0.19 269.37 us 9.01 us 29850.52 us 26442 ENTRYLK
0.26 942.72 us 125.40 us 11143.54 us 10149 WRITE
0.52 569.83 us 8.92 us 3526.39 us 33889 GETXATTR
1.14 2122.15 us 8.75 us 170951.32 us 19775 FLUSH
1.33 920.75 us 0.81 us 27584.76 us 53271 OPENDIR
95.86 23866.45 us 76.91 us 66659.49 us 148246 READDIRP
0.00 0.00 us 0.00 us 0.00 us 341 UPCALL
0.00 0.00 us 0.00 us 0.00 us 91 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 30 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 250 CI_FORGET
Duration: 5691 seconds
Data Read: 42401792 bytes
Data Written: 9455665152 bytes
Interval 0 Stats:
Block Size: 4096b+ 32768b+ 65536b+
No. of Reads: 0 0 7
No. of Writes: 1 24 3023
Block Size: 131072b+ 262144b+ 524288b+
No. of Reads: 320 0 0
No. of Writes: 0 41 436
Block Size: 1048576b+ 2097152b+
No. of Reads: 0 0
No. of Writes: 4856 1768
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 249 FORGET
0.00 0.00 us 0.00 us 0.00 us 7069 RELEASE
0.00 0.00 us 0.00 us 0.00 us 53292 RELEASEDIR
0.00 163.77 us 143.80 us 187.63 us 3 SETXATTR
0.00 539.28 us 199.35 us 909.49 us 3 MKDIR
0.00 1861.11 us 734.86 us 2433.60 us 3 RMDIR
0.00 34.80 us 14.08 us 120.65 us 215 IPC
0.00 47.22 us 16.75 us 265.51 us 426 FSETXATTR
0.00 105.54 us 34.69 us 438.87 us 474 OPEN
0.00 58.10 us 9.99 us 2784.86 us 906 READDIR
0.00 193.84 us 78.00 us 1070.84 us 338 READ
0.01 78.11 us 27.12 us 267.42 us 6606 FGETXATTR
0.02 62.29 us 15.49 us 2094.30 us 13240 FINODELK
0.03 71.48 us 13.41 us 2627.65 us 14792 INODELK
0.03 146.99 us 13.72 us 18406.23 us 7828 STATFS
0.06 144.89 us 37.17 us 1728.42 us 14262 FSTAT
0.06 294.14 us 95.08 us 1298.49 us 7338 SETATTR
0.09 501.65 us 171.12 us 3673.36 us 6606 CREATE
0.10 277.18 us 83.58 us 2255.78 us 13240 FXATTROP
0.11 618.23 us 182.22 us 31669.04 us 6606 UNLINK
0.19 247.20 us 12.45 us 56440.26 us 27624 LOOKUP
0.19 269.37 us 9.01 us 29850.52 us 26442 ENTRYLK
0.26 942.72 us 125.40 us 11143.54 us 10149 WRITE
0.52 569.83 us 8.92 us 3526.39 us 33889 GETXATTR
1.14 2122.15 us 8.75 us 170951.32 us 19775 FLUSH
1.33 920.75 us 0.81 us 27584.76 us 53271 OPENDIR
95.86 23866.52 us 76.91 us 66659.49 us 148246 READDIRP
0.00 0.00 us 0.00 us 0.00 us 341 UPCALL
0.00 0.00 us 0.00 us 0.00 us 91 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 30 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 250 CI_FORGET
Duration: 5691 seconds
Data Read: 42401792 bytes
Data Written: 9455665152 bytes
Gluster7.0 (Ctime disabled)
==========================
ToDo
Attachment:
64KiB_x_6000_8T.fio
Description: Binary data
Attachment:
10MiB_x_1200_8T.fio
Description: Binary data
Attachment:
1MiB_x_6000_8T.fio
Description: Binary data
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/118564314 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/118564314 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
