Re: Performance is falling rapidly when updating from v5.5 to v7.0

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 





Am Di., 5. Nov. 2019 um 12:06 Uhr schrieb RAFI KC <rkavunga@xxxxxxxxxx>:


On 11/4/19 8:46 PM, David Spisla wrote:
Dear Gluster Community,

I also have a issue concerning performance. The last days I updated our test cluster from GlusterFS v5.5 to v7.0 . The setup in general:

2 HP DL380 Servers with 10Gbit NICs, 1 Distribute-Replica 2 Volume with 2 Replica Pairs. Client is SMB Samba (access via vfs_glusterfs) . I did several tests to ensure that Samba don't causes the fall.
The setup ist completely the same except the Gluster Version
Here are my results:
64KiB           1MiB             10MiB            (Filesize)
3,49             47,41            300,50          (Values in MiB/s with GlusterFS v5.5)
0,16              2,61             76,63            (Values in MiB/s with GlusterFS v7.0)


Can you please share the profile information [1] for both versions?  Also it would be really helpful if you can mention the io patterns that used for this tests.

[1] : https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/

Hello Rafi,
thank you for your help.

* First more information about the io patterns: As a client we use a DL360 Windws Server 2017 machine with 10Gbit NIC connected to the storage machines. The share will be mounted via SMB and the tests writes with fio. We use this job files (see attachment). Each job file will be executed separetely and there is a sleep about 60s between each test run to calm down the system before starting a new test.

* Attached below you find the profile output from the tests with v5.5 (ctime enabled), v7.0 (ctime enabled).

* Beside of the tests with Samba I did also some fio tests directly on the FUSE Mounts (locally on one of the storage nodes). The results show that there is only a small decrease of performance between v5.5 and v7.0
(All values in MiB/s)
64KiB    1MiB     10MiB
50,09     679,96   1023,02 (v5.5)
47,00     656,46    977,60 (v7.0)

It seems to be that the combination of samba + gluster7.0 has a lot of problems, or not?



We use this volume options (GlusterFS 7.0):

Volume Name: archive1
Type: Distributed-Replicate
Volume ID: 44c17844-0bd4-4ca2-98d8-a1474add790c
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
Brick2: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
Brick3: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
Brick4: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
user.smb: disable
features.read-only: off
features.worm: off
features.worm-file-level: on
features.retention-mode: enterprise
features.default-retention-period: 120
network.ping-timeout: 10
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.nl-cache: on
performance.nl-cache-timeout: 600
client.event-threads: 32
server.event-threads: 32
cluster.lookup-optimize: on
performance.stat-prefetch: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
performance.cache-samba-metadata: on
performance.cache-ima-xattrs: on
performance.io-thread-count: 64
cluster.use-compound-fops: on
performance.cache-size: 512MB
performance.cache-refresh-timeout: 10
performance.read-ahead: off
performance.write-behind-window-size: 4MB
performance.write-behind: on
storage.build-pgfid: on
features.ctime: on
cluster.quorum-type: fixed
cluster.quorum-count: 1
features.bitrot: on
features.scrub: Active
features.scrub-freq: daily

For GlusterFS 5.5 its nearly the same except the fact that there were 2 options to enable ctime feature.



Ctime stores additional metadata information as an extended attributes which sometimes exceeds the default inode size. In such scenarios the additional xattrs won't fit into the default size. This will result in additional blocks to be used to store xattrs in the inide, which will effect the latency. This is purely based on the i/o operations and the total xattrs size stored in the inode.

Is it possible for you to repeat the test by disabling ctime or increasing the inode size to a higher value say 1024KB?

I will do so but for today I could not finish tests with ctime disabled (or higher inode value) because it takes a lot of time with v7.0 due to the low performance and I will perform it tomorrow. As soon as possible I give you the results.
By the way: You really mean inode size on xfs layer 1024KB? Or do you mean 1024Bytes? We use per default 512Bytes, because this is the recommended size until now . But it seems to be that there is a need for a new recommendation when using ctime feature as a default. I can not image that this is the real cause for the low performance because in v5.5 we also use ctime feature with inode size 512Bytes.

Regards
David


Our optimization for Samba looks like this (for every version):

[global]
workgroup = SAMBA
netbios name = CLUSTER
kernel share modes = no
aio read size = 1
aio write size = 1
kernel oplocks = no
max open files = 100000
nt acl support = no
security = user
server min protocol = SMB2
store dos attributes = no
strict locking = no
full_audit:failure = pwrite_send pwrite_recv pwrite offload_write_send offload_write_recv create_file open unlink connect disconnect rename chown fchown lchown chmod fchmod mkdir rmdir ntimes ftruncate fallocate
full_audit:success = pwrite_send pwrite_recv pwrite offload_write_send offload_write_recv create_file open unlink connect disconnect rename chown fchown lchown chmod fchmod mkdir rmdir ntimes ftruncate fallocate
full_audit:facility = local5
durable handles = yes
posix locking = no
log level = 2
max log size = 100000
debug pid = yes

What can be the cause for this rapid falling of the performance for small files? Are some of our vol options not recommended anymore?
There were some patches concerning performance for small files in v6.0 und v7.0 :

#1670031: performance regression seen with smallfile workload tests

#1659327: 43% regression in small-file sequential read performance

And one patch for the io-cache:

#1659869: improvements to io-cache

Regards

David Spisla


________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
Gluster5.5 (Ctime enabled)
==========================

fs-dl380-c1-n1:~ # gluster vo profile repo1 info
Brick: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                13722                125700                 15237 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3236                   746                    30 
 
   Block Size:               4096b+               16384b+               32768b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    2                  9051                  9050 
 
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    7                   163                     0 
No. of Writes:                 3034                214018                    99 
 
   Block Size:             524288b+             1048576b+             2097152b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  431                  4898                  1732 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          19937      FORGET
      0.00       0.00 us       0.00 us       0.00 us          28745     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          27033  RELEASEDIR
      0.00     372.52 us     372.52 us     372.52 us              1     XATTROP
      0.00     155.97 us     148.40 us     163.74 us              3    SETXATTR
      0.00     355.85 us     319.36 us     392.34 us              3       MKDIR
      0.00      35.10 us      14.17 us      58.11 us             39         IPC
      0.00    2092.77 us     759.52 us    2785.95 us              3       RMDIR
      0.00      57.69 us      13.35 us     514.30 us            135     READDIR
      0.01     894.26 us      34.03 us   56491.66 us            182        READ
      0.01     374.15 us      22.21 us   19577.39 us            827      STATFS
      0.03      69.78 us      21.13 us     352.94 us           7778       FSTAT
      0.03     282.54 us      17.73 us   23688.04 us           2268   FSETXATTR
      0.04      60.61 us      14.01 us   10210.17 us          13348    FINODELK
      0.04     355.90 us      31.44 us   56158.91 us           2325        OPEN
      0.05     306.29 us     109.78 us     674.82 us           3665     SETATTR
      0.10     276.25 us      21.09 us 1058896.03 us           7498     INODELK
      0.16     507.07 us     181.63 us    5291.74 us           6607      CREATE
      0.18     297.21 us      90.56 us   11128.32 us          13316    FXATTROP
      0.24     773.47 us     179.92 us   27742.67 us           6606      UNLINK
      0.33     265.41 us      12.03 us   62361.64 us          26444     ENTRYLK
      0.88     234.65 us      18.24 us   15480.79 us          80510      LOOKUP
      0.93     750.11 us      12.26 us   14390.35 us          26600    GETXATTR
      1.04    1127.56 us      11.10 us  227020.48 us          19784       FLUSH
      1.09     881.86 us       1.22 us   33956.54 us          26502     OPENDIR
      4.70     595.48 us      68.68 us  230191.32 us         168825       WRITE
     90.14   33977.47 us      84.81 us  129099.85 us          56766    READDIRP
      0.00       0.00 us       0.00 us       0.00 us        1958138      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        1958010     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us           2261   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            128   CI_FORGET
 
    Duration: 6513 seconds
   Data Read: 21823488 bytes
Data Written: 38141710505 bytes
 
Interval 1 Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                13722                125700                 15237 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3236                   746                    30 
 
   Block Size:               4096b+               65536b+              131072b+ 
 No. of Reads:                    0                     7                   163 
No. of Writes:                    2                  3034                     2 
 
   Block Size:             262144b+              524288b+             1048576b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   55                   431                  4898 
 
   Block Size:            2097152b+ 
 No. of Reads:                    0 
No. of Writes:                 1732 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            126      FORGET
      0.00       0.00 us       0.00 us       0.00 us           8927     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          26500  RELEASEDIR
      0.00     372.52 us     372.52 us     372.52 us              1     XATTROP
      0.00     155.97 us     148.40 us     163.74 us              3    SETXATTR
      0.00     355.85 us     319.36 us     392.34 us              3       MKDIR
      0.00      36.18 us      14.17 us      58.11 us             34         IPC
      0.00    2092.77 us     759.52 us    2785.95 us              3       RMDIR
      0.00      57.02 us      13.35 us     514.30 us            127     READDIR
      0.01     894.26 us      34.03 us   56491.66 us            182        READ
      0.01     380.67 us      22.21 us   19577.39 us            806      STATFS
      0.03      69.78 us      21.13 us     352.94 us           7778       FSTAT
      0.03     282.54 us      17.73 us   23688.04 us           2268   FSETXATTR
      0.04      60.61 us      14.01 us   10210.17 us          13348    FINODELK
      0.04     355.90 us      31.44 us   56158.91 us           2325        OPEN
      0.05     306.29 us     109.78 us     674.82 us           3665     SETATTR
      0.10     276.25 us      21.09 us 1058896.03 us           7498     INODELK
      0.16     507.07 us     181.63 us    5291.74 us           6607      CREATE
      0.18     297.21 us      90.56 us   11128.32 us          13316    FXATTROP
      0.24     773.47 us     179.92 us   27742.67 us           6606      UNLINK
      0.33     265.41 us      12.03 us   62361.64 us          26444     ENTRYLK
      0.88     234.65 us      18.24 us   15480.79 us          80496      LOOKUP
      0.93     750.43 us      12.26 us   14390.35 us          26587    GETXATTR
      1.04    1127.56 us      11.10 us  227020.48 us          19784       FLUSH
      1.09     882.00 us       1.22 us   33956.54 us          26498     OPENDIR
      4.70     595.47 us      68.68 us  230191.32 us         168825       WRITE
     90.14   33978.50 us      84.81 us  129099.85 us          56766    READDIRP
      0.00       0.00 us       0.00 us       0.00 us        1958138      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        1958010     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us           2261   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            128   CI_FORGET
 
    Duration: 789 seconds
   Data Read: 21823488 bytes
Data Written: 9485570217 bytes
 
Brick: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:              16384b+               32768b+               65536b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 8952                  8950                  2966 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:               217821                    88                   484 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 4982                  1846 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          20194      FORGET
      0.00       0.00 us       0.00 us       0.00 us          26431     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          26795  RELEASEDIR
      0.00     164.43 us     161.90 us     167.62 us              3    SETXATTR
      0.00      30.57 us      17.79 us      44.45 us             21         IPC
      0.00     390.62 us     383.31 us     397.93 us              2     XATTROP
      0.00      56.79 us      26.70 us     152.48 us             23   FSETXATTR
      0.00     447.69 us     400.71 us     510.30 us              3       MKDIR
      0.00     129.81 us      55.26 us     289.15 us             28        OPEN
      0.00      75.71 us      14.13 us     816.87 us             75     READDIR
      0.00    2188.47 us     796.44 us    2993.06 us              3       RMDIR
      0.01     160.12 us      21.94 us    6000.03 us            813      STATFS
      0.06      70.99 us      24.27 us   29788.04 us           7349     INODELK
      0.07      73.26 us      24.65 us     677.95 us           7784       FSTAT
      0.09      60.50 us      17.42 us     521.01 us          13194    FINODELK
      0.13     312.52 us     117.48 us     565.10 us           3652     SETATTR
      0.39     508.69 us     179.00 us    1052.82 us           6594      CREATE
      0.44     292.42 us      90.47 us    4546.87 us          13194    FXATTROP
      0.77     337.08 us      12.36 us   56472.78 us          19800       FLUSH
      0.81    1067.61 us     218.87 us   50116.71 us           6594      UNLINK
      1.08     356.53 us       1.54 us   22107.25 us          26472     OPENDIR
      1.12     368.18 us      13.26 us   68738.68 us          26406     ENTRYLK
      1.19    1003.64 us     166.98 us    6085.26 us          10330       WRITE
      2.29     753.23 us      12.07 us    5793.53 us          26507    GETXATTR
      2.44     253.59 us      16.41 us   13225.37 us          83900      LOOKUP
     89.11   12640.00 us      20.25 us  145809.04 us          61411    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            425      UPCALL
      0.00       0.00 us       0.00 us       0.00 us             35     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             27   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            390   CI_FORGET
 
    Duration: 6513 seconds
   Data Read: 0 bytes
Data Written: 38958530560 bytes
 
Interval 1 Stats:
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 2966                     3                    49 
 
   Block Size:             524288b+             1048576b+             2097152b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  484                  4982                  1846 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            387      FORGET
      0.00       0.00 us       0.00 us       0.00 us           6622     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          26472  RELEASEDIR
      0.00     164.43 us     161.90 us     167.62 us              3    SETXATTR
      0.00      30.40 us      17.79 us      44.45 us             19         IPC
      0.00     390.62 us     383.31 us     397.93 us              2     XATTROP
      0.00      56.79 us      26.70 us     152.48 us             23   FSETXATTR
      0.00     447.69 us     400.71 us     510.30 us              3       MKDIR
      0.00     129.81 us      55.26 us     289.15 us             28        OPEN
      0.00      77.61 us      14.13 us     816.87 us             71     READDIR
      0.00    2188.47 us     796.44 us    2993.06 us              3       RMDIR
      0.01     161.55 us      21.94 us    6000.03 us            792      STATFS
      0.06      70.99 us      24.27 us   29788.04 us           7349     INODELK
      0.07      73.26 us      24.65 us     677.95 us           7784       FSTAT
      0.09      60.50 us      17.42 us     521.01 us          13194    FINODELK
      0.13     312.52 us     117.48 us     565.10 us           3652     SETATTR
      0.39     508.69 us     179.00 us    1052.82 us           6594      CREATE
      0.44     292.42 us      90.47 us    4546.87 us          13194    FXATTROP
      0.77     337.08 us      12.36 us   56472.78 us          19800       FLUSH
      0.81    1067.61 us     218.87 us   50116.71 us           6594      UNLINK
      1.08     356.55 us       1.54 us   22107.25 us          26470     OPENDIR
      1.12     368.18 us      13.26 us   68738.68 us          26406     ENTRYLK
      1.19    1003.64 us     166.98 us    6085.26 us          10330       WRITE
      2.29     753.43 us      12.07 us    5793.53 us          26499    GETXATTR
      2.44     253.59 us      16.41 us   13225.37 us          83890      LOOKUP
     89.11   12640.71 us      20.25 us  145809.04 us          61411    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            425      UPCALL
      0.00       0.00 us       0.00 us       0.00 us             35     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             27   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            390   CI_FORGET
 
    Duration: 789 seconds
   Data Read: 0 bytes
Data Written: 9811918848 bytes
 
Brick: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:              16384b+               32768b+               65536b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 8952                  8950                  2966 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                   24                     0                     0 
No. of Writes:               217821                    88                   484 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 4982                  1846 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          26404      FORGET
      0.00       0.00 us       0.00 us       0.00 us          26426     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          27037  RELEASEDIR
      0.00     242.57 us     242.57 us     242.57 us              1    READDIRP
      0.00     270.43 us     224.72 us     316.13 us              2 REMOVEXATTR
      0.00     376.30 us     357.25 us     395.35 us              2     XATTROP
      0.00     411.18 us     363.84 us     462.52 us              3       MKDIR
      0.00      36.88 us      25.54 us      54.42 us             36         IPC
      0.00     176.85 us      32.37 us     492.36 us             15   FSETXATTR
      0.01     206.12 us     108.73 us     412.08 us             21        OPEN
      0.01     197.53 us      85.44 us     451.06 us             27        READ
      0.01    1995.17 us     760.64 us    2693.43 us              3       RMDIR
      0.01    1366.94 us     127.39 us    3611.57 us              5    SETXATTR
      0.02      88.63 us      25.71 us     863.15 us            135     READDIR
      0.13     116.18 us      34.05 us    2167.66 us            828      STATFS
      0.78      78.60 us      26.05 us   52960.99 us           7352     INODELK
      1.19      67.15 us      20.02 us    2608.91 us          13194    FINODELK
      1.58      59.32 us      18.09 us    2981.27 us          19800       FLUSH
      1.64     334.42 us     118.08 us     855.90 us           3654     SETATTR
      2.57      72.45 us      19.16 us     732.36 us          26406     ENTRYLK
      3.84     108.04 us       2.38 us     677.07 us          26502     OPENDIR
      4.56     515.76 us     191.84 us     883.80 us           6594      CREATE
      5.32     300.53 us      96.16 us    6531.99 us          13194    FXATTROP
      7.63     862.82 us     248.89 us    4093.19 us           6594      UNLINK
     14.59    1052.78 us     173.50 us    3653.35 us          10330       WRITE
     26.87     753.92 us      21.93 us    5703.54 us          26567    GETXATTR
     29.25     276.32 us      32.87 us    3104.49 us          78907      LOOKUP
      0.00       0.00 us       0.00 us       0.00 us           6646      UPCALL
      0.00       0.00 us       0.00 us       0.00 us             30     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             17   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           6616   CI_FORGET
 
    Duration: 6596 seconds
   Data Read: 3145728 bytes
Data Written: 38958530560 bytes
 
Interval 1 Stats:
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    0                    24                     0 
No. of Writes:                 2966                     3                    49 
 
   Block Size:             524288b+             1048576b+             2097152b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  484                  4982                  1846 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6597      FORGET
      0.00       0.00 us       0.00 us       0.00 us           6615     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          26500  RELEASEDIR
      0.00     242.57 us     242.57 us     242.57 us              1    READDIRP
      0.00     270.43 us     224.72 us     316.13 us              2 REMOVEXATTR
      0.00     376.30 us     357.25 us     395.35 us              2     XATTROP
      0.00     411.18 us     363.84 us     462.52 us              3       MKDIR
      0.00      37.24 us      25.54 us      54.42 us             34         IPC
      0.00     176.85 us      32.37 us     492.36 us             15   FSETXATTR
      0.01     206.12 us     108.73 us     412.08 us             21        OPEN
      0.01     197.53 us      85.44 us     451.06 us             27        READ
      0.01    1995.17 us     760.64 us    2693.43 us              3       RMDIR
      0.01    1366.94 us     127.39 us    3611.57 us              5    SETXATTR
      0.02      88.84 us      25.71 us     863.15 us            127     READDIR
      0.13     116.49 us      34.05 us    2167.66 us            806      STATFS
      0.78      78.60 us      26.05 us   52960.99 us           7352     INODELK
      1.19      67.15 us      20.02 us    2608.91 us          13194    FINODELK
      1.58      59.32 us      18.09 us    2981.27 us          19800       FLUSH
      1.64     334.42 us     118.08 us     855.90 us           3654     SETATTR
      2.57      72.45 us      19.16 us     732.36 us          26406     ENTRYLK
      3.84     108.06 us       2.38 us     677.07 us          26498     OPENDIR
      4.56     515.76 us     191.84 us     883.80 us           6594      CREATE
      5.32     300.53 us      96.16 us    6531.99 us          13194    FXATTROP
      7.63     862.82 us     248.89 us    4093.19 us           6594      UNLINK
     14.59    1052.78 us     173.50 us    3653.35 us          10330       WRITE
     26.87     754.21 us      21.93 us    5703.54 us          26555    GETXATTR
     29.25     276.33 us      32.87 us    3104.49 us          78893      LOOKUP
      0.00       0.00 us       0.00 us       0.00 us           6646      UPCALL
      0.00       0.00 us       0.00 us       0.00 us             30     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             17   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           6616   CI_FORGET
 
    Duration: 789 seconds
   Data Read: 3145728 bytes
Data Written: 9811918848 bytes
 
Brick: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                13722                125700                 15237 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3236                   746                    30 
 
   Block Size:               4096b+               16384b+               32768b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    2                  9051                  9050 
 
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    7                   139                     0 
No. of Writes:                 3034                214018                    99 
 
   Block Size:             524288b+             1048576b+             2097152b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  431                  4898                  1732 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          26420      FORGET
      0.00       0.00 us       0.00 us       0.00 us          28806     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          27035  RELEASEDIR
      0.00     373.31 us     373.31 us     373.31 us              1    READDIRP
      0.00     410.11 us     410.11 us     410.11 us              1     XATTROP
      0.00     172.89 us     140.25 us     229.22 us              3    SETXATTR
      0.00     348.41 us     319.63 us     404.97 us              3       MKDIR
      0.00      40.45 us      31.64 us      58.41 us             36         IPC
      0.00    2020.28 us     690.06 us    2754.73 us              3       RMDIR
      0.01      74.65 us      18.21 us     868.57 us            135     READDIR
      0.02     170.34 us      80.49 us     293.60 us            155        READ
      0.07     107.76 us      30.67 us    5006.30 us            827      STATFS
      0.14      77.87 us      27.17 us   17142.15 us           2329   FSETXATTR
      0.25     136.56 us      42.42 us    1019.27 us           2386        OPEN
      0.36      61.10 us      16.56 us    3013.27 us           7498     INODELK
      0.60      57.67 us      18.69 us     337.84 us          13355    FINODELK
      0.78      50.74 us      13.71 us     556.16 us          19784       FLUSH
      0.90     315.86 us     109.57 us    1634.45 us           3665     SETATTR
      1.26      61.32 us      17.83 us     759.14 us          26444     ENTRYLK
      2.03      98.28 us       2.27 us    4248.68 us          26502     OPENDIR
      2.58     500.80 us     174.65 us    1841.95 us           6607      CREATE
      3.16     304.42 us      93.33 us   46617.14 us          13316    FXATTROP
      4.43     859.31 us     203.35 us    3433.33 us           6606      UNLINK
     13.58     230.56 us      25.82 us    1441.09 us          75517      LOOKUP
     15.45     745.15 us      20.75 us    6030.94 us          26595    GETXATTR
     54.36     412.94 us      74.85 us    5222.73 us         168825       WRITE
      0.00       0.00 us       0.00 us       0.00 us        1966855      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        1957919     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us           2325   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           8936   CI_FORGET
 
    Duration: 6596 seconds
   Data Read: 18677760 bytes
Data Written: 38141710505 bytes
 
Interval 1 Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                13722                125700                 15237 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3236                   746                    30 
 
   Block Size:               4096b+               65536b+              131072b+ 
 No. of Reads:                    0                     7                   139 
No. of Writes:                    2                  3034                     2 
 
   Block Size:             262144b+              524288b+             1048576b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   55                   431                  4898 
 
   Block Size:            2097152b+ 
 No. of Reads:                    0 
No. of Writes:                 1732 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6609      FORGET
      0.00       0.00 us       0.00 us       0.00 us           8988     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          26500  RELEASEDIR
      0.00     373.31 us     373.31 us     373.31 us              1    READDIRP
      0.00     410.11 us     410.11 us     410.11 us              1     XATTROP
      0.00     172.89 us     140.25 us     229.22 us              3    SETXATTR
      0.00     348.41 us     319.63 us     404.97 us              3       MKDIR
      0.00      40.85 us      31.64 us      58.41 us             34         IPC
      0.00    2020.28 us     690.06 us    2754.73 us              3       RMDIR
      0.01      74.66 us      18.21 us     868.57 us            127     READDIR
      0.02     170.34 us      80.49 us     293.60 us            155        READ
      0.07     107.62 us      30.67 us    5006.30 us            806      STATFS
      0.14      77.87 us      27.17 us   17142.15 us           2329   FSETXATTR
      0.25     136.56 us      42.42 us    1019.27 us           2386        OPEN
      0.36      61.10 us      16.56 us    3013.27 us           7498     INODELK
      0.60      57.67 us      18.69 us     337.84 us          13355    FINODELK
      0.78      50.74 us      13.71 us     556.16 us          19784       FLUSH
      0.90     315.86 us     109.57 us    1634.45 us           3665     SETATTR
      1.26      61.32 us      17.83 us     759.14 us          26444     ENTRYLK
      2.03      98.29 us       2.27 us    4248.68 us          26498     OPENDIR
      2.58     500.80 us     174.65 us    1841.95 us           6607      CREATE
      3.16     304.42 us      93.33 us   46617.14 us          13316    FXATTROP
      4.43     859.31 us     203.35 us    3433.33 us           6606      UNLINK
     13.58     230.56 us      25.82 us    1441.09 us          75503      LOOKUP
     15.45     745.45 us      20.75 us    6030.94 us          26582    GETXATTR
     54.37     412.94 us      74.85 us    5222.73 us         168825       WRITE
      0.00       0.00 us       0.00 us       0.00 us        1966855      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        1957919     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us           2325   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           8936   CI_FORGET
 
    Duration: 789 seconds
   Data Read: 18677760 bytes
Data Written: 9485570217 bytes


Gluster7.0 (Ctime enabled)
==========================

fs-dl380-c1-n1:~ # gluster vo profile repo1 info
Brick: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                23977                143222                 10220 
 
   Block Size:                512b+                1024b+                8192b+ 
 No. of Reads:                    0                     0                     1 
No. of Writes:                 2067                    51                     0 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                    0                     8                   171 
No. of Writes:                   26                  2954                     0 
 
   Block Size:             262144b+              524288b+             1048576b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   50                   480                  5002 
 
   Block Size:            2097152b+ 
 No. of Reads:                    0 
No. of Writes:                 1841 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6590      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7432     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53292  RELEASEDIR
      0.00     129.79 us     114.92 us     140.65 us              3    SETXATTR
      0.00     386.13 us     331.50 us     440.75 us              2    READDIRP
      0.00     292.58 us     249.60 us     360.18 us              3     XATTROP
      0.00     576.10 us     446.41 us     775.50 us              3       MKDIR
      0.00    1857.67 us     820.07 us    2405.74 us              3       RMDIR
      0.00      33.78 us      14.96 us     120.77 us            216         IPC
      0.02      73.47 us      11.54 us     315.66 us            412   FSETXATTR
      0.02     169.11 us      43.46 us     405.47 us            188        READ
      0.04      69.85 us       9.31 us    2344.93 us            906     READDIR
      0.05      79.66 us       7.34 us     267.55 us           1151    GETXATTR
      0.06     123.14 us      28.68 us     517.29 us            846        OPEN
      0.32      85.70 us      25.91 us     399.66 us           6594   FGETXATTR
      0.48     107.57 us      25.52 us   14326.25 us           7828      STATFS
      0.49      43.59 us      11.85 us    2791.32 us          19801       FLUSH
      0.50      62.56 us      13.76 us    1445.20 us          14181    FINODELK
      0.67     177.44 us      56.71 us     687.67 us           6594       FSTAT
      0.92      61.48 us      11.24 us    2399.86 us          26408     ENTRYLK
      1.31     315.70 us      96.90 us     915.84 us           7305     SETATTR
      1.97     525.68 us     173.64 us    3103.01 us           6595      CREATE
      2.20     277.03 us      68.90 us   13473.77 us          13942    FXATTROP
      2.70     214.72 us       9.56 us    2757.47 us          22114      LOOKUP
      3.01     802.37 us     182.07 us    5259.34 us           6594      UNLINK
      3.08     101.50 us       0.65 us    4039.32 us          53271     OPENDIR
     36.60    4043.67 us      15.93 us  994013.08 us          15909     INODELK
     45.55     421.68 us      64.99 us    9380.82 us         189890       WRITE
      0.00       0.00 us       0.00 us       0.00 us        3677977      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        3671345     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             36   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           6632   CI_FORGET
 
    Duration: 5698 seconds
   Data Read: 22977840 bytes
Data Written: 9841829137 bytes
 
Interval 0 Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                23977                143222                 10220 
 
   Block Size:                512b+                1024b+                8192b+ 
 No. of Reads:                    0                     0                     1 
No. of Writes:                 2067                    51                     0 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                    0                     8                   171 
No. of Writes:                   26                  2954                     0 
 
   Block Size:             262144b+              524288b+             1048576b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   50                   480                  5002 
 
   Block Size:            2097152b+ 
 No. of Reads:                    0 
No. of Writes:                 1841 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6590      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7432     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53292  RELEASEDIR
      0.00     129.79 us     114.92 us     140.65 us              3    SETXATTR
      0.00     386.13 us     331.50 us     440.75 us              2    READDIRP
      0.00     292.58 us     249.60 us     360.18 us              3     XATTROP
      0.00     576.10 us     446.41 us     775.50 us              3       MKDIR
      0.00    1857.67 us     820.07 us    2405.74 us              3       RMDIR
      0.00      33.78 us      14.96 us     120.77 us            216         IPC
      0.02      73.47 us      11.54 us     315.66 us            412   FSETXATTR
      0.02     169.11 us      43.46 us     405.47 us            188        READ
      0.04      69.85 us       9.31 us    2344.93 us            906     READDIR
      0.05      79.66 us       7.34 us     267.55 us           1151    GETXATTR
      0.06     123.14 us      28.68 us     517.29 us            846        OPEN
      0.32      85.70 us      25.91 us     399.66 us           6594   FGETXATTR
      0.48     107.70 us      25.52 us   14326.25 us           7828      STATFS
      0.49      43.59 us      11.85 us    2791.32 us          19801       FLUSH
      0.50      62.56 us      13.76 us    1445.20 us          14181    FINODELK
      0.67     177.44 us      56.71 us     687.67 us           6594       FSTAT
      0.92      61.48 us      11.24 us    2399.86 us          26408     ENTRYLK
      1.31     315.70 us      96.90 us     915.84 us           7305     SETATTR
      1.97     525.68 us     173.64 us    3103.01 us           6595      CREATE
      2.20     277.03 us      68.90 us   13473.77 us          13942    FXATTROP
      2.70     214.72 us       9.56 us    2757.47 us          22114      LOOKUP
      3.01     802.37 us     182.07 us    5259.34 us           6594      UNLINK
      3.08     101.50 us       0.65 us    4039.32 us          53271     OPENDIR
     36.60    4043.67 us      15.93 us  994013.08 us          15909     INODELK
     45.55     421.68 us      64.99 us    9380.82 us         189890       WRITE
      0.00       0.00 us       0.00 us       0.00 us        3677977      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        3671345     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             36   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           6632   CI_FORGET
 
    Duration: 5698 seconds
   Data Read: 22977840 bytes
Data Written: 9841829137 bytes
 
Brick: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:               4096b+               32768b+               65536b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    1                    24                  3023 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                  320                     0                     0 
No. of Writes:                    0                    41                   436 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 4856                  1768 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6605      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7075     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53292  RELEASEDIR
      0.00     138.84 us     112.92 us     160.96 us              3    SETXATTR
      0.00     399.64 us     344.42 us     454.87 us              2    READDIRP
      0.00     571.87 us     398.94 us     844.47 us              3       MKDIR
      0.01    2045.86 us     777.17 us    2709.66 us              3       RMDIR
      0.01      35.10 us      12.94 us      93.54 us            216         IPC
      0.06      72.26 us      12.14 us     334.52 us            432   FSETXATTR
      0.10     113.49 us      26.00 us     484.88 us            473        OPEN
      0.10     174.90 us      87.44 us     453.04 us            324        READ
      0.11      65.07 us      11.03 us    4413.29 us            907     READDIR
      0.11      67.32 us      13.89 us    1223.39 us            918    GETXATTR
      1.06      87.26 us      26.59 us     194.28 us           6606   FGETXATTR
      1.34      93.17 us      20.71 us    8258.97 us           7828      STATFS
      1.37      56.19 us      14.38 us     588.75 us          13240    FINODELK
      1.74      47.75 us       9.44 us    1288.06 us          19775       FLUSH
      2.27     186.16 us      58.73 us    1957.66 us           6606       FSTAT
      2.90      59.56 us      15.95 us    2550.84 us          26442     ENTRYLK
      4.20     310.72 us      92.26 us     920.97 us           7338     SETATTR
      6.31     517.93 us     194.94 us    3140.66 us           6606      CREATE
      6.84     279.99 us      71.20 us    2480.15 us          13240    FXATTROP
      8.44     214.97 us      15.99 us    3525.82 us          21284      LOOKUP
      9.83     807.18 us     187.73 us    3510.05 us           6606      UNLINK
      9.91     100.86 us       1.15 us    2942.26 us          53271     OPENDIR
     19.45    1039.19 us     126.76 us    5976.65 us          10149       WRITE
     23.82     873.49 us      13.82 us  996890.14 us          14792     INODELK
      0.00       0.00 us       0.00 us       0.00 us           6707      UPCALL
      0.00       0.00 us       0.00 us       0.00 us             82     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             21   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           6625   CI_FORGET
 
    Duration: 5698 seconds
   Data Read: 41943040 bytes
Data Written: 9455665152 bytes
 
Interval 0 Stats:
   Block Size:               4096b+               32768b+               65536b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    1                    24                  3023 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                  320                     0                     0 
No. of Writes:                    0                    41                   436 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 4856                  1768 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6605      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7075     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53292  RELEASEDIR
      0.00     138.84 us     112.92 us     160.96 us              3    SETXATTR
      0.00     399.64 us     344.42 us     454.87 us              2    READDIRP
      0.00     571.87 us     398.94 us     844.47 us              3       MKDIR
      0.01    2045.86 us     777.17 us    2709.66 us              3       RMDIR
      0.01      35.10 us      12.94 us      93.54 us            216         IPC
      0.06      72.26 us      12.14 us     334.52 us            432   FSETXATTR
      0.10     113.49 us      26.00 us     484.88 us            473        OPEN
      0.10     174.90 us      87.44 us     453.04 us            324        READ
      0.11      65.07 us      11.03 us    4413.29 us            907     READDIR
      0.11      67.32 us      13.89 us    1223.39 us            918    GETXATTR
      1.06      87.26 us      26.59 us     194.28 us           6606   FGETXATTR
      1.34      93.17 us      20.71 us    8258.97 us           7828      STATFS
      1.37      56.19 us      14.38 us     588.75 us          13240    FINODELK
      1.74      47.75 us       9.44 us    1288.06 us          19775       FLUSH
      2.27     186.16 us      58.73 us    1957.66 us           6606       FSTAT
      2.90      59.56 us      15.95 us    2550.84 us          26442     ENTRYLK
      4.20     310.72 us      92.26 us     920.97 us           7338     SETATTR
      6.31     517.93 us     194.94 us    3140.66 us           6606      CREATE
      6.84     279.99 us      71.20 us    2480.15 us          13240    FXATTROP
      8.44     214.97 us      15.99 us    3525.82 us          21284      LOOKUP
      9.83     807.18 us     187.73 us    3510.05 us           6606      UNLINK
      9.91     100.86 us       1.15 us    2942.26 us          53271     OPENDIR
     19.45    1039.20 us     126.76 us    5976.65 us          10149       WRITE
     23.82     873.49 us      13.82 us  996890.14 us          14792     INODELK
      0.00       0.00 us       0.00 us       0.00 us           6707      UPCALL
      0.00       0.00 us       0.00 us       0.00 us             82     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             21   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           6625   CI_FORGET
 
    Duration: 5698 seconds
   Data Read: 41943040 bytes
Data Written: 9455665152 bytes
 
Brick: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                23977                143222                 10220 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                    0                     0                     1 
No. of Writes:                 2067                    51                     0 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                    0                     1                   170 
No. of Writes:                   26                  2954                     0 
 
   Block Size:             262144b+              524288b+             1048576b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   50                   480                  5002 
 
   Block Size:            2097152b+ 
 No. of Reads:                    0 
No. of Writes:                 1841 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            944      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7430     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53292  RELEASEDIR
      0.00     134.44 us      75.05 us     193.89 us              3    SETXATTR
      0.00     268.64 us     259.90 us     278.55 us              3     XATTROP
      0.00     439.49 us     203.80 us     682.65 us              3       MKDIR
      0.00    1853.60 us     823.20 us    2393.82 us              3       RMDIR
      0.00      34.05 us      13.77 us     101.66 us            215         IPC
      0.00      50.12 us      19.28 us     294.14 us            410   FSETXATTR
      0.00     176.78 us      47.52 us     268.32 us            173        READ
      0.00      67.16 us       9.59 us    4569.92 us            906     READDIR
      0.00     135.83 us      38.87 us   18143.17 us            837        OPEN
      0.02      76.76 us      23.00 us     192.51 us           6594   FGETXATTR
      0.04      65.77 us      10.26 us   19823.69 us          14227    FINODELK
      0.04      68.49 us      11.99 us   18024.04 us          15909     INODELK
      0.05     155.32 us      16.76 us   21625.26 us           7828      STATFS
      0.08     142.52 us      37.26 us    4159.63 us          14358       FSTAT
      0.08     296.31 us      85.27 us    2190.78 us           7305     SETATTR
      0.13     509.05 us     179.26 us    6280.47 us           6595      CREATE
      0.15     274.40 us      79.84 us   13303.23 us          13942    FXATTROP
      0.18     233.24 us       8.82 us   53288.69 us          19801       FLUSH
      0.19     738.29 us     191.82 us   26804.67 us           6594      UNLINK
      0.44     416.08 us      11.09 us   31352.67 us          27762      LOOKUP
      0.55     551.61 us      10.11 us   46117.06 us          26408     ENTRYLK
      0.74     367.23 us       0.64 us   34196.75 us          53271     OPENDIR
      2.73     372.74 us       8.60 us   61916.08 us         192531    GETXATTR
      4.06     560.96 us      62.01 us   62168.04 us         189890       WRITE
     90.53   13985.76 us      15.94 us   60188.98 us         170026    READDIRP
      0.00       0.00 us       0.00 us       0.00 us        3672463      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        3671509     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             37   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            954   CI_FORGET
 
    Duration: 5691 seconds
   Data Read: 22379229 bytes
Data Written: 9841829137 bytes
 
Interval 0 Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                23977                143222                 10220 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                    0                     0                     1 
No. of Writes:                 2067                    51                     0 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                    0                     1                   170 
No. of Writes:                   26                  2954                     0 
 
   Block Size:             262144b+              524288b+             1048576b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   50                   480                  5002 
 
   Block Size:            2097152b+ 
 No. of Reads:                    0 
No. of Writes:                 1841 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            944      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7430     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53292  RELEASEDIR
      0.00     134.44 us      75.05 us     193.89 us              3    SETXATTR
      0.00     268.64 us     259.90 us     278.55 us              3     XATTROP
      0.00     439.49 us     203.80 us     682.65 us              3       MKDIR
      0.00    1853.60 us     823.20 us    2393.82 us              3       RMDIR
      0.00      34.05 us      13.77 us     101.66 us            215         IPC
      0.00      50.12 us      19.28 us     294.14 us            410   FSETXATTR
      0.00     176.78 us      47.52 us     268.32 us            173        READ
      0.00      67.16 us       9.59 us    4569.92 us            906     READDIR
      0.00     135.83 us      38.87 us   18143.17 us            837        OPEN
      0.02      76.76 us      23.00 us     192.51 us           6594   FGETXATTR
      0.04      65.77 us      10.26 us   19823.69 us          14227    FINODELK
      0.04      68.49 us      11.99 us   18024.04 us          15909     INODELK
      0.05     155.32 us      16.76 us   21625.26 us           7828      STATFS
      0.08     142.52 us      37.26 us    4159.63 us          14358       FSTAT
      0.08     296.31 us      85.27 us    2190.78 us           7305     SETATTR
      0.13     509.05 us     179.26 us    6280.47 us           6595      CREATE
      0.15     274.40 us      79.84 us   13303.23 us          13942    FXATTROP
      0.18     233.24 us       8.82 us   53288.69 us          19801       FLUSH
      0.19     738.29 us     191.82 us   26804.67 us           6594      UNLINK
      0.44     416.08 us      11.09 us   31352.67 us          27762      LOOKUP
      0.55     551.61 us      10.11 us   46117.06 us          26408     ENTRYLK
      0.74     367.23 us       0.64 us   34196.75 us          53271     OPENDIR
      2.73     372.74 us       8.60 us   61916.08 us         192531    GETXATTR
      4.06     560.96 us      62.01 us   62168.04 us         189890       WRITE
     90.53   13985.90 us      15.94 us   60188.98 us         170026    READDIRP
      0.00       0.00 us       0.00 us       0.00 us        3672463      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        3671509     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             37   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            954   CI_FORGET
 
    Duration: 5691 seconds
   Data Read: 22379229 bytes
Data Written: 9841829137 bytes
 
Brick: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:               4096b+               32768b+               65536b+ 
 No. of Reads:                    0                     0                     7 
No. of Writes:                    1                    24                  3023 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                  320                     0                     0 
No. of Writes:                    0                    41                   436 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 4856                  1768 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            249      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7069     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53292  RELEASEDIR
      0.00     163.77 us     143.80 us     187.63 us              3    SETXATTR
      0.00     539.28 us     199.35 us     909.49 us              3       MKDIR
      0.00    1861.11 us     734.86 us    2433.60 us              3       RMDIR
      0.00      34.80 us      14.08 us     120.65 us            215         IPC
      0.00      47.22 us      16.75 us     265.51 us            426   FSETXATTR
      0.00     105.54 us      34.69 us     438.87 us            474        OPEN
      0.00      58.10 us       9.99 us    2784.86 us            906     READDIR
      0.00     193.84 us      78.00 us    1070.84 us            338        READ
      0.01      78.11 us      27.12 us     267.42 us           6606   FGETXATTR
      0.02      62.29 us      15.49 us    2094.30 us          13240    FINODELK
      0.03      71.48 us      13.41 us    2627.65 us          14792     INODELK
      0.03     146.99 us      13.72 us   18406.23 us           7828      STATFS
      0.06     144.89 us      37.17 us    1728.42 us          14262       FSTAT
      0.06     294.14 us      95.08 us    1298.49 us           7338     SETATTR
      0.09     501.65 us     171.12 us    3673.36 us           6606      CREATE
      0.10     277.18 us      83.58 us    2255.78 us          13240    FXATTROP
      0.11     618.23 us     182.22 us   31669.04 us           6606      UNLINK
      0.19     247.20 us      12.45 us   56440.26 us          27624      LOOKUP
      0.19     269.37 us       9.01 us   29850.52 us          26442     ENTRYLK
      0.26     942.72 us     125.40 us   11143.54 us          10149       WRITE
      0.52     569.83 us       8.92 us    3526.39 us          33889    GETXATTR
      1.14    2122.15 us       8.75 us  170951.32 us          19775       FLUSH
      1.33     920.75 us       0.81 us   27584.76 us          53271     OPENDIR
     95.86   23866.45 us      76.91 us   66659.49 us         148246    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            341      UPCALL
      0.00       0.00 us       0.00 us       0.00 us             91     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             30   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            250   CI_FORGET
 
    Duration: 5691 seconds
   Data Read: 42401792 bytes
Data Written: 9455665152 bytes
 
Interval 0 Stats:
   Block Size:               4096b+               32768b+               65536b+ 
 No. of Reads:                    0                     0                     7 
No. of Writes:                    1                    24                  3023 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                  320                     0                     0 
No. of Writes:                    0                    41                   436 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 4856                  1768 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            249      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7069     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53292  RELEASEDIR
      0.00     163.77 us     143.80 us     187.63 us              3    SETXATTR
      0.00     539.28 us     199.35 us     909.49 us              3       MKDIR
      0.00    1861.11 us     734.86 us    2433.60 us              3       RMDIR
      0.00      34.80 us      14.08 us     120.65 us            215         IPC
      0.00      47.22 us      16.75 us     265.51 us            426   FSETXATTR
      0.00     105.54 us      34.69 us     438.87 us            474        OPEN
      0.00      58.10 us       9.99 us    2784.86 us            906     READDIR
      0.00     193.84 us      78.00 us    1070.84 us            338        READ
      0.01      78.11 us      27.12 us     267.42 us           6606   FGETXATTR
      0.02      62.29 us      15.49 us    2094.30 us          13240    FINODELK
      0.03      71.48 us      13.41 us    2627.65 us          14792     INODELK
      0.03     146.99 us      13.72 us   18406.23 us           7828      STATFS
      0.06     144.89 us      37.17 us    1728.42 us          14262       FSTAT
      0.06     294.14 us      95.08 us    1298.49 us           7338     SETATTR
      0.09     501.65 us     171.12 us    3673.36 us           6606      CREATE
      0.10     277.18 us      83.58 us    2255.78 us          13240    FXATTROP
      0.11     618.23 us     182.22 us   31669.04 us           6606      UNLINK
      0.19     247.20 us      12.45 us   56440.26 us          27624      LOOKUP
      0.19     269.37 us       9.01 us   29850.52 us          26442     ENTRYLK
      0.26     942.72 us     125.40 us   11143.54 us          10149       WRITE
      0.52     569.83 us       8.92 us    3526.39 us          33889    GETXATTR
      1.14    2122.15 us       8.75 us  170951.32 us          19775       FLUSH
      1.33     920.75 us       0.81 us   27584.76 us          53271     OPENDIR
     95.86   23866.52 us      76.91 us   66659.49 us         148246    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            341      UPCALL
      0.00       0.00 us       0.00 us       0.00 us             91     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             30   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            250   CI_FORGET
 
    Duration: 5691 seconds
   Data Read: 42401792 bytes
Data Written: 9455665152 bytes

Gluster7.0 (Ctime disabled)
==========================
ToDo

Attachment: 64KiB_x_6000_8T.fio
Description: Binary data

Attachment: 10MiB_x_1200_8T.fio
Description: Binary data

Attachment: 1MiB_x_6000_8T.fio
Description: Binary data

________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux