Re: Possible SHD stalling

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

On 2020/05/08 12:21, Ravishankar N wrote:
Hi,
On 08/05/20 1:03 pm, Jaco Kroon wrote:

Hi,

On 2020/05/08 06:32, Ravishankar N wrote:

On 08/05/20 1:10 am, Jaco Kroon wrote:

Hi,

So upgrading to 7.5 made a difference in that it takes a bit longer to cause chaos, but it still does the exact same thing.

To be precise:

I've now had to set:

gluster volume set vpbx_shared cluster.heal-wait-queue-length 1
gluster volume set vpbx_shared cluster.shd-wait-qlength 1
gluster volume set vpbx_shared cluster.shd-max-threads 1

With any of these settings >1 I end up in a situation where simple directory listings of < 100 entries in a folder takes upwards of a minute to complete *from time to time*.

Hi Jaco, these options are to be increased only if your hardware is capable of handling the load.It was contributed initially by facebook as it worked for them. See https://bugzilla.redhat.com/show_bug.cgi?id=1221737#c0.  So I think this is expected. Unless you are saying that the `ls` slowness happens even when there is no self-heal going on, in which case we need to see what the issue is.

The defaults for the queue lengths are >1 (1024) if I recall.

The default for max threads are 1.

Even with all those at a value of 1 we reached an ls latency of 10s overnight, and I specicifically switched off backups before letting this run overnight to ensure that it would not interfere.

Can the hardware handle it?  Well, let's look at the capacity status.  During threads above set to 16, and queue lengths at 10240.  All three hosts have approximately equal load values, with zoidberg (two data bricks compared to the other hosts one data brick and one "healing" arbiter that's newly added) being marginally (~2% on IO utilization) higher loaded, CPU was very similar.  I suspect this is because it's supplying two source bricks compared to the other hosts each have one source and one sink.

During these stalls the IO wait is <10% on all hosts and according to iotop that's almost all glusterfsd.  This goes up slightly if there are LVM snapshots in place for backup purposes, which takes about an hour.

CPU on glusterfsd < 50% according to top (ie, half a core's capacity).  16 cores available.

64GB RAM total, and as of writing, free -m is reporting:

              total        used        free      shared  buff/cache   available
Mem:          64413        6600       29011         370       28801       56517

RAID controller on each host:

                description: RAID bus controller
                product: MegaRAID SAS 2108 [Liberator]
                vendor: Broadcom / LSI

                   description: SCSI Disk
                   product: PERC H700
                   vendor: DELL

Looking at the queue depth for the exposed drive:

flexo /sys/class/block/sda # cat device/queue_depth
256

The disks themselves are in a RAID-1 configuration, 4TB NL-SAS drives from Seagate.

On top of the RAID-exposed drive we run LVM and the gluster data is sitting in an ext4 filesystem on top of a 3TB LV inside of the single PV that makes up the VG

I don't feel that there should be a problem with threads set to 4 at least, but probably even 8 or 16 should be OK.  I mean, until the IO capacity doesn't get to at least 70-80% it really should make little difference in my experience.  Technically even at 100% utilization it could just be a single IO request at any point in time (as we've seen cloning large disks) so in theory you can then still increase concurrency, but from experience we know that overall performance starts to degrade quite sharply at a certain tipping point and then you're in trouble with little to no warning.

Under non-healing conditions IO utilization (iostat -dmx 1, iotop etc ... ie, percentage of time that has outstanding requests ignoring multi-concurrency) is generally about 0.3%.

So somewhere there is definitely a bottleneck.  I don't mind deploying a custom "debug" patch on top of gluserfs 7.5 for the weekend but I generally need optimal performance again by Monday morning 6:30 SAST (GMT+2), then pushing the heal as hard as I possibly can overnight to get lock statistics or something.  If I can just know what options ./configure should be on/off, and which volume options needs to be set to get the right statistics, that's also helpful.  I'm even happy to then be pointed at the code and told "it's this process where the problem's at" (bottleneck, as it doesn't look like anything is broken really, just terribly unacceptably slow) - in other words, I'm happy to go attempt and contribute a patch on the issue, but I need a starting point of where to start scratching.  Can even do this Saturday after 15:00 till Sunday morning again.

I don't have a patch to offer but the entry point for the self-heal-daemon code itself is in syncop_mt_dir_scan()  which gets called for each of the 3 sub folders under /brick/.glusterfs/indices/ (for each brick). It uses gluster's synctask framework to readdir the entries in these sub folders,  and eventually calls afr_shd_index_heal() for each of the entries. So if you think shd is stuck, maybe you can see at what point it is stuck - perhaps while afr_shd_selfheal() for a given entry is stuck on an inodelk held by the client. Maybe look at the pstack of the shd process.

I'm not sure "stuck" is the right word, but looking at the "statistics heal-count" values it goes into a form of "go slow" mode, and ends up adding more entries for heal in some cases at a rate of about 2 every second, sometimes 4 at worst (based on ~10-15 second intervals depending on how long it takes to gather the heal-counts).

My guess currently is that since the log entries that scroll by relates to metadata and entry heals, these are when it's healing files, and when it hits a folder it somehow locks something, then iterates through the folder (very) slowly for some reason, and whilst this is happening, directory listings are outright unusable.

Is there lock profiling code available in glusterfs?

I saw this "issue":  https://github.com/gluster/glusterfs/issues/275

And there was some related PR I can't track down now which aims to wrap pthread mutex_* calls with some other library to time locks etc ... having looked at that code it made sense but could trivially introduce extra latency of a few micro-seconds even in the optimal case, and at worst could change lock behaviour (all _lock first does _trylock then re-issues _lock) - although I haven't fully analysed the code change.  And one would need to modify everywhere where mutex's are called, alternatively LD_PRELOAD hacks are useful, but then obtaining code locations where locks are initialised and used becomes much, much harder, but it would make an analysis of whether locks are contended on for extended periods of time (more than a few milliseconds) or held for extended times quite trivial, and I'm guessing backtrace() from execinfo could be helpful here to at least get an idea of which locks are involved.

This looks like the approach used by mutrace:  http://0pointer.de/blog/projects/mutrace.html so might just end up using that.  Ironically enough if the author of that project just added a little bit more graph tracking code he could turn it into a deadlock detection tool as well.

I also see from the profile info on your first email that XATTROP fop has the highest latency. Can you do an strace of the brick processes to see if there is any syscall (setxattr in particular, for the XATTROP fop) that is taking higher than usual times?

I'm familiar with strace.  What am I looking for?  How will I identify XATTROP fops and their associated system calls in this trace?

For the slow ls from the client, an ongoing selfheal should not affect it, since readdir, stat etc are all read-only operations that do not take any locks that can compete with heal, and there is no hardware bottleneck at the brick side to process requests from what you say. Maybe a tcp dump between the client and the server can help find if the slowness is from the server side by looking at the request and response times of the FOPS in the dump.

Well, it does.  I've samples a few hundred ls's since my email this morning.  All of them were sub 0.5s.  The only difference is that all three SHDs were killed.  Almost without a doubt in my mind there has to be some unforeseen interaction.  We've been having a few discussions, and a very, very long time ago, we effectively actioned a full heal with "find /path/to/fuse-mount -exec stat {} \;" and other times with "find /path/to/fuse-mount -print0 | xargs -0 -n50 -P5 -- stat" - and if we recall correctly we've seen similar there.  We thought at that time it was the FUSE process causing trouble, and it's entirely possible that that is indeed the case, but we are wondering now whether that too not maybe related to the heal process that was just happening due to the stats (heal on stat).  We generally run with cluster.*-self-heal off nowadays and rather rely purely on the SHD for performance reasons.

Since you're saying hardware is not saturated despite increasing the shd threads and wait-queues, you could try increasing performance.io-thread-count on the brick side from the default 16.

This sounds sensible.


Regards,
Ravi

Kind Regards,
Jaco


Regards,

Ravi

It seems that when the logs

With these the maximum stall I can reproduce is usually around 3 seconds before the directory listing will complete, but about 5s worst case.  I've now reverted these as well, which are known for us to cause issues under heal required conditions:

cluster.metadata-self-heal              on                                     
cluster.data-self-heal                  on                                     
cluster.entry-self-heal                 on                                     

Not sure if I need to remount for these to become effective but I've done so just to be on the safe side.

I'm still seeing these errors:

[2020-05-07 19:20:12.763311] W [MSGID: 114031] [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] 0-vpbx_shared-client-1: remote operation failed [Invalid argument]
[2020-05-07 19:20:12.995675] W [MSGID: 114031] [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] 0-vpbx_shared-client-1: remote operation failed [Invalid argument]
[2020-05-07 19:20:59.199776] W [MSGID: 114031] [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] 0-vpbx_shared-client-1: remote operation failed [Invalid argument]

And the start of my issues still seem to co-incide with that.  Sample case:

# time ls
...
real    0m5.140s
user    0m0.009s
sys    0m0.000s
# f=(*); echo "${#f[@]}"
165

This just does not seem right.  Note, again, if I set *any* of the above parameters >1 these delays get far, far worse (I've had over a minute for this exact same ls).

Any advise would be greatly appreciated.  To me, it looks like some lock is being taken somewhere, and takes a while to release.  I don't understand the glusterfs internals nearly well enough to understand exactly what's going on here, nor do I know what information to provide to help drill down on this.  Happy to go look at code, if I can get pointed in a direction.

Kind Regards,
Jaco

On 2020/05/05 08:43, Jaco Kroon wrote:

Hi Artem,

I'll do that.  In the meantime consider this (inferred from the changelogs).

It seems like glusterfs SHD is queueing into a bounded queue under lock conditions, and waiting for space to open up.  At least, this is what's hinted in the changelogs, I've been trying to grasp the SHD code for two days now and am not getting to wrap my head around that.  If this is true, then the only way to avoid lock-ups are to ensure that the queue is larger than the total number of files that can ever require healing (effectively making the queue unbounded).

Kind Regards,
Jaco

On 2020/05/04 21:56, Artem Russakovskii wrote:
I reported something similar recently to this list, and I highly advise you to update to 7.5 (in my case it was 5.11 -> 5.13), then see if it keeps happening. The self heal issues should be fixed there properly at least.

Sincerely,
Artem

--
Founder, Android PoliceAPK Mirror, Illogical Robot LLC


On Sat, May 2, 2020 at 2:04 PM Jaco Kroon <jaco@xxxxxxxxx> wrote:

Hi All,

So we've got a cluster currently running on glusterfs 7.3

Heal performance is absolutely terrible (it starts of great, runs for a while then completely trips up).

Once the SHD trips up, access speeds to the cluster also becomes poor.  And with poor I mean directory listings that normally takes well under 500ms takes upwards of 10s, but I've seen upwards of 40s (although a follow up ls immediately after is nice and fast again, temporarily).  With all SHDs killed this particular ls is as per normal (around 30-50ms).

The SHD logs doesn't provide any usable clues, the only variance in logs is this, just before all daemons across the cluster crawls to a near halt.

[2020-05-02 19:55:58.627036] I [MSGID: 108026] [afr-self-heal-common.c:1747:afr_log_selfheal] 0-vpbx_shared-replicate-0: Completed entry selfheal on 5b9aa3d8-5085-46e2-9500-145b8cbf9a61. sources= sinks=0 1  
[2020-05-02 19:55:58.627058] I [MSGID: 108026] [afr-self-heal-common.c:1747:afr_log_selfheal] 0-vpbx_shared-replicate-0: Completed entry selfheal on c736c9b6-f116-468b-adbc-f102f5b57d01. sources= sinks=0 1  
 
[2020-05-02 19:55:59.125001] W [MSGID: 114031] [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] 0-vpbx_shared-client-1: remote operation failed [Invalid argument]
 
[2020-05-02 19:55:59.339803] I [MSGID: 108026] [afr-self-heal-common.c:1747:afr_log_selfheal] 0-vpbx_shared-replicate-0: Completed entry selfheal on 2e128eeb-e991-4544-8316-4ddfcb4942da. sources= sinks=0 1 
[2020-05-02 19:55:59.490387] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-vpbx_shared-replicate-0: performing metadata selfheal on 527a63a3-9acd-4c3f-b95c-5c719325c973

A few more entries will complete overall and then it'll be a file every other second.  Once I saw this I ran a few more tests, and on every test just prior to slowing to a crawl, that entry is there.

There is a perceived improvement when lowering the thread count, but not significant enough to confirm a correlation.  Even with cluster.shd-max-threads=1 this still happens (gets a few tens thousand files completed instead of few hundred, at 32 it gets to maybe 400 to 500 before it stalls, 16 does maybe a thousand).

There are probably performant bursts hereafter, because from experience we know this process takes about two months.  But this time around it seems to be much, much worse.  Might be because that was healing a single brick to another brick, so less locking going?  At the moment the replicas are basically trying to populate the newly added arbiters.  Also, we haven't messed as greatly with the health check settings (queues and threads and locks nearly as much as since the arbiter addition because performance has simply dropped to non-acceptable levels.

Unfortunately granular entry heal isn't on at the moment (nor can I switch it on due to a few million outstanding heals).  This used to be a 2x2 cluster.  We then moved one of the bricks to a newly added server using replace-brick.  What a frightening process this is.  We then proceeded to add the arbiters.

Some of the settings below was absolute shots in the dark, based on the little information we've been able to track down so far.

Kind Regards,
Jaco

The current gluster volume info (this used to be 2x2 distributed-replicate, aiming to get it to 3x (2 + 1) ... at current rates we might get there by 2021):

Volume Name: glvol
Type: Distributed-Replicate
Volume ID: 87cfd38f-d72e-4cc2-b644-9e772d2059bb
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (2 + 1) = 6
Transport-type: tcp
Bricks:
Brick1: zoidberg:/mnt/gluster/a
Brick2: flexo:/mnt/gluster/b
Brick3: bender:/mnt/gluster/c (arbiter)
Brick4: bender:/mnt/gluster/a
Brick5: zoidberg:/mnt/gluster/b
Brick6: flexo:/mnt/gluster/c (arbiter)
Options Reconfigured:
cluster.heal-timeout: 5 <-- hoping this would at least shorten the blockages.
cluster.locking-scheme: granular
cluster.heal-wait-queue-length: 10240
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on <-- was hoping profiling might provide some insight.  Going to switch this off now.
cluster.background-self-heal-count: 128
cluster.shd-wait-qlength: 131072
cluster.shd-max-threads: 32  <-- even with this disk IO never goes over 10% utilization.  CPU goes to about 2 cores.  So it's lock contention that's slowing it down we reckon.
cluster.metadata-self-heal: off
cluster.entry-self-heal: off
cluster.data-self-heal: off <-- these kill performance.
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: on
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.stat-prefetch: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
cluster.self-heal-daemon: yes <-- disabling/enabling this doesn't help, have to kill the shd and start ... force to get things moving along again.
performance.io-thread-count: 16
performance.high-prio-threads: 16
performance.normal-prio-threads: 16
performance.low-prio-threads: 16
performance.least-prio-threads: 4
cluster.self-heal-window-size: 512
cluster.self-heal-readdir-size: 131072
client.event-threads: 8
server.event-threads: 16
network.ping-timeout: 2
cluster.data-self-heal-algorithm: full <-- most files are small (<1MB), on dual 1gbit links (so up to 2gbit, but b/w never goes over 5mbit anyway).

performance info:

Brick: zoidberg:/mnt/gluster/a
-----------------------------------------
Cumulative Stats:
   Block Size:                  1b+                   8b+                  16b+
 No. of Reads:                    0                     0                     0
No. of Writes:                  117                   195                   367
 
   Block Size:                 32b+                  64b+                 128b+
 No. of Reads:                  306                     0                     6
No. of Writes:                 1016                     9                   134
 
   Block Size:                256b+                 512b+                1024b+
 No. of Reads:                  107                    58                   485
No. of Writes:                  731                    73                   168
 
   Block Size:               2048b+                4096b+                8192b+
 No. of Reads:                   43                    59                   383
No. of Writes:                  295                   475                 10562
 
   Block Size:              16384b+               32768b+               65536b+
 No. of Reads:                  347                   380                   634
No. of Writes:                 1487                  6369                  3970
 
   Block Size:             131072b+
 No. of Reads:                35704
No. of Writes:                12398
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         957293      FORGET
      0.00       0.00 us       0.00 us       0.00 us        1455091     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          69951  RELEASEDIR
      0.00      16.20 us       5.43 us     100.95 us            113         IPC
      0.00     687.33 us      80.53 us   15414.03 us             30 REMOVEXATTR
      0.00     840.76 us      72.71 us   37648.39 us             52   FTRUNCATE
      0.00     138.49 us      36.66 us   25280.24 us           1385        STAT
      0.00     364.93 us      63.45 us   15957.66 us            810   FGETXATTR
      0.00    2208.82 us     127.23 us  267092.17 us            137        LINK
      0.00   37595.33 us    1677.46 us  218313.32 us             12       FSYNC
      0.00     675.61 us     106.82 us  114351.01 us            698    TRUNCATE
      0.00    3995.66 us     147.00 us  109544.46 us            188      RENAME
      0.00     453.11 us      53.37 us  104848.94 us           1680     SETATTR
      0.00    6004.32 us     229.33 us  151322.58 us            127       MKNOD
      0.00      68.74 us      16.11 us   61050.93 us          11412    FINODELK
      0.00    3793.44 us      68.61 us  411523.70 us            221    SETXATTR
      0.00   28669.99 us     227.28 us  138988.41 us             30     SYMLINK
      0.00      56.18 us       7.31 us   41484.37 us          22640       FLUSH
      0.00     105.64 us      19.34 us  351055.72 us          21693      STATFS
      0.00   23642.90 us     244.19 us  234912.25 us            178       MKDIR
      0.00     537.38 us      31.73 us  296911.67 us           9653       FSTAT
      0.00     214.87 us      25.71 us 1712020.99 us          39054        READ
      0.01     147.08 us       1.47 us  241611.80 us          69951     OPENDIR
      0.01    3899.84 us     117.22 us  503422.21 us           3180       RMDIR
      0.01     327.08 us      55.69 us  420376.80 us          38366       WRITE
      0.02    5580.36 us     210.11 us 1029702.30 us           6169      CREATE
      0.05      92.69 us      29.94 us  687955.86 us        1159936    READLINK
      0.40     543.50 us      36.45 us 2430653.54 us        1449307        OPEN
      0.48   87643.67 us      70.20 us  661999.51 us          10787    FXATTROP
      0.53    2342.72 us      20.91 us 4157542.43 us         447821      UNLINK
      0.53  558734.79 us      24.40 us 1399487.63 us           1889          LK
      0.78  145786.31 us      19.06 us 37490825.73 us          10687     READDIR
      0.99    2426.23 us      12.47 us 24176123.13 us         809430    GETXATTR
      1.45     676.83 us       9.30 us 193177067.25 us        4260021     ENTRYLK
      1.76   38325.38 us      21.44 us 46693575.93 us          91439    READDIRP
      4.44    1200.31 us       9.10 us 2559377.90 us        7352811     INODELK
      9.30    1849.24 us       9.28 us 35299050.00 us        9989221      LOOKUP
     79.24   47791.46 us      76.77 us 10488824.26 us        3294642     XATTROP
      0.00       0.00 us       0.00 us       0.00 us        3194806      UPCALL
      0.00       0.00 us       0.00 us       0.00 us           5311     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             54    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us           1907   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us        3189457   CI_FORGET
 
    Duration: 209524 seconds
   Data Read: 4775365531 bytes
Data Written: 2354108664 bytes
 
Interval 0 Stats:
   Block Size:                  1b+                   8b+                  16b+
 No. of Reads:                    0                     0                     0
No. of Writes:                  117                   195                   367
 
   Block Size:                 32b+                  64b+                 128b+
 No. of Reads:                  306                     0                     6
No. of Writes:                 1016                     9                   134
 
   Block Size:                256b+                 512b+                1024b+
 No. of Reads:                  107                    58                   485
No. of Writes:                  731                    73                   168
 
   Block Size:               2048b+                4096b+                8192b+
 No. of Reads:                   43                    59                   383
No. of Writes:                  295                   475                 10562
 
   Block Size:              16384b+               32768b+               65536b+
 No. of Reads:                  347                   380                   634
No. of Writes:                 1487                  6369                  3970
 
   Block Size:             131072b+
 No. of Reads:                35704
No. of Writes:                12398
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         957293      FORGET
      0.00       0.00 us       0.00 us       0.00 us        1455091     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          69951  RELEASEDIR
      0.00      16.20 us       5.43 us     100.95 us            113         IPC
      0.00     687.33 us      80.53 us   15414.03 us             30 REMOVEXATTR
      0.00     840.76 us      72.71 us   37648.39 us             52   FTRUNCATE
      0.00     138.49 us      36.66 us   25280.24 us           1385        STAT
      0.00     363.20 us      63.45 us   15957.66 us            810   FGETXATTR
      0.00    2208.82 us     127.23 us  267092.17 us            137        LINK
      0.00   37595.33 us    1677.46 us  218313.32 us             12       FSYNC
      0.00     675.61 us     106.82 us  114351.01 us            698    TRUNCATE
      0.00    3995.66 us     147.00 us  109544.46 us            188      RENAME
      0.00     453.11 us      53.37 us  104848.94 us           1680     SETATTR
      0.00    6004.32 us     229.33 us  151322.58 us            127       MKNOD
      0.00      68.74 us      16.11 us   61050.93 us          11412    FINODELK
      0.00    3793.44 us      68.61 us  411523.70 us            221    SETXATTR
      0.00   28669.99 us     227.28 us  138988.41 us             30     SYMLINK
      0.00      56.19 us       7.31 us   41484.37 us          22640       FLUSH
      0.00     105.61 us      19.34 us  351055.72 us          21693      STATFS
      0.00   23642.90 us     244.19 us  234912.25 us            178       MKDIR
      0.00     537.38 us      31.73 us  296911.67 us           9653       FSTAT
      0.00     214.91 us      25.71 us 1712020.99 us          39054        READ
      0.01     147.08 us       1.47 us  241611.80 us          69951     OPENDIR
      0.01    3899.84 us     117.22 us  503422.21 us           3180       RMDIR
      0.01     327.13 us      55.69 us  420376.80 us          38366       WRITE
      0.02    5580.36 us     210.11 us 1029702.30 us           6169      CREATE
      0.05      92.69 us      29.94 us  687955.86 us        1159936    READLINK
      0.40     543.49 us      36.45 us 2430653.54 us        1449307        OPEN
      0.48   87643.67 us      70.20 us  661999.51 us          10787    FXATTROP
      0.53    2342.57 us      20.91 us 4157542.43 us         447821      UNLINK
      0.53  558416.10 us      24.40 us 1399487.63 us           1889          LK
      0.78  145786.31 us      19.06 us 37490825.73 us          10687     READDIR
      0.99    2426.24 us      12.47 us 24176123.13 us         809430    GETXATTR
      1.45     676.75 us       9.30 us 193177067.25 us        4260021     ENTRYLK
      1.76   38324.59 us      21.44 us 46693575.93 us          91439    READDIRP
      4.44    1200.37 us       9.10 us 2559377.90 us        7352811     INODELK
      9.29    1848.94 us       9.28 us 35299050.00 us        9989221      LOOKUP
     79.24   47792.42 us      76.77 us 10488824.26 us        3294642     XATTROP
      0.00       0.00 us       0.00 us       0.00 us        3194806      UPCALL
      0.00       0.00 us       0.00 us       0.00 us           5311     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             54    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us           1907   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us        3189457   CI_FORGET
 
    Duration: 209524 seconds
   Data Read: 4775365531 bytes
Data Written: 2354108664 bytes
 
Brick: zoidberg:/mnt/gluster/b
-----------------------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   8b+
 No. of Reads:                    0                     0                     0
No. of Writes:                   53                     1                    60
 
   Block Size:                 16b+                  32b+                  64b+
 No. of Reads:                    0                   213                     2
No. of Writes:                   14                  1057                    15
 
   Block Size:                128b+                 256b+                 512b+
 No. of Reads:                    0                    41                   146
No. of Writes:                   59                   715                    80
 
   Block Size:               1024b+                2048b+                4096b+
 No. of Reads:                  224                     9                    15
No. of Writes:                  163                   282                   404
 
   Block Size:               8192b+               16384b+               32768b+
 No. of Reads:                   52                    74                   210
No. of Writes:                 7415                  1430                  6860
 
   Block Size:              65536b+              131072b+              262144b+
 No. of Reads:                  299                 15313                     1
No. of Writes:                 2458                 12378                     1
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         686733      FORGET
      0.00       0.00 us       0.00 us       0.00 us         837434     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          87338  RELEASEDIR
      0.00      22.69 us       7.16 us     332.61 us            116         IPC
      0.00     119.93 us      70.56 us     272.28 us             57   FTRUNCATE
      0.00     152.25 us      86.82 us     340.63 us             65   FGETXATTR
      0.00    2336.06 us      90.05 us   20509.81 us             23 REMOVEXATTR
      0.00     631.26 us     123.17 us   51803.77 us            121        LINK
      0.00     369.27 us      22.64 us  145090.48 us            490          LK
      0.00     360.85 us      45.14 us   46846.98 us            540        STAT
      0.00    1190.96 us     117.40 us   69781.85 us            200    TRUNCATE
      0.00     245.72 us      54.91 us  123985.76 us           1482     SETATTR
      0.00   42139.10 us    3313.56 us  312273.21 us             12       FSYNC
      0.00      55.83 us       8.26 us   49649.35 us          10047       FLUSH
      0.00      57.52 us      14.62 us     820.59 us           9977    FINODELK
      0.00    3191.60 us     143.06 us   96214.75 us            188      RENAME
      0.00   32862.86 us     215.38 us   83851.13 us             25     SYMLINK
      0.00   11392.03 us     230.70 us  105471.90 us            118       MKNOD
      0.00     134.63 us      28.08 us   36326.43 us          12587       FSTAT
      0.00     112.32 us      19.91 us  381241.34 us          21695      STATFS
      0.00     307.86 us      28.00 us  292666.83 us          16841        READ
      0.00   29625.60 us     239.57 us  239532.26 us            178       MKDIR
      0.00     319.24 us      51.19 us  136353.39 us          33445       WRITE
      0.01     175.83 us       1.57 us  513635.22 us          87338     OPENDIR
      0.01     249.96 us      58.42 us  594774.86 us          85089    SETXATTR
      0.01   11001.25 us     119.87 us  612765.03 us           3180       RMDIR
      0.04   14240.66 us     203.89 us 1376758.99 us           6067      CREATE
      0.05      89.45 us      30.14 us  687961.60 us        1219542    READLINK
      0.39    1114.30 us      44.29 us 2412169.89 us         831371        OPEN
      0.51  127078.45 us      78.55 us  694364.09 us           9493    FXATTROP
      0.63     399.16 us       8.43 us 3815689.36 us        3764714     ENTRYLK
      0.86   91973.00 us      19.89 us 24258016.33 us          22253     READDIR
      1.06    6913.71 us      11.58 us 36968532.72 us         364963    GETXATTR
      1.29   62130.12 us      36.89 us 8649871.11 us          49472    READDIRP
      2.89    1477.53 us      11.10 us 2807637.64 us        4650500     INODELK
      4.12   23681.31 us      20.98 us 23291414.50 us         414352      UNLINK
     16.09    5000.31 us       9.04 us 23866016.88 us        7655621      LOOKUP
     72.04   68953.31 us      68.40 us 27542499.65 us        2486126     XATTROP
      0.00       0.00 us       0.00 us       0.00 us        2433405      UPCALL
      0.00       0.00 us       0.00 us       0.00 us           2453     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             69    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us           1544   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us        2430885   CI_FORGET
 
    Duration: 209524 seconds
   Data Read: 2051388524 bytes
Data Written: 2208261499 bytes
 
Interval 0 Stats:
   Block Size:                  1b+                   2b+                   8b+
 No. of Reads:                    0                     0                     0
No. of Writes:                   53                     1                    60
 
   Block Size:                 16b+                  32b+                  64b+
 No. of Reads:                    0                   213                     2
No. of Writes:                   14                  1057                    15
 
   Block Size:                128b+                 256b+                 512b+
 No. of Reads:                    0                    41                   146
No. of Writes:                   59                   715                    80
 
   Block Size:               1024b+                2048b+                4096b+
 No. of Reads:                  224                     9                    15
No. of Writes:                  163                   282                   404
 
   Block Size:               8192b+               16384b+               32768b+
 No. of Reads:                   52                    74                   210
No. of Writes:                 7415                  1430                  6860
 
   Block Size:              65536b+              131072b+              262144b+
 No. of Reads:                  299                 15313                     1
No. of Writes:                 2458                 12378                     1
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         686733      FORGET
      0.00       0.00 us       0.00 us       0.00 us         837434     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          87338  RELEASEDIR
      0.00      22.69 us       7.16 us     332.61 us            116         IPC
      0.00     119.93 us      70.56 us     272.28 us             57   FTRUNCATE
      0.00     152.25 us      86.82 us     340.63 us             65   FGETXATTR
      0.00    2336.06 us      90.05 us   20509.81 us             23 REMOVEXATTR
      0.00     631.26 us     123.17 us   51803.77 us            121        LINK
      0.00     369.27 us      22.64 us  145090.48 us            490          LK
      0.00     360.85 us      45.14 us   46846.98 us            540        STAT
      0.00    1190.96 us     117.40 us   69781.85 us            200    TRUNCATE
      0.00     245.72 us      54.91 us  123985.76 us           1482     SETATTR
      0.00   42139.10 us    3313.56 us  312273.21 us             12       FSYNC
      0.00      55.83 us       8.26 us   49649.35 us          10047       FLUSH
      0.00      57.52 us      14.62 us     820.59 us           9977    FINODELK
      0.00    3191.60 us     143.06 us   96214.75 us            188      RENAME
      0.00   32862.86 us     215.38 us   83851.13 us             25     SYMLINK
      0.00   11392.03 us     230.70 us  105471.90 us            118       MKNOD
      0.00     134.63 us      28.08 us   36326.43 us          12587       FSTAT
      0.00     112.32 us      19.91 us  381241.34 us          21695      STATFS
      0.00     307.91 us      28.00 us  292666.83 us          16841        READ
      0.00   29625.60 us     239.57 us  239532.26 us            178       MKDIR
      0.00     319.22 us      51.19 us  136353.39 us          33445       WRITE
      0.01     175.83 us       1.57 us  513635.22 us          87338     OPENDIR
      0.01     249.95 us      58.42 us  594774.86 us          85089    SETXATTR
      0.01   11001.25 us     119.87 us  612765.03 us           3180       RMDIR
      0.04   14240.66 us     203.89 us 1376758.99 us           6067      CREATE
      0.05      89.46 us      30.14 us  687961.60 us        1219542    READLINK
      0.39    1114.30 us      44.29 us 2412169.89 us         831371        OPEN
      0.51  127078.45 us      78.55 us  694364.09 us           9493    FXATTROP
      0.63     399.16 us       8.43 us 3815689.36 us        3764714     ENTRYLK
      0.86   91973.00 us      19.89 us 24258016.33 us          22253     READDIR
      1.06    6913.26 us      11.58 us 36968532.72 us         364963    GETXATTR
      1.29   62121.95 us      36.89 us 8649871.11 us          49472    READDIRP
      2.89    1477.58 us      11.10 us 2807637.64 us        4650500     INODELK
      4.12   23681.29 us      20.98 us 23291414.50 us         414352      UNLINK
     16.09    5000.02 us       9.04 us 23866016.88 us        7655621      LOOKUP
     72.04   68956.14 us      68.40 us 27542499.65 us        2486126     XATTROP
      0.00       0.00 us       0.00 us       0.00 us        2433405      UPCALL
      0.00       0.00 us       0.00 us       0.00 us           2453     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             69    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us           1544   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us        2430885   CI_FORGET
 
    Duration: 209524 seconds
   Data Read: 2051388524 bytes
Data Written: 2208261499 bytes
 
Brick: flexo:/mnt/gluster/b
--------------------------------------
Cumulative Stats:
   Block Size:                  1b+                   8b+                  16b+
 No. of Reads:                    0                     0                     0
No. of Writes:                  117                   195                   367
 
   Block Size:                 32b+                  64b+                 128b+
 No. of Reads:                  130                     0                     3
No. of Writes:                 1012                     9                   134
 
   Block Size:                256b+                 512b+                1024b+
 No. of Reads:                   47                     8                   257
No. of Writes:                  729                    73                   168
 
   Block Size:               2048b+                4096b+                8192b+
 No. of Reads:                   10                    27                   240
No. of Writes:                  294                   466                 10472
 
   Block Size:              16384b+               32768b+               65536b+
 No. of Reads:                  139                   119                   135
No. of Writes:                 1480                  6273                  3935
 
   Block Size:             131072b+
 No. of Reads:                 7892
No. of Writes:                12327
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         955058      FORGET
      0.00       0.00 us       0.00 us       0.00 us        1450749     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          81377  RELEASEDIR
      0.00      15.55 us       5.35 us      31.88 us            108         IPC
      0.00     125.85 us      59.77 us     224.15 us             44   FTRUNCATE
      0.00     198.38 us      81.68 us    1332.45 us             30 REMOVEXATTR
      0.00    4254.16 us     621.52 us   10602.99 us              4       FSYNC
      0.00     795.08 us     190.30 us   16518.64 us             30     SYMLINK
      0.00     195.90 us     111.02 us     953.39 us            137        LINK
      0.00     286.07 us     179.64 us     437.82 us            119       MKNOD
      0.00     162.61 us      65.62 us    3864.77 us            221    SETXATTR
      0.00     117.29 us      48.50 us     317.17 us            354        STAT
      0.00      70.35 us      23.10 us     317.32 us            777          LK
      0.00     461.72 us     114.54 us   16013.81 us            188      RENAME
      0.00     150.40 us      75.59 us     749.54 us            810   FGETXATTR
      0.00     267.80 us      99.67 us     994.58 us            698    TRUNCATE
      0.00    1176.00 us     194.89 us   78261.12 us            178       MKDIR
      0.00     439.83 us      54.38 us   75270.51 us           1665     SETATTR
      0.00      65.89 us      17.39 us     354.77 us          11372    FINODELK
      0.00     131.57 us      32.68 us    2107.52 us           7398       FSTAT
      0.00     120.71 us      31.43 us     847.97 us           9094        READ
      0.00      60.75 us       7.25 us   22520.98 us          22614       FLUSH
      0.00      73.03 us      22.41 us     750.17 us          21656      STATFS
      0.00     323.63 us     166.56 us   23034.44 us           6145      CREATE
      0.00     217.55 us      50.60 us   30734.99 us          38051       WRITE
      0.01     146.39 us       1.80 us  318612.45 us          81377     OPENDIR
      0.01    5641.37 us     108.84 us  468960.33 us           3177       RMDIR
      0.03     100.05 us      30.93 us   49756.93 us         432439    READLINK
      0.13     483.47 us      16.81 us 2608013.14 us         448381      UNLINK
      0.21    7101.56 us      22.26 us 2554390.49 us          52185    READDIRP
      0.35     414.05 us      36.58 us 1038229.93 us        1444606        OPEN
      0.49   78518.02 us      73.39 us  373999.31 us          10745    FXATTROP
      0.72     776.21 us      11.24 us 4271792.57 us        1605822    GETXATTR
      0.77   97829.38 us      18.84 us 9477446.11 us          13532     READDIR
      0.79     320.44 us       7.71 us 2369511.06 us        4257297     ENTRYLK
      1.97     464.02 us       9.38 us 1038460.80 us        7333722     INODELK
     13.73    2372.50 us       8.33 us 6258159.87 us        9979799      LOOKUP
     80.79   42372.04 us      61.49 us 2238308.13 us        3289103     XATTROP
      0.00       0.00 us       0.00 us       0.00 us        2250268      UPCALL
      0.00       0.00 us       0.00 us       0.00 us           3756     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us            167    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us           1908   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us        2246352   CI_FORGET
 
    Duration: 209334 seconds
   Data Read: 1059777395 bytes
Data Written: 2336921014 bytes
 
Interval 0 Stats:
   Block Size:                  1b+                   8b+                  16b+
 No. of Reads:                    0                     0                     0
No. of Writes:                  117                   195                   367
 
   Block Size:                 32b+                  64b+                 128b+
 No. of Reads:                  130                     0                     3
No. of Writes:                 1012                     9                   134
 
   Block Size:                256b+                 512b+                1024b+
 No. of Reads:                   47                     8                   257
No. of Writes:                  729                    73                   168
 
   Block Size:               2048b+                4096b+                8192b+
 No. of Reads:                   10                    27                   240
No. of Writes:                  294                   466                 10472
 
   Block Size:              16384b+               32768b+               65536b+
 No. of Reads:                  139                   119                   135
No. of Writes:                 1480                  6273                  3935
 
   Block Size:             131072b+
 No. of Reads:                 7892
No. of Writes:                12327
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         955058      FORGET
      0.00       0.00 us       0.00 us       0.00 us        1450749     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          81377  RELEASEDIR
      0.00      15.55 us       5.35 us      31.88 us            108         IPC
      0.00     125.85 us      59.77 us     224.15 us             44   FTRUNCATE
      0.00     198.38 us      81.68 us    1332.45 us             30 REMOVEXATTR
      0.00    4254.16 us     621.52 us   10602.99 us              4       FSYNC
      0.00     795.08 us     190.30 us   16518.64 us             30     SYMLINK
      0.00     195.90 us     111.02 us     953.39 us            137        LINK
      0.00     286.07 us     179.64 us     437.82 us            119       MKNOD
      0.00     162.61 us      65.62 us    3864.77 us            221    SETXATTR
      0.00     117.29 us      48.50 us     317.17 us            354        STAT
      0.00      70.35 us      23.10 us     317.32 us            777          LK
      0.00     461.72 us     114.54 us   16013.81 us            188      RENAME
      0.00     150.43 us      75.59 us     749.54 us            810   FGETXATTR
      0.00     267.80 us      99.67 us     994.58 us            698    TRUNCATE
      0.00    1176.00 us     194.89 us   78261.12 us            178       MKDIR
      0.00     439.83 us      54.38 us   75270.51 us           1665     SETATTR
      0.00      65.89 us      17.39 us     354.77 us          11372    FINODELK
      0.00     131.57 us      32.68 us    2107.52 us           7398       FSTAT
      0.00     120.74 us      31.43 us     847.97 us           9094        READ
      0.00      60.75 us       7.25 us   22520.98 us          22614       FLUSH
      0.00      73.03 us      22.41 us     750.17 us          21656      STATFS
      0.00     323.63 us     166.56 us   23034.44 us           6145      CREATE
      0.00     217.50 us      50.60 us   30734.99 us          38051       WRITE
      0.01     146.39 us       1.80 us  318612.45 us          81377     OPENDIR
      0.01    5641.37 us     108.84 us  468960.33 us           3177       RMDIR
      0.03     100.06 us      30.93 us   49756.93 us         432439    READLINK
      0.13     483.47 us      16.81 us 2608013.14 us         448381      UNLINK
      0.21    7101.56 us      22.26 us 2554390.49 us          52185    READDIRP
      0.35     414.01 us      36.58 us 1038229.93 us        1444606        OPEN
      0.49   78518.02 us      73.39 us  373999.31 us          10745    FXATTROP
      0.72     776.24 us      11.24 us 4271792.57 us        1605822    GETXATTR
      0.77   97829.38 us      18.84 us 9477446.11 us          13532     READDIR
      0.79     320.45 us       7.71 us 2369511.06 us        4257297     ENTRYLK
      1.97     464.07 us       9.38 us 1038460.80 us        7333722     INODELK
     13.72    2372.42 us       8.33 us 6258159.87 us        9979799      LOOKUP
     80.79   42373.50 us      61.49 us 2238308.13 us ��      3289103     XATTROP
      0.00       0.00 us       0.00 us       0.00 us        2250268      UPCALL
      0.00       0.00 us       0.00 us       0.00 us           3756     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us            167    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us           1908   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us        2246352   CI_FORGET
 
    Duration: 209334 seconds
   Data Read: 1059777395 bytes
Data Written: 2336921014 bytes
 
Brick: flexo:/mnt/gluster/c
--------------------------------------
Cumulative Stats:
   Block Size:                  1b+
 No. of Reads:                    0
No. of Writes:                32843
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         686562      FORGET
      0.00       0.00 us       0.00 us       0.00 us         834596     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          93757  RELEASEDIR
      0.00      18.50 us       8.21 us      50.20 us            107         IPC
      0.00     205.83 us     126.30 us     412.06 us             36        LINK
      0.00     167.33 us      75.01 us     272.47 us             65   FGETXATTR
      0.00     203.58 us     117.35 us     648.28 us             88      RENAME
      0.00     203.20 us      66.60 us     809.96 us            153    TRUNCATE
      0.00      67.06 us      23.64 us     555.59 us            491          LK
      0.01      51.73 us       7.70 us    1174.64 us           9339       FLUSH
      0.01      62.41 us      15.13 us     636.35 us           9011    FINODELK
      0.03     126.23 us      43.15 us    4760.14 us          10824       FSTAT
      0.03     571.94 us      94.85 us   51697.03 us           2594       MKDIR
      0.05     264.87 us      81.58 us   53152.06 us           9009    FXATTROP
      0.05     423.05 us      38.26 us  548686.55 us           5754      CREATE
      0.05      78.14 us      19.17 us     678.32 us          32843       WRITE
      0.06     526.39 us     130.84 us   41405.46 us           5798     SYMLINK
      0.10     133.73 us      16.25 us  676494.72 us          34904     READDIR
      0.12    2054.48 us      99.28 us   88061.83 us           2732       RMDIR
      0.21     110.60 us       1.19 us   16833.78 us          93757     OPENDIR
      0.72     205.89 us      33.28 us  261023.99 us         170292      UNLINK
      0.85    2971.45 us      21.24 us  257839.19 us          13840    READDIRP
      1.94     200.88 us      25.49 us  461654.32 us         468842   FTRUNCATE
      2.13     212.44 us      54.02 us  461878.34 us         486882 REMOVEXATTR
      2.27     223.01 us      52.03 us  178897.40 us         493240    SETXATTR
      3.53     155.31 us       7.96 us   67873.00 us        1101338    GETXATTR
      3.63     212.32 us      44.42 us  551264.02 us         829002        OPEN
      5.77      76.36 us       7.76 us  962634.73 us        3660425     ENTRYLK
      6.23     304.93 us      42.51 us  661394.43 us         990644     SETATTR
      7.81     382.20 us     107.42 us  846502.78 us         990766     XATTROP
     12.39     549.11 us     135.57 us 1659697.83 us        1093387       MKNOD
     12.47     139.78 us       9.76 us  552067.56 us        4325281     INODELK
     39.53     206.94 us       8.56 us  964823.33 us        9259429      LOOKUP
      0.00       0.00 us       0.00 us       0.00 us        1856377      UPCALL
      0.00       0.00 us       0.00 us       0.00 us         933300     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us            251    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us            293   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us         923077   CI_FORGET
 
    Duration: 209334 seconds
   Data Read: 0 bytes
Data Written: 32843 bytes
 
Interval 0 Stats:
   Block Size:                  1b+
 No. of Reads:                    0
No. of Writes:                32843
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         686562      FORGET
      0.00       0.00 us       0.00 us       0.00 us         834596     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          93757  RELEASEDIR
      0.00      18.50 us       8.21 us      50.20 us            107         IPC
      0.00     205.83 us     126.30 us     412.06 us             36        LINK
      0.00     167.33 us      75.01 us     272.47 us             65   FGETXATTR
      0.00     203.58 us     117.35 us     648.28 us             88      RENAME
      0.00     203.20 us      66.60 us     809.96 us            153    TRUNCATE
      0.00      67.06 us      23.64 us     555.59 us            491          LK
      0.01      51.71 us       7.70 us    1174.64 us           9339       FLUSH
      0.01      62.41 us      15.13 us     636.35 us           9011    FINODELK
      0.03     126.23 us      43.15 us    4760.14 us          10824       FSTAT
      0.03     571.94 us      94.85 us   51697.03 us           2594       MKDIR
      0.05     264.87 us      81.58 us   53152.06 us           9009    FXATTROP
      0.05     423.05 us      38.26 us  548686.55 us           5754      CREATE
      0.05      78.14 us      19.17 us     678.32 us          32843       WRITE
      0.06     526.39 us     130.84 us   41405.46 us           5798     SYMLINK
      0.10     133.73 us      16.25 us  676494.72 us          34904     READDIR
      0.12    2054.48 us      99.28 us   88061.83 us           2732       RMDIR
      0.21     110.60 us       1.19 us   16833.78 us          93757     OPENDIR
      0.72     205.89 us      33.28 us  261023.99 us         170292      UNLINK
      0.85    2971.45 us      21.24 us  257839.19 us          13840    READDIRP
      1.94     200.87 us      25.49 us  461654.32 us         468842   FTRUNCATE
      2.13     212.44 us      54.02 us  461878.34 us         486882 REMOVEXATTR
      2.27     223.00 us      52.03 us  178897.40 us         493240    SETXATTR
      3.53     155.32 us       7.96 us   67873.00 us        1101338    GETXATTR
      3.63     212.32 us      44.42 us  551264.02 us         829002        OPEN
      5.77      76.36 us       7.76 us  962634.73 us        3660425     ENTRYLK
      6.23     304.88 us      42.51 us  661394.43 us         990644     SETATTR
      7.81     382.20 us     107.42 us  846502.78 us         990766     XATTROP
     12.39     549.25 us     135.57 us 1659697.83 us        1093387       MKNOD
     12.47     139.78 us       9.76 us  552067.56 us        4325281     INODELK
     39.53     206.95 us       8.56 us  964823.33 us        9259429      LOOKUP
      0.00       0.00 us       0.00 us       0.00 us        1856377      UPCALL
      0.00       0.00 us       0.00 us       0.00 us         933300     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us            251    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us            293   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us         923077   CI_FORGET
 
    Duration: 209334 seconds
   Data Read: 0 bytes
Data Written: 32843 bytes
 
Brick: bender:/mnt/gluster/c
---------------------------------------
Cumulative Stats:
   Block Size:                  1b+
 No. of Reads:                    0
No. of Writes:                37050
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
  ��   0.00       0.00 us       0.00 us       0.00 us         928119      FORGET
      0.00       0.00 us       0.00 us       0.00 us        1401524     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          70092  RELEASEDIR
      0.00      19.49 us       8.28 us     180.61 us            100         IPC
      0.00     238.88 us     163.97 us     486.93 us             38        LINK
      0.00     306.43 us     196.67 us     788.36 us             30     SYMLINK
      0.00    2670.18 us     765.93 us    4151.64 us              4       FSYNC
      0.00     350.65 us     141.92 us   10573.64 us            122      RENAME
      0.00      67.02 us      20.66 us     453.90 us            777          LK
      0.00     158.32 us      64.22 us     532.21 us            810   FGETXATTR
      0.00     229.87 us      66.47 us     826.31 us            603    TRUNCATE
      0.01    1742.85 us     202.81 us   47471.71 us            302       MKDIR
      0.01      59.05 us      15.40 us    1813.56 us          10060    FINODELK
      0.01     118.54 us      36.65 us    4002.58 us           6409       FSTAT
      0.01      49.38 us       8.08 us    1264.41 us          21719       FLUSH
      0.03      74.76 us      20.86 us    2008.41 us          37050       WRITE
      0.04     553.12 us      40.81 us  386485.29 us           5901      CREATE
      0.06     534.01 us     100.86 us  527770.44 us          10057    FXATTROP
      0.12     138.96 us       1.00 us  601208.70 us          70092     OPENDIR
      0.12     947.76 us      18.54 us  712568.33 us          10779     READDIR
      0.15    4099.07 us     124.69 us  421939.47 us           3008       RMDIR
      0.61     465.30 us      20.67 us  936959.86 us         109478      UNLINK
      1.57     182.41 us      49.54 us   73938.09 us         722669 REMOVEXATTR
      1.61     171.41 us       6.98 us   83809.88 us         789294    GETXATTR
      1.86     217.88 us      26.40 us  556446.19 us         718835   FTRUNCATE
      1.89     219.47 us      49.92 us  506767.78 us         722856    SETXATTR
      3.02     181.66 us      44.40 us  344656.69 us        1395957        OPEN
      5.56     322.08 us      43.37 us  634069.61 us        1450160     SETATTR
      5.84     115.88 us       7.67 us  747387.51 us        4233520     ENTRYLK
      6.53     378.33 us     122.84 us  766939.14 us        1449601     XATTROP
      9.75     115.48 us      10.70 us  556675.13 us        7097226     INODELK
     19.37    1104.98 us     101.28 us 1612346.75 us        1472813       MKNOD
     41.85     295.61 us       8.98 us  996805.54 us       11896091      LOOKUP
      0.00       0.00 us       0.00 us       0.00 us        3531139      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        1862470     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us            244    CI_XATTR
 ��    0.00       0.00 us       0.00 us       0.00 us            632   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us        1668669   CI_FORGET
 
    Duration: 208643 seconds
   Data Read: 0 bytes
Data Written: 37050 bytes
 
Interval 0 Stats:
   Block Size:                  1b+
 No. of Reads:                    0
No. of Writes:                37050
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         928119      FORGET
      0.00       0.00 us       0.00 us       0.00 us        1401524     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          70092  RELEASEDIR
      0.00      19.49 us       8.28 us     180.61 us            100         IPC
      0.00     238.88 us     163.97 us     486.93 us             38        LINK
      0.00     306.43 us     196.67 us     788.36 us             30     SYMLINK
      0.00    2670.18 us     765.93 us    4151.64 us              4       FSYNC
      0.00     350.65 us     141.92 us   10573.64 us            122      RENAME
      0.00      67.02 us      20.66 us     453.90 us            777          LK
      0.00     158.32 us      64.22 us     532.21 us            810   FGETXATTR
      0.00     229.87 us      66.47 us     826.31 us            603    TRUNCATE
      0.01    1742.85 us     202.81 us   47471.71 us            302       MKDIR
      0.01      59.05 us      15.40 us    1813.56 us          10060    FINODELK
      0.01     118.54 us      36.65 us    4002.58 us           6409       FSTAT
      0.01      49.39 us       8.08 us    1264.41 us          21719       FLUSH
      0.03      74.77 us      20.86 us    2008.41 us          37050       WRITE
      0.04     553.12 us      40.81 us  386485.29 us           5901      CREATE
      0.06     534.01 us     100.86 us  527770.44 us          10057    FXATTROP
      0.12     138.96 us       1.00 us  601208.70 us          70092     OPENDIR
      0.12     947.76 us      18.54 us  712568.33 us          10779     READDIR
      0.15    4099.07 us     124.69 us  421939.47 us           3008       RMDIR
      0.61     465.30 us      20.67 us  936959.86 us         109478      UNLINK
      1.57     182.41 us      49.54 us   73938.09 us         722669 REMOVEXATTR
      1.61     171.41 us       6.98 us   83809.88 us         789294    GETXATTR
      1.86     217.88 us      26.40 us  556446.19 us         718835   FTRUNCATE
      1.89     219.48 us      49.92 us  506767.78 us         722856    SETXATTR
      3.02     181.66 us      44.40 us  344656.69 us        1395957        OPEN
      5.56     322.09 us      43.37 us  634069.61 us        1450160     SETATTR
      5.84     115.88 us       7.67 us  747387.51 us        4233520     ENTRYLK
      6.53     378.33 us     122.84 us  766939.14 us        1449601     XATTROP
      9.75     115.48 us      10.70 us  556675.13 us        7097226     INODELK
     19.37    1105.06 us     101.28 us 1612346.75 us        1472813       MKNOD
     41.84     295.54 us       8.98 us  996805.54 us       11896091      LOOKUP
      0.00       0.00 us       0.00 us       0.00 us        3531139      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        1862470     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us            244    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us            632   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us        1668669   CI_FORGET
 
    Duration: 208643 seconds
   Data Read: 0 bytes
Data Written: 37050 bytes
��
Brick: bender:/mnt/gluster/a
---------------------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   8b+
 No. of Reads:                    0                     0                     0
No. of Writes:                   53                     1                    60
 
   Block Size:                 16b+                  32b+                  64b+
 No. of Reads:                    0                   235                     4
No. of Writes:                   14                  1033                    15
 
   Block Size:                128b+                 256b+                 512b+
 No. of Reads:                    4                   115                    81
No. of Writes:                   58                   703                    80
 
   Block Size:               1024b+                2048b+                4096b+
 No. of Reads:                  181                    26                    74
No. of Writes:                  163                   278                   397
 
   Block Size:               8192b+               16384b+               32768b+
 No. of Reads:                  131                   161                   267
No. of Writes:                 7264                  1398                  6742
 
   Block Size:              65536b+              131072b+              262144b+
 No. of Reads:                  443                 25528                     0
No. of Writes:                 2426                 12240                     1
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         684942      FORGET
      0.00       0.00 us       0.00 us       0.00 us         837092     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          87297  RELEASEDIR
      0.00     219.22 us     121.08 us     306.12 us              8 REMOVEXATTR
      0.00      21.70 us       7.24 us     196.07 us             97         IPC
      0.00     152.77 us      78.90 us    1059.82 us             46   FTRUNCATE
      0.00     295.73 us     167.10 us     606.96 us             65   FGETXATTR
      0.00     243.66 us     140.50 us    3392.16 us            121        LINK
      0.00    1216.47 us     194.37 us   23411.42 us             25     SYMLINK
      0.00     327.87 us     234.27 us     551.71 us            107       MKNOD
      0.00   36094.61 us   36094.61 us   36094.61 us              1       FSYNC
      0.00     219.63 us     147.34 us     783.10 us            188      RENAME
      0.00     104.27 us      40.34 us     595.29 us            745        STAT
      0.00     531.71 us     113.57 us   53973.08 us            200    TRUNCATE
      0.00     289.05 us      63.10 us   70107.41 us           1444     SETATTR
      0.00    2430.15 us     232.78 us   68120.54 us            177       MKDIR
      0.00      52.68 us       7.28 us     826.01 us           9890       FLUSH
      0.00      60.96 us      17.54 us    1393.95 us           9792    FINODELK
      0.00      74.26 us      20.00 us    1522.39 us          21521      STATFS
      0.00     129.21 us      34.15 us    9068.67 us          13094       FSTAT
      0.00     456.99 us     195.50 us  130182.02 us           5967      CREATE
      0.00     112.06 us      28.14 us    9545.53 us          27474        READ
      0.01     313.76 us      52.98 us   71635.74 us          32926       WRITE
      0.01     135.84 us       1.52 us  671373.08 us          87297     OPENDIR
      0.01    5225.28 us     131.28 us  951323.69 us           3177       RMDIR
      0.01     203.34 us      56.13 us  486756.90 us          85073    SETXATTR
      0.03    3521.00 us     145.28 us 3396306.00 us          15090    READDIRP
      0.04  113758.50 us      26.72 us 1147074.44 us            564          LK
      0.05      92.18 us      29.71 us   39069.99 us         850938    READLINK
      0.23     453.79 us      41.84 us  655999.23 us         831212        OPEN
      0.41   29425.34 us      19.20 us 1526187.08 us          22482     READDIR
      0.50    1954.70 us      21.51 us 1526772.28 us         415226      UNLINK
      0.54    1272.83 us       9.66 us 1604368.93 us         686945    GETXATTR
      0.55     237.57 us       7.91 us 1277047.91 us        3739718     ENTRYLK
      0.60  103888.49 us      89.32 us  761509.35 us           9311    FXATTROP
      2.60     909.04 us       9.68 us 2272778.49 us        4649291     INODELK
      5.82    1240.20 us       8.78 us 1526868.50 us        7625383      LOOKUP
     88.60   58005.98 us      67.31 us 1448810.06 us        2481914     XATTROP
      0.00       0.00 us       0.00 us       0.00 us        1396875      UPCALL
      0.00       0.00 us       0.00 us       0.00 us           1854     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             28    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us           1051   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us        1394996   CI_FORGET
 
    Duration: 208643 seconds
   Data Read: 3409342848 bytes
Data Written: 2180686507 bytes
 
Interval 0 Stats:
   Block Size:                  1b+                   2b+                   8b+
 No. of Reads:                    0                     0                     0
No. of Writes:                   53                     1                    60
 
   Block Size:                 16b+                  32b+                  64b+
 No. of Reads:                    0                   235                     4
No. of Writes:                   14                  1033                    15
 
   Block Size:                128b+                 256b+                 512b+
 No. of Reads:                    4                   115                    81
No. of Writes:                   58                   703                    80
 
   Block Size:               1024b+                2048b+                4096b+
 No. of Reads:                  181                    26                    74
No. of Writes:                  163                   278                   397
 
   Block Size:               8192b+               16384b+               32768b+
 No. of Reads:                  131                   161                   267
No. of Writes:                 7264                  1398                  6742
 
   Block Size:              65536b+              131072b+              262144b+
 No. of Reads:                  443                 25528                     0
No. of Writes:                 2426                 12240                     1
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         684942      FORGET
      0.00       0.00 us       0.00 us       0.00 us         837092     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          87297  RELEASEDIR
      0.00     219.22 us     121.08 us     306.12 us              8 REMOVEXATTR
      0.00      21.70 us       7.24 us     196.07 us             97         IPC
      0.00     152.77 us      78.90 us    1059.82 us             46   FTRUNCATE
      0.00     295.73 us     167.10 us     606.96 us             65   FGETXATTR
      0.00     243.66 us     140.50 us    3392.16 us            121        LINK
      0.00    1216.47 us     194.37 us   23411.42 us             25     SYMLINK
      0.00     327.87 us     234.27 us     551.71 us            107       MKNOD
      0.00   36094.61 us   36094.61 us   36094.61 us              1       FSYNC
      0.00     219.63 us     147.34 us     783.10 us            188      RENAME
      0.00     104.27 us      40.34 us     595.29 us            745        STAT
      0.00     531.71 us     113.57 us   53973.08 us            200    TRUNCATE
      0.00     289.05 us      63.10 us   70107.41 us           1444     SETATTR
      0.00    2430.15 us     232.78 us   68120.54 us            177       MKDIR
      0.00      52.69 us       7.28 us     826.01 us           9890       FLUSH
      0.00      60.96 us      17.54 us    1393.95 us           9792    FINODELK
      0.00      74.25 us      20.00 us    1522.39 us          21521      STATFS
      0.00     129.23 us      34.15 us    9068.67 us          13094       FSTAT
      0.00     456.99 us     195.50 us  130182.02 us           5967      CREATE
      0.00     112.45 us      28.14 us    9545.53 us          27474        READ
      0.01     313.77 us      52.98 us   71635.74 us          32926       WRITE
      0.01     135.84 us       1.52 us  671373.08 us          87297     OPENDIR
      0.01    5225.28 us     131.28 us  951323.69 us           3177       RMDIR
      0.01     203.34 us      56.13 us  486756.90 us          85073    SETXATTR
      0.03    3521.00 us     145.28 us 3396306.00 us          15090    READDIRP
      0.04  113758.50 us      26.72 us 1147074.44 us            564          LK
      0.05      92.18 us      29.71 us   39069.99 us         850938    READLINK
      0.23     453.79 us      41.84 us  655999.23 us         831212        OPEN
      0.41   29425.34 us      19.20 us 1526187.08 us          22482     READDIR
      0.50    1954.70 us      21.51 us 1526772.28 us         415226      UNLINK
      0.54    1272.79 us       9.66 us 1604368.93 us         686945    GETXATTR
      0.55     237.57 us       7.91 us 1277047.91 us        3739718     ENTRYLK
      0.60  103888.49 us      89.32 us  761509.35 us           9311    FXATTROP
      2.60     909.04 us       9.68 us 2272778.49 us        4649291     INODELK
      5.82    1240.10 us       8.78 us 1526868.50 us        7625383      LOOKUP
     88.60   58006.02 us      67.31 us 1448810.06 us        2481914     XATTROP
      0.00       0.00 us       0.00 us       0.00 us        1396875      UPCALL
      0.00       0.00 us       0.00 us       0.00 us           1854     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             28    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us           1051   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us        1394996   CI_FORGET
 
    Duration: 208643 seconds
   Data Read: 3409342848 bytes
Data Written: 2180686507 bytes
 

________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux