Fwd: Re[2]: missing files

"David F. Robinson" <david.robinson@xxxxxxxxxxxxx> · Wed, 11 Feb 2015 09:22:29 -0500

David  (Sent from mobile)
===============================
David F. Robinson, Ph.D. 
President - Corvid Technologies
704.799.6944 x101 [office]
704.252.1310      [cell]
704.799.7974      [fax]
David.Robinson@xxxxxxxxxxxxx
http://www.corvidtechnologies.com

Begin forwarded message:

From: "David F. Robinson" <david.robinson@xxxxxxxxxxxxx>
Date: February 10, 2015 at 3:52:41 PM EST
To: Shyam <srangana@xxxxxxxxxx>
Subject: Re[2]: [Gluster-devel] missing files
Reply-To: "David F. Robinson" <david.robinson@xxxxxxxxxxxxx>

files attached this time...

David

------ Original Message ------
From: "Shyam" <srangana@xxxxxxxxxx>
To: "David F. Robinson" <david.robinson@xxxxxxxxxxxxx>
Sent: 2/10/2015 3:25:13 PM
Subject: Re:  missing files

Ouch1 I think one file is missing io-stats-post-slow.txt could you check and send that across.

As of now from the strace data, the slow volume is slow in open and mkdir, which is causing the overall slowness in the tar operation.

Data is:
syscall #count avg.increase(in secs) total contribution (in secs)
open 4061 0.0368440096 149.6235229856
mkdir 384 0.0741610521 28.4778440064

Now to check what io-stats reflects in terms of numbers to glean some internal gluster specifics.

Shyam

On 02/10/2015 02:18 PM, David F. Robinson wrote:
files attached...

David

------ Original Message ------
From: "Shyam" <srangana@xxxxxxxxxx>
To: "David F. Robinson" <david.robinson@xxxxxxxxxxxxx>
Sent: 2/10/2015 1:23:58 PM
Subject: Re:  missing files

Hi David,

I am attempting to get a system with 20TB space to see if I can
recreate the issue in house. I have the machines, I need to configure
the same and get the data generation going.

In the meantime would it be possible to get the following output from
both the fast and slow volumes (the job need not complete on the slow
volume), each run should include the previous output, i.e run strace
first, then strace with the gluster io-stats enabled, etc.

1) strace -tt tar -xPf <boost tar file>

2) Enable io-stats opions and get the folowing information,

2.1) Enable io-stats options:
  - gluster volume set <volname> diagnostics.latency-measurement on
  - gluster volume set <volname> diagnostics.count-fop-hits on

2.2) Get an initial dump before starting the tar operation
  - setfattr -n trusted.io-stats-dump -v /tmp/io-stats-pre.txt
<gluster mount point>

2.3) run with strace as in (1)

2.3) Get a final dump after the tar operation
  - As in 2.2, replace file name with /tmp/io-stats-post.txt

2.4) Turn off io-stats
  - gluster volume set <volname> diagnostics.latency-measurement off
  - gluster volume set <volname> diagnostics.count-fop-hits off

I am fine with a call, just that I do not have any questions as of
now, or the need to poke around the volume etc. I am hoping that I can
reproduce this in house and see the results for myself (and in the
meantime gather the above information).

Thanks, for coming on the IRC channel, for any quick conversation on
the same.

Shyam

On 02/10/2015 12:29 PM, David F. Robinson wrote:

I have some questions from the logs, just to confirm I am seeing the
right things here (below). Some other observations as follows,

a) The stale linkto files should not cause a slowness as observed,
from the code standpoint it should not be the reason (although it is
too early to rule out other possibilities at present).

b) From the current logs I do not have any clue on the problem, other
than time stamps, as a result the approach forward maybe,
  - Reproduce this in house and check (mostly easiest as I would have
total control of the systems)
  - I may request you for some additional logs (thinking what is best
at present)

Onto the information from the logs:

1) backup-homegfs.log is the mount /backup/homegfs that is always
showing up a slow performance, correct? This is the primary production
volume, correct?

- The logs are neat, just captures the extract portions of activity
making it easier to look at them, thanks.
Yes, /backup/homegfs is the mount point for the homegfs_bkp volume and
it was showing the performance problems. I created a new volume for
testing (same machine, same bricks, etc...) called test2brick. The
performance issue was gone. My first guess was that somehow the
homegfs_bkp volume was corrupted and my second guess was that the
performance improvement on test2brick was due to it having no data. To
test, I did a 'cp -ar /backup/homegfs/xyz /test2brick' for roughly 10TB
of files. I then re-ran my test on test2brick and it showed the
performance issue. I then did a 'rm -rf /test2brick/xyz' and reran the
tests and the performance recovered.

2) Did the tar job complete on backup-homegfs? I ask this, as the
following are the start and end logs of the extract and this is
incorrect when compared to the two other logs:

[2015-02-09 22:44:57.872633] I [MSGID: 109036]
[dht-common.c:6222:dht_log_new_layout_for_dir_selfheal]
0-homegfs_bkp-dht: Setting layout of /boost_1_57_0 with [Subvol_name:
homegfs_bkp-client-0, Err: -1 , Start: 2105878482 , Stop: 4294967295
], [Subvol_name: homegfs_bkp-client-1, Err: -1 , Start: 0 , Stop:
2105878481 ],

[2015-02-09 22:48:38.898166] I [MSGID: 109036]
[dht-common.c:6222:dht_log_new_layout_for_dir_selfheal]
0-homegfs_bkp-dht: Setting layout of
/boost_1_57_0/boost/math/special_functions/detail with [Subvol_name:
homegfs_bkp-client-0, Err: -1 , Start: 0 , Stop: 2105878481 ],
[Subvol_name: homegfs_bkp-client-1, Err: -1 , Start: 2105878482 ,
Stop: 4294967295 ],

Note the last log ends with the name of the directory as,
"/boost_1_57_0/boost/math/special_functions/detail", whereas in the
test2 and test3 bricks these end at
"/boost_1_57_0/tools/regression/xsl_reports/xsl/v2/html"
I ran it many times on both volumes to make sure that it wasn't an
anomaly. The tar extraction on the homegfs_bkp volume takes 19-minutes.
  I have let it complete several times, but I also have killed it after
5-10 minutes because after 5-minutes, I know there is still a
performance problem. So, the log you were looking at was probably one
of the runs where I killed the extraction.

3) Various tests on test3brick details:

[2015-02-10 14:22:52.403546] I [MSGID: 109036] - Start
[2015-02-10 14:23:51.082752] I [MSGID: 109036] - Stop

Time taken: 00:59

[2015-02-10 14:24:20.590523] I [MSGID: 109036] - Start
[2015-02-10 14:25:14.113263] I [MSGID: 109036] - End

Time taken: 00:54

This did not have any other data on the bricks, correct?
I haven't done much testing on brick3 yet. I was going under the
assumption that the gluster developers were going to want me to test
some options and I didn't want to have to wait for 10TB to copy over
prior to each test. So, I created a test2brick that will contain
roughly 10TB of data and I created a test3brick that is empty. Copying
over the 10TB of data takes several hours.

4) Various tests on test2brick:

(i)
[2015-02-09 22:42:43.215401] I [MSGID: 109036] - Start
[2015-02-09 22:44:16.270895] I [MSGID: 109036] - End

Time taken on empty volume: 1:33 (with no data on volume, correct?)
This sounds about right, although with 0-data the time comes out around
50-58 seconds.

(ii)
[2015-02-10 02:24:32.497366] I [MSGID: 109036] - Start
[2015-02-10 02:26:35.599863] I [MSGID: 109036] - Stop

Time taken with some data in the volume: 2:03

Added data, /users/carpentr
Correct. I started testing as the cp was ongoing to see what the effect
of the cp was on performance. It was slower (as expected), but nothing
like the 19-minutes I was getting on homegfs_bkp.

(iii)

[2015-02-10 13:55:32.496258] I [MSGID: 109036] - Start
[2015-02-10 13:58:49.158800] I [MSGID: 109036] - Stop

Time taken with some _more_ data in the volume: 3:17

Added data, /users/former_employees
Correct. As the amount of data on test2brick increases, the time
increases.

(iv)
[2015-02-10 14:15:48.617364] I [MSGID: 109036] - Start
[2015-02-10 14:22:48.178021] I [MSGID: 109036] - Stop

Time taken with no change between (iii) and (iv) in terms of volume
data: 7:00

Are the above observations right?
The time when the full copy had finished was approximately 7-minutes. I
only copied over approximately 10-TB to test2brick. Homegfs_bkp
contains roughly 40-TB. I am assuming that if I copied over the full
40TB, the test2brick would reproduce the 19-minute runtime of
homegfs_bkp.

I am working on getting on IIRC now. I also am availble at 704.252.1310
if you want to discuss.

David

Thanks,
Shyam

On 02/10/2015 10:51 AM, David F. Robinson wrote:
I am not on IRC, but I could get on it. I can be available for a call
as well. You can call my cell 704.252.1310 anytime. Or, if you want
multiple people on the call, I have a Telecon # setup that I could
send out.

David (Sent from mobile)

===============================
David F. Robinson, Ph.D.
President - Corvid Technologies
704.799.6944 x101 [office]
704.252.1310 [cell]
704.799.7974 [fax]
David.Robinson@xxxxxxxxxxxxx
http://www.corvidtechnologies.com

On Feb 10, 2015, at 10:12 AM, Shyam <srangana@xxxxxxxxxx> wrote:

Hi,

Are you on IRC? (if so nick and channels please (#gluster-devel
would be nice))

Can we get into a call or talk somehow? Maybe faster for me to catch
up on certain things and also to look at what the problem could be.

Thanks,
Shyam

On 02/10/2015 12:11 AM, David Robinson wrote:
When I created a new volume on the same storage system my
performance
problem goes away completely.
The only difference is that the new volume doesn't have all of the
broken links in .glusterfs and the new volume was created with
3.6.2.
The old volume was created using an older version of gluster prior
to my
upgrades to 3.6.

I even tried a gluster volume reset to make sure all of the
performance
options were the same. The new volume (test2brick) is 20x faster for
small file operations.

Email train below.

===============================
David F. Robinson, Ph.D.
President - Corvid Technologies
704.799.6944 x101 [office]
704.252.1310 [cell]
704.799.7974 [fax]
David.Robinson@xxxxxxxxxxxxx <mailto:David.Robinson@xxxxxxxxxxxxx>
http://www.corvidtechnologies.com

Begin forwarded message:

*From:* "David F. Robinson" <david.robinson@xxxxxxxxxxxxx
<mailto:david.robinson@xxxxxxxxxxxxx>>
*Date:* February 9, 2015 at 5:52:00 PM EST
*To:* "Benjamin Turner" <bennyturns@xxxxxxxxx
<mailto:bennyturns@xxxxxxxxx>>
*Subject:* *Re[5]:  missing files*
*Reply-To:* "David F. Robinson" <david.robinson@xxxxxxxxxxxxx
<mailto:david.robinson@xxxxxxxxxxxxx>>

Ben,
I cleared the logs and rebooted the machine. Same issue.
homegfs_bkp
takes 19-minutes and test2brick (the new volume) takes 1-minute.
Is it possible that some old parameters are still set for
homegfs_bkp
that are no longer in use? I tried a gluster volume reset for
homegfs_bkp, but it didn't have any effect.
I have attached the full logs.
David
------ Original Message ------
From: "David F. Robinson" <david.robinson@xxxxxxxxxxxxx
<mailto:david.robinson@xxxxxxxxxxxxx>>
To: "Benjamin Turner" <bennyturns@xxxxxxxxx
<mailto:bennyturns@xxxxxxxxx>>
Sent: 2/9/2015 5:39:18 PM
Subject: Re[4]:  missing files
Ben,
I have traced this out to a point where I can rule out many
issues.
I was hoping you could help me from here.
I went with the "tar -xPf boost.tar" as my test case, which on my
old
storage system took about 1-minute to extract. On my backup system
and my primary storage (both gluster), it takes roughly
19-minutes.
First step was to create a new storage system (striped RAID, two
sets
of 3-drives). All was good here with a gluster extraction time of
1-minute. I then went to my backup system and created another
partition using only one of the two bricks on that system. Still
1-minute. I went to a two brick setup and it stayed at 1-minute.
At this point, I have recreated using the same parameters on a
test2brick volume that should be identical to my homegfs_bkp
volume. Everything is the same including how I mounted the volume.
The only different is that the homegfs_bkp has 30-TB of data
and the
test2brick is blank. I didn't think that performance would be
affected by putting data on the volume.
Can you help? Do you have any suggestions? Do you think upgrading
gluster from 3.5 to 3.6.1 to 3.6.2 somehow message up
homegfs_bkp? My
layout is shown below. These should give identical speeds.
[root@gfs01bkp test2brick]# gluster volume info homegfs_bkp
Volume Name: homegfs_bkp
Type: Distribute
Volume ID: 96de8872-d957-4205-bf5a-076e3f35b294
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: gfsib01bkp.corvidtec.com
<http://gfsib01bkp.corvidtec.com>:/data/brick01bkp/homegfs_bkp
Brick2: gfsib01bkp.corvidtec.com
<http://gfsib01bkp.corvidtec.com>:/data/brick02bkp/homegfs_bkp
[root@gfs01bkp test2brick]# gluster volume info test2brick

Volume Name: test2brick
Type: Distribute
Volume ID: 123259b2-3c61-4277-a7e8-27c7ec15e550
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: gfsib01bkp.corvidtec.com
<http://gfsib01bkp.corvidtec.com>:/data/brick01bkp/test2brick
Brick2: gfsib01bkp.corvidtec.com
<http://gfsib01bkp.corvidtec.com>:/data/brick02bkp/test2brick
[root@gfs01bkp brick02bkp]# mount | grep test2brick
gfsib01bkp.corvidtec.com
<http://gfsib01bkp.corvidtec.com>:/test2brick.tcp on /test2brick
type
fuse.glusterfs (rw,allow_other,max_read=131072)
[root@gfs01bkp brick02bkp]# mount | grep homegfs_bkp
gfsib01bkp.corvidtec.com
<http://gfsib01bkp.corvidtec.com>:/homegfs_bkp.tcp on
/backup/homegfs
type fuse.glusterfs (rw,allow_other,max_read=131072)
[root@gfs01bkp brick02bkp]# df -h
Filesystem Size Used Avail Use%
Mounted on
/dev/mapper/vg00-lv_root 20G 1.7G 18G 9% /
tmpfs 16G 0 16G 0%
/dev/shm
/dev/md126p1 1008M 110M 848M 12% /boot
/dev/mapper/vg00-lv_opt 5.0G 220M 4.5G 5% /opt
/dev/mapper/vg00-lv_tmp 5.0G 139M 4.6G 3% /tmp
/dev/mapper/vg00-lv_usr 20G 2.7G 17G 15% /usr
/dev/mapper/vg00-lv_var 40G 4.4G 34G 12% /var
/dev/mapper/vg01-lvol1 88T 22T 67T 25%
/data/brick01bkp
/dev/mapper/vg02-lvol1 88T 22T 67T 25%
/data/brick02bkp
gfsib01bkp.corvidtec.com
<http://gfsib01bkp.corvidtec.com>:/homegfs_bkp.tcp 175T 43T
133T 25% /backup/homegfs
gfsib01bkp.corvidtec.com
<http://gfsib01bkp.corvidtec.com>:/test2brick.tcp 175T 43T
133T 25% /test2brick
------ Original Message ------
From: "Benjamin Turner" <bennyturns@xxxxxxxxx
<mailto:bennyturns@xxxxxxxxx>>
To: "David F. Robinson" <david.robinson@xxxxxxxxxxxxx
<mailto:david.robinson@xxxxxxxxxxxxx>>
Sent: 2/6/2015 12:52:58 PM
Subject: Re: Re[2]:  missing files
Hi David. Lets start with the basics and go from there. IIRC you
are using LVM with thick provisioning, lets verify the following:

1. You have everything properly aligned for your RAID stripe
size,
etc. I have attached the script we package with RHS that I am in
the process of updating. I want to double check you created
the PV
/ VG / LV with the proper variables. Have a look at the
create_pv,
create_vg, and create_lv(old) functions. You will need to know
the
stripe size of your raid and the number of stripe elements(data
disks, not hotspares). Also make sure you mkfs.xfs with:

echo "mkfs -t xfs -f -K -i size=$inode_size -d
sw=$stripe_elements,su=$stripesize -n size=$fs_block_size
/dev/$vgname/$lvname"

We use 512k inodes because some workload use more than the
default
inode size and you don't want xattrs bleeding over inodes.

2. Are you running RHEL or Centos? If so I would
recommend tuned_profile=rhs-high-throughput. If you don't have
that
tuned profile I'll get you everything it sets.

3. For small files we we recommend the following:

# RAID related variables.
# stripesize - RAID controller stripe unit size
# stripe_elements - the number of data disks
# The --dataalignment option is used while creating the physical
volumeTo
# align I/O at LVM layer
# dataalign -
# RAID6 is recommended when the workload has predominantly larger
# files ie not in kilobytes.
# For RAID6 with 12 disks and 128K stripe element size.
stripesize=128k
stripe_elements=10
dataalign=1280k

# RAID10 is recommended when the workload has predominantly
smaller
files
# i.e in kilobytes.
# For RAID10 with 12 disks and 256K stripe element size,
uncomment the
# lines below.
# stripesize=256k
# stripe_elements=6
# dataalign=1536k

4. Jumbo frames everywhere! Check out the effect of jumbo frames,
make sure they are setup properly on your switch and add the
MTU=9000 to your ifcfg files(unless you have it already):

https://rhsummit.files.wordpress.com/2013/07/england_th_0450_rhs_perf_practices-4_neependra.pdf

(see the jumbo frames section here, the whole thing is a good
read)
https://rhsummit.files.wordpress.com/2014/04/bengland_h_1100_rhs_performance.pdf

(this is updated for 2014)

5. There is a smallfile enhancement that just landed in master
that
is showing me a 60% improvement in writes. This is called multi
threaded epoll and it is looking VERY promising WRT smallfile
performance. Here is a summary:

Hi all. I see alot of discussion on $subject and I wanted to
take a
minute to talk about it and what we can do to test / observe the
effects of it. Lets start with a bit of background:

**Background**

-Currently epoll is single threaded on both clients and servers.
*This leads to a "hot thread" which consumes 100% of a CPU core.
*This can be observed by running BenE's smallfile benchmark to
create files, running top(on both clients and servers), and
pressing
H to show threads.
*You will be able to see a single glusterfs thread eating 100% of
the CPU:

2871 root 20 0 746m 24m 3004 S100.0 0.1 14 <callto:100.0
0.1 14>:35.89 glusterfsd
4522 root 20 0 747m 24m 3004 S5.3 0.1 0 <callto:5.3 0.1
0>:02.25 glusterfsd
4507 root 20 0 747m 24m 3004 S5.0 0.1 0 <callto:5.0 0.1
0>:05.91 glusterfsd
21200 root 20 0 747m 24m 3004 S4.6 0.1 0 <callto:4.6 0.1
0>:21.16 glusterfsd

-Single threaded epoll is a bottlenck for high IOP / low metadata
workloads(think smallfile). With single threaded epoll we are CPU
bound by the single thread pegging out a CPU.

So the proposed solution to this problem is to make epoll multi
threaded on both servers and clients. Here is a link to the
upstream proposal:

http://www.gluster.org/community/documentation/index.php/Features/Feature_Smallfile_Perf#multi-thread-epoll

Status: [http://review.gluster.org/#/c/3842/based on Anand
Avati's
patch ]

Why: remove single-thread-per-brick barrier to higher CPU
utilization by servers

Use case: multi-client and multi-thread applications

Improvement: measured 40% with 2 epoll threads and 100% with 4
epoll
threads for small file creates to an SSD

Disadvantage: conflicts with support for SSL sockets, may require
significant code change to support both.

Note: this enhancement also helps high-IOPS applications such as
databases and virtualization which are not metadata-intensive.
This
has been measured already using a Fusion I/O SSD performing
random
reads and writes -- it was necessary to define multiple bricks
per
SSD device to get Gluster to the same order of magnitude IOPS
as a
local filesystem. But this workaround is problematic for users,
because storage space is not properly measured when there are
multiple bricks on the same filesystem.

Multi threaded epoll is part of a larger page that talks about
smallfile performance enhancements, proposed and happening.

Goal: if successful, throughput bottleneck should be either the
network or the brick filesystem!
What it doesn't do: multi-thread-epoll does not solve the
excessive-round-trip protocol problems that Gluster has.
What it should do: allow Gluster to exploit the mostly
untapped CPU
resources on the Gluster servers and clients.
How it does it: allow multiple threads to read protocol messages
and
process them at the same time.
How to observe: multi-thread-epoll should be configurable (how to
configure? gluster command?), with thread count 1 it should be
same
as RHS 3.0, with thread count 2-4 it should show significantly
more
CPU utilization (threads visible with "top -H"), resulting in
higher
throughput.

**How to observe**

Here are the commands needed to setup an environment to test
in on
RHS 3.0.3:
rpm -e glusterfs-api glusterfs glusterfs-libs glusterfs-fuse
glusterfs-geo-replication glusterfs-rdma glusterfs-server
glusterfs-cli gluster-nagios-common samba-glusterfs vdsm-gluster
--nodeps
rhn_register
yum groupinstall "Development tools"
git clonehttps://github.com/gluster/glusterfs.git
git branch test
git checkout test
git
fetchhttp://review.gluster.org/glusterfsrefs/changes/42/3842/17
&& git cherry-pick FETCH_HEAD
git
fetchhttp://review.gluster.org/glusterfsrefs/changes/88/9488/2
&& git cherry-pick FETCH_HEAD
yum install openssl openssl-devel
wgetftp://fr2.rpmfind.net/linux/epel/6/x86_64/cmockery2-1.3.8-2.el6.x86_64.rpm

wgetftp://fr2.rpmfind.net/linux/epel/6/x86_64/cmockery2-devel-1.3.8-2.el6.x86_64.rpm

yum install cmockery2-1.3.8-2.el6.x86_64.rpm
cmockery2-devel-1.3.8-2.el6.x86_64.rpm libxml2-devel
./autogen.sh
./configure
make
make install

Verify you are using the upstream with:

# gluster -- version

To enable set multithreaded epoll run the following commands:

 From the patch:
{ .key = "client.event-threads", 839
.voltype = "protocol/client", 840
.op_version = GD_OP_VERSION_3_7_0, 841
},
{ .key = "server.event-threads", 946
.voltype = "protocol/server", 947
.op_version = GD_OP_VERSION_3_7_0, 948
},

# gluster v set <volname> server.event-threads 4
# gluster v set <volname> client.event-threads 4

Also grab smallfile:

https://github.com/bengland2/smallfile

After git cloneing smallfile run:

python /small-files/smallfile/smallfile_cli.py --operation create
--threads 8 --file-size 64 --files 10000 --top /gluster-mount
--pause 1000 --host-set "client1 client2"

Again we will be looking at top + show threads(press H). With 4
threads on both clients and servers you should see something
similar
to(this isnt exact, I coped and pasted):

2871 root 20 0 746m 24m 3004 S 35.0 0.1 14 <callto:25.0
0.1 14>:35.89 glusterfsd
2872 root 20 0 746m 24m 3004 S 51.0 0.1 14 <callto:25.0
0.1 14>:35.89 glusterfsd
2873 root 20 0 746m 24m 3004 S 43.0 0.1 14 <callto:25.0
0.1 14>:35.89 glusterfsd
2874 root 20 0 746m 24m 3004 S 65.0 0.1 14 <callto:25.0
0.1 14>:35.89 glusterfsd
4522 root 20 0 747m 24m 3004 S5.3 0.1 0 <callto:5.3 0.1
0>:02.25 glusterfsd
4507 root 20 0 747m 24m 3004 S5.0 0.1 0 <callto:5.0 0.1
0>:05.91 glusterfsd
21200 root 20 0 747m 24m 3004 S4.6 0.1 0 <callto:4.6 0.1
0>:21.16 glusterfsd

If you have a test env I would be interested to see how multi
threaded epoll performs, but I am 100% sure its not ready for
production yet. RH will be supporting it with our 3.0.4(the next
one) release unless we find show stopping bugs. My testing looks
very promising though.

Smallfile performance enhancements are one of the key focuses for
our 3.1 release this summer, we are working very hard to improve
this as this is the use case for the majority of people.

On Fri, Feb 6, 2015 at 11:59 AM, David F. Robinson
<david.robinson@xxxxxxxxxxxxx
<mailto:david.robinson@xxxxxxxxxxxxx>>
wrote:

    Ben,
    I was hoping you might be able to help with two performance
    questions. I was doing some testing of my rsync where I am
    backing up my primary gluster system (distributed +
replicated)
    to my backup gluster system (distributed). I tried three
tests
    where I rsynced from one of my primary sytems (gfsib02b)
to my
    backup machine. The test directory contains roughly 5500
files,
    most of which are small. The script I ran is shown below
which
    repeats the tests 3x for each section to check variability in
    timing.
    1) Writing to the local disk is drastically faster than
writing
    to gluster. So, my writes to the backup gluster system are
what
    is slowing me down, which makes sense.
    2) When I write to the backup gluster system
(/backup/homegfs),
    the timing goes from 35-seconds to 1min40seconds. The
question
    here is whether you could recommend any settings for this
volume
    that would improve performance for small file writes? I have
    included the output of 'gluster volume info" below.
    3) When I did the same tests on the Source_bkp volume, it is
    almost 3x as slow as the homegfs_bkp volume. However,
these are
    just different volumes on the same storage system. The volume
    parameters are identical (see below). The performance of
these
    two should be identical. Any idea why they wouldn't be? And
    any suggestions for how to fix this? The only thing that I
see
    different between the two is the order of the "Options
    reconfigured" section. I assume order of options doesn't
matter.
    *Backup to local hard disk (no gluster writes)*
    /time /usr/local/bin/rsync -av --numeric-ids --delete
    --block-size=131072 -e "ssh -T -c arcfour -o
Compression=no -x"
    gfsib02b:/homegfs/test /temp1
     time /usr/local/bin/rsync -av --numeric-ids --delete
    --block-size=131072 -e "ssh -T -c arcfour -o
Compression=no -x"
    gfsib02b:/homegfs/test /temp2
     time /usr/local/bin/rsync -av --numeric-ids --delete
    --block-size=131072 -e "ssh -T -c arcfour -o
Compression=no -x"
    gfsib02b:/homegfs/test /temp3 /
    //
    / real 0m35.579s
            user 0m31.290s
            sys 0m12.282s
    ///
    //
    / real 0m38.035s
            user 0m31.622s
            sys 0m10.907s
    /
    //
    / real 0m38.313s
            user 0m31.458s
            sys 0m10.891s
    /
    *Backup to gluster backup system on volume homegfs_bkp*
    / time /usr/local/bin/rsync -av --numeric-ids --delete
    --block-size=131072 -e "ssh -T -c arcfour -o
Compression=no -x"
    gfsib02b:/homegfs/test /backup/homegfs/temp1
     time /usr/local/bin/rsync -av --numeric-ids --delete
    --block-size=131072 -e "ssh -T -c arcfour -o
Compression=no -x"
    gfsib02b:/homegfs/test /backup/homegfs/temp2
     time /usr/local/bin/rsync -av --numeric-ids --delete
    --block-size=131072 -e "ssh -T -c arcfour -o
Compression=no -x"
    gfsib02b:/homegfs/test /backup/homegfs/temp3 /
    //
    /
    / real 1m42.026s
            user 0m32.604s
            sys 0m9.967s
    /
    / real 1m45.480s
            user 0m32.577s
            sys 0m11.994s
    /
    / real 1m40.436s
            user 0m32.521s
            sys 0m11.240s/
    /
    //
    *Backup to gluster backup system on volume Source_bkp*
    / time /usr/local/bin/rsync -av --numeric-ids --delete
    --block-size=131072 -e "ssh -T -c arcfour -o
Compression=no -x"
    gfsib02b:/homegfs/test /backup/Source/temp1
     time /usr/local/bin/rsync -av --numeric-ids --delete
    --block-size=131072 -e "ssh -T -c arcfour -o
Compression=no -x"
    gfsib02b:/homegfs/test /backup/Source/temp2
     time /usr/local/bin/rsync -av --numeric-ids --delete
    --block-size=131072 -e "ssh -T -c arcfour -o
Compression=no -x"
    gfsib02b:/homegfs/test /backup/Source/temp3 /
    / real 3m30.491s
            user 0m32.676s
            sys 0m10.776s
    ///
    //
    / real 3m26.076s
            user 0m32.588s
            sys 0m11.048s
    /
    //
    / real 3m7.460s
            user 0m32.763s
            sys 0m11.687s
    /

    /Volume Name: *Source_bkp*
    Type: Distribute
    Volume ID: 1d4c210d-a731-4d39-a0c5-ea0546592c1d
    Status: Started
    Number of Bricks: 2
    Transport-type: tcp
    Bricks:
    Brick1: gfsib01bkp.corvidtec.com
<http://gfsib01bkp.corvidtec.com>:/data/brick01bkp/Source_bkp
    Brick2: gfsib01bkp.corvidtec.com
<http://gfsib01bkp.corvidtec.com>:/data/brick02bkp/Source_bkp
    Options Reconfigured:
    performance.cache-size: 128MB
    performance.io-thread-count: 32
    server.allow-insecure: on
    network.ping-timeout: 10
    storage.owner-gid: 100
    performance.write-behind-window-size: 128MB
    server.manage-gids: on
    changelog.rollover-time: 15
    changelog.fsync-interval: 3

    Volume Name: *homegfs_bkp*
    Type: Distribute
    Volume ID: 96de8872-d957-4205-bf5a-076e3f35b294
    Status: Started
    Number of Bricks: 2
    Transport-type: tcp
    Bricks:
    Brick1: gfsib01bkp.corvidtec.com
<http://gfsib01bkp.corvidtec.com>:/data/brick01bkp/homegfs_bkp
    Brick2: gfsib01bkp.corvidtec.com
<http://gfsib01bkp.corvidtec.com>:/data/brick02bkp/homegfs_bkp
    Options Reconfigured:
    storage.owner-gid: 100
    performance.io-thread-count: 32
    server.allow-insecure: on
    network.ping-timeout: 10
    performance.cache-size: 128MB
    performance.write-behind-window-size: 128MB
    server.manage-gids: on
    changelog.rollover-time: 15
    changelog.fsync-interval: 3
    /
    /
    /
    //
    //
    /
    /
    //
    //
    //
    ------ Original Message ------
    From: "Benjamin Turner" <bennyturns@xxxxxxxxx
    <mailto:bennyturns@xxxxxxxxx>>
    To: "David F. Robinson" <david.robinson@xxxxxxxxxxxxx
    <mailto:david.robinson@xxxxxxxxxxxxx>>
    Cc: "Gluster Devel" <gluster-devel@xxxxxxxxxxx
    <mailto:gluster-devel@xxxxxxxxxxx>>;
"gluster-users@xxxxxxxxxxx
    <mailto:gluster-users@xxxxxxxxxxx>"
<gluster-users@xxxxxxxxxxx
    <mailto:gluster-users@xxxxxxxxxxx>>
    Sent: 2/3/2015 7:12:34 PM
    Subject: Re:  missing files
    It sounds to me like the files were only copied to one
replica,
    werent there for the initial for the initial ls which
triggered
    a self heal, and were there for the last ls because they
were
    healed. Is there any chance that one of the replicas was
down
    during the rsync? It could be that you lost a brick during
    copy or something like that. To confirm I would look for
    disconnects in the brick logs as well as checking
    glusterfshd.log to verify the missing files were actually
healed.

    -b

    On Tue, Feb 3, 2015 at 5:37 PM, David F. Robinson
    <david.robinson@xxxxxxxxxxxxx
    <mailto:david.robinson@xxxxxxxxxxxxx>> wrote:

        I rsync'd 20-TB over to my gluster system and noticed
that
        I had some directories missing even though the rsync
        completed normally.
        The rsync logs showed that the missing files were
transferred.
        I went to the bricks and did an 'ls -al
        /data/brick*/homegfs/dir/*' the files were on the
bricks.
        After I did this 'ls', the files then showed up on the
FUSE
        mounts.
        1) Why are the files hidden on the fuse mount?
        2) Why does the ls make them show up on the FUSE mount?
        3) How can I prevent this from happening again?
        Note, I also mounted the gluster volume using NFS and
saw
        the same behavior. The files/directories were not shown
        until I did the "ls" on the bricks.
        David
        ===============================
        David F. Robinson, Ph.D.
        President - Corvid Technologies
        704.799.6944 x101 <tel:704.799.6944%20x101> [office]
        704.252.1310 <tel:704.252.1310> [cell]
        704.799.7974 <tel:704.799.7974> [fax]
        David.Robinson@xxxxxxxxxxxxx
        <mailto:David.Robinson@xxxxxxxxxxxxx>
        http://www.corvidtechnologies.com/

        _______________________________________________
        Gluster-devel mailing list
        Gluster-devel@xxxxxxxxxxx
<mailto:Gluster-devel@xxxxxxxxxxx>
        http://www.gluster.org/mailman/listinfo/gluster-devel

=== Cumulative stats ===
      Duration : 80629 secs
     BytesRead : 5465816563903
  BytesWritten : 104984439407

Block Size   :               1B+               2B+               4B+
Read Count   :                 1                 3                10
Write Count  :                 0                 5                 2

Block Size   :               8B+              16B+              32B+
Read Count   :                21                25               106
Write Count  :                 9                29                51

Block Size   :              64B+             128B+             256B+
Read Count   :               583               174               729
Write Count  :                79               144               564

Block Size   :             512B+            1024B+            2048B+
Read Count   :               649              1354              2614
Write Count  :              3280              4464              4509

Block Size   :            4096B+            8192B+           16384B+
Read Count   :              5573             10445             21909
Write Count  :              4159              6989               355

Block Size   :           32768B+           65536B+          131072B+
Read Count   :             38193            111438          41599470
Write Count  :                56                 0            799938

Block Size   :          262144B+
Read Count   :                 0
Write Count  :                31

Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
---           ----------    -----------    -----------    -----------
STAT                  12       22.58 us        9.00 us       96.00 us
MKDIR              10213    26212.14 us       88.00 us   582978.00 us
UNLINK             10439    10966.20 us       39.00 us   701286.00 us
RMDIR              10213      216.30 us       41.00 us    78288.00 us
READ              801312       74.21 us       19.00 us    94552.00 us
WRITE             817604      115.22 us       10.00 us    83186.00 us
STATFS              1131       18.97 us        9.00 us      103.00 us
FLUSH              11564       13.10 us        5.00 us      150.00 us
SETXATTR           10215       47.47 us       16.00 us      192.00 us
OPENDIR            20463       17.15 us       11.00 us      159.00 us
CREATE             10439    25071.92 us       67.00 us  4369403.00 us
FSTAT                660       28.34 us        8.00 us      115.00 us
LOOKUP             60897       82.09 us       16.00 us    62328.00 us
SETATTR            33770       34.39 us       13.00 us      205.00 us
READDIRP           31289      930.23 us       10.00 us   226729.00 us
FORGET             25752           0 us           0 us           0 us
RELEASE            14744           0 us           0 us           0 us
RELEASEDIR         69034           0 us           0 us           0 us
------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----

Current open fd's: 0 Max open fd's: 1 time 2015-02-09 22:44:58.229611

==========Open File Stats========

COUNT:  	  FILE NAME

==========Read File Stats========

COUNT:  	  FILE NAME
800000       /myfile.txt

==========Write File Stats========

COUNT:  	  FILE NAME
799969       /myfile.txt
64           /boost_1_57_0/boost/phoenix/core/detail/preprocessed/function_eval_50.hpp
63           /boost_1_57_0/boost/phoenix/core/detail/preprocessed/function_eval_50.hpp
55           /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/try_catch_eval_50.hpp
53           /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/try_catch_eval_50.hpp
46           /boost_1_57_0/boost/phoenix/scope/preprocessed/lambda_40.hpp
45           /boost_1_57_0/boost/phoenix/scope/preprocessed/lambda_40.hpp
43           /boost_1_57_0/boost/phoenix/core/detail/preprocessed/function_eval_40.hpp
43           /boost_1_57_0/boost/phoenix/core/detail/preprocessed/function_eval_40.hpp
37           /boost_1_57_0/boost/phoenix/scope/detail/preprocessed/make_locals_50.hpp
36           /boost_1_57_0/boost/phoenix/scope/detail/preprocessed/make_locals_50.hpp
30           /boost_1_57_0/boost/phoenix/core/detail/preprocessed/actor_operator_40.hpp
29           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_vector50.hpp
29           /boost_1_57_0/boost/phoenix/core/detail/preprocessed/actor_operator_40.hpp
29           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_vector50.hpp
29           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_vector50.hpp
28           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_vector50.hpp
27           /boost_1_57_0/boost/phoenix/scope/preprocessed/lambda_30.hpp
27           /boost_1_57_0/boost/fusion/container/map/detail/cpp03/preprocessed/as_map50.hpp
27           /boost_1_57_0/boost/fusion/container/map/detail/cpp03/preprocessed/as_map50.hpp
27           /boost_1_57_0/boost/fusion/container/map/detail/cpp03/preprocessed/as_map50.hpp
26           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/map_tie40.hpp
26           /boost_1_57_0/boost/phoenix/scope/preprocessed/lambda_30.hpp
26           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/map_tie40.hpp
26           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/map_tie40.hpp
25           /boost_1_57_0/boost/phoenix/core/detail/preprocessed/function_eval_30.hpp
25           /boost_1_57_0/boost/math/octonion.hpp
25           /boost_1_57_0/boost/phoenix/core/detail/preprocessed/function_eval_30.hpp
25           /boost_1_57_0/boost/math/octonion.hpp
25           /boost_1_57_0/boost/math/octonion.hpp
24           /boost_1_57_0/boost/fusion/container/set/detail/preprocessed/as_set50.hpp
24           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque50.hpp
24           /boost_1_57_0/boost/fusion/container/set/detail/preprocessed/as_set50.hpp
24           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque50.hpp
24           /boost_1_57_0/boost/fusion/container/set/detail/preprocessed/as_set50.hpp
24           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque50.hpp
24           /boost_1_57_0/boost/fusion/container/map/detail/cpp03/preprocessed/as_map50.hpp
24           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque50.hpp
23           /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/catch_push_back_40.hpp
22           /boost_1_57_0/boost/numeric/ublas/matrix_expression.hpp
22           /boost_1_57_0/boost/numeric/ublas/matrix.hpp
22           /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/catch_push_back_40.hpp
22           /boost_1_57_0/boost/fusion/container/set/detail/preprocessed/as_set50.hpp
21           /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/try_catch_eval_30.hpp
21           /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/try_catch_eval_30.hpp
21           /boost_1_57_0/boost/numeric/ublas/matrix.hpp
20           /boost_1_57_0/boost/phoenix/core/preprocessed/expression_40.hpp
20           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/map_tie40.hpp
19           /boost_1_57_0/boost/phoenix/object/detail/preprocessed/new_eval_50.hpp
19           /boost_1_57_0/boost/phoenix/object/detail/preprocessed/new_eval_50.hpp
19           /boost_1_57_0/boost/phoenix/core/preprocessed/expression_40.hpp
19           /boost_1_57_0/boost/numeric/ublas/matrix_expression.hpp
19           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector20.hpp
19           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_list40.hpp
19           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector20.hpp
19           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_list40.hpp
19           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector20.hpp
19           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_list40.hpp
18           /boost_1_57_0/boost/phoenix/core/detail/preprocessed/phx2_result_50.hpp
18           /boost_1_57_0/boost/fusion/tuple/detail/preprocessed/make_tuple50.hpp
18           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector20.hpp
18           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_set40.hpp
18           /boost_1_57_0/boost/fusion/tuple/detail/preprocessed/make_tuple50.hpp
18           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_set40.hpp
18           /boost_1_57_0/boost/fusion/tuple/detail/preprocessed/make_tuple50.hpp
18           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_set40.hpp
17           /boost_1_57_0/boost/fusion/container/set/detail/preprocessed/as_set40.hpp
17           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_list40.hpp
17           /boost_1_57_0/boost/phoenix/core/detail/preprocessed/phx2_result_50.hpp
17           /boost_1_57_0/boost/fusion/container/set/detail/preprocessed/as_set40.hpp
17           /boost_1_57_0/boost/fusion/container/set/detail/preprocessed/as_set40.hpp
17           /boost_1_57_0/boost/fusion/tuple/detail/preprocessed/make_tuple50.hpp
17           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_set40.hpp
16           /boost_1_57_0/boost/graph/distributed/adjacency_list.hpp
16           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque40.hpp
16           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/as_deque40.hpp
16           /boost_1_57_0/boost/circular_buffer/base.hpp
16           /boost_1_57_0/boost/graph/distributed/adjacency_list.hpp
16           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque40.hpp
16           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/as_deque40.hpp
16           /boost_1_57_0/boost/circular_buffer/base.hpp
16           /boost_1_57_0/boost/graph/distributed/adjacency_list.hpp
16           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque40.hpp
16           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/as_deque40.hpp
16           /boost_1_57_0/boost/circular_buffer/base.hpp
16           /boost_1_57_0/boost/graph/distributed/adjacency_list.hpp
16           /boost_1_57_0/boost/fusion/container/set/detail/preprocessed/as_set40.hpp
16           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque40.hpp
16           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/as_deque40.hpp
16           /boost_1_57_0/boost/circular_buffer/base.hpp
15           /boost_1_57_0/boost/polygon/detail/scan_arbitrary.hpp
15           /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/catch_push_back_30.hpp
15           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/map_tie30.hpp
15           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque_keyed_values40.hpp
15           /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/catch_push_back_30.hpp
15           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/map_tie30.hpp
15           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque_keyed_values40.hpp
15           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/map_tie30.hpp
15           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/map_tie30.hpp
15           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque_keyed_values40.hpp

==========Directory open stats========

COUNT:  	  DIRECTORY NAME
7            /boost_1_57_0/boost
7            /boost_1_57_0/boost
5            /boost_1_57_0/boost
5            /boost_1_57_0/boost
5            /boost_1_57_0/boost
3            /boost_1_57_0
3            /boost_1_57_0/boost/fusion
3            /boost_1_57_0/boost/local_function
3            /boost_1_57_0
3            /boost_1_57_0/boost/fusion
3            /boost_1_57_0/boost/local_function
3            /boost_1_57_0
3            /boost_1_57_0/boost/fusion
3            /boost_1_57_0/boost/local_function
3            /boost_1_57_0
3            /boost_1_57_0/boost/fusion
3            /boost_1_57_0/boost/local_function
3            /boost_1_57_0
3            /boost_1_57_0/boost/fusion
3            /boost_1_57_0/boost/local_function
2            /boost_1_57_0/boost/phoenix
2            /boost_1_57_0/boost/phoenix/stl
2            /boost_1_57_0/boost/phoenix/stl/algorithm
2            /boost_1_57_0/boost/phoenix/stl/algorithm/detail
2            /boost_1_57_0/boost/phoenix/stl/container
2            /boost_1_57_0/boost/phoenix/stl/container/detail
2            /boost_1_57_0/boost/phoenix/statement
2            /boost_1_57_0/boost/phoenix/statement/detail
2            /boost_1_57_0/boost/phoenix/statement/detail/preprocessed
2            /boost_1_57_0/boost/phoenix/scope
2            /boost_1_57_0/boost/phoenix/scope/detail
2            /boost_1_57_0/boost/phoenix/scope/detail/preprocessed
2            /boost_1_57_0/boost/phoenix/scope/preprocessed
2            /boost_1_57_0/boost/phoenix/function
2            /boost_1_57_0/boost/phoenix/function/detail
2            /boost_1_57_0/boost/phoenix/function/detail/preprocessed
2            /boost_1_57_0/boost/phoenix/core
2            /boost_1_57_0/boost/phoenix/core/preprocessed
2            /boost_1_57_0/boost/phoenix/core/detail
2            /boost_1_57_0/boost/phoenix/core/detail/preprocessed
2            /boost_1_57_0/boost/phoenix/bind
2            /boost_1_57_0/boost/phoenix/bind/preprocessed
2            /boost_1_57_0/boost/phoenix/bind/detail
2            /boost_1_57_0/boost/phoenix/bind/detail/preprocessed
2            /boost_1_57_0/boost/phoenix/support
2            /boost_1_57_0/boost/phoenix/support/preprocessed
2            /boost_1_57_0/boost/phoenix/support/preprocessor
2            /boost_1_57_0/boost/phoenix/support/detail
2            /boost_1_57_0/boost/phoenix/operator
2            /boost_1_57_0/boost/phoenix/operator/detail
2            /boost_1_57_0/boost/phoenix/operator/detail/preprocessed
2            /boost_1_57_0/boost/phoenix/operator/preprocessed
2            /boost_1_57_0/boost/phoenix/object
2            /boost_1_57_0/boost/phoenix/object/detail
2            /boost_1_57_0/boost/phoenix/object/detail/preprocessed
2            /boost_1_57_0/boost/phoenix/fusion
2            /boost_1_57_0/boost/multi_index
2            /boost_1_57_0/boost/multi_index/detail
2            /boost_1_57_0/boost/move
2            /boost_1_57_0/boost/move/detail
2            /boost_1_57_0/boost/logic
2            /boost_1_57_0/boost/log
2            /boost_1_57_0/boost/log/support
2            /boost_1_57_0/boost/log/sinks
2            /boost_1_57_0/boost/log/expressions
2            /boost_1_57_0/boost/log/expressions/predicates
2            /boost_1_57_0/boost/log/expressions/formatters
2            /boost_1_57_0/boost/log/utility
2            /boost_1_57_0/boost/log/utility/type_dispatch
2            /boost_1_57_0/boost/log/utility/setup
2            /boost_1_57_0/boost/log/utility/manipulators
2            /boost_1_57_0/boost/log/utility/functional
2            /boost_1_57_0/boost/log/sources
2            /boost_1_57_0/boost/log/keywords
2            /boost_1_57_0/boost/log/detail
2            /boost_1_57_0/boost/log/core
2            /boost_1_57_0/boost/log/attributes
2            /boost_1_57_0/boost/lockfree
2            /boost_1_57_0/boost/lockfree/detail
2            /boost_1_57_0/boost/locale
2            /boost_1_57_0/boost/locale/boundary
2            /boost_1_57_0/boost/lambda
2            /boost_1_57_0/boost/lambda/detail
2            /boost_1_57_0/boost/iostreams
2            /boost_1_57_0/boost/iostreams/filter
2            /boost_1_57_0/boost/iostreams/device
2            /boost_1_57_0/boost/iostreams/detail
2            /boost_1_57_0/boost/iostreams/detail/streambuf
2            /boost_1_57_0/boost/iostreams/detail/adapter
2            /boost_1_57_0/boost/iostreams/detail/vc6
2            /boost_1_57_0/boost/iostreams/detail/config
2            /boost_1_57_0/boost/iostreams/detail/broken_overload_resolution
2            /boost_1_57_0/boost/intrusive
2            /boost_1_57_0/boost/intrusive/detail
2            /boost_1_57_0/boost/heap
2            /boost_1_57_0/boost/heap/detail
2            /boost_1_57_0/boost/graph
2            /boost_1_57_0/boost/graph/planar_detail
2            /boost_1_57_0/boost/graph/detail
2            /boost_1_57_0/boost/graph/property_maps

========Directory readdirp Stats=======

COUNT:  	  DIRECTORY NAME
8            /boost_1_57_0/boost
8            /boost_1_57_0/boost
7            /boost_1_57_0/boost/fusion/include
7            /boost_1_57_0/boost/mpl
7            /boost_1_57_0/boost/asio/detail
7            /boost_1_57_0/boost/fusion/include
7            /boost_1_57_0/boost/mpl
7            /boost_1_57_0/boost/asio/detail
7            /gfstest
7            /boost_1_57_0/boost
7            /boost_1_57_0/boost/fusion/include
7            /boost_1_57_0/boost/asio/detail
7            /boost_1_57_0/boost
7            /boost_1_57_0/boost/fusion/include
7            /boost_1_57_0/boost/asio/detail
7            /boost_1_57_0/boost/fusion/include
7            /boost_1_57_0/boost/asio/detail
6            /boost_1_57_0/boost/graph
6            /boost_1_57_0/boost/graph
6            /boost_1_57_0/boost/graph
6            /boost_1_57_0/boost/graph
6            /boost_1_57_0/boost
6            /boost_1_57_0/boost/graph
5            /boost_1_57_0/boost/mpl/aux_
5            /boost_1_57_0/boost/math/tools/detail
5            /boost_1_57_0/boost/detail
5            /boost_1_57_0/boost/asio
5            /boost_1_57_0/boost/mpl/aux_
5            /boost_1_57_0/boost/math/tools/detail
5            /boost_1_57_0/boost/detail
5            /boost_1_57_0/boost/asio
5            /gfstest/6dir
5            /gfstest/5dir
5            /gfstest/1dir
5            /gfstest/4dir
5            /gfstest/3dir
5            /gfstest/0dir
5            /boost_1_57_0/boost/mpl/aux_
5            /boost_1_57_0/boost/math/tools/detail
5            /boost_1_57_0/boost/detail
5            /boost_1_57_0/boost/asio
5            /boost_1_57_0/boost/detail
5            /boost_1_57_0/boost/asio
5            /boost_1_57_0/boost/detail
5            /boost_1_57_0/boost/asio
4            /boost_1_57_0/boost/phoenix/stl/container
4            /boost_1_57_0/boost/phoenix/scope/detail
4            /boost_1_57_0/boost/phoenix/function/detail
4            /boost_1_57_0/boost/phoenix/bind
4            /boost_1_57_0/boost/phoenix/bind/detail
4            /boost_1_57_0/boost/phoenix/operator/detail
4            /boost_1_57_0/boost/multi_index/detail
4            /boost_1_57_0/boost/log/detail
4            /boost_1_57_0/boost/iostreams
4            /boost_1_57_0/boost/iostreams/detail
4            /boost_1_57_0/boost/intrusive/detail
4            /boost_1_57_0/boost/graph/distributed
4            /boost_1_57_0/boost/gil
4            /boost_1_57_0/boost/geometry
4            /boost_1_57_0/boost/geometry/policies
4            /boost_1_57_0/boost/geometry/algorithms/detail
4            /boost_1_57_0/boost/geometry/algorithms/detail/overlay
4            /boost_1_57_0/boost/geometry/multi
4            /boost_1_57_0/boost/geometry/multi/algorithms/detail
4            /boost_1_57_0/boost/geometry/index/detail/rtree
4            /boost_1_57_0/boost/fusion/sequence
4            /boost_1_57_0/boost/fusion/functional
4            /boost_1_57_0/boost/fusion/container/generation/detail
4            /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed
4            /boost_1_57_0/boost/fusion/adapted/struct
4            /boost_1_57_0/boost/date_time
4            /boost_1_57_0/boost/chrono
4            /boost_1_57_0/boost/preprocessor/array
4            /boost_1_57_0/boost/numeric/odeint
4            /boost_1_57_0/boost/numeric/ublas
4            /boost_1_57_0/boost/mpl/vector/aux_/preprocessed
4            /boost_1_57_0/boost/mpl/aux_/preprocessed
4            /boost_1_57_0/boost/mpl/aux_/preprocessed/plain
4            /boost_1_57_0/boost/mpl/aux_/preprocessed/msvc60
4            /boost_1_57_0/boost/mpl/aux_/preprocessed/no_ttp
4            /boost_1_57_0/boost/mpl/aux_/preprocessed/gcc
4            /boost_1_57_0/boost/mpl/aux_/preprocessed/bcc551
4            /boost_1_57_0/boost/mpl/map/aux_/preprocessed
4            /boost_1_57_0/boost/math/special_functions
4            /boost_1_57_0/boost/math/special_functions/detail
4            /boost_1_57_0/boost/local_function/aux_
4            /boost_1_57_0/boost/local_function/aux_/preprocessor/traits/decl_sign_
4            /boost_1_57_0/boost/interprocess/sync
4            /boost_1_57_0/boost/interprocess/detail
4            /boost_1_57_0/boost/functional/overloaded_function
4            /boost_1_57_0/boost/function_types/detail
4            /boost_1_57_0/boost/coroutine
4            /boost_1_57_0/boost/asio/local
4            /boost_1_57_0/boost/asio/detail/impl
4            /boost_1_57_0/boost/archive
4            /boost_1_57_0/boost/align
4            /boost_1_57_0/boost/accumulators/statistics
4            /boost_1_57_0/boost/accumulators/framework
4            /boost_1_57_0/boost/phoenix/stl/container
4            /boost_1_57_0/boost/phoenix/scope/detail

========Read Throughput File Stats=====

TIMESTAMP 			 THROUGHPUT(KBPS)	FILE NAME
1970-01-01 00:00:04.000000 	 6898.00     	  /myfile.txt

======Write Throughput File Stats======

TIMESTAMP 			 THROUGHPUT(KBPS)	FILE NAME
1970-01-01 00:30:41.000000 	 4681.00     	  /myfile.txt
1970-01-01 00:01:37.000000 	 1059.00     	  /boost_1_57_0/boost/asio/detail/impl/socket_ops.ipp
1970-01-01 00:02:29.000000 	 1024.00     	  /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque_keyed_values40.hpp
1970-01-01 00:36:01.000000 	 890.00      	  /boost_1_57_0/boost/polygon/detail/polygon_45_formation.hpp
1970-01-01 00:00:33.000000 	 890.00      	  /boost_1_57_0/boost/phoenix/scope/preprocessed/lambda_40.hpp
1970-01-01 00:02:13.000000 	 890.00      	  /boost_1_57_0/boost/phoenix/scope/detail/preprocessed/dynamic_40.hpp
1970-01-01 00:01:41.000000 	 890.00      	  /boost_1_57_0/boost/intrusive/rbtree.hpp
1970-01-01 00:15:13.000000 	 853.00      	  /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/try_catch_eval_50.hpp
1970-01-01 00:02:25.000000 	 853.00      	  /boost_1_57_0/boost/phoenix/scope/detail/preprocessed/make_locals_50.hpp
1970-01-01 00:01:37.000000 	 853.00      	  /boost_1_57_0/boost/math/bindings/mpreal.hpp
1970-01-01 00:25:37.000000 	 853.00      	  /boost_1_57_0/boost/intrusive/unordered_set.hpp
1970-01-01 00:02:25.000000 	 853.00      	  /boost_1_57_0/boost/asio/basic_socket.hpp
1970-01-01 00:05:53.000000 	 819.00      	  /boost_1_57_0/boost/phoenix/stl/algorithm/transformation.hpp
1970-01-01 01:59:29.000000 	 819.00      	  /boost_1_57_0/boost/mpl/vector/aux_/preprocessed/no_ctps/vector30.hpp
1970-01-01 00:01:37.000000 	 819.00      	  /boost_1_57_0/boost/asio/basic_seq_packet_socket.hpp
1970-01-01 00:01:41.000000 	 816.00      	  /boost_1_57_0/boost/math/bindings/e_float.hpp
1970-01-01 00:01:57.000000 	 808.00      	  /boost_1_57_0/boost/interprocess/streams/vectorstream.hpp
1970-01-01 00:02:29.000000 	 787.00      	  /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/catch_push_back_10.hpp
1970-01-01 01:15:13.000000 	 787.00      	  /boost_1_57_0/boost/math/cstdfloat/cstdfloat_cmath.hpp
1970-01-01 00:19:29.000000 	 787.00      	  /boost_1_57_0/boost/lambda/detail/bind_functions.hpp
1970-01-01 00:09:53.000000 	 787.00      	  /boost_1_57_0/boost/asio/basic_socket_acceptor.hpp
1970-01-01 00:52:33.000000 	 771.00      	  /boost_1_57_0/boost/math/special_functions/lanczos.hpp
1970-01-01 00:02:29.000000 	 758.00      	  /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/catch_push_back_40.hpp
1970-01-01 00:01:57.000000 	 758.00      	  /boost_1_57_0/boost/phoenix/scope/preprocessed/lambda_30.hpp
1970-01-01 01:57:05.000000 	 758.00      	  /boost_1_57_0/boost/mpl/vector/aux_/preprocessed/no_ctps/vector40.hpp
1970-01-01 00:48:01.000000 	 758.00      	  /boost_1_57_0/boost/intrusive/sg_set.hpp
1970-01-01 00:23:29.000000 	 749.00      	  /boost_1_57_0/boost/polygon/detail/boolean_op.hpp
1970-01-01 00:01:57.000000 	 737.00      	  /boost_1_57_0/boost/math/concepts/distributions.hpp
1970-01-01 00:02:13.000000 	 731.00      	  /boost_1_57_0/boost/phoenix/core/detail/preprocessed/phx2_result_50.hpp
1970-01-01 00:00:33.000000 	 731.00      	  /boost_1_57_0/boost/interprocess/detail/intersegment_ptr.hpp
1970-01-01 00:02:41.000000 	 731.00      	  /boost_1_57_0/boost/function/function_template.hpp
1970-01-01 00:14:09.000000 	 716.00      	  /boost_1_57_0/boost/interprocess/allocators/node_allocator.hpp
1970-01-01 00:02:29.000000 	 715.00      	  /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_list30.hpp
1970-01-01 01:20:17.000000 	 707.00      	  /boost_1_57_0/boost/polygon/voronoi_diagram.hpp
1970-01-01 00:02:13.000000 	 706.00      	  /boost_1_57_0/boost/phoenix/object/detail/preprocessed/new_40.hpp
1970-01-01 00:01:57.000000 	 706.00      	  /boost_1_57_0/boost/numeric/ublas/matrix_expression.hpp
1970-01-01 00:02:13.000000 	 692.00      	  /boost_1_57_0/boost/lambda/detail/operator_return_type_traits.hpp
1970-01-01 00:02:41.000000 	 691.00      	  /boost_1_57_0/boost/numeric/ublas/vector_proxy.hpp
1970-01-01 00:05:37.000000 	 682.00      	  /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/try_catch_eval_30.hpp
1970-01-01 00:48:17.000000 	 682.00      	  /boost_1_57_0/boost/phoenix/core/preprocessed/actor_40.hpp
1970-01-01 00:01:41.000000 	 682.00      	  /boost_1_57_0/boost/math/quaternion.hpp
1970-01-01 00:46:25.000000 	 679.00      	  /boost_1_57_0/boost/icl/detail/interval_set_algo.hpp
1970-01-01 00:02:29.000000 	 660.00      	  /boost_1_57_0/boost/phoenix/function/detail/preprocessed/function_operator_20.hpp
1970-01-01 00:13:05.000000 	 660.00      	  /boost_1_57_0/boost/numeric/ublas/matrix.hpp
1970-01-01 00:01:37.000000 	 660.00      	  /boost_1_57_0/boost/intrusive/bstree.hpp
1970-01-01 00:02:29.000000 	 653.00      	  /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_list40.hpp
1970-01-01 00:01:57.000000 	 653.00      	  /boost_1_57_0/boost/interprocess/detail/managed_memory_impl.hpp
1970-01-01 00:01:57.000000 	 648.00      	  /boost_1_57_0/boost/lexical_cast/detail/converter_lexical.hpp
1970-01-01 00:24:49.000000 	 647.00      	  /boost_1_57_0/boost/mpl/vector/aux_/preprocessed/plain/vector20.hpp
1970-01-01 00:06:25.000000 	 640.00      	  /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/try_catch_eval_50.hpp
1970-01-01 00:01:25.000000 	 640.00      	  /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector20.hpp
1970-01-01 00:00:33.000000 	 640.00      	  /boost_1_57_0/boost/fusion/container/map/detail/cpp03/preprocessed/as_map50.hpp
1970-01-01 00:10:09.000000 	 640.00      	  /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/map_tie40.hpp
1970-01-01 00:02:29.000000 	 640.00      	  /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_vector50.hpp
1970-01-01 00:02:29.000000 	 640.00      	  /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque50.hpp
1970-01-01 00:01:57.000000 	 640.00      	  /boost_1_57_0/boost/polygon/detail/scan_arbitrary.hpp
1970-01-01 00:02:29.000000 	 640.00      	  /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/catch_push_back_30.hpp
1970-01-01 00:01:21.000000 	 640.00      	  /boost_1_57_0/boost/phoenix/core/detail/preprocessed/function_eval_50.hpp
1970-01-01 02:05:53.000000 	 640.00      	  /boost_1_57_0/boost/numeric/ublas/vector_sparse.hpp
1970-01-01 00:01:57.000000 	 640.00      	  /boost_1_57_0/boost/intrusive/bstree_algorithms.hpp
1970-01-01 00:09:05.000000 	 640.00      	  /boost_1_57_0/boost/container/flat_map.hpp
1970-01-01 00:03:33.000000 	 640.00      	  /boost_1_57_0/boost/math/special_functions/prime.hpp
1970-01-01 00:01:25.000000 	 640.00      	  /boost_1_57_0/boost/circular_buffer/base.hpp
1970-01-02 12:45:05.000000 	 631.00      	  /boost_1_57_0/boost/algorithm/string/erase.hpp
1970-01-01 02:27:29.000000 	 628.00      	  /boost_1_57_0/boost/graph/stanford_graph.hpp
1970-01-01 00:33:53.000000 	 627.00      	  /boost_1_57_0/boost/math/bindings/rr.hpp
1970-01-01 00:17:21.000000 	 620.00      	  /boost_1_57_0/boost/dynamic_bitset/dynamic_bitset.hpp
1970-01-01 00:02:13.000000 	 620.00      	  /boost_1_57_0/boost/phoenix/core/preprocessed/actor_50.hpp
1970-01-01 00:01:25.000000 	 609.00      	  /boost_1_57_0/boost/log/detail/light_function_pp.hpp
1970-01-01 00:47:45.000000 	 604.00      	  /boost_1_57_0/boost/interprocess/smart_ptr/shared_ptr.hpp
1970-01-01 06:21:05.000000 	 602.00      	  /boost_1_57_0/boost/polygon/voronoi_diagram.hpp
1970-01-01 00:01:57.000000 	 602.00      	  /boost_1_57_0/boost/polygon/polygon_set_data.hpp
1970-01-01 00:22:41.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/scope/detail/preprocessed/make_locals_50.hpp
1970-01-01 00:01:21.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/core/detail/preprocessed/phx2_result_50.hpp
1970-01-01 00:17:37.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/core/detail/preprocessed/function_eval_50.hpp
1970-01-01 00:01:25.000000 	 602.00      	  /boost_1_57_0/boost/mpl/vector/aux_/preprocessed/no_ctps/vector40.hpp
1970-01-01 00:01:25.000000 	 602.00      	  /boost_1_57_0/boost/mpl/vector/aux_/preprocessed/no_ctps/vector30.hpp
1970-01-01 00:54:09.000000 	 602.00      	  /boost_1_57_0/boost/math/octonion.hpp
1970-01-01 00:12:01.000000 	 602.00      	  /boost_1_57_0/boost/intrusive/slist.hpp
1970-01-01 00:01:37.000000 	 602.00      	  /boost_1_57_0/boost/intrusive/list.hpp
1970-01-01 00:05:53.000000 	 602.00      	  /boost_1_57_0/boost/fusion/container/set/detail/preprocessed/as_set40.hpp
1970-01-01 00:10:09.000000 	 602.00      	  /boost_1_57_0/boost/fusion/container/set/detail/preprocessed/as_set30.hpp
1970-01-01 00:01:25.000000 	 602.00      	  /boost_1_57_0/boost/fusion/container/map/detail/cpp03/preprocessed/map50.hpp
1970-01-01 00:00:49.000000 	 602.00      	  /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/vector_tie50.hpp
1970-01-01 00:02:29.000000 	 602.00      	  /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/vector_tie30.hpp
1970-01-01 00:01:25.000000 	 602.00      	  /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_vector30.hpp
1970-01-01 00:03:33.000000 	 602.00      	  /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_list30.hpp
1970-01-01 00:02:29.000000 	 602.00      	  /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_deque20.hpp
1970-01-01 00:07:45.000000 	 602.00      	  /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/list_tie40.hpp
1970-01-01 00:05:05.000000 	 602.00      	  /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/deque_tie50.hpp
1970-01-01 00:02:29.000000 	 602.00      	  /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque40.hpp
1970-01-01 00:01:05.000000 	 602.00      	  /boost_1_57_0/boost/asio/basic_socket.hpp
1970-01-01 00:01:57.000000 	 602.00      	  /boost_1_57_0/boost/polygon/polygon_45_set_concept.hpp
1970-01-01 00:09:05.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/support/preprocessed/vector_30.hpp
1970-01-01 00:02:25.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/support/preprocessed/vector_20.hpp
1970-01-01 00:02:29.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/try_catch_expression_50.hpp
1970-01-01 00:02:13.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/try_catch_eval_20.hpp
1970-01-01 00:13:21.000000 	 602.00      	  /boost_1_57_0/boost/math/special_functions/prime.hpp
1970-01-01 07:34:25.000000 	 602.00      	  /boost_1_57_0/boost/math/octonion.hpp
1970-01-01 00:01:57.000000 	 602.00      	  /boost_1_57_0/boost/math/bindings/detail/big_lanczos.hpp

=== Interval 2 stats ===
      Duration : 200 secs
     BytesRead : 262144
  BytesWritten : 34406743

Block Size   :               2B+               4B+               8B+
Read Count   :                 0                 0                 0
Write Count  :                 1                 1                 3

Block Size   :              16B+              32B+              64B+
Read Count   :                 0                 0                 0
Write Count  :                 7                13                23

Block Size   :             128B+             256B+             512B+
Read Count   :                 0                 0                 0
Write Count  :                37               143               827

Block Size   :            1024B+            2048B+            4096B+
Read Count   :                 0                 0                 0
Write Count  :              1131              1153              1065

Block Size   :            8192B+           16384B+          131072B+
Read Count   :                 0                 0                 2
Write Count  :              2236                 7                 0

Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
---           ----------    -----------    -----------    -----------
STAT                   2        9.50 us        9.00 us       10.00 us
MKDIR                585    20055.67 us       97.00 us    92578.00 us
UNLINK              3417      142.80 us       45.00 us    21958.00 us
RMDIR                585       55.12 us       45.00 us      102.00 us
READ                   2       61.00 us       49.00 us       73.00 us
WRITE               6647       26.64 us       10.00 us      121.00 us
STATFS               182       15.53 us       10.00 us       48.00 us
FLUSH               3417        9.72 us        5.00 us       51.00 us
SETXATTR             586       31.29 us       17.00 us      139.00 us
OPENDIR             1178       16.90 us       13.00 us       46.00 us
CREATE              3417    22210.28 us       69.00 us  4369403.00 us
FSTAT                168       24.99 us       17.00 us       64.00 us
LOOKUP             12339       28.99 us       17.00 us      171.00 us
SETATTR            11991       30.89 us       13.00 us      166.00 us
READDIRP            1836      170.48 us       11.00 us     1195.00 us
FORGET              4002           0 us           0 us           0 us
RELEASE             3417           0 us           0 us           0 us
RELEASEDIR          1178           0 us           0 us           0 us
------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----

=== Cumulative stats ===
      Duration : 80429 secs
     BytesRead : 5700283543640
  BytesWritten : 100050626

Block Size   :               1B+               2B+               4B+
Read Count   :                 1                 2                 7
Write Count  :                 0                 0                 6

Block Size   :               8B+              16B+              32B+
Read Count   :                23                16                91
Write Count  :                 8                15                57

Block Size   :              64B+             128B+             256B+
Read Count   :               589               206               760
Write Count  :                71               175               382

Block Size   :             512B+            1024B+            2048B+
Read Count   :               640              1345              2573
Write Count  :              2468              3248              3521

Block Size   :            4096B+            8192B+           16384B+
Read Count   :              5477             10514             21796
Write Count  :              3179              5481               365

Block Size   :           32768B+           65536B+          131072B+
Read Count   :             38283            111152          43388707
Write Count  :                39                 0                 0

Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
---           ----------    -----------    -----------    -----------
STAT                  10       10.70 us        9.00 us       12.00 us
MKDIR               9628    27069.69 us       91.00 us  1633942.00 us
UNLINK              6967    15847.52 us       38.00 us   173531.00 us
RMDIR               9628      235.50 us       43.00 us    64920.00 us
READ                   2      233.50 us      201.00 us      266.00 us
WRITE              11570       31.49 us       10.00 us      163.00 us
STATFS               949       28.75 us        9.00 us    11060.00 us
FLUSH               8065       10.99 us        5.00 us      111.00 us
SETXATTR            9629       35.29 us       16.00 us      138.00 us
OPENDIR            19285       17.74 us       12.00 us      531.00 us
CREATE              6967    26181.05 us       67.00 us   528654.00 us
FSTAT                  4       79.50 us       67.00 us       85.00 us
LOOKUP             48995      159.26 us       17.00 us    84036.00 us
SETATTR            21666       31.95 us       12.00 us      183.00 us
READDIRP           29493      938.44 us       10.00 us   208823.00 us
FORGET             21676           0 us           0 us           0 us
RELEASE            11253           0 us           0 us           0 us
RELEASEDIR         67856           0 us           0 us           0 us
------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----

Current open fd's: 0 Max open fd's: 1 time 2015-02-09 22:44:58.272315

==========Open File Stats========

COUNT:  	  FILE NAME

==========Read File Stats========

COUNT:  	  FILE NAME

==========Write File Stats========

COUNT:  	  FILE NAME
118          /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/switch_50.hpp
77           /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/switch_40.hpp
69           /boost_1_57_0/boost/phoenix/scope/preprocessed/lambda_50.hpp
48           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector50.hpp
48           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector50.hpp
46           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector50.hpp
45           /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/switch_30.hpp
45           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_map50.hpp
45           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_map50.hpp
45           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_map50.hpp
44           /boost_1_57_0/boost/phoenix/core/detail/preprocessed/actor_operator_50.hpp
40           /boost_1_57_0/boost/phoenix/bind/preprocessed/bind_member_function_50.hpp
39           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector40.hpp
39           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/map_tie50.hpp
39           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector40.hpp
39           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/map_tie50.hpp
38           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/map_tie50.hpp
37           /boost_1_57_0/boost/phoenix/operator/preprocessed/member_50.hpp
37           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector40.hpp
36           /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/try_catch_eval_40.hpp
33           /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/catch_push_back_50.hpp
29           /boost_1_57_0/boost/phoenix/core/preprocessed/expression_50.hpp
29           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector30.hpp
29           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_map40.hpp
29           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector30.hpp
29           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_map40.hpp
29           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_map40.hpp
28           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_set50.hpp
28           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_list50.hpp
28           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_deque50.hpp
28           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_set50.hpp
28           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_list50.hpp
28           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_deque50.hpp
28           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector30.hpp
28           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_set50.hpp
28           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_list50.hpp
28           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_deque50.hpp
27           /boost_1_57_0/boost/phoenix/bind/preprocessed/bind_member_function_40.hpp
25           /boost_1_57_0/boost/phoenix/scope/detail/preprocessed/make_locals_40.hpp
25           /boost_1_57_0/boost/phoenix/operator/preprocessed/member_40.hpp
24           /boost_1_57_0/boost/numeric/ublas/matrix_sparse.hpp
24           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/as_vector50.hpp
24           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/as_deque50.hpp
24           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/as_vector50.hpp
24           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/as_deque50.hpp
23           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/as_vector50.hpp
23           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/as_deque50.hpp
22           /boost_1_57_0/boost/phoenix/operator/detail/preprocessed/mem_fun_ptr_gen_50.hpp
22           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector50.hpp
21           /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/switch_20.hpp
21           /boost_1_57_0/boost/phoenix/scope/detail/preprocessed/dynamic_50.hpp
21           /boost_1_57_0/boost/numeric/ublas/matrix_proxy.hpp
21           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque_keyed_values50.hpp
21           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque_keyed_values50.hpp
21           /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque_keyed_values50.hpp
20           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/map_tie50.hpp
20           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_map50.hpp
20           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_map40.hpp
19           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_deque40.hpp
19           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_deque40.hpp
19           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_deque40.hpp
18           /boost_1_57_0/boost/phoenix/core/detail/preprocessed/actor_operator_30.hpp
18           /boost_1_57_0/boost/fusion/container/map/detail/cpp03/preprocessed/as_map40.hpp
18           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_vector40.hpp
18           /boost_1_57_0/boost/fusion/container/map/detail/cpp03/preprocessed/as_map40.hpp
18           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_vector40.hpp
18           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_vector40.hpp
17           /boost_1_57_0/boost/phoenix/core/detail/preprocessed/call_50.hpp
17           /boost_1_57_0/boost/phoenix/bind/preprocessed/bind_member_function_30.hpp
17           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/as_vector40.hpp
17           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_map30.hpp
17           /boost_1_57_0/boost/fusion/algorithm/transformation/detail/preprocessed/zip50.hpp
17           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/as_vector40.hpp
17           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_map30.hpp
17           /boost_1_57_0/boost/fusion/algorithm/transformation/detail/preprocessed/zip50.hpp
17           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/as_vector40.hpp
17           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_map30.hpp
17           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_deque50.hpp
16           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector40.hpp
15           /boost_1_57_0/boost/phoenix/scope/detail/preprocessed/make_locals_30.hpp
15           /boost_1_57_0/boost/phoenix/operator/preprocessed/member_30.hpp
15           /boost_1_57_0/boost/phoenix/operator/detail/preprocessed/mem_fun_ptr_gen_40.hpp
15           /boost_1_57_0/boost/msm/front/euml/container.hpp
15           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vvector50.hpp
15           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vvector50.hpp
15           /boost_1_57_0/boost/fusion/container/map/detail/cpp03/preprocessed/as_map40.hpp
15           /boost_1_57_0/boost/fusion/algorithm/transformation/detail/preprocessed/zip50.hpp
15           /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_list50.hpp
14           /boost_1_57_0/boost/polygon/detail/polygon_arbitrary_formation.hpp
14           /boost_1_57_0/boost/phoenix/object/detail/preprocessed/construct_eval_50.hpp
14           /boost_1_57_0/boost/intrusive/hashtable.hpp
14           /boost_1_57_0/boost/intrusive/hashtable.hpp
14           /boost_1_57_0/boost/fusion/tuple/detail/preprocessed/tuple50.hpp
14           /boost_1_57_0/boost/intrusive/hashtable.hpp
14           /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vvector50.hpp
13           /boost_1_57_0/boost/phoenix/support/preprocessed/vector_50.hpp
13           /boost_1_57_0/boost/phoenix/scope/preprocessed/lambda_20.hpp
13           /boost_1_57_0/boost/phoenix/function/detail/preprocessed/function_operator_50.hpp
13           /boost_1_57_0/boost/phoenix/core/detail/preprocessed/function_eval_20.hpp
13           /boost_1_57_0/boost/phoenix/bind/preprocessed/bind_function_50.hpp

==========Directory open stats========

COUNT:  	  DIRECTORY NAME
7            /boost_1_57_0/boost
5            /boost_1_57_0/boost
5            /boost_1_57_0/boost
5            /boost_1_57_0/boost
3            /boost_1_57_0
3            /boost_1_57_0/boost/fusion
3            /boost_1_57_0/boost/local_function
3            /boost_1_57_0
3            /boost_1_57_0/boost/fusion
3            /boost_1_57_0/boost/local_function
3            /boost_1_57_0
3            /boost_1_57_0/boost/fusion
3            /boost_1_57_0/boost/local_function
3            /boost_1_57_0
3            /boost_1_57_0/boost/fusion
3            /boost_1_57_0/boost/local_function
2            /boost_1_57_0/boost/phoenix
2            /boost_1_57_0/boost/phoenix/stl
2            /boost_1_57_0/boost/phoenix/stl/algorithm
2            /boost_1_57_0/boost/phoenix/stl/algorithm/detail
2            /boost_1_57_0/boost/phoenix/stl/container
2            /boost_1_57_0/boost/phoenix/stl/container/detail
2            /boost_1_57_0/boost/phoenix/statement
2            /boost_1_57_0/boost/phoenix/statement/detail
2            /boost_1_57_0/boost/phoenix/statement/detail/preprocessed
2            /boost_1_57_0/boost/phoenix/scope
2            /boost_1_57_0/boost/phoenix/scope/detail
2            /boost_1_57_0/boost/phoenix/scope/detail/preprocessed
2            /boost_1_57_0/boost/phoenix/scope/preprocessed
2            /boost_1_57_0/boost/phoenix/function
2            /boost_1_57_0/boost/phoenix/function/detail
2            /boost_1_57_0/boost/phoenix/function/detail/preprocessed
2            /boost_1_57_0/boost/phoenix/core
2            /boost_1_57_0/boost/phoenix/core/preprocessed
2            /boost_1_57_0/boost/phoenix/core/detail
2            /boost_1_57_0/boost/phoenix/core/detail/preprocessed
2            /boost_1_57_0/boost/phoenix/bind
2            /boost_1_57_0/boost/phoenix/bind/preprocessed
2            /boost_1_57_0/boost/phoenix/bind/detail
2            /boost_1_57_0/boost/phoenix/bind/detail/preprocessed
2            /boost_1_57_0/boost/phoenix/support
2            /boost_1_57_0/boost/phoenix/support/preprocessed
2            /boost_1_57_0/boost/phoenix/support/preprocessor
2            /boost_1_57_0/boost/phoenix/support/detail
2            /boost_1_57_0/boost/phoenix/operator
2            /boost_1_57_0/boost/phoenix/operator/detail
2            /boost_1_57_0/boost/phoenix/operator/detail/preprocessed
2            /boost_1_57_0/boost/phoenix/operator/preprocessed
2            /boost_1_57_0/boost/phoenix/object
2            /boost_1_57_0/boost/phoenix/object/detail
2            /boost_1_57_0/boost/phoenix/object/detail/preprocessed
2            /boost_1_57_0/boost/phoenix/fusion
2            /boost_1_57_0/boost/multi_index
2            /boost_1_57_0/boost/multi_index/detail
2            /boost_1_57_0/boost/move
2            /boost_1_57_0/boost/move/detail
2            /boost_1_57_0/boost/logic
2            /boost_1_57_0/boost/log
2            /boost_1_57_0/boost/log/support
2            /boost_1_57_0/boost/log/sinks
2            /boost_1_57_0/boost/log/expressions
2            /boost_1_57_0/boost/log/expressions/predicates
2            /boost_1_57_0/boost/log/expressions/formatters
2            /boost_1_57_0/boost/log/utility
2            /boost_1_57_0/boost/log/utility/type_dispatch
2            /boost_1_57_0/boost/log/utility/setup
2            /boost_1_57_0/boost/log/utility/manipulators
2            /boost_1_57_0/boost/log/utility/functional
2            /boost_1_57_0/boost/log/sources
2            /boost_1_57_0/boost/log/keywords
2            /boost_1_57_0/boost/log/detail
2            /boost_1_57_0/boost/log/core
2            /boost_1_57_0/boost/log/attributes
2            /boost_1_57_0/boost/lockfree
2            /boost_1_57_0/boost/lockfree/detail
2            /boost_1_57_0/boost/locale
2            /boost_1_57_0/boost/locale/boundary
2            /boost_1_57_0/boost/lambda
2            /boost_1_57_0/boost/lambda/detail
2            /boost_1_57_0/boost/iostreams
2            /boost_1_57_0/boost/iostreams/filter
2            /boost_1_57_0/boost/iostreams/device
2            /boost_1_57_0/boost/iostreams/detail
2            /boost_1_57_0/boost/iostreams/detail/streambuf
2            /boost_1_57_0/boost/iostreams/detail/adapter
2            /boost_1_57_0/boost/iostreams/detail/vc6
2            /boost_1_57_0/boost/iostreams/detail/config
2            /boost_1_57_0/boost/iostreams/detail/broken_overload_resolution
2            /boost_1_57_0/boost/intrusive
2            /boost_1_57_0/boost/intrusive/detail
2            /boost_1_57_0/boost/heap
2            /boost_1_57_0/boost/heap/detail
2            /boost_1_57_0/boost/graph
2            /boost_1_57_0/boost/graph/planar_detail
2            /boost_1_57_0/boost/graph/detail
2            /boost_1_57_0/boost/graph/property_maps
2            /boost_1_57_0/boost/graph/parallel
2            /boost_1_57_0/boost/graph/parallel/detail
2            /boost_1_57_0/boost/graph/distributed
2            /boost_1_57_0/boost/graph/distributed/detail

========Directory readdirp Stats=======

COUNT:  	  DIRECTORY NAME
8            /boost_1_57_0/boost
8            /gfstest
7            /boost_1_57_0/boost/graph
7            /boost_1_57_0/boost/fusion/include
7            /boost_1_57_0/boost/mpl
7            /boost_1_57_0/boost/asio/detail
7            /boost_1_57_0/boost
7            /boost_1_57_0/boost/graph
7            /boost_1_57_0/boost/fusion/include
7            /boost_1_57_0/boost/asio/detail
7            /boost_1_57_0/boost/graph
7            /boost_1_57_0/boost/fusion/include
7            /boost_1_57_0/boost/asio/detail
7            /boost_1_57_0/boost
7            /boost_1_57_0/boost/graph
7            /boost_1_57_0/boost/fusion/include
7            /boost_1_57_0/boost/asio/detail
6            /boost_1_57_0/boost
5            /boost_1_57_0/boost/geometry/algorithms/detail
5            /boost_1_57_0/boost/mpl/aux_
5            /boost_1_57_0/boost/math/tools/detail
5            /boost_1_57_0/boost/icl
5            /gfstest
5            /gfstest/10dir
5            /gfstest/9dir
5            /gfstest/7dir
5            /gfstest/2dir
5            /gfstest/8dir
5            /boost_1_57_0/boost/geometry/algorithms/detail
5            /boost_1_57_0/boost/math/tools/detail
5            /boost_1_57_0/boost/icl
5            /boost_1_57_0/boost/geometry/algorithms/detail
5            /boost_1_57_0/boost/icl
5            /boost_1_57_0/boost/geometry/algorithms/detail
5            /boost_1_57_0/boost/icl
4            /boost_1_57_0/boost/phoenix
4            /boost_1_57_0/boost/phoenix/stl
4            /boost_1_57_0/boost/phoenix/core/detail
4            /boost_1_57_0/boost/multi_index/detail
4            /boost_1_57_0/boost/log
4            /boost_1_57_0/boost/log/detail
4            /boost_1_57_0/boost/intrusive
4            /boost_1_57_0/boost/geometry/iterators/detail
4            /boost_1_57_0/boost/geometry/io
4            /boost_1_57_0/boost/geometry/algorithms
4            /boost_1_57_0/boost/geometry/multi/geometries
4            /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed
4            /boost_1_57_0/boost/fusion/container/deque/detail/cpp03
4            /boost_1_57_0/boost/fusion/adapted
4            /boost_1_57_0/boost/fusion/adapted/adt
4            /boost_1_57_0/boost/date_time
4            /boost_1_57_0/boost/chrono/io
4            /boost_1_57_0/boost/chrono/detail/inlined
4            /boost_1_57_0/boost/bimap
4            /boost_1_57_0/boost/atomic/detail
4            /boost_1_57_0/boost/assign
4            /boost_1_57_0/boost/polygon
4            /boost_1_57_0/boost/numeric
4            /boost_1_57_0/boost/numeric/odeint/stepper
4            /boost_1_57_0/boost/numeric/odeint/external
4            /boost_1_57_0/boost/numeric/conversion
4            /boost_1_57_0/boost/multiprecision
4            /boost_1_57_0/boost/mpl/set/aux_/preprocessed
4            /boost_1_57_0/boost/mpl/list/aux_/preprocessed
4            /boost_1_57_0/boost/mpl/aux_/preprocessed/no_ctps
4            /boost_1_57_0/boost/mpl/aux_/preprocessed/mwcw
4            /boost_1_57_0/boost/mpl/aux_/preprocessed/msvc70
4            /boost_1_57_0/boost/mpl/aux_/preprocessed/bcc
4            /boost_1_57_0/boost/mpl/aux_/preprocessed/dmc
4            /boost_1_57_0/boost/mpl/aux_/preprocessed/bcc_pre590
4            /boost_1_57_0/boost/mpl/map/aux_
4            /boost_1_57_0/boost/math/special_functions
4            /boost_1_57_0/boost/local_function
4            /boost_1_57_0/boost/local_function/aux_/preprocessor
4            /boost_1_57_0/boost/local_function/aux_/preprocessor/traits/decl_
4            /boost_1_57_0/boost/local_function/aux_/macro
4            /boost_1_57_0/boost/icl/type_traits
4            /boost_1_57_0/boost/compatibility
4            /boost_1_57_0/boost/asio
4            /boost_1_57_0/boost/asio/detail/impl
4            /boost_1_57_0/boost/archive
4            /boost_1_57_0/boost/algorithm/searching
4            /boost_1_57_0/boost/accumulators/statistics
4            /gfstest/28dir
4            /gfstest/28dir/8dir
4            /gfstest/28dir/4dir
4            /gfstest/28dir/10dir
4            /gfstest/28dir/1dir
4            /gfstest/27dir
4            /gfstest/27dir/10dir
4            /gfstest/27dir/9dir
4            /gfstest/27dir/2dir
4            /gfstest/26dir
4            /gfstest/26dir/8dir
4            /gfstest/26dir/3dir
4            /gfstest/26dir/6dir
4            /gfstest/26dir/5dir
4            /gfstest/26dir/2dir
4            /gfstest/25dir
4            /gfstest/25dir/6dir

========Read Throughput File Stats=====

TIMESTAMP 			 THROUGHPUT(KBPS)	FILE NAME

======Write Throughput File Stats======

TIMESTAMP 			 THROUGHPUT(KBPS)	FILE NAME
1970-01-01 00:01:41.000000 	 930.00      	  /boost_1_57_0/boost/math/bindings/mpfr.hpp
1970-01-01 00:14:41.000000 	 890.00      	  /boost_1_57_0/boost/numeric/ublas/banded.hpp
1970-01-01 00:01:57.000000 	 890.00      	  /boost_1_57_0/boost/math/special_functions/zeta.hpp
1970-01-01 00:01:57.000000 	 890.00      	  /boost_1_57_0/boost/math/special_functions/math_fwd.hpp
1970-01-01 00:04:01.000000 	 890.00      	  /boost_1_57_0/boost/intrusive/set.hpp
1970-01-01 00:07:29.000000 	 853.00      	  /boost_1_57_0/boost/phoenix/support/preprocessed/vector_50.hpp
1970-01-01 00:03:33.000000 	 853.00      	  /boost_1_57_0/boost/math/special_functions/zeta.hpp
1970-01-01 00:22:41.000000 	 853.00      	  /boost_1_57_0/boost/math/special_functions/bessel.hpp
1970-01-01 00:01:25.000000 	 853.00      	  /boost_1_57_0/boost/chrono/io/time_point_io.hpp
1970-01-01 00:25:37.000000 	 819.00      	  /boost_1_57_0/boost/math/special_functions/bessel.hpp
1970-01-01 00:01:25.000000 	 819.00      	  /boost_1_57_0/boost/fusion/tuple/detail/preprocessed/tuple30.hpp
1970-01-01 00:03:13.000000 	 819.00      	  /boost_1_57_0/boost/atomic/detail/ops_msvc_arm.hpp
1970-01-01 00:51:13.000000 	 819.00      	  /boost_1_57_0/boost/atomic/detail/atomic_template.hpp
1970-01-01 00:13:53.000000 	 787.00      	  /boost_1_57_0/boost/phoenix/scope/preprocessed/lambda_20.hpp
1970-01-01 00:01:41.000000 	 787.00      	  /boost_1_57_0/boost/numeric/ublas/assignment.hpp
1970-01-01 00:01:25.000000 	 787.00      	  /boost_1_57_0/boost/mpl/vector/aux_/preprocessed/plain/vector50.hpp
1970-01-01 00:01:25.000000 	 787.00      	  /boost_1_57_0/boost/container/map.hpp
1970-01-01 00:29:37.000000 	 786.00      	  /boost_1_57_0/boost/multiprecision/cpp_int/bitwise.hpp
1970-01-01 00:02:57.000000 	 781.00      	  /boost_1_57_0/boost/graph/astar_search.hpp
1970-01-01 00:03:33.000000 	 758.00      	  /boost_1_57_0/boost/phoenix/bind/preprocessed/bind_member_function_40.hpp
1970-01-01 00:25:53.000000 	 731.00      	  /boost_1_57_0/boost/phoenix/core/preprocessed/expression_50.hpp
1970-01-01 00:00:33.000000 	 720.00      	  /boost_1_57_0/boost/geometry/algorithms/detail/overlay/intersection_insert.hpp
1970-01-01 00:44:01.000000 	 706.00      	  /boost_1_57_0/boost/msm/front/euml/container.hpp
1970-01-01 01:45:37.000000 	 682.00      	  /boost_1_57_0/boost/numeric/odeint/external/openmp/openmp_range_algebra.hpp
1970-01-01 00:01:25.000000 	 682.00      	  /boost_1_57_0/boost/multi_index/hashed_index.hpp
1970-01-01 00:01:25.000000 	 682.00      	  /boost_1_57_0/boost/math/tools/detail/rational_horner3_16.hpp
1970-01-01 00:01:25.000000 	 682.00      	  /boost_1_57_0/boost/graph/dijkstra_shortest_paths.hpp
N/A.000000 	 682.00      	  /boost_1_57_0/boost/fusion/tuple/detail/preprocessed/tuple40.hpp
1970-01-01 00:02:25.000000 	 682.00      	  /boost_1_57_0/boost/intrusive/hashtable.hpp
1970-01-01 00:33:21.000000 	 660.00      	  /boost_1_57_0/boost/multiprecision/detail/functions/pow.hpp
1970-01-01 00:01:25.000000 	 660.00      	  /boost_1_57_0/boost/graph/distributed/compressed_sparse_row_graph.hpp
1970-01-01 00:02:29.000000 	 660.00      	  /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque_keyed_values30.hpp
1970-01-01 00:30:25.000000 	 657.00      	  /boost_1_57_0/boost/math/tools/detail/rational_horner3_16.hpp
1970-01-01 00:01:25.000000 	 653.00      	  /boost_1_57_0/boost/numeric/ublas/storage_sparse.hpp
1970-01-01 01:54:57.000000 	 647.00      	  /boost_1_57_0/boost/math/tools/detail/rational_horner2_19.hpp
1970-01-01 00:18:25.000000 	 643.00      	  /boost_1_57_0/boost/geometry/algorithms/centroid.hpp
1970-01-01 00:02:13.000000 	 640.00      	  /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/switch_40.hpp
1970-01-01 00:02:25.000000 	 640.00      	  /boost_1_57_0/boost/phoenix/scope/preprocessed/lambda_50.hpp
N/A.000000 	 640.00      	  /boost_1_57_0/boost/phoenix/scope/detail/preprocessed/dynamic_50.hpp
1970-01-01 00:06:09.000000 	 640.00      	  /boost_1_57_0/boost/phoenix/operator/preprocessed/member_30.hpp
1970-01-01 00:01:25.000000 	 640.00      	  /boost_1_57_0/boost/phoenix/core/detail/preprocessed/call_40.hpp
1970-01-01 00:01:25.000000 	 640.00      	  /boost_1_57_0/boost/numeric/ublas/triangular.hpp
1970-01-01 00:09:53.000000 	 640.00      	  /boost_1_57_0/boost/numeric/ublas/matrix_sparse.hpp
1970-01-01 00:01:25.000000 	 640.00      	  /boost_1_57_0/boost/math/special_functions/math_fwd.hpp
1970-01-01 00:03:33.000000 	 640.00      	  /boost_1_57_0/boost/math/special_functions/detail/igamma_large.hpp
1970-01-01 00:09:05.000000 	 640.00      	  /boost_1_57_0/boost/gil/gil_concept.hpp
1970-01-01 00:01:25.000000 	 640.00      	  /boost_1_57_0/boost/geometry/algorithms/detail/relate/areal_areal.hpp
1970-01-01 00:02:29.000000 	 640.00      	  /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_map50.hpp
1970-01-01 00:02:29.000000 	 640.00      	  /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_deque50.hpp
1970-01-01 00:01:41.000000 	 640.00      	  /boost_1_57_0/boost/locale/boundary/index.hpp
1970-01-01 00:03:45.000000 	 640.00      	  /boost_1_57_0/boost/interprocess/mem_algo/detail/multi_simple_seq_fit_impl.hpp
1970-01-01 00:42:57.000000 	 640.00      	  /boost_1_57_0/boost/interprocess/detail/win32_api.hpp
1970-01-01 00:01:25.000000 	 640.00      	  /boost_1_57_0/boost/icl/interval_base_map.hpp
1970-01-01 00:01:25.000000 	 640.00      	  /boost_1_57_0/boost/icl/concept/interval_associator.hpp
1970-01-01 00:02:57.000000 	 640.00      	  /boost_1_57_0/boost/graph/vf2_sub_graph_iso.hpp
1970-01-01 00:01:25.000000 	 640.00      	  /boost_1_57_0/boost/graph/mcgregor_common_subgraphs.hpp
1970-01-01 00:01:25.000000 	 640.00      	  /boost_1_57_0/boost/bind/bind.hpp
1970-01-01 00:01:05.000000 	 640.00      	  /boost_1_57_0/boost/atomic/detail/ops_gcc_arm.hpp
1970-01-01 03:24:49.000000 	 626.00      	  /boost_1_57_0/boost/numeric/odeint/stepper/bulirsch_stoer.hpp
1970-01-01 00:00:49.000000 	 620.00      	  /boost_1_57_0/boost/graph/distributed/named_graph.hpp
1970-01-01 00:01:25.000000 	 620.00      	  /boost_1_57_0/boost/graph/distributed/compressed_sparse_row_graph.hpp
1970-01-01 00:01:21.000000 	 615.00      	  /boost_1_57_0/boost/gil/bit_aligned_pixel_reference.hpp
1970-01-01 00:04:33.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/try_catch_eval_40.hpp
1970-01-01 00:59:13.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/switch_50.hpp
1970-01-01 00:01:09.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/switch_30.hpp
1970-01-01 00:12:17.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/switch_20.hpp
1970-01-01 00:02:29.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/statement/detail/preprocessed/catch_push_back_50.hpp
1970-01-01 00:19:29.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/scope/detail/preprocessed/make_locals_30.hpp
1970-01-01 00:02:25.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/operator/preprocessed/member_50.hpp
1970-01-01 00:16:49.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/operator/preprocessed/member_40.hpp
1970-01-01 00:06:57.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/operator/detail/preprocessed/mem_fun_ptr_gen_50.hpp
1970-01-01 00:03:45.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/operator/detail/preprocessed/mem_fun_ptr_gen_40.hpp
1970-01-01 00:09:05.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/object/detail/preprocessed/construct_eval_50.hpp
1970-01-01 00:01:25.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/core/preprocessed/expression_20.hpp
1970-01-01 00:10:57.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/core/detail/preprocessed/phx2_result_40.hpp
1970-01-01 00:01:21.000000 	 602.00      	  /boost_1_57_0/boost/phoenix/core/detail/preprocessed/actor_operator_50.hpp
1970-01-01 00:01:41.000000 	 602.00      	  /boost_1_57_0/boost/numeric/ublas/traits.hpp
1970-01-01 00:08:01.000000 	 602.00      	  /boost_1_57_0/boost/numeric/ublas/matrix_proxy.hpp
1970-01-01 00:01:25.000000 	 602.00      	  /boost_1_57_0/boost/multiprecision/cpp_dec_float.hpp
1970-01-01 08:29:05.000000 	 602.00      	  /boost_1_57_0/boost/multiprecision/cpp_bin_float/io.hpp
1970-01-01 00:14:41.000000 	 602.00      	  /boost_1_57_0/boost/multi_index/sequenced_index.hpp
1970-01-01 00:11:29.000000 	 602.00      	  /boost_1_57_0/boost/mpl/vector/aux_/preprocessed/plain/vector40.hpp
1970-01-01 00:03:33.000000 	 602.00      	  /boost_1_57_0/boost/mpl/vector/aux_/preprocessed/plain/vector30.hpp
1970-01-01 00:02:13.000000 	 602.00      	  /boost_1_57_0/boost/mpl/vector/aux_/preprocessed/no_ctps/vector20.hpp
1970-01-01 00:01:25.000000 	 602.00      	  /boost_1_57_0/boost/mpl/vector/aux_/preprocessed/no_ctps/vector10.hpp
1970-01-01 00:01:25.000000 	 602.00      	  /boost_1_57_0/boost/math/special_functions/detail/unchecked_bernoulli.hpp
1970-01-01 00:01:25.000000 	 602.00      	  /boost_1_57_0/boost/graph/max_cardinality_matching.hpp
1970-01-01 00:01:25.000000 	 602.00      	  /boost_1_57_0/boost/graph/distributed/named_graph.hpp
1970-01-01 00:19:45.000000 	 602.00      	  /boost_1_57_0/boost/graph/distributed/dehne_gotz_min_spanning_tree.hpp
1970-01-01 00:17:53.000000 	 602.00      	  /boost_1_57_0/boost/graph/boykov_kolmogorov_max_flow.hpp
1970-01-01 00:01:41.000000 	 602.00      	  /boost_1_57_0/boost/gil/algorithm.hpp
1970-01-01 00:02:57.000000 	 602.00      	  /boost_1_57_0/boost/geometry/algorithms/detail/overlay/get_turns.hpp
1970-01-01 00:02:41.000000 	 602.00      	  /boost_1_57_0/boost/geometry/algorithms/detail/overlay/get_turn_info_la.hpp
1970-01-01 00:02:29.000000 	 602.00      	  /boost_1_57_0/boost/geometry/algorithms/detail/buffer/buffered_piece_collection.hpp
N/A.000000 	 602.00      	  /boost_1_57_0/boost/fusion/tuple/detail/preprocessed/tuple50.hpp
1970-01-01 00:01:41.000000 	 602.00      	  /boost_1_57_0/boost/fusion/tuple/detail/preprocessed/make_tuple40.hpp
1970-01-01 00:24:33.000000 	 602.00      	  /boost_1_57_0/boost/fusion/container/vector/detail/preprocessed/vector50.hpp
1970-01-01 00:02:29.000000 	 602.00      	  /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/map_tie50.hpp
1970-01-01 00:02:57.000000 	 602.00      	  /boost_1_57_0/boost/fusion/container/generation/detail/preprocessed/make_list50.hpp
1970-01-01 01:02:57.000000 	 602.00      	  /boost_1_57_0/boost/fusion/container/deque/detail/cpp03/preprocessed/deque_keyed_values50.hpp

=== Interval 1 stats ===
      Duration : 5662 secs
     BytesRead : 356
  BytesWritten : 60435863

Block Size   :               4B+               8B+              16B+
Read Count   :                 0                 0                 0
Write Count  :                 4                 4                 9

Block Size   :              32B+              64B+             128B+
Read Count   :                 0                 0                 2
Write Count  :                32                39               104

Block Size   :             256B+             512B+            1024B+
Read Count   :                 0                 0                 0
Write Count  :               215              1416              1869

Block Size   :            2048B+            4096B+            8192B+
Read Count   :                 0                 0                 0
Write Count  :              2095              1884              3851

Block Size   :           16384B+
Read Count   :                 0
Write Count  :                48

Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
---           ----------    -----------    -----------    -----------
STAT                  10       10.70 us        9.00 us       12.00 us
MKDIR               9628    27069.69 us       91.00 us  1633942.00 us
UNLINK              6967    15847.52 us       38.00 us   173531.00 us
RMDIR               9628      235.50 us       43.00 us    64920.00 us
READ                   2      233.50 us      201.00 us      266.00 us
WRITE              11570       31.49 us       10.00 us      163.00 us
STATFS               949       28.75 us        9.00 us    11060.00 us
FLUSH               8065       10.99 us        5.00 us      111.00 us
SETXATTR            9629       35.29 us       16.00 us      138.00 us
OPENDIR            19285       17.74 us       12.00 us      531.00 us
CREATE              6967    26181.05 us       67.00 us   528654.00 us
FSTAT                  4       79.50 us       67.00 us       85.00 us
LOOKUP             48995      159.26 us       17.00 us    84036.00 us
SETATTR            21666       31.95 us       12.00 us      183.00 us
READDIRP           29493      938.44 us       10.00 us   208823.00 us
FORGET             16595           0 us           0 us           0 us
RELEASE             6967           0 us           0 us           0 us
RELEASEDIR         19285           0 us           0 us           0 us
------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----

Attachment:
strace_slow1.dat.gz

Description: GNU Zip compressed data
Attachment:
strace_slow2.dat.gz

Description: GNU Zip compressed data
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel