Re: Question about compile performance over GlusterFS

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Amar S. Tumballi wrote:
Hi Craig,
 I will be looking into this issue. Btw, is there any reason you are not
using ib-verbs? instead using ib-sdp?
 Let me get back to you regarding this. Give me few days.



For streaming bandwidth, verbs and ib-sdp did not provide any different
results.  However, I didn't really think about it after getting it setup.  I
will re-test with ib-verbs today.

Craig


Regards,
Amar

On Thu, Mar 13, 2008 at 8:41 AM, Craig Tierney <Craig.Tierney@xxxxxxxx>
wrote:

Amar S. Tumballi wrote:
Hi Craig,
 Thanks for a nice comparison  between GlusterFS and other network file
systems. But sure, before concluding about the performance, I would
suggest
few improvement to your GlusterFS setup.

1. Try with client/protocol instead of having unify with only one
subvolume.
(Unify makes sense when you have more than one subvolume, but when there
is
only one subvolume, its a extra layer which may count as overhead),
below
io-thread volume.

2. in io-cache on client side, as during kernel compile lot of *.h files
are
re-read, you can give preference to *h files only.

volume ioc
  type performance/io-cache
  subvolumes wb
  option priority *.h:100
end-volume

I changed the io-cache settings to those above and eliminated the use of
the Unify
subvolume (my scripts generate server/client configs automatically, and in
most
cases multiple servers are used, in this case they weren't).    The
compile time
went down, but not by much.  The latest test finished in 1042 seconds.

What I didn't test this time is the compile directly on the storage that
is exported
by Gluster.  The runtime there is 399 seconds, so the underlying
filesystem is fast.

I am not making any conclusions about the performance based on these
numbers.
Things are going great so far, and this should be a solveable problem
based on the
other performance characteristics I have seen.

Craig



Regards,
Amar

On Wed, Mar 12, 2008 at 3:55 PM, Craig Tierney <Craig.Tierney@xxxxxxxx>
wrote:

I have been testing out my GlusterFS setup.  I have been
very happy with the streaming IO performance and scalability.
We have some users on the system now and they are seeing
very good performance (fast and consistent) as compared
to our other filesystem.

I have a test that I created that tries to measure metadata
performance by building the linux kernel.  What I have
found is that GlusterFS is slower than local disk, NFS,
and Panasas.  The compile time on those three systems
is roughly 500 seconds.  For GlusterFS (1.3.7), the
compile time is roughly 1200 seconds.  My GlusterFS filesystem
is using ramdisks on the servers and communicating using
IB-Verbs.  My server and client configs are below.

Note I did not implement both write-behind and not read-behind
based on some benchmarks I saw on the list on how it affects
re-write.

So, is this just because mmap isn't (yet) supported in FUSE?
Or, is there something else I should be looking at.

Thanks,
Craig


server.cfg
----------

volume brick
  type storage/posix                   # POSIX FS translator
  option directory /tmp/scratch/export        # Export this directory
end-volume

volume server
  type protocol/server
  subvolumes brick
  option transport-type ib-sdp/server     # For TCP/IP transport
  option auth.ip.brick.allow *
end-volume

client.cfgvolume client-ns
  type protocol/client
  option transport-type ib-sdp/client
  option remote-host w8-ib0
  option remote-subvolume brick-ns
end-volume



volume client-w8
  type protocol/client
  option transport-type ib-sdp/client
  option remote-host w8-ib0
  option remote-subvolume brick
end-volume

volume unify
        type cluster/unify
        subvolumes  client-w8
        option namespace client-ns
        option scheduler rr
end-volume

volume iot
        type performance/io-threads
        subvolumes unify
        option thread-count 4
end-volume

volume wb
        type performance/write-behind
        subvolumes iot
end-volume

volume ioc
        type performance/io-cache
        subvolumes wb
end-volume

----------




--
Craig Tierney (craig.tierney@xxxxxxxx)




--
Craig Tierney (craig.tierney@xxxxxxxx)






--
Craig Tierney (craig.tierney@xxxxxxxx)




[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux