Amar S. Tumballi wrote:
Hi Craig,
I will be looking into this issue. Btw, is there any reason you are not
using ib-verbs? instead using ib-sdp?
Let me get back to you regarding this. Give me few days.
For streaming bandwidth, verbs and ib-sdp did not provide any different
results. However, I didn't really think about it after getting it setup. I
will re-test with ib-verbs today.
Craig
Regards,
Amar
On Thu, Mar 13, 2008 at 8:41 AM, Craig Tierney <Craig.Tierney@xxxxxxxx>
wrote:
Amar S. Tumballi wrote:
Hi Craig,
Thanks for a nice comparison between GlusterFS and other network file
systems. But sure, before concluding about the performance, I would
suggest
few improvement to your GlusterFS setup.
1. Try with client/protocol instead of having unify with only one
subvolume.
(Unify makes sense when you have more than one subvolume, but when there
is
only one subvolume, its a extra layer which may count as overhead),
below
io-thread volume.
2. in io-cache on client side, as during kernel compile lot of *.h files
are
re-read, you can give preference to *h files only.
volume ioc
type performance/io-cache
subvolumes wb
option priority *.h:100
end-volume
I changed the io-cache settings to those above and eliminated the use of
the Unify
subvolume (my scripts generate server/client configs automatically, and in
most
cases multiple servers are used, in this case they weren't). The
compile time
went down, but not by much. The latest test finished in 1042 seconds.
What I didn't test this time is the compile directly on the storage that
is exported
by Gluster. The runtime there is 399 seconds, so the underlying
filesystem is fast.
I am not making any conclusions about the performance based on these
numbers.
Things are going great so far, and this should be a solveable problem
based on the
other performance characteristics I have seen.
Craig
Regards,
Amar
On Wed, Mar 12, 2008 at 3:55 PM, Craig Tierney <Craig.Tierney@xxxxxxxx>
wrote:
I have been testing out my GlusterFS setup. I have been
very happy with the streaming IO performance and scalability.
We have some users on the system now and they are seeing
very good performance (fast and consistent) as compared
to our other filesystem.
I have a test that I created that tries to measure metadata
performance by building the linux kernel. What I have
found is that GlusterFS is slower than local disk, NFS,
and Panasas. The compile time on those three systems
is roughly 500 seconds. For GlusterFS (1.3.7), the
compile time is roughly 1200 seconds. My GlusterFS filesystem
is using ramdisks on the servers and communicating using
IB-Verbs. My server and client configs are below.
Note I did not implement both write-behind and not read-behind
based on some benchmarks I saw on the list on how it affects
re-write.
So, is this just because mmap isn't (yet) supported in FUSE?
Or, is there something else I should be looking at.
Thanks,
Craig
server.cfg
----------
volume brick
type storage/posix # POSIX FS translator
option directory /tmp/scratch/export # Export this directory
end-volume
volume server
type protocol/server
subvolumes brick
option transport-type ib-sdp/server # For TCP/IP transport
option auth.ip.brick.allow *
end-volume
client.cfgvolume client-ns
type protocol/client
option transport-type ib-sdp/client
option remote-host w8-ib0
option remote-subvolume brick-ns
end-volume
volume client-w8
type protocol/client
option transport-type ib-sdp/client
option remote-host w8-ib0
option remote-subvolume brick
end-volume
volume unify
type cluster/unify
subvolumes client-w8
option namespace client-ns
option scheduler rr
end-volume
volume iot
type performance/io-threads
subvolumes unify
option thread-count 4
end-volume
volume wb
type performance/write-behind
subvolumes iot
end-volume
volume ioc
type performance/io-cache
subvolumes wb
end-volume
----------
--
Craig Tierney (craig.tierney@xxxxxxxx)
--
Craig Tierney (craig.tierney@xxxxxxxx)
--
Craig Tierney (craig.tierney@xxxxxxxx)
