Hi!
After some tests with the newest version of glusterfs. We have some
testing results.
The new version is a lot faster (even without io-cache). The
throughput is about 1.5 times higher (from 20MBit output to 32MBit
output) than with the old version we used before.
We were note able to use io-cache because the glusterfs process then
uses more and more cpu power and the system is therefore slower than
without io-cache. It seems that the purging algorithm in io-cache is
not optimal. Maybe it is called too often or the underlaying data
structure for the cache is not optimal. We tried different settings
for page-count and page-size. Like page-count 32 and page-size 64MB or
page-count 128 and page-size 1MB. After some time the cache seems to
be full and from now on the purge algorithm seems to work without
sleep. Maybe something like an upper and a lower threshold could solve
that problem. This is also done in squid. It purges the cache, when it
is over about 90% of full size and purges it as long as the size is
larger than 75% of the full size.
Another problem we discovered was the following segfault after some
time. I just can't explain the reason for it. It seems to be load
dependant.
#0 0x00002b5f8c363eb2 in pthread_spin_lock () from /lib/libpthread.so.0
#1 0x00000000004094fb in fetch_cbk (frame=<value optimized out>,
cookie=0x60c3e0, this=0x60ff80, op_ret=0, op_errno=6341600,
spec_data=0x6107fc "### file: client-volume.spec.sample\n\n", '#'
<repeats 46 times>, "\n### GlusterFS Client Volume Specification
##\n", '#' <repeats 46 times>, "\n\n#### CONFIG FILE RULE"...) at
../../libglusterfs/src/stack.h:99
#2 0x00002b5f8cad376b in client_getspec_cbk (frame=0x60fce0, args=0x60fda0)
at client-protocol.c:4058
#3 0x00002b5f8cad77b8 in notify (this=0x60c300, event=<value optimized out>,
data=0x60d1e0) at client-protocol.c:4364
#4 0x00002b5f8bd2f4f2 in sys_epoll_iteration (ctx=<value optimized out>)
at epoll.c:53
#5 0x0000000000409400 in fetch_spec (ctx=0x7fff1efa4900,
remote_host=<value optimized out>, remote_port=0x60c3c0 "9999",
transport=<value optimized out>) at fetch-spec.c:131
#6 0x000000000040329d in main (argc=5, argv=0x7fff1efa4a78) at glusterfs.c:128
The config we used in that case is the following:
for the server:
volume brick
type storage/posix # POSIX FS translator
option directory /media/storage # Export this directory
end-volume
volume iothreads #iothreads can give performance a boost
type performance/io-threads
option thread-count 16
subvolumes brick
end-volume
### Add network serving capability to above brick.
volume server
type protocol/server
option transport-type tcp/server # For TCP/IP transport
option listen-port 9999 # Default is 6996
option client-volume-filename
/opt/glusterfs-1.3.0/etc/glusterfs/client.vol
subvolumes iothreads
option auth.ip.iothreads.allow * # Allow access to "brick" volume
end-volume
for the client:
volume client1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
option remote-host 10.1.1.13 # IP address of the remote brick
option remote-port 9999 # default server port is 6996
option remote-subvolume iothreads # name of the remote volume
end-volume
### Add client feature and attach to remote subvolume
volume client2
type protocol/client
option transport-type tcp/client # for TCP/IP transport
option remote-host 10.1.1.14 # IP address of the remote brick
option remote-port 9999 # default server port is 6996
option remote-subvolume iothreads # name of the remote volume
end-volume
volume afrbricks
type cluster/afr
subvolumes client1 client2
option replicate *:2
option self-heal off
end-volume
volume iothreads #iothreads can give performance a boost
type performance/io-threads
option thread-count 16
subvolumes afrbricks
end-volume
### Add writeback feature
volume writeback
type performance/write-behind
option aggregate-size 0 # unit in bytes
subvolumes iothreads
end-volume
#volume bricks
# type performance/io-cache
# option page-count 128 #128 is default option
# option page-size 128KB #128KB is default option
# option priority */imagescache/galerie/*.jpg:3,*3.jpg:2,*0.jpg:1
# subvolumes writeback
#end-volume
Again mainy thanks for your help!
Cheers,
Bernhard J. M. Grün
