I've configured a cluster with replication that uses most of the
advanced features you've implemented including io-threads, afr,
readahead, and writebehind. I am very satisfied with the write
performance, but the file creation performance leaves much to be
desired. What can we do to speed this up?
Creating 100 empty files
# time for i in {1..100}; do touch test.$i;done
real 0m46.913s
user 0m0.023s
sys 0m0.067s
That's about 0.500 seconds just to create an empty file.
In general, what do you advise for tuning the performance of
reading/writing tons of tiny files. Can the client use io-threads to
improve performance? Right now, our application stuffs all the tiny
files in a single directory. Eventually, we were planning on hashing
them out to directories. Would hashing them out into multiple
directories positively and significantly affect the performance of
GlusterFS?
Best,
Erik Osterman
For what it's worth, here are my configurations:
#
# Master
#
volume posix0
type storage/posix # POSIX FS translator
option directory /home/glusterfs # Export this directory
end-volume
volume brick0
type performance/io-threads
option thread-count 8
option queue-limit 1024
subvolumes posix0
end-volume
### Add network serving capability to above brick.
volume server
type protocol/server
option transport-type tcp/server # For TCP/IP transport
# option bind-address 192.168.1.10 # Default is to listen on all
interfaces
option listen-port 6996 # Default is 6996
option client-volume-filename /etc/glusterfs/client.vol
subvolumes brick0
option auth.ip.brick0.allow * # access to "brick" volume
end-volume
#
# Mirror
#
volume posix0
type storage/posix # POSIX FS translator
option directory /home/glusterfs # Export this directory
end-volume
volume mirror0
type performance/io-threads
option thread-count 8
option queue-limit 1024
subvolumes posix0
end-volume
### Add network serving capability to above brick.
volume server
type protocol/server
option transport-type tcp/server # For TCP/IP transport
# option bind-address 192.168.1.11 # Default is to listen on all
interfaces
option listen-port 6996 # Default is 6996
option client-volume-filename /etc/glusterfs/client.vol
subvolumes mirror0
option auth.ip.mirror0.allow * # access to "brick" volume
end-volume
#
# Client
#
### Add client feature and attach to remote subvolume of server
volume brick0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
option remote-host 216.182.237.155 # IP address of the remote brick
server
option remote-port 6996 # default server port is 6996
option remote-subvolume brick0 # name of the remote volume
end-volume
### Add client feature and attach to remote mirror of brick0
volume mirror0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
option remote-host 216.55.170.26 # IP address of the remote
mirror server
option remote-port 6996 # default server port is 6996
option remote-subvolume mirror0 # name of the remote volume
end-volume
### Add AFR feature to brick
volume afr0
type cluster/afr
subvolumes brick0 mirror0
option replicate *:2 # All files 2 copies (RAID-1)
end-volume
### Add unify feature to cluster the servers. Associate an
### appropriate scheduler that matches your I/O demand.
volume bricks
type cluster/unify
subvolumes afr0
### ** Round Robin (RR) Scheduler **
option scheduler rr
option rr.limits.min-free-disk 2GB
end-volume
### Add performance feature
volume writebehind
type performance/write-behind
option aggregate-size 131072 # aggregate block size in bytes
subvolumes bricks
end-volume
### Add performance feature
volume readahead
type performance/read-ahead
option page-size 131072
option page-count 16
subvolumes writebehind
end-volume
