Am Dienstag 08 Januar 2008 06:06:30 schrieb Anand Avati: > > > Brandon, > > > who does the copy is decided where the AFR translator is loaded. if > > > you have AFR loaded on the client side, then the client does the two > > > writes. > > > > you > > > > > can also have AFR loaded on the server side, and handle server do the > > > replication. Translators can be loaded anywhere (client or server, > > > > anywhere > > > > > in the graph). You need to think more on the lines on how you can > > > > 'program > > > > > glusterfs' rather than how to 'configure glusterfs'. > > > > Performance-wise, which is better? Or does it make sense one way vs. > > the other based on number of clients? > > Depends, if the interconnect between server and client is precious, then > have the servers replicate (load afr on server side) with replication > happening on a seperate network. This is also good if you have servers > interconnected with high speed networks like infiniband. > > If your servers are having just one network interface (no seperate network > for replication), and your client apps are IO bound, then it does not > matter where you load AFR; they all would give the same performance. > > avati i did a simple test recently, which suggests that there is a significant performance difference: I did a comparison of client vs. server side afr with bonnie, for a one client and two servers setup with tla patch628, connected over GB Ethernet; please see my results below. There also was a posting on this list with a lot of test results, suggesting that server side afr is fastest: http://lists.nongnu.org/archive/html/gluster-devel/2007-08/msg00136.html In my own results though, client-side afr seems to be better in most of the test; I should note that I'm not sure if the chosen setup has a negative impact on the performance (two servers afr-ing each other), so any comments on this would be highly appreciate (I add the configs for the tests below). server side afr (I hope it stays readable): Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP stf-db22 31968M 31438 43 35528 0 990 0 32375 43 41107 1 38.1 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 34 0 416 0 190 0 35 0 511 0 227 0 client side afr: Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP stf-db22 31968M 27583 38 31518 0 862 0 49522 63 56388 2 28.0 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 418 0 2225 1 948 1 455 0 2305 1 947 0 server side afr config: glusterfs-server.vol.server_afr: volume fsbrick1 type storage/posix option directory /data1 end-volume volume fsbrick2 type storage/posix option directory /data2 end-volume volume nsfsbrick1 type storage/posix option directory /data-ns1 end-volume volume brick1 type performance/io-threads option thread-count 8 option queue-limit 1024 subvolumes fsbrick1 end-volume volume brick2 type performance/io-threads option thread-count 8 option queue-limit 1024 subvolumes fsbrick2 end-volume volume brick1r type protocol/client option transport-type tcp/client option remote-host 10.10.1.99 option remote-subvolume brick2 end-volume volume afr1 type cluster/afr subvolumes brick1 brick1r # option replicate *:2 # obsolete with tla snapshot end-volume ### Add network serving capability to above bricks. volume server type protocol/server option transport-type tcp/server # For TCP/IP transport option listen-port 6996 # Default is 6996 option client-volume-filename /etc/glusterfs/glusterfs-client.vol subvolumes afr1 nsfsbrick1 option auth.ip.afr1.allow * # Allow access to "brick" volume option auth.ip.brick2.allow * # Allow access to "brick" volume option auth.ip.nsfsbrick1.allow * # Allow access to "brick" volume end-volume glusterfs-client.vol.test.server_afr: volume fsc1 type protocol/client option transport-type tcp/client option remote-host 10.10.1.10 option remote-subvolume afr1 end-volume volume fsc2 type protocol/client option transport-type tcp/client option remote-host 10.10.1.99 option remote-subvolume afr1 end-volume volume ns1 type protocol/client option transport-type tcp/client option remote-host 10.10.1.10 option remote-subvolume nsfsbrick1 end-volume volume ns2 type protocol/client option transport-type tcp/client option remote-host 10.10.1.99 option remote-subvolume nsfsbrick1 end-volume volume afrns type cluster/afr subvolumes ns1 ns2 end-volume volume bricks type cluster/unify subvolumes fsc1 fsc2 option namespace afrns option scheduler alu option alu.limits.min-free-disk 5% # Stop creating files when free-space lt 5 % option alu.limits.max-open-files 10000 option alu.order disk-usage:read-usage:write-usage:open-files-usage:disk-speed-usage option alu.disk-usage.entry-threshold 2GB # Units in KB, MB and GB are allowed option alu.disk-usage.exit-threshold 60MB # Units in KB, MB and GB are allowed option alu.open-files-usage.entry-threshold 1024 option alu.open-files-usage.exit-threshold 32 option alu.stat-refresh.interval 10sec end-volume volume readahead type performance/read-ahead option page-size 256KB option page-count 2 subvolumes bricks end-volume volume write-behind type performance/write-behind option aggregate-size 1MB subvolumes readahead end-volume ----------------------------------------------------------------------- client side afr config: glusterfs-server.vol.client_afr: volume fsbrick1 type storage/posix option directory /data1 end-volume volume fsbrick2 type storage/posix option directory /data2 end-volume volume nsfsbrick1 type storage/posix option directory /data-ns1 end-volume volume brick1 type performance/io-threads option thread-count 8 option queue-limit 1024 subvolumes fsbrick1 end-volume volume brick2 type performance/io-threads option thread-count 8 option queue-limit 1024 subvolumes fsbrick2 end-volume ### Add network serving capability to above bricks. volume server type protocol/server option transport-type tcp/server # For TCP/IP transport option listen-port 6996 # Default is 6996 option client-volume-filename /etc/glusterfs/glusterfs-client.vol subvolumes brick1 brick2 nsfsbrick1 option auth.ip.brick1.allow * # Allow access to "brick" volume option auth.ip.brick2.allow * # Allow access to "brick" volume option auth.ip.nsfsbrick1.allow * # Allow access to "brick" volume end-volume glusterfs-client.vol.test.client_afr: volume fsc1 type protocol/client option transport-type tcp/client option remote-host 10.10.1.10 option remote-subvolume brick1 end-volume volume fsc1r type protocol/client option transport-type tcp/client option remote-host 10.10.1.10 option remote-subvolume brick2 end-volume volume fsc2 type protocol/client option transport-type tcp/client option remote-host 10.10.1.99 option remote-subvolume brick1 end-volume volume fsc2r type protocol/client option transport-type tcp/client option remote-host 10.10.1.99 option remote-subvolume brick2 end-volume volume afr1 type cluster/afr subvolumes fsc1 fsc2r # option replicate *:2 # obsolete with tla snapshot end-volume volume afr2 type cluster/afr subvolumes fsc2 fsc1r # option replicate *:2 # obsolete with tla snapshot end-volume volume ns1 type protocol/client option transport-type tcp/client option remote-host 10.10.1.10 option remote-subvolume nsfsbrick1 end-volume volume ns2 type protocol/client option transport-type tcp/client option remote-host 10.10.1.99 option remote-subvolume nsfsbrick1 end-volume volume afrns type cluster/afr subvolumes ns1 ns2 # option replicate *:2 # obsolete with tla snapshot end-volume volume bricks type cluster/unify subvolumes afr1 afr2 option namespace afrns option scheduler alu option alu.limits.min-free-disk 5% # Stop creating files when free-space lt 5 % option alu.limits.max-open-files 10000 option alu.order disk-usage:read-usage:write-usage:open-files-usage:disk-speed-usage option alu.disk-usage.entry-threshold 2GB # Units in KB, MB and GB are allowed option alu.disk-usage.exit-threshold 60MB # Units in KB, MB and GB are allowed option alu.open-files-usage.entry-threshold 1024 option alu.open-files-usage.exit-threshold 32 option alu.stat-refresh.interval 10sec end-volume volume readahead type performance/read-ahead option page-size 256KB option page-count 2 subvolumes bricks end-volume volume write-behind type performance/write-behind option aggregate-size 1MB subvolumes readahead end-volume Cheers, Sascha
