Krishna Srinivas wrote:
On Nov 21, 2007 1:05 AM, Anand Avati <avati@xxxxxxxxxxxxx> wrote:
Have you tried chaining AFR volumes? There's quite a few ways I can
imagine reducing line saturation if that's the problem. Here's one:
server1 has an afr of a local volume and a volume from server2
server3 has an afr of a local volume and a volume from server4
client afr of server1's afr and server2's afr
This should allow a single write from a client at 1000/8/2 = 62.5
MByte/s. Client writes only twice (to server1 and server3) halving the
bandwidth. Server1 and server3 write only once each to server2 and
server4 respectively and receive the writes from the client, halving
their bandwidth.
Note: I have no idea how well this will perform in reality. There may
be enough lag in glusterfs chaining writes that the gains aren't worth
it, but I suspect since it is effectively pipelining the writes along
there won't be too much lag.
Good thought kevan, it sounds like a good in-between solution.
Jerker, this might work even better for you if the server-server
interconnects are on a seperate physical media not cascaded with the
server-client interconnect.
True. Actually if afr is on the server side, write performance will
increase only
if the server-server interconnect is on a different network (or VLAN)
than the server-client network. If they are on same LAN, regardless
of where the AFR is loaded (either client or server) there should be
very little difference in performance (atleast theoretically) because
even if AFR is loaded on the server it will still have to do the writes
to all the other servers.
Yeah, that's why I was careful to state that the client should AFR the
servers it connects to directly. That way, each server has at most two
network operations per file operation.
You could, theoretically, get gigabit performance if every system in the
glusterfs cluster has two nics, and you have switch ports, vlans and
private netblocks to spare.
E.g.
Client[ETH0]->Server(N)[ETH0],Server(N)[ETH1]->Server(N+1)[ETH0],..,Server((N/2)-1)[ETH1]->Server(N/2)[ETH0]
Client[ETH1]->Server((N/2)+1)[ETH0],Server((N/2)+1)[ETH1]->Server((N/2)+2)[ETH0],..,Server(N-1)[ETH1]->Server(N)[ETH0]
I suspect adding additional servers to the chain beyond 1 or 2 would
introduce some serious lag, and thus make it not very useful. You would
also be taxing the backplane of most switches fairly quickly if you used
vlans and few switches.
--
-Kevan Benson
-A-1 Networks
