Stephan von Krawczynski wrote:
On Sun, 31 Jan 2010 00:29:55 +0000
Gordan Bobic <gordan@xxxxxxxxxx> wrote:
Stephan von Krawczynski wrote:
On Fri, 29 Jan 2010 18:41:10 +0000
Gordan Bobic <gordan@xxxxxxxxxx> wrote:
I'm seeing things like this in the logs, coupled with things locking up
for a while until the timeout is complete:
[2010-01-29 18:29:01] E
[client-protocol.c:415:client_ping_timer_expired] home2: Server
10.2.0.10:6997 has not responded in the last 42 seconds, disconnecting.
[2010-01-29 18:29:01] E
[client-protocol.c:415:client_ping_timer_expired] home2: Server
10.2.0.10:6997 has not responded in the last 42 seconds, disconnecting.
The thing is, I know for a fact that there is no network outage of any
sort. All the machines are on a local gigabit ethernet, and there is no
connectivity loss observed anywhere else. ssh sessions going to the
machines that are supposedly "not responding" remain alive and well,
with no lag.
What you're seeing here is exactly what made us increase the ping-timeout to
120.
To us it is obvious that the keep alive strategy does not cope with minimal
packet loss. On _every_ network you can see packet loss (read the docs of your
switch carefully). We had the impression that the strategy implemented is not
aware of the fact that a lost ping packet is no proof for a disconnected
server but only a hint for a closer look.
It sounds like there needs to be more heartbeats/minute. 1 packet per 10
seconds might be a good figure to start with, but I cannot see that even
1 packet / second would be harmful unless the number of nodes starts to
get very large, and disconnection should be triggered only after some
threshold number (certainly > 1) of those get lost in a row. Are there
options to tune such parameters in the volume spec file?
Really, if you walk that way you should definitely have tuneable parameters,
because 1 per second is probably no good idea over (slow) wan. I have found
none so far ...
Indeed, 1/second over a slow WAN would be OTT, but the performance over
a WAN (or anything with > 5ms latencies) would be unusable anyway. But
this is why the heartbeating parameters should be tunable for different
environments. If the defaults don't work on a local Gb etherhet WAN with
only 5 nodes on it, they will are never likely to work for any common
environment.
Slightly offtopic I would like to ask if you, too, experienced glusterfs using
a lot more bandwith than a comparable nfs connection on the server network
side. It really looks a bit like a waste of resources to me...
I haven't noticed bandwidth going "missing", if that's what you mean. I
do my replication server-side, so the server replicates the writes n-1
times for n servers, and my cacti graphs are broadly in line with the
bandwidth usage expected. If I disconnect all the mirrors except the
server I'm connecting to,the bandwidth usage between the client and the
server is similar to NFS.
What bandwidth "leakage" are you observing?
Gordan
