On 07/27/2016 02:38 AM, Vijay Bellur wrote:
On 07/26/2016 05:56 AM, Soumya Koduri wrote:
Hi Vijay,
On 07/26/2016 12:13 AM, Vijay Bellur wrote:
On 07/22/2016 08:44 AM, Soumya Koduri wrote:
Hi,
In certain scenarios (esp.,in highly available environments), the
application may have to fail-over/connect to a different glusterFS
client while the I/O is happening. In such cases until there is a ping
timer expiry and glusterFS server cleans up the locks held by the older
glusterFS client, the application will not be able to reclaim their
lost
locks. To avoid that we need support in Gluster to let clients reclaim
the existing locks provided lkwoner and the lock range matches.
If the server detects a disconnection, it goes about cleaning up the
locks held by the disconnected client. Only if the failover connection
happens before this server cleanup the outlined scheme would work.Since
there is no ping timer on the server, do you propose to have a grace
timer on the server?
But we are looking for a solution which can work in active-active
configuration as well. We need to handle cases where in the connection
between server and the old-client is still in use, which can happen
during load-balancing or failback.
Different cases which I can outline are:
Application Client - (AC)
Application/GlusterClient 1 - GC1
Application/GlusterClient 2 - GC2
Gluster Server (GS)
1) Active-Passive config (service gone down)
AC ----> GC1 ----> GS (GC2 is not active)
| (failover)
v
AC ----> GC2 ----> GS (GC1 connection gets dropped and GC2 establishes
connection)
In this case, we can have grace timer to allow reclaims only for certain
time post GC2 (any) rpc connection establishment.
2) Active-Active config (service gone down)
AC ----> GC1 ----> GS
^
|
GC2 -------
| (failover)
v
AC ----> GC2 ----> GS (GC1 connection gets dropped)
The grace timer then shall not get triggered in this case. But at-least
the locks from GC1 gets cleaned post its connection cleanup.
grace timer is not required if lock reclamation can happen before the
old connection between GC1 & GS gets dropped. Is this guaranteed to
happen every time?
Not all the time but more likely since failover time is usually lesser
than ping timer / rpc connection expiry time.
3) Active-Active config (both the services active/load-balancing)
This is the trick one.
AC ----> GC1 ----> GS
^
|
GC2 -------
| (load-balancing/failback)
v
GC1 ----> GS
^
|
AC ----> GC2 -------
The locks taken by GC1 shall end up being on the server for ever unless
we restart either GC1 or the server.
Yes, this is trickier. The behavior is dependent on how the application
performs a failback. How do we handle this with Ganesha today? Since the
connection between nfs client and Ganesha/GC1 is broken, would it not
send cleanup requests on locks it held on behalf of that client?
Yes. I checked within NFS-Ganesha community too. There seems to be a
provision in NFS-Ganesha to trigger an event upon receiving which it can
flush the locks associated with an IP. We could send this event to the
active servers (in this case GC1) while triggering fail-back. So from
NFS-Ganesha perspective, this seems to be taken care of. Unless some
other application (SMB3 handles?) has this use-case, we may for now can
ignore it.
Considering above cases, looks like we may need to allow reclaim of the
locks all the time. Please suggest if I have missed out any details.
I agree that lock reclamation is needed. Grace timeout behavior does
need more thought for all these cases. Given the involved nature of this
problem, it might be better to write down a more detailed spec that
discusses all these cases for a more thorough review.
Sure. I will open up a spec.
Thanks,
Soumya
For client-side support, I am thinking if we can integrate with the new
lock API being introduced as part of mandatory lock support in gfapi
[2]
Is glfs_file_lock() planned to be used here? If so, how do we specify
that it is a reclaim lock in this api?
Yes. We have been discussing on that patch-set if we can use the same
API. We should either have a separate field to pass reclaim flag or if
we choose not to change its definition, then probably can have
additional lock types -
GLFS_LK_ADVISORY
GLFS_LK_MANDATORY
New lock-types
GLFS_LK_RECLAIM_ADVISORY
GLFS_LK_RECLAIM_MANDATORY
Either approach seems reasonable to me.
We also would need to pass the reclaim_lock flag over rpc.
To avoid new fop/rpc changes, I was considering to take xdata approach
(similar to the way lock mode is passed in xdata for mandatory lock
support) since the processing of reclamation doesn't differ much from
the existing lk fop except for conflicting lock checks.
This looks ok to me.
Thanks,
Vijay
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel
