Thanks for your inputs. Replies inline.
-Soumya
On 04/20/2015 07:48 AM, Joseph Fernandes wrote:
Adding more to Dan's Reply,
In tiering we lose the heat of the file(collected on the source brick) when the file gets migrated, by the DHT-Rebalancer during the rebalancer withing the tier.
We like to leverage the common solution infra for passing this extra metadata to destination.
----- Original Message -----
From: "Dan Lambright" <dlambrig@xxxxxxxxxx>
To: "Niels de Vos" <ndevos@xxxxxxxxxx>
Cc: "Joseph Fernandes" <josferna@xxxxxxxxxx>, "gluster Devel" <gluster-devel@xxxxxxxxxxx>, "Soumya Koduri" <skoduri@xxxxxxxxxx>
Sent: Monday, April 20, 2015 4:01:12 AM
Subject: Re: Upcall state + Data Tiering
----- Original Message -----
From: "Niels de Vos" <ndevos@xxxxxxxxxx>
To: "Dan Lambright" <dlambrig@xxxxxxxxxx>, "Joseph Fernandes" <josferna@xxxxxxxxxx>
Cc: "gluster Devel" <gluster-devel@xxxxxxxxxxx>, "Soumya Koduri" <skoduri@xxxxxxxxxx>
Sent: Sunday, April 19, 2015 9:01:56 AM
Subject: Re: Upcall state + Data Tiering
On Thu, Apr 16, 2015 at 04:58:29PM +0530, Soumya Koduri wrote:
Hi Dan/Joseph,
As part of upcall support on the server-side, we maintain certain state to
notify clients of the cache-invalidation and recall-leaselk events.
We have certain known limitations with Rebalance and Self-Heal. Details in
the below link -
http://www.gluster.org/community/documentation/index.php/Features/Upcall-infrastructure#Limitations
In case of Cache-invalidation,
upcall state is not migrated and once the rebalance is finished, the file
is
deleted and we may falsely notify the client that file is deleted when in
reality it isn't.
In case of Lease-locks,
As the case with posix locks, we do not migrate lease-locks as well but
will
end-up recalling lease-lock.
Here rebalance is an admin driven job, but that is not the case with
respect
to Data tiering.
We would like to know when the files are moved from hot to cold tiers or
vice-versa or rather when a file is considered to be migrated from cold to
hot tier, where we see potential issues.
Is it the first fop which triggers it? and
where are the further fops processed - on hot tier or cold tier?
Data tiering's basic design has been to reuse DHT's data migration algorithms. In this case, we see the same problem exists with DHT, but is a known limitation controlled by management operations. And therefore (if I follow) they may not tackle this problem right away, hence tiering may not be able to leverage their solution. It is of course desirable for data tiering to solve the problem in order to use the new upcall mechanisms.
Migration of a file is a multi-state process. I/O is accepted at the same time migration is underway. I believe the upcall manager and the migration manager (for lack of better words) would have to coordinate. The former subsystem understands locks, and the later how to move files.
With that "coordination" in mind, a basic strategy might be something like this:
On the source, when a file is ready to be moved, the migration manager informs the upcall manager.
The upcall manager packages relevant lock information and returns it to the migrator. The information reflects the state of posix or lease locks.
The migration manager moves the file.
The migration manager then sends the lock information as a virtual extended attribute.
On the destination server, the upcall manager is invoked. It is passed the contents of the virtual attributes. The upcall manager rebuilds the lock state and puts the file into proper order.
This approach sounds good. The same has been followed to migrate locks
incase of graph switch on the client-side
("glfs_migrate_fd_locks_safe"). Rebalance process (maybe self-heal
daemon) too could transfer the state using similar xattrs.
Only at that point, does the setxattr RPC return, and only then, shall the file be declared "migrated".
We would have to handle any changes to the lock state that occur when the file is in the middle of being migrated. Probably, the upcall manager would change the contents of the "package".
This sounds tricky. I do not know atm how the data changes on the files
which are getting rebalanced are handled but maybe we could follow on
the same lines.
It is desirable to invent something that would work with both DHT and tiering (i.e. be implemented at the core dht-rebalance.c layer). And in fact, the mechanism I describe could be useful for other meta-data transfer applications.
Agree. Thanks again for your inputs. I shall open a BZ capturing all
these details.
This is just a high level sketch, to provoke discussion and checkpoint if this is the right direction. It would take much time to sort through the details. Other ideas are welcome.
My understanding is the following:
- when a file is "cold" and gets accessed, the 1st FOP will mark the
file for migration to the "hot" tier
- migration is async, so the initial responses on FOPs would come from
the "cold" tier
- upon migration (similar to rebalance) locking state and upcall
tracking is lost
I think this is a problem. There seems to be a window where a client can
get (posix) locks while the file is on the "cold" tier. After migrating
the file from "cold" to "hot", these locks would get lost. The same
counts for tracking access in the upcall xlator.
Please provide your inputs on the same. We may need to document the same or
provide suggestions to the customers while deploying this solution.
Some ideas on how this can get solved would be most welcome.
Thanks,
Niels
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel
