Hi,
Here is an update on DHT2 progress as we exit our self imposed first 3
week iteration on the same.
The soft update:
----------------
(if you are looking at technical details skip to the next section :) )
Target: The target of the iteration was to exit with some FOPs
implemented both in the new POSIX xlator and in DHT2. Also, to firm up
the design to a large extent.
Reality: POSIX xlator changes are WIP, none of the FOPs implemented here
are tested. DHT2 empty xlator including its required configuration
options is created. The design itself has come a long way, but is about
50-60% done.
Next iteration plan:
- Implement basic FOPs for a proof of concept DH2 implementation
- Mature the design further to decide on which approach to take (see
P1/2 below
- Identify and understand how to resolve dependencies across other xlators
The technical notes:
--------------------
DHT2 has 2 possible implementation options, one where the file name and
its inode is separated to an MDS and a DS. The other where, in addition
to the file name and inode, its data extents are also separated, into
the MDS and DS. Each has its own advantages and possible shortcomings.
These 2 approaches are defined as Phase1 (P1) and Phase2 (P2) of the
DHT2 prototyping effort. With P1 being the former as stated above.
Here are more details on the design,
P1:
- Design of DHT2:
https://raw.githubusercontent.com/ShyamsundarR/glusterfs/gl40_dht_playground/xlators/cluster/dht2/docs/DHT2_FirstPrototypeDesign.md
- Functional spec of POSIX: https://review.gerrithub.io/#/c/243885/
P2:
- Initial thoughts on DHT2-P2: https://review.gerrithub.io/#/c/244752/
Here are some interesting problems that arise from DHT2:
1. Quota xlator as it exists today, cannot path walk and update
accounting information in real time. As it may need to traverse across
MDS nodes to do the same.
2. Small file creation scale considerations
See the WSR for details on these and some other issues here, [1] [2]
How to track/follow/contribute to the project:
----------------------------------------------
We are hosting the design, code and other artifacts for DHT2 in the
following gerrithub project,
https://review.gerrithub.io/#/admin/projects/ShyamsundarR/glusterfs
This tracks the github project,
https://github.com/ShyamsundarR/glusterfs/tree/gl40_dht_playground
Most of the initial commits are separated as design, WSR (weekly status
reports) and RFC (code patches), in the gerrithub project above.
Those interested can start above, and ping us on gluster-devel in case
information is lacking on any front.
Thanks,
KP, Venky, Shyam
[1]
https://raw.githubusercontent.com/ShyamsundarR/glusterfs/gl40_dht_playground/xlators/cluster/dht2/docs/DHT2_WSR_Week002_Aug_20_2015.md
[2]
https://raw.githubusercontent.com/ShyamsundarR/glusterfs/gl40_dht_playground/xlators/cluster/dht2/docs/DHT2_WSR_Week003_Aug_28_2015.md
On 08/10/2015 01:29 PM, Shyam wrote:
Hi,
DHT2 initial prototype development is targeted in gerrithub [1] with its
backed github repos from [2].
The git repos at [1/2] should serve as the single point for all DHT2
related information going forward.
Here are the important directories to monitor for documentation and code
for DHT2,
- xlator/cluster/dht2/docs
- xlator/cluster/dht2/src
The initial prototype design is derived from the design documents
previously posted at [3, 4] and, can be found here [5].
The milestones and plans would also be put into the same directory as
[5] and we intend to maintain weekly update reports on progress et. al.
at the same place (i.e we would open a review, update it and commit it
into the repo.)
The current work is to write POSIX layer to suit the on disk layout as
needed by DHT2, so that we can then start writing DHT2 code.
Work till now can be found in the gerrithub project.
Shyam
[1] https://review.gerrithub.io/#/admin/projects/ShyamsundarR/glusterfs
[2] https://github.com/ShyamsundarR/glusterfs/tree/gl40_dht_playground
[3] http://www.gluster.org/pipermail/gluster-devel/2015-June/045449.html
[4] http://www.gluster.org/pipermail/gluster-devel/2015-June/045591.html
[5]
https://github.com/ShyamsundarR/glusterfs/blob/gl40_dht_playground/xlators/cluster/dht2/docs/DHT2_FirstPrototypeDesign.md
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel
