Hello everyone!
Following a discussion I had with @Susant Palai some time ago, we have decided to look into an option to improve the rebalance process in the DHT layer by modifying the underlying mechanism. Currently, dir-tree crawling is done recursively, by a single thread, which is likely slow and also poses the risk of stack overflow. An iterative multithreaded solution might improve performance and also stability (by eliminating the risk of stack overflow). I have prepared a POC doc on the matter, including a sample implementation of the iterative multithreaded solution. The doc can be found at:
https://docs.google.com/document/d/1JCl0T9zeagOcFFpgVQF8zNyhlR54VqkNAZ7TJb42egE/edit
Apart from the rebalance process, maybe this approach can be useful for other use-cases where dir-tree crawl is being performed? Any comments on the concept, the design of the solution and the implementation are welcome.
--
Following a discussion I had with @Susant Palai some time ago, we have decided to look into an option to improve the rebalance process in the DHT layer by modifying the underlying mechanism. Currently, dir-tree crawling is done recursively, by a single thread, which is likely slow and also poses the risk of stack overflow. An iterative multithreaded solution might improve performance and also stability (by eliminating the risk of stack overflow). I have prepared a POC doc on the matter, including a sample implementation of the iterative multithreaded solution. The doc can be found at:
https://docs.google.com/document/d/1JCl0T9zeagOcFFpgVQF8zNyhlR54VqkNAZ7TJb42egE/edit
Apart from the rebalance process, maybe this approach can be useful for other use-cases where dir-tree crawl is being performed? Any comments on the concept, the design of the solution and the implementation are welcome.
--
Barak Sason Rofman
Gluster Storage Development
_______________________________________________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-devel
