At the last Ceph Developer Monthly (CDM) meeting, there was some discussion about "Mantle" [0], a "a programmable metadata balancer built into the MDS" which would use Lua for controlling the load balancing policies. I wanted to raise a few questions that are important from the testing/release/maintenance point of view in hopes that we can clarify what the best path is moving forward and how we can have a robust support for this feature. Using Python to create a good portion of the tooling for Ceph, we have found there is a constant struggle for testing: * virtualenvs can (and do) break * Python packaging can be cumbersome to work with and the Ceph source has a good amount of workarounds to provide an easier path for development) * Python tools in the Ceph tree are not real Python packages which forces the whole tooling to workaround this as well * Dependencies are difficult to deal with (ceph-disk has only one: argparse) There is a tendency to embed dependencies in the Ceph project which is a big problem for maintainers (there are many reasons for this, including security patches, and keeping up to date with forks). For Mantle, this is the case again: it depends on a Pull Request [1] that embeds a *forked* version of the official Lua repository that includes custom changes. Here are a few concerns I have: * The fork is already modified and has diverged from the upstream Lua [2], who is going to keep up with upstream changes? (pulling/rebasing etc...) * Who (or whom) is going to be responsible for keeping track of CVE's and their corresponding patches? * Embedding Lua in this form makes it *impossible* for distro maintainers to package Ceph: they would most likely rip out and figure out how to try and make it work with the distro-provided version. *This is very frowned upon and creates a gigantic amount of work for package maintainers*. * Modifying the make files to accommodate for this is fine, but that is a tiny corner of the impact this would have on everything else that deals with Ceph. These would certainly need attention (and there are lots more): - Continuous Integration scripts (make check) - Release scripts and process/workflow - Testing (both functional and unit tests) - Packaging rules Even if the Lua interpreter was out-of-tree, my concern is how to provide these patches (e.g. the cmsgpack module) so that it works in the expected manner? Is it not possible to make this a completely separate component of Ceph that doesn't require the project to embed the modified forked or depend in any way with the modifications done to the Lua interpreter? [0] Mantle BluePrint https://github.com/michaelsevilla/mantle-popper/wiki/Blueprint [1] Lua object class support Pull Request https://github.com/ceph/ceph/pull/7338 [2] https://github.com/ceph/lua/commits/lua-5.3-ceph -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
