We then moved on to running teuthology in non-sepia labs. It turns out all four companies present have done this, with varying levels of trouble. The biggest issue at an institutional level is just that the teuthology code base still embeds a lot of assumptions about accessing the sepia lab — even after the work to support running in openstack clouds, it expects to be querying ceph.com subdomains for packages and a bunch of other things, and many of them are hard-coded into the source rather than being configurable. (Even leaving aside the teuthology core code, I know that I have certainly written tests that pull random tarballs from ceph.com/qa, and although that ought to function it means you need external network bandwidth sufficient to download them on every run!) So far every group has done the tedious patching themselves to change this. Hopefully some of them can share the patches they’ve made so we can get an idea of the problem size, and if nobody else volunteers sooner the next group who needs an install can make it configurable instead of simply swapping out URLs? :) PROBLEM TOPIC: replace hard-coded Sepia lab URLs with configurables One of the bigger-picture questions I had was whether we need to rethink the way we use or build teuthology, given the number of components it has these days (Shaman, Pulpito, beanstalkd for job scheduling, Jenkins for building and some other stuff, I think a “Chacra” but I don’t know what for, etc). It would be great if we could share resources with other open-source projects doing similar things. After discussion, it sounds like for institutions it is not a problem to install and keep those pieces running (at least, in comparison to the other challenges of actually developing and testing Ceph). I asked specifically about replacing beanstalkd, since I know that’s been a long-term todo item and when there is lab contention that is not a sufficient scheduler. The room seemed to think that might be nice but isn’t a big issue unless the lab doesn’t have enough capacity for the testing we need. Moreover, any replacement would have to be a full job scheduling system and that would require a lot more (teuthology-based) custom coding and (site-based) configuration. We would like to do more test framework sharing with other groups, but the gains for us are limited unless we manage to build out a serious community. The Sepia maintainers have explored sharing with the CentOS CI system in the past, and Joao brought up OpenQA, but all the systems we’ve explored so far don’t handle the build volume we need and mostly replicate smaller components of our overall system. (Both of those projects are focused on building existing packages, installing them on a single node, and making sure they can be turned on; they don’t handle cross-system orchestration, deep testing, or any kind of job scheduling beyond “there is a machine to build the package now”.) John suggested the best areas for collaboration are around building packages and displaying test results if we want to explore that, but in terms of distributed testing we would be donating far more time than we could plausibly get back. -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
