Hey folks, here's some background on a couple ideas for developing ceph with local container images that we were talking about in recent teuthology meetings. The goal is to be able to run any teuthology test locally, just like you would kick off a suite in the sepia lab. Junior's teuthology dev setup is the first step towards this: https://github.com/ceph/teuthology/pull/1665
One aspect of this is generating container images from local builds. There are a couple ways to do this that save hours by avoiding the usual package + container build process:
1) Copying everything from a build into an existing container image
This is what the cpatch/cstart scripts do - https://docs.ceph.com/en/pacific/dev/cephadm/developing-cephadm/#cstart-and-cpatch
If teuthology could use these images, this could be a fast path from local testing to test runs, even without other pieces. No need to push to ceph-ci and wait for any package or complete container builds. This would require getting the rest of teuthology tests using containers as well - some pieces like clients are still using packages even in cephadm-based tests.
2) Bind mounting/symlinking/otherwise referencing existing files from a container image that doesn't need updating with each build
I prototyped this when Sage created cpatch and tried to automate listing which files to copy or reference:
https://github.com/ceph/ceph/pull/34328#issuecomment-608926509
This is what the cpatch/cstart scripts do - https://docs.ceph.com/en/pacific/dev/cephadm/developing-cephadm/#cstart-and-cpatch
If teuthology could use these images, this could be a fast path from local testing to test runs, even without other pieces. No need to push to ceph-ci and wait for any package or complete container builds. This would require getting the rest of teuthology tests using containers as well - some pieces like clients are still using packages even in cephadm-based tests.
2) Bind mounting/symlinking/otherwise referencing existing files from a container image that doesn't need updating with each build
I prototyped this when Sage created cpatch and tried to automate listing which files to copy or reference:
https://github.com/ceph/ceph/pull/34328#issuecomment-608926509
Essentially both of these approaches skip hours of the package and container build process by modifying existing containers. There are a couple caveats:
- the scripts would need adjustment whenever new installed files are added that doesn't match an existing regex. This is similar to what we need to do when adding new files to packaging, so it's not a big deal, but is something to be aware of.
- the build would need to be in the same environment as the container - these are all centos 8 stream currently. This could be done directly, by developing in a shell in the container, or with wrapper scripts hiding that detail from you.
A third approach would be entirely redoing the way ceph containers are created to not rely on packages. This would effectively get us approach (1), however it would be a massive effort and not worth it imo. The same caveats as above would apply to this too.
Any other ideas? Thoughts on this?
Josh
_______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
