On 1/18/22 2:56 PM, Josh Durgin wrote:
Hey folks, here's some background on a couple ideas for developing
ceph with local container images that we were talking about in recent
teuthology meetings. The goal is to be able to run any teuthology test
locally, just like you would kick off a suite in the sepia lab.
Junior's teuthology dev setup is the first step towards this:
https://github.com/ceph/teuthology/pull/1665
<https://github.com/ceph/teuthology/pull/1665>
One aspect of this is generating container images from local builds.
There are a couple ways to do this that save hours by avoiding
the usual package + container build process:
1) Copying everything from a build into an existing container image
This is what the cpatch/cstart scripts do -
https://docs.ceph.com/en/pacific/dev/cephadm/developing-cephadm/#cstart-and-cpatch
<https://docs.ceph.com/en/pacific/dev/cephadm/developing-cephadm/#cstart-and-cpatch>
If teuthology could use these images, this could be a fast path from
local testing to test runs, even without other pieces. No need to push
to ceph-ci and wait for any package or complete container builds. This
would require getting the rest of teuthology tests using containers as
well - some pieces like clients are still using packages even in
cephadm-based tests.
2) Bind mounting/symlinking/otherwise referencing existing files from
a container image that doesn't need updating with each build
I prototyped this when Sage created cpatch and tried to automate
listing which files to copy or reference:
https://github.com/ceph/ceph/pull/34328#issuecomment-608926509
<https://github.com/ceph/ceph/pull/34328#issuecomment-608926509>
https://gist.github.com/jdurgin/3215df1295d7bfee0e91b203eae71dce
<https://gist.github.com/jdurgin/3215df1295d7bfee0e91b203eae71dce>
Essentially both of these approaches skip hours of the package and
container build process by modifying existing containers. There are a
couple caveats:
- the scripts would need adjustment whenever new installed files are
added that doesn't match an existing regex. This is similar to what we
need to do when adding new files to packaging, so it's not a big deal,
but is something to be aware of.
- the build would need to be in the same environment as the container
- these are all centos 8 stream currently. This could be done
directly, by developing in a shell in the container, or with wrapper
scripts hiding that detail from you.
A third approach would be entirely redoing the way ceph containers are
created to not rely on packages. This would effectively get us
approach (1), however it would be a massive effort and not worth it
imo. The same caveats as above would apply to this too.
Any other ideas? Thoughts on this?
Does the container build process take significantly longer than the
normal rpm build process? I've seen some references to things like
buildah and (docker) build caches being able to speed up container
creation, but I don't really know anything about it. My preference
would probably be to just take some preexisting base container image and
compile/dump ceph into a /usr/local equivalent given that the container
is sort of the package anyway already, but I imagine that might not be
super popular. ;)
Josh
_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx
_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx
