On Tue, Jan 18, 2022 at 1:14 PM Mark Nelson <mnelson@xxxxxxxxxx> wrote: > > > On 1/18/22 2:56 PM, Josh Durgin wrote: > > Hey folks, here's some background on a couple ideas for developing > > ceph with local container images that we were talking about in recent > > teuthology meetings. The goal is to be able to run any teuthology test > > locally, just like you would kick off a suite in the sepia lab. > > Junior's teuthology dev setup is the first step towards this: > > https://github.com/ceph/teuthology/pull/1665 > > <https://github.com/ceph/teuthology/pull/1665> > > > > One aspect of this is generating container images from local builds. > > There are a couple ways to do this that save hours by avoiding > > the usual package + container build process: > > > > 1) Copying everything from a build into an existing container image > > > > This is what the cpatch/cstart scripts do - > > https://docs.ceph.com/en/pacific/dev/cephadm/developing-cephadm/#cstart-and-cpatch > > <https://docs.ceph.com/en/pacific/dev/cephadm/developing-cephadm/#cstart-and-cpatch> > > > > If teuthology could use these images, this could be a fast path from > > local testing to test runs, even without other pieces. No need to push > > to ceph-ci and wait for any package or complete container builds. This > > would require getting the rest of teuthology tests using containers as > > well - some pieces like clients are still using packages even in > > cephadm-based tests. > > > > 2) Bind mounting/symlinking/otherwise referencing existing files from > > a container image that doesn't need updating with each build > > > > I prototyped this when Sage created cpatch and tried to automate > > listing which files to copy or reference: > > https://github.com/ceph/ceph/pull/34328#issuecomment-608926509 > > <https://github.com/ceph/ceph/pull/34328#issuecomment-608926509> > > https://gist.github.com/jdurgin/3215df1295d7bfee0e91b203eae71dce > > <https://gist.github.com/jdurgin/3215df1295d7bfee0e91b203eae71dce> > > > > Essentially both of these approaches skip hours of the package and > > container build process by modifying existing containers. There are a > > couple caveats: > > > > - the scripts would need adjustment whenever new installed files are > > added that doesn't match an existing regex. This is similar to what we > > need to do when adding new files to packaging, so it's not a big deal, > > but is something to be aware of. > > > > - the build would need to be in the same environment as the container > > - these are all centos 8 stream currently. This could be done > > directly, by developing in a shell in the container, or with wrapper > > scripts hiding that detail from you. > > > > A third approach would be entirely redoing the way ceph containers are > > created to not rely on packages. This would effectively get us > > approach (1), however it would be a massive effort and not worth it > > imo. The same caveats as above would apply to this too. > > > > Any other ideas? Thoughts on this? > > > Does the container build process take significantly longer than the > normal rpm build process? I've seen some references to things like > buildah and (docker) build caches being able to speed up container > creation, but I don't really know anything about it. My preference > would probably be to just take some preexisting base container image and > compile/dump ceph into a /usr/local equivalent given that the container > is sort of the package anyway already, but I imagine that might not be > super popular. ;) That's pretty much what these scripts do, but they put things in the same place as usual. The containers are built based on rpms, so it's strictly longer than package builds. _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
