On Wed, Jan 19, 2022 at 11:32 AM John Mulligan <jmulliga@xxxxxxxxxx> wrote: > > On Tuesday, January 18, 2022 3:56:34 PM EST Josh Durgin wrote: > > Hey folks, here's some background on a couple ideas for developing ceph > > with local container images that we were talking about in recent teuthology > > meetings. The goal is to be able to run any teuthology test locally, just > > like you would kick off a suite in the sepia lab. Junior's teuthology dev > > setup is the first step towards this: > > https://github.com/ceph/teuthology/pull/1665 > > > > One aspect of this is generating container images from local builds. There > > are a couple ways to do this that save hours by avoiding the usual > > package + container build process: > > > > 1) Copying everything from a build into an existing container image > > > > This is what the cpatch/cstart scripts do - > > https://docs.ceph.com/en/pacific/dev/cephadm/developing-cephadm/#cstart-and-> cpatch > > > > If teuthology could use these images, this could be a fast path from local > > testing to test runs, even without other pieces. No need to push to ceph-ci > > and wait for any package or complete container builds. This would require > > getting the rest of teuthology tests using containers as well - some pieces > > like clients are still using packages even in cephadm-based tests. > > > > 2) Bind mounting/symlinking/otherwise referencing existing files from a > > container image that doesn't need updating with each build > > > > I prototyped this when Sage created cpatch and tried to automate listing > > which files to copy or reference: > > https://github.com/ceph/ceph/pull/34328#issuecomment-608926509 > > https://gist.github.com/jdurgin/3215df1295d7bfee0e91b203eae71dce > > > > Essentially both of these approaches skip hours of the package and > > container build process by modifying existing containers. There are a > > couple caveats: > > > > - the scripts would need adjustment whenever new installed files are added > > that doesn't match an existing regex. This is similar to what we need to do > > when adding new files to packaging, so it's not a big deal, but is > > something to be aware of. > > > > - the build would need to be in the same environment as the container - > > these are all centos 8 stream currently. This could be done directly, by > > developing in a shell in the container, or with wrapper scripts hiding that > > detail from you. > > > > A third approach would be entirely redoing the way ceph containers are > > created to not rely on packages. This would effectively get us approach > > (1), however it would be a massive effort and not worth it imo. The same > > caveats as above would apply to this too. > > > > Any other ideas? Thoughts on this? > > > As someone who's only recently started contributing to cephadm but familiar > with containers a while now this idea is very appealing to me. > I have used the cpatch tool a few times already and it has worked well for the > things I was testing, despite a few small workarounds that I needed to do. > > Regarding building the binaries on a platform matching that of the ceph > containers. One approach that may work is to use a multi stage build. Eg: > ``` > FROM centos:8 AS builder > ARG BUILD_OPTIONS=default > > # do build stuff, with the requiremnt that the dev's working > # tree is mounted into the container > > FROM ceph-base > > COPY --from builder <various artifacts> > > # "patch" the container similarly to what cpatch does today > ``` Interesting, using a base container like that sounds helpful. It reminds me of another possible benefit of a container-based dev environment: we could have pre-built container images of a dev environment. If we built these periodically with master, you would have a lot less to build when making changes - just the incremental pieces since the last dev image was created. Why spend all that developer hardware time on rebuilding everything when we're building master centrally already? _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
