On Tuesday, January 18, 2022 3:56:34 PM EST Josh Durgin wrote: > Hey folks, here's some background on a couple ideas for developing ceph > with local container images that we were talking about in recent teuthology > meetings. The goal is to be able to run any teuthology test locally, just > like you would kick off a suite in the sepia lab. Junior's teuthology dev > setup is the first step towards this: > https://github.com/ceph/teuthology/pull/1665 > > One aspect of this is generating container images from local builds. There > are a couple ways to do this that save hours by avoiding the usual > package + container build process: > > 1) Copying everything from a build into an existing container image > > This is what the cpatch/cstart scripts do - > https://docs.ceph.com/en/pacific/dev/cephadm/developing-cephadm/#cstart-and-> cpatch > > If teuthology could use these images, this could be a fast path from local > testing to test runs, even without other pieces. No need to push to ceph-ci > and wait for any package or complete container builds. This would require > getting the rest of teuthology tests using containers as well - some pieces > like clients are still using packages even in cephadm-based tests. > > 2) Bind mounting/symlinking/otherwise referencing existing files from a > container image that doesn't need updating with each build > > I prototyped this when Sage created cpatch and tried to automate listing > which files to copy or reference: > https://github.com/ceph/ceph/pull/34328#issuecomment-608926509 > https://gist.github.com/jdurgin/3215df1295d7bfee0e91b203eae71dce > > Essentially both of these approaches skip hours of the package and > container build process by modifying existing containers. There are a > couple caveats: > > - the scripts would need adjustment whenever new installed files are added > that doesn't match an existing regex. This is similar to what we need to do > when adding new files to packaging, so it's not a big deal, but is > something to be aware of. > > - the build would need to be in the same environment as the container - > these are all centos 8 stream currently. This could be done directly, by > developing in a shell in the container, or with wrapper scripts hiding that > detail from you. > > A third approach would be entirely redoing the way ceph containers are > created to not rely on packages. This would effectively get us approach > (1), however it would be a massive effort and not worth it imo. The same > caveats as above would apply to this too. > > Any other ideas? Thoughts on this? As someone who's only recently started contributing to cephadm but familiar with containers a while now this idea is very appealing to me. I have used the cpatch tool a few times already and it has worked well for the things I was testing, despite a few small workarounds that I needed to do. Regarding building the binaries on a platform matching that of the ceph containers. One approach that may work is to use a multi stage build. Eg: ``` FROM centos:8 AS builder ARG BUILD_OPTIONS=default # do build stuff, with the requiremnt that the dev's working # tree is mounted into the container FROM ceph-base COPY --from builder <various artifacts> # "patch" the container similarly to what cpatch does today ``` I agree that a wrapper tool such as cpatch or something like it would be desirable to hide some of the complexities of setting up the container, but the result would be very useful. _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
