Josh, in case it helps: at Dashboard team we're nightly-building Ceph container images from the latest Shaman RPMs for dev purposes: https://github.com/rhcs-dashboard/ceph-dev/actions/runs/1726646812
For Python-only (ceph-mgr) development it's enough to mount the external src/pybind/mgr and voilà you can interactively test your changes in a running container. For compiled code, you could also mount individual files instead.
Kind Regards,
ErnestoOn Thu, Jan 20, 2022 at 11:57 PM Josh Durgin <jdurgin@xxxxxxxxxx> wrote:
On Wed, Jan 19, 2022 at 11:32 AM John Mulligan <jmulliga@xxxxxxxxxx> wrote:
>
> On Tuesday, January 18, 2022 3:56:34 PM EST Josh Durgin wrote:
> > Hey folks, here's some background on a couple ideas for developing ceph
> > with local container images that we were talking about in recent teuthology
> > meetings. The goal is to be able to run any teuthology test locally, just
> > like you would kick off a suite in the sepia lab. Junior's teuthology dev
> > setup is the first step towards this:
> > https://github.com/ceph/teuthology/pull/1665
> >
> > One aspect of this is generating container images from local builds. There
> > are a couple ways to do this that save hours by avoiding the usual
> > package + container build process:
> >
> > 1) Copying everything from a build into an existing container image
> >
> > This is what the cpatch/cstart scripts do -
> > https://docs.ceph.com/en/pacific/dev/cephadm/developing-cephadm/#cstart-and-> cpatch
> >
> > If teuthology could use these images, this could be a fast path from local
> > testing to test runs, even without other pieces. No need to push to ceph-ci
> > and wait for any package or complete container builds. This would require
> > getting the rest of teuthology tests using containers as well - some pieces
> > like clients are still using packages even in cephadm-based tests.
> >
> > 2) Bind mounting/symlinking/otherwise referencing existing files from a
> > container image that doesn't need updating with each build
> >
> > I prototyped this when Sage created cpatch and tried to automate listing
> > which files to copy or reference:
> > https://github.com/ceph/ceph/pull/34328#issuecomment-608926509
> > https://gist.github.com/jdurgin/3215df1295d7bfee0e91b203eae71dce
> >
> > Essentially both of these approaches skip hours of the package and
> > container build process by modifying existing containers. There are a
> > couple caveats:
> >
> > - the scripts would need adjustment whenever new installed files are added
> > that doesn't match an existing regex. This is similar to what we need to do
> > when adding new files to packaging, so it's not a big deal, but is
> > something to be aware of.
> >
> > - the build would need to be in the same environment as the container -
> > these are all centos 8 stream currently. This could be done directly, by
> > developing in a shell in the container, or with wrapper scripts hiding that
> > detail from you.
> >
> > A third approach would be entirely redoing the way ceph containers are
> > created to not rely on packages. This would effectively get us approach
> > (1), however it would be a massive effort and not worth it imo. The same
> > caveats as above would apply to this too.
> >
> > Any other ideas? Thoughts on this?
>
>
> As someone who's only recently started contributing to cephadm but familiar
> with containers a while now this idea is very appealing to me.
> I have used the cpatch tool a few times already and it has worked well for the
> things I was testing, despite a few small workarounds that I needed to do.
>
> Regarding building the binaries on a platform matching that of the ceph
> containers. One approach that may work is to use a multi stage build. Eg:
> ```
> FROM centos:8 AS builder
> ARG BUILD_OPTIONS=default
>
> # do build stuff, with the requiremnt that the dev's working
> # tree is mounted into the container
>
> FROM ceph-base
>
> COPY --from builder <various artifacts>
>
> # "patch" the container similarly to what cpatch does today
> ```
Interesting, using a base container like that sounds helpful. It
reminds me of another possible benefit of a container-based dev
environment: we could have pre-built container images of a dev
environment. If we built these periodically with master, you
would have a lot less to build when making changes - just the
incremental pieces since the last dev image was created. Why
spend all that developer hardware time on rebuilding everything
when we're building master centrally already?
_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx
_______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
