Re: ceph containers for a faster dev + test cycle

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




On 1/18/22 2:56 PM, Josh Durgin wrote:
Hey folks, here's some background on a couple ideas for developing ceph with local container images that we were talking about in recent teuthology meetings. The goal is to be able to run any teuthology test locally, just like you would kick off a suite in the sepia lab. Junior's teuthology dev setup is the first step towards this: https://github.com/ceph/teuthology/pull/1665 <https://github.com/ceph/teuthology/pull/1665>

One aspect of this is generating container images from local builds. There are a couple ways to do this that save hours by avoiding the usual package + container build process:

1) Copying everything from a build into an existing container image

This is what the cpatch/cstart scripts do - https://docs.ceph.com/en/pacific/dev/cephadm/developing-cephadm/#cstart-and-cpatch <https://docs.ceph.com/en/pacific/dev/cephadm/developing-cephadm/#cstart-and-cpatch>

If teuthology could use these images, this could be a fast path from local testing to test runs, even without other pieces. No need to push to ceph-ci and wait for any package or complete container builds. This would require getting the rest of teuthology tests using containers as well - some pieces like clients are still using packages even in cephadm-based tests.

2) Bind mounting/symlinking/otherwise referencing existing files from a container image that doesn't need updating with each build

I prototyped this when Sage created cpatch and tried to automate listing which files to copy or reference: https://github.com/ceph/ceph/pull/34328#issuecomment-608926509 <https://github.com/ceph/ceph/pull/34328#issuecomment-608926509> https://gist.github.com/jdurgin/3215df1295d7bfee0e91b203eae71dce <https://gist.github.com/jdurgin/3215df1295d7bfee0e91b203eae71dce>

Essentially both of these approaches skip hours of the package and container build process by modifying existing containers. There are a couple caveats:

- the scripts would need adjustment whenever new installed files are added that doesn't match an existing regex. This is similar to what we need to do when adding new files to packaging, so it's not a big deal, but is something to be aware of.

- the build would need to be in the same environment as the container - these are all centos 8 stream currently. This could be done directly, by developing in a shell in the container, or with wrapper scripts hiding that detail from you.

A third approach would be entirely redoing the way ceph containers are created to not rely on packages. This would effectively get us approach (1), however it would be a massive effort and not worth it imo. The same caveats as above would apply to this too.

Any other ideas? Thoughts on this?


Does the container build process take significantly longer than the normal rpm build process?  I've seen some references to things like buildah and (docker) build caches being able to speed up container creation, but I don't really know anything about it.  My preference would probably be to just take some preexisting base container image and compile/dump ceph into a /usr/local equivalent given that the container is sort of the package anyway already, but I imagine that might not be super popular. ;)



Josh

_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx

_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx




[Index of Archives]     [CEPH Users]     [Ceph Devel]     [Ceph Large]     [Information on CEPH]     [Linux BTRFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux