I run a small ceph cluster(3 mon on 3 node, 7 osd on 2 node) at home with custom setup, and I think cephadm is the future, so I want to convert this cluster to cephadm. My cluster setup is complex compare to standard deployment, the cluster is created in early days, so the it is deployed manually, and later I make all ceph daemons run inside container(using ceph/daemon) with podman to decouple with the host system(is NixOS), and manage container startup with NixOS using systemd service(service file is generated with nix expression). I think some OS files need to be mutable to make cephadm work properly, for example, /etc/ceph/ceph.conf need to be writable by cephadm. This is how we config most Linux distros, but not NixOS, which is basically all system files is immutable, include /etc. So I plan to run cephadm in a container, with "--privileged=true" and "--net=host", and ssh listen on port '23' to avoid conflict with host, and create a dummy 'ntp.service' which only run 'sleep inf' to cheat cephadm, because I have chrony on host system. Maybe /dev need to bind mount from host. I have already build the image and successfully run 'cephadm check-host' in the container. Official document for cephadm adoption process(https://docs.ceph.com/en/latest/cephadm/adoption/) lack details so I am not sure whether my unusual setup cluster can be convert successfully or not. so I need some suggestion for further steps of convertion. Below is some details of what I have already done: Dockerfile: ``` FROM fedora:36 RUN dnf -y install \ systemd openssh-server openssh-clients cephadm podman containernetworking-plugins && \ dnf clean all RUN ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa && \ ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519 && \ sed -i -e 's/^.*pam_loginuid.so.*$/session optional pam_loginuid.so/' /etc/pam.d/sshd && \ sed -i -e 's/^.*Port 22/Port 23/' /etc/ssh/sshd_config EXPOSE 23 RUN (for i in \ systemd-network-generator.service \ rpmdb-migrate.service \ rpmdb-rebuild.service \ getty@tty1.service \ remote-fs.target \ systemd-resolved.service \ systemd-oomd.service \ systemd-network-generator.service \ dnf-makecache.timer \ fstrim.timer; do \ rm -f /etc/systemd/system/*.wants/$i; \ done) COPY ./ntp.service /etc/systemd/system RUN (cd /etc/systemd/system/multi-user.target.wants; ln -s ../ntp.service) RUN mkdir -p /etc/ceph && \ mkdir -p /var/lib/containers && \ mkdir -p /var/lib/ceph && \ mkdir -p /var/log/ceph && \ mkdir -p /root/.ssh && chown 700 /root/.ssh VOLUME /etc/ceph VOLUME /var/lib/containers VOLUME /var/lib/ceph VOLUME /var/log/ceph VOLUME /root/.ssh CMD ["/sbin/init"] ``` and below is ntp.service file: ``` [Unit] After=network.target [Service] ExecStart=/bin/sleep inf Restart=always Type=simple ``` I start tag the image build from above Dockerfile with name 'cephadm', and "--security-opt=seccomp=unconfined" option is necessary for podman build to work. Then I start container with below script: ``` #!/bin/sh mkdir -p /var/log/ceph mkdir -p /etc/ceph/ssh podman run --rm -d \ --net=host \ --privileged=true \ --name=cephadm \ -v /var/lib/containers:/var/lib/containers:z \ -v /var/lib/ceph:/var/lib/ceph:z \ -v /var/log/ceph:/var/log/ceph:z \ -v /etc/ceph:/etc/ceph:z \ -v /etc/ceph/ssh:/root/.ssh:z \ cephadm ``` finally run "podman exec -it cephadm cephadm host-check" will generate below output: ``` podman (/usr/bin/podman) version 4.0.3 is present systemctl is present lvcreate is present Unit ntp.service is enabled and running Host looks OK ``` and logs in /var/log/ceph/cephadm.log is also looks good. -- Best wishes ~ _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |