I wiped all the cephadm stuff off the nodes when I flipped them back to direct installations then rebooted as part of that process, so all the system logs were rotated and the messages are lost. Here is the one I posted earlier from the Rocky computer trying to run Octopus monitor container ( it could run the mgr container though, which was odd to me ): 2022-06-25 21:51:34,427 7f4748727b80 DEBUG stat: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2022-06-25 21:51:34,647 7f4748727b80 DEBUG stat: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2022-06-25 21:51:34,652 7f4748727b80 DEBUG stat: Copying blob sha256:731c3beff4deece7d4e54bc26ecf6d99988b19ea8414524277d83bc5a5d6eb70 2022-06-25 21:51:59,006 7f4748727b80 DEBUG stat: Copying config sha256:2cf504fded3980c76b59a354fca8f301941f86e369215a08752874d1ddb69b73 2022-06-25 21:51:59,008 7f4748727b80 DEBUG stat: Writing manifest to image destination 2022-06-25 21:51:59,008 7f4748727b80 DEBUG stat: Storing signatures 2022-06-25 21:51:59,239 7f4748727b80 DEBUG stat: 167 167 2022-06-25 21:51:59,703 7f4748727b80 DEBUG /usr/bin/ceph-mon: too many arguments: [--default-log-to-journald=true,--default-mon-cluster-log-to-journald=true] 2022-06-25 21:51:59,797 7f4748727b80 INFO Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=tpixmon5 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/33ca8009-79d6-45cf-a67e-9753ab4dc861:/var/log/ceph:z -v /var/lib/ceph/33ca8009-79d6-45cf-a67e-9753ab4dc861/mon.tpixmon5:/var/lib/cep h/mon/ceph-tpixmon5:z -v /tmp/ceph-tmp7xmra8lk:/tmp/keyring:z -v /tmp/ceph-tmp7mid2k57:/tmp/config:z docker.io/ceph/ceph:v15 --mkfs -i tpixmon5 --fsid 33ca8009-79d6-45cf-a67e-9753ab4dc861 -c /tmp/config --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false 2022-06-25 21:51:59,798 7f4748727b80 INFO /usr/bin/ceph-mon: stderr too many arguments: [--default-log-to-journald=true,--default-mon-cluster-log-to-journald=true] I didn't capture the overlay one from the centos 7 machine but the only message had to do with the overlay. My search history showed this: "overlayfs: unrecognized mount option "volatile" or missing value". I also tried to load podman 3.1 which I cobbled together from around the internet and it didn't work with the Octopus containers :( -Brent -----Original Message----- From: Nico Schottelius <nico.schottelius@xxxxxxxxxxx> Sent: Monday, August 8, 2022 3:09 AM To: Brent Kennedy <bkennedy@xxxxxxxxxx> Cc: ceph-users@xxxxxxx Subject: Re: Re: Upgrade paths beyond octopus on Centos7 Hey Brent, thanks a lot for following up on this. Would it be possible to send the error messsages that you get in both cases? While I do have my reservations about cephadm (based on experience with ceph-deploy, ceph-ansible and friends), I would like to drill down to the core of the problem, as containers *should* indeed be running on "any" CRI. If they don't, I would expect the usage of parameters that are unknown to either podman version, however being in the container specification and unrelated to the actual image. Do you mind posting the cephadm and podman versions and the corresponding error messages that you have received with Octopus / Quincy? Best regards, Nico "Brent Kennedy" <bkennedy@xxxxxxxxxx> writes: > All I can say is that its been impossible this past month to upgrade > past octopus using cephadm on centos 7. I thought if I spun up new > servers and started containers on those using the Octopus cephadm script, I would be ok. > But both Rocky and Centos 8 stream wont run the older Octopus containers. > When the containers start on podman 4, they show an error regarding groups. > Searching on the web for that error only returns posts saying you can > ignore it, but the container/service wont start. I thought upgrading > to quincy would solve this, but then the quincy containers wont run on > centos 7, they throw an overlay error. Which is how I ended up with > cluster that was limping along with one monitor and 132 OSDs. Just > today, I went back and manually installed ceph octopus on all the > nodes(bare metal install) and that got me back to working again. > Based on another post, it seems the best way to proceed from here is > to upgrade the remaining centos 7 servers to centos stream 8 or > wipe/install rocky and load octopus bare metal. Then once that is > done, upgrade to quincy as bare metal. Final step would be then > moving to containers(cephadm). Unfortunately, I had already adopted > all the OSD containers, so hopefully I can swap them back to bare metal without too much hassle. > > This podman issue basically shows the flaw in the thinking that > containers solve the OS issue( I ran into this with Docker and > mesosphere, so I kinda knew what I was in for ). As much as I > appreciate the Dev team here at ceph and like container methodology, > the way this went down is a shame ( unless I am missing something ). > I only held back upgrading because of the lack of upgrade path and > then the centos stream situation, we normally upgrade things within 6 > months of release. BTW, I tried to upgrade centos 7 to stream 8 and > it said all the ceph modules conflicted with upgrade components, thus > I had to remove them, hence why I am starting fresh with each machine( its also quicker with VM images, at least for the VMs ). > > The upgrade path discussion I am referring to is titled: "Migration > from CentOS7/Nautilus to CentOS Stream/Pacific" > > -Brent > > -----Original Message----- > From: Marc <Marc@xxxxxxxxxxxxxxxxx> > Sent: Sunday, August 7, 2022 5:25 AM > To: Nico Schottelius <nico.schottelius@xxxxxxxxxxx>; > ceph-users@xxxxxxx > Subject: Re: Upgrade paths beyond octopus on Centos7 > > >> Reading your mails I am double puzzled, as I thought that cephadm >> would actually solve these kind of issues in the first place and I >> would > > It is being advocated like this. My opinion is, that it is primarily > being used as a click next next install tool so a broader audience can be reached. > If the focus is on this, problems such as the one below are imminent. > >> expect it to be be especially stable on RH/Centos. > > I thought I would give CentOS 9 stream a chance upgrading the office server. > Converting applications to containers, so I am less dependant in the > future on the os. On the 10th day or so some container crashed, > crashed the whole server, and then strangely enough all containers > would not start because of a little damage in one container layer (not > shared with others) of the new container. > Unfortunately mounted all on the root fs, so I had to do fsck of the > root fs. > > Afaik is podman also a fork of the docker code, and fwiiw there have > developers that coded there own containerizer because they thought the > docker implementation was not stable. > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an > email to ceph-users-leave@xxxxxxx > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an > email to ceph-users-leave@xxxxxxx -- Sustainable and modern Infrastructures by ungleich.ch _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |