Hello, More info about this. Am running ubuntu 18.04. Looks all the servers with the osd the issue has double entry of ceph-volume in systemd. I dont know what caused double entry in some servers Eg /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-30-4589d067-cdb7-44de-a484-497ca13cc32d.service /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-30-8a70ad95-1c79-4502-a9a3-d5d7b9df84b6.service With one server i managed to bring osd back online by deleting wrong entry and running ceph-volume lvm activate --all before rebooting the server With Another server osds are not coming online because i deleted the wrong entry and then rebooted the server. then i run ceph-volume lvm activate --all after reboot but osds cant come up Anyone with way around this? On Tue, Mar 31, 2020 at 5:39 PM Lomayani S. Laizer <lomlaizer@xxxxxxxxx> wrote: > Hello, > I have upgraded nautilus cluster to octopus few days ago. the cluster was > running ok and even after to octopus everything was running ok > > the issue came when i rebooted the servers for updating the kernel. Two > servers out of 6 osd's servers osd cant start. No error reported in > ceph-volume.log and ceph-volume-systemd.log > > Starting osd with /usr/bin/ceph-osd -f --cluster ceph --id 30 --setuser > ceph --setgroup ceph works just fine. the issue is starting osd in systemd > > ceph-volume-systemd.log > > 16:36:28,193][systemd][WARNING] failed activating OSD, retries left: 30 > [2020-03-31 16:36:28,196][systemd][WARNING] command returned non-zero exit > status: 1 > [2020-03-31 16:36:28,196][systemd][WARNING] failed activating OSD, retries > left: 30 > [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: > lvm-28-7f4113c8-c5cf-4f70-9f7a-7a32de9d6587 > [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: > lvm-30-8a70ad95-1c79-4502-a9a3-d5d7b9df84b6 > [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: > lvm-31-a8efb7db-686b-4789-a9c4-01442c28577f > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra > data: 28-7f4113c8-c5cf-4f70-9f7a-7a32de9d6587 > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra > data: 30-8a70ad95-1c79-4502-a9a3-d5d7b9df84b6 > [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: > lvm-33-7d688fc1-ed7b-45ae-ac0e-7b1787e0b64f > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra > data: 31-a8efb7db-686b-4789-a9c4-01442c28577f > [2020-03-31 16:41:25,068][systemd][INFO ] raw systemd input received: > lvm-29-3e52d340-5416-46e6-b697-c15ca85f6883 > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra > data: 33-7d688fc1-ed7b-45ae-ac0e-7b1787e0b64f > [2020-03-31 16:41:25,068][systemd][INFO ] raw systemd input received: > lvm-32-3841a62d-d6bc-404a-8762-163530b2d5d4 > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra > data: 29-3e52d340-5416-46e6-b697-c15ca85f6883 > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra > data: 32-3841a62d-d6bc-404a-8762-163530b2d5d4 > [2020-03-31 16:41:25,108][ceph_volume.process][INFO ] Running command: > /usr/sbin/ceph-volume lvm trigger 29-3e52d340-5416-46e6-b697-c15ca85f6883 > > ceph-volume.log > 2-163530b2d5d4 > [2020-03-31 17:17:23,679][ceph_volume.process][INFO ] Running command: > /bin/systemctl enable --runtime ceph-osd@31 > [2020-03-31 17:17:23,863][ceph_volume.process][INFO ] Running command: > /bin/systemctl enable --runtime ceph-osd@30 > [2020-03-31 17:17:24,045][ceph_volume.process][INFO ] Running command: > /bin/systemctl enable --runtime ceph-osd@33 > [2020-03-31 17:17:24,241][ceph_volume.process][INFO ] Running command: > /bin/systemctl enable --runtime ceph-osd@32 > [2020-03-31 17:17:24,449][ceph_volume.process][INFO ] Running command: > /bin/systemctl enable --runtime ceph-osd@28 > [2020-03-31 17:17:24,629][ceph_volume.process][INFO ] Running command: > /bin/systemctl enable --runtime ceph-osd@29 > [2020-03-31 17:17:24,652][ceph_volume.process][INFO ] stderr Created > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@31.service → > /lib/systemd/system/ceph-osd@.service. > [2020-03-31 17:17:24,664][ceph_volume.process][INFO ] stderr Created > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@30.service → > /lib/systemd/system/ceph-osd@.service. > [2020-03-31 17:17:24,872][ceph_volume.process][INFO ] Running command: > /bin/systemctl start ceph-osd@31 > [2020-03-31 17:17:24,875][ceph_volume.process][INFO ] stderr Created > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@33.service → > /lib/systemd/system/ceph-osd@.service. > [2020-03-31 17:17:25,072][ceph_volume.process][INFO ] stderr Created > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@32.service → > /lib/systemd/system/ceph-osd@.service. > [2020-03-31 17:17:25,075][ceph_volume.process][INFO ] Running command: > /bin/systemctl start ceph-osd@30 > [2020-03-31 17:17:25,282][ceph_volume.process][INFO ] Running command: > /bin/systemctl start ceph-osd@33 > [2020-03-31 17:17:25,497][ceph_volume.process][INFO ] stderr Created > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@28.service → > /lib/systemd/system/ceph-osd@.service. > [2020-03-31 17:17:25,499][ceph_volume.process][INFO ] Running command: > /bin/systemctl start ceph-osd@32 > [2020-03-31 17:17:25,520][ceph_volume.process][INFO ] stderr Created > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@29.service → > /lib/systemd/system/ceph-osd@.service. > [2020-03-31 17:17:25,705][ceph_volume.process][INFO ] Running command: > /bin/systemctl start ceph-osd@28 > [2020-03-31 17:17:25,887][ceph_volume.process][INFO ] Running command: > /bin/systemctl start ceph-osd@29 > > -- > Lomayani > > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |