Hi all, we had to bring the OSDs back up prior to an upgrade from octopus to pacific. Unfortunately, instructions we found for an off-line update of the osdmap did not work. We first tried a command like ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-556/ --op set-osdmap --file osd.map but this gave an error; see full session log below. We found that --force should force the update and it "worked" without an error message. However, when we started the OSDs they ignored the injected OSD map and started to get into their endless "try to catch up" loop spamming the log partition to death. In the end we did a zap+recreate and got them up. Afterwards we found that we might have missed an important flag. In tracker https://tracker.ceph.com/issues/15520 the command also has the --epoch flag that we overlooked (and doesn't use the --file flag): ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-556/ –op set-osdmap <osdmap-file> --epoch <epoch-number> --force It would be great if someone could provide a correct and complete example for how to set an osdmap with ceph-objectstore-tool, the documentation at https://docs.ceph.com/en/pacific/man/8/ceph-objectstore-tool/#description (and also for latest) is "a bit sparse" here. Working examples for such operations would be great in the docs. My best bet after failing is: ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-556/ --op set-osdmap --file osd.map --epoch <epoch-number> [--force] Can't confirm though. Here our session log: [root@ceph-adm:ceph-13 ~]# ceph-objectstore-tool --no-mon-config --data-path /var/lib/ceph/osd/ceph-1000/ --op set-osdmap --file osd.map osdmap (#-1:bdf1d99b:::osdmap.3145505:0#) does not exist. [root@ceph-adm:ceph-13 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1000/ --op set-osdmap --file osd.map osdmap (#-1:bdf1d99b:::osdmap.3145505:0#) does not exist. [root@ceph-adm:ceph-13 ~]# ceph osd getmap -o osd.map got osdmap epoch 3145529 [root@ceph-adm:ceph-13 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1000/ --op set-osdmap --file osd.map osdmap (#-1:b9b9d99b:::osdmap.3145529:0#) does not exist. [root@ceph-adm:ceph-13 ~]# echo $? 1 [root@ceph-adm:ceph-13 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1000/ --op get-osdmap --file osd.map.2 osdmap#2971464 exported. [root@ceph-adm:ceph-13 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1000/ --op set-osdmap --file osd.map osdmap (#-1:b9b9d99b:::osdmap.3145529:0#) does not exist. [root@ceph-adm:ceph-13 ~]# ceph-objectstore-tool --force --data-path /var/lib/ceph/osd/ceph-1000/ --op set-osdmap --file osd.map osdmap (#-1:b9b9d99b:::osdmap.3145529:0#) does not exist. Creating a new epoch. [root@ceph-adm:ceph-13 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1000/ --op get-osdmap --file osd.map.2 osdmap#2971464 exported. [root@ceph-adm:ceph-13 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1000/ --op set-osdmap --file osd.map --force [root@ceph-adm:ceph-13 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1000/ --op get-osdmap --file osd.map.2 osdmap#2971464 exported. [root@ceph-adm:ceph-13 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1000/ --op set-osdmap --file osd.map [root@ceph-adm:ceph-13 ~]# ceph osd getmap -o osd.map got osdmap epoch 3145556 [root@ceph-adm:ceph-13 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1004/ --op set-osdmap --file osd.map --force osdmap (#-1:bc85d99b:::osdmap.3145556:0#) does not exist. Creating a new epoch. [root@ceph-adm:ceph-13 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-990/ --op set-osdmap --file osd.map --force osdmap (#-1:bc85d99b:::osdmap.3145556:0#) does not exist. Creating a new epoch. [root@ceph-adm:ceph-13 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-995/ --op set-osdmap --file osd.map --force osdmap (#-1:bc85d99b:::osdmap.3145556:0#) does not exist. Creating a new epoch. [root@ceph-adm:ceph-13 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1000/ --op set-osdmap --file osd.map --force osdmap (#-1:bc85d99b:::osdmap.3145556:0#) does not exist. Creating a new epoch. [root@ceph-adm:ceph-13 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1004/ --op set-osdmap --file osd.map --force Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |