Zitat von Jens-U. Mozdzen
Hi Alfredo,
thank you for your comments:
Zitat von Alfredo Deza <adeza@xxxxxxxxxx>:
On Wed, Jan 10, 2018 at 8:57 AM, Jens-U. Mozdzen <jmozdzen@xxxxxx> wrote:
Dear *,
has anybody been successful migrating Filestore OSDs to Bluestore OSDs,
keeping the OSD number? There have been a number of messages on the list,
reporting problems, and my experience is the same. (Removing the existing
OSD and creating a new one does work for me.)
I'm working on an Ceph 12.2.2 cluster and tried following
http://docs.ceph.com/docs/master/rados/operations/add-or-rm-osds/#replacing-an-osd
- this basically says
1. destroy old OSD
2. zap the disk
3. prepare the new OSD
4. activate the new OSD
I never got step 4 to complete. The closest I got was by doing the
following
steps (assuming OSD ID "999" on /dev/sdzz):
1. Stop the old OSD via systemd (osd-node # systemctl stop
ceph-osd@999.service)
2. umount the old OSD (osd-node # umount /var/lib/ceph/osd/ceph-999)
3a. if the old OSD was Bluestore with LVM, manually clean up the old OSD's
volume group
3b. zap the block device (osd-node # ceph-volume lvm zap /dev/sdzz)
4. destroy the old OSD (osd-node # ceph osd destroy 999
--yes-i-really-mean-it)
5. create a new OSD entry (osd-node # ceph osd new $(cat
/var/lib/ceph/osd/ceph-999/fsid) 999)
Step 5 and 6 are problematic if you are going to be trying ceph-volume
later on, which takes care of doing this for you.
6. add the OSD secret to Ceph authentication (osd-node # ceph auth add
osd.999 mgr 'allow profile osd' osd 'allow *' mon 'allow profile osd' -i
/var/lib/ceph/osd/ceph-999/keyring)
I at first tried to follow the documented steps (without my steps 5
and 6), which did not work for me. The documented approach failed with
"init authentication >> failed: (1) Operation not permitted", because
actually ceph-volume did not add the auth entry for me.
But even after manually adding the authentication, the "ceph-volume"
approach failed, as the OSD was still marked "destroyed" in the osdmap
epoch as used by ceph-osd (see the commented messages from
ceph-osd.999.log below).
7. prepare the new OSD (osd-node # ceph-volume lvm prepare --bluestore
--osd-id 999 --data /dev/sdzz)
You are going to hit a bug in ceph-volume that is preventing you from
specifying the osd id directly if the ID has been destroyed.
See http://tracker.ceph.com/issues/22642
If I read that bug description correctly, you're confirming why I
needed step #6 above (manually adding the OSD auth entry. But even if
ceph-volume had added it, the ceph-osd.log entries suggest that
starting the OSD would still have failed, because of accessing the
wrong osdmap epoch.
To me it seems like I'm hitting a bug outside of ceph-volume [...]
just for the records (and search engines), this was confirmed to be a
bug, see http://tracker.ceph.com/issues/22673
Regards,
Jens
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
