First you stop the service, then make sure they're down, out, crush remove, auth del, and finally osd rm. You had it almost in the right order, but you were down and outing them before you stopped them. That would allow them to mark themselves back up and in. The down and out commands don't need the 'osd.', just the ${n}.
In any case, by this point the cluster definitely believes them to be down, out, and removed from the cluster. I swear I remember having phantom devices in my crush map like this before, but I thought it was because the osd hadn't been rm'd from the cluster... which doesn't seem to be the case.
Does anyone else have any thoughts?
On Fri, Feb 16, 2018 at 4:22 PM Karsten Becker <karsten.becker@xxxxxxxxxxx> wrote:
Here is what I did - bash history:
> 1897 for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph osd down osd.$n; done> 1920 for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph osd out
osd.$n; done
> 1921 for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph osd down osd.$n; done
> 1923 for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do systemctl stop ceph-osd@$n.service; done
> 1925 for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph osd crush remove osd.${n}; done
> 1926 for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph auth del osd.${n}; done
> 1927 for n in 6 7 14 15 16 17 18 19 3 9 10 11 12 20; do ceph osd rm ${n}; done
I assume that I did the right steps...
On 16.02.2018 21:56, David Turner wrote:
> What is the output of `ceph osd stat`? My guess is that they are still
> considered to be part of the cluster and going through the process of
> removing OSDs from your cluster is what you need to do. In particular
> `ceph osd rm 19`.
>
> On Fri, Feb 16, 2018 at 2:31 PM Karsten Becker
> <karsten.becker@xxxxxxxxxxx <mailto:karsten.becker@xxxxxxxxxxx>> wrote:
>
> Hi.
>
> during the reorgainzation of my cluster I removed some OSDs. Obviously
> something went wrong for 2 of them, osd.19 and osd.20.
>
> If I get my current Crush map, decompile and edit them, I see 2
> orphaned/stale entries for the former OSDs:
>
> > device 16 osd.16 class hdd
> > device 17 osd.17 class hdd
> > device 18 osd.18 class hdd
> > device 19 device19
> > device 20 device20
> > device 21 osd.21 class hdd
> > device 22 osd.22 class hdd
> > device 23 osd.23 class hdd
>
> If I delete them from the Crush map (file), recompile it and set it
> productive - they appear again... if I get the current map again and
> decompile them, they are in again.
>
> So how to get rid of these entries?
>
> Best from Berlin/Germany
> Karsten
>
> Ecologic Institut gemeinnuetzige GmbH
> Pfalzburger Str. 43/44, D-10717 Berlin
> Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
> Sitz der Gesellschaft / Registered Office: Berlin (Germany)
> Registergericht / Court of Registration: Amtsgericht Berlin
> (Charlottenburg), HRB 57947
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
Ecologic Institut gemeinnuetzige GmbH
Pfalzburger Str. 43/44, D-10717 Berlin
Geschaeftsfuehrerin / Director: Dr. Camilla Bausch
Sitz der Gesellschaft / Registered Office: Berlin (Germany)
Registergericht / Court of Registration: Amtsgericht Berlin (Charlottenburg), HRB 57947
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
