|
On 2017-09-29 11:31, Maged Mokhtar wrote:
On 2017-09-29 10:44, Adrian Saul wrote:
Do you mean that after you delete and remove the crush and auth entries for the OSD, when you go to create another OSD later it will re-use the previous OSD ID that you have destroyed in the past? Because I have seen that behaviour as well - but only for previously allocated OSD IDs that have been osd rm/crush rm/auth del.
-----Original Message-----
From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of
Luis Periquito
Sent: Friday, 29 September 2017 6:01 PM
To: Ceph Users <ceph-users@xxxxxxxxxxxxxx>
Subject: osd create returns duplicate ID's
Hi all,
I use puppet to deploy and manage my clusters.
Recently, as I have been doing a removal of old hardware and adding of new
I've noticed that sometimes the "ceph osd create" is returning repeated IDs.
Usually it's on the same server, but yesterday I saw it in different servers.
I was expecting the OSD ID's to be unique, and when they come on the same
server puppet starts spewing errors - which is desirable - but when it's in
different servers it broke those OSDs in Ceph. As they hadn't backfill any full
PGs I just wiped, removed and started anew.
As for the process itself: The OSDs are marked out and removed from crush,
when empty they are auth del and osd rm. After building the server puppet
will osd create, and use the generated ID for crush move and mkfs.
Unfortunately I haven't been able to reproduce in isolation, and being a
production cluster logging is tuned way down.
This has happened in several different clusters, but they are all running
10.2.7.
Any ideas?
thanks,
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Confidentiality: This email and any attachments are confidential and may be subject to copyright, legal or some other professional privilege. They are intended solely for the attention and use of the named addressee(s). They may only be copied, distributed or disclosed with the consent of the copyright owner. If you have received this email by mistake or by breach of the confidentiality clause, please notify the sender immediately by return email and delete or destroy all copies of the email. Any confidentiality, privilege or copyright is not waived or lost because this email has been sent to you by mistake. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxxhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Ceph will re-use previous deleted osd ids. this is desirable to minimize data rebalancing. What is not correct is having duplicate active ids and i am not sure how this is happening but i would suggest avoid add/remove osds simultaneously ie should add them one at a time, if you can do it manually check that the osd was added in crush and process is up and running before trying to add a new one..if that still produces duplicates then there is a serious issue. If adding via script double check it is not trying to do several tasks at once.
One more thing if you are using a script to add osds, try to add a small sleep/pause to allow the new osd to get activated via udev and register itself in crush before starting to create a new one.
|
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
