For some reason GMail wasn't showing me any of the other responses. I saw what appeared to be an unanswered question. As soon as I hit send, the rest of the discussion showed up. Sorry about that.
On Tue, Oct 21, 2014 at 8:25 AM, Chad Seys <cwseys@xxxxxxxxxxxxxxxx> wrote:
Hi Craig,
> It's part of the way the CRUSH hashing works. Any change to the CRUSH map
> causes the algorithm to change slightly.
Dan@cern could not replicate my observations, so I plan to follow his
procedure (fake create an OSD, wait for rebalance, remove fake OSD) in the
near future to see if I can replicate his! :)
I haven't actually tried your OSD+HOST test, but I know that
ceph osd rm <osd-id>
# Wait for data movement to finish
ceph osd crush remove <osd-id>
# More data movement
The discussion about that indicated that any changes to the crushmap will cause some data movement, but it's possible that I'm mis-remembering.
> BTW, it's safer to remove OSDs and hosts by first marking the OSDs UP and
> OUT (ceph osd out OSDID). That will trigger the remapping, while keeping
> the OSDs in the pool so you have all of your replicas.
I am under the impression that the procedure I posted does leave the OSDs in
the pool while an additional replication takes place: After "ceph osd crush
remove osd.osdnum" I see that the used % on the removed OSD slowly decreases
as the relocation of blocks takes place.
Now that I see the rest of the discussion... you can check the ceph pg query stats during backfill. You should see n+1 OSDs in the acting section.
BUT, I think that to keep an OSD out after using "ceph osd out OSDID" one
needs to turn off "auto in" or something.
As long as it doesn't restart, it should stay OUT.
I don't want to turn that off b/c in the past I had some slow drives which
would occasionally be marked "out". If they stayed "out" that could increase
load on other drives, making them unresponsive, getting them marked "out" as
well, leading to a domino effect where too many drives get marked "out" and
the cluster goes down.
You might want to tweak mon osd down out interval, mon osd min down reporters, and mon osd min down reports. I bumped up the down out interval, because I had some OSDs that would get kicked out for being unresponsive, and they would hang for 15 minutes until they hit the suicide timeout and restarted. The min down reporters and reports are less relevant, but I had similar problems with one slow host marking all of the other OSDs down. I bumped up reports and reporters so that it requires more than one host to get OSDs kicked out.
> Ceph is also really good at handling multiple changes in a row. For
> example, I had to reformat all of my OSDs because I chose my mkfs.xfs
> parameters poorly. I removed the OSDS, without draining them first, which
> caused a lot of remapping. I then quickly formatted the OSDs, and put them
> back in. The CRUSH map went back to what it started with, and the only
> remapping required was to re-populate the newly formatted OSDs.
In this case you'd be living with num_replicas-1 for a while. Sounds
exciting! :)
Yeah, not something I'd generally recommend. I have federation setup, so I'm able to recover any lost data from the other cluster. I was also pretty desperate, the XFS params I used were causing massive cluster instability. When I started, half of my OSDs were flapping.
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
