One of the things you need to be aware of when doing this is that the crush map is, more or less, stupid in knowing your network setup. You can configure your crush map with racks, datacenters, etc, but it has no idea where anything is. You have to tell it. You can use placement rules to help when adding things to the cluster, but for now you just need to create the buckets and move the hosts into them.
Sage explains a lot of the crush map here.
https://www.slideshare.net/mobile/sageweil1/a-crash-course-in-crush
On Wed, Jul 19, 2017, 2:43 AM Laszlo Budai <laszlo@xxxxxxxxxxxxxxxx> wrote:
Hi David,
thank you for pointing this out. Google wasn't able to find it ...
As far as I understand that thread is talking about a situation when you add hosts to an existing CRUSH bucket. That sounds good, and probably that will be our solution for cluster2.
I wonder whether there are any recommendations how to perform a migration from a CRUSH map that only has OSD-host-root to a new one that has OSD-host-chassis-root?
Thank you,
Laszlo
On 18.07.2017 20:05, David Turner wrote:
> This was recently covered on the mailing list. I believe this will cover all of your questions.
>
> https://www.spinics.net/lists/ceph-users/msg37252.html
>
>
> On Tue, Jul 18, 2017, 9:07 AM Laszlo Budai <laszlo@xxxxxxxxxxxxxxxx <mailto:laszlo@xxxxxxxxxxxxxxxx>> wrote:
>
> Dear all,
>
> we are planning to add new hosts to our existing hammer clusters, and I'm looking for best practices recommendations.
>
> currently we have 2 clusters with 72 OSDs and 6 nodes each. We want to add 3 more nodes (36 OSDs) to each cluster, and we have some questions about what would be the best way to do it. Currently the two clusters have different CRUSH maps.
>
> Cluster 1
> The CRUSH map only has OSDs, hosts and the root bucket. Failure domain is host.
> Our final desired state would be:
> OSD - hosts - chassis - root where each chassis has 3 hosts, each host has 12 OSDs, and the failure domain would be chassis.
>
> What would be the recommended way to achieve this without downtime for client operations?
> I have read about the possibility to throttle down the recovery/backfill using
> osd max backfills = 1
> osd recovery max active = 1
> osd recovery max single start = 1
> osd recovery op priority = 1
> osd recovery threads = 1
> osd backfill scan max = 16
> osd backfill scan min = 4
>
> but we wonder about the situation when, in a worst case scenario, all the replicas belonging to one pg have to be migrated to new locations according to the new CRUSH map. How will ceph behave in such situation?
>
>
> Cluster 2
> the crush map already contains chassis. Currently we have 3 chassis (c1, c2, c3) and 6 hosts:
> - x1, x2 in chassis c1
> - y1, y2 in chassis c2
> - x3, y3 in chassis c3
>
> We are adding hosts z1, z2, z3 and our desired CRUSH map would look like this:
> - x1, x2, x3 in c1
> - y1, y2, y3 in c2
> - z1, z2, z3 in c3
>
> Again, what would be the recommended way to achieve this while the clients are still accessing the data?
>
> Is it safe to add more OSDs at a time? or we should add them one by one?
>
> Thank you in advance for any suggestions, recommendations.
>
> Kind regards,
> Laszlo
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
