Re: Another OSD Crush question.

Robert LeBlanc <robert@xxxxxxxxxxxxx> · Thu, 23 Apr 2015 09:50:26 -0600

If you force CRUSH to put copies in each rack, then you will be limited by the smallest rack. You can have some sever limitations if you try to keep your copies to two racks (see the thread titles "CRUSH rule for 3 replicas across 2 hosts") for some of my explanation about this.
If I were you, I would install almost all the new hardware and hold out a few pieces. Get the new hardware up and running, then take down some of the original hardware and relocate it in the other cabinets so that you even out the older lower capacity nodes and new higher capacity nodes in each cabinet. That would give you the best of redundancy and performance (not all PGs would have to have a replica on the potentially slower hardware). This would allow you to have replication level three and able to lose a rack.

Another options if you have the racks is to spread the new hardware over 3 racks instead of 2 so that your cluster is over 4 racks. CRUSH will give a preference to the newer hardware (assuming the CRUSH weights reflect the size of the disk) and you would no longer be limited by the older smaller rack.

On Thu, Apr 23, 2015 at 3:20 AM, Rogier Dikkes <rogier.dikkes@xxxxxxxxxxx> wrote:
Hello all, 

At this moment we have a scenario where i would like your opinion on. 

Scenario: 
Currently we have a ceph environment with 1 rack of hardware, this rack contains a couple of OSD nodes with 4T disks. In a few months time we will deploy 2 more racks with OSD nodes, these nodes have 6T disks and 1 node more per rack. 

Short overview: 
rack1: 4T OSD
rack2: 6T OSD
rack3: 6T OSD

At this moment we are playing around with the idea to use the CRUSH map to make ceph 'rack aware' and ensure to have data replicated between racks. However from documentation i gathered i found that when you enforce data replication between buckets then your max storage size will be the lowest bucket value. My understanding: enforce the objects (size=3) to be replicated to 3 racks, the moment the rack with 4T OSD's is full we cannot store data anymore. 

Is this assumption correct?

The current idea we play with: 

- Create 2 rack buckets
- Create a ruleset to create 2 object replica’s for the 2x 6T buckets
- Create a ruleset to create 1 object replica over all the hosts.

This would result in 3 replicas of the object. Where we are sure that 2 objects at least are in different racks. In the unlikely event of a rack failure we would have at least 1 or 2 replica’s left.

Our idea is to have a crush rule with config that looks like: 
device 0 osd.0
device 1 osd.1
device 2 osd.2
device 3 osd.3
device 4 osd.4
device 5 osd.5
device 6 osd.6
device 7 osd.7
device 8 osd.8
device 9 osd.9

      host r01-cn01 {
              id -1
              alg straw
              hash 0
              item osd.0 weight 4.00
      }

      host r01-cn02 {
              id -2
              alg straw
              hash 0
              item osd.1 weight 4.00
      }

      host r01-cn03 {
              id -3
              alg straw
              hash 0
              item osd.3 weight 4.00
      }

      host r02-cn04 {
              id -4
              alg straw
              hash 0
              item osd.4 weight 6.00
      }

      host r02-cn05 {
              id -5
              alg straw
              hash 0
              item osd.5 weight 6.00
      }

      host r02-cn06 {
              id -6
              alg straw
              hash 0
              item osd.6 weight 6.00
      }

      host r03-cn07 {
              id -7
              alg straw
              hash 0
              item osd.7 weight 6.00
      }

      host r03-cn08 {
              id -8
              alg straw
              hash 0
              item osd.8 weight 6.00
      }

      host r03-cn09 {
              id -9
              alg straw
              hash 0
              item osd.9 weight 6.00
      }

      rack r02 {
              id -10
              alg straw
              hash 0
              item r02-cn04 weight 6.00
              item r02-cn05 weight 6.00
              item r02-cn06 weight 6.00
      }      

      rack r03 {
              id -11
              alg straw
              hash 0
              item r03-cn07 weight 6.00
              item r03-cn08 weight 6.00
              item r03-cn09 weight 6.00
      }

      root 6t {
              id -12
              alg straw
              hash 0
              item r02 weight 18.00
              item r03 weight 18.00
      }

      rule one {
              ruleset 1
              type replicated
              min_size 1
              max_size 10
              step take 6t
              step chooseleaf firstn 2 type rack
              step chooseleaf firstn 1 type host
              step emit
      }
Is this the right approach and would this cause limitations in regards of performance or usability? Do you have suggestions? 

Another interesting situation we have now is: We are going to move the hardware to new locations next year, the rack layout will change and thus the crush map will be altered. When changing a CRUSH map that theoretically would change the 2x 6T racks into 4 racks, would we need to take any special actions into consideration?

Thank you for your answers, they are much appreciated! 

Rogier Dikkes
System Programmer Hadoop & HPC Cloud
SURFsara | Science Park 140 | 1098 XG Amsterdam

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com