I have created a cluster with 2 nodes and 6 osds each. I would like to verify if my pgs are being placed on the correct nodes based on my crushmap. I would like to make sure that my replication (x2, default) is not placed on the same host. Osd.0 through 5 is on host0 and osd.6 through osd.11 is on host1. Here is my crushmap. # begin crush map # devices device 0 device0 device 1 device1 device 2 device2 device 3 device3 device 4 device4 device 5 device5 device 6 device6 device 7 device7 device 8 device8 device 9 device9 device 10 device10 device 11 device11 # types type 0 device type 1 host type 2 root # buckets host host0 { id -1 # do not change unnecessarily # weight 6.000 alg straw hash 0 # rjenkins1 item device0 weight 1.000 item device1 weight 1.000 item device2 weight 1.000 item device3 weight 1.000 item device4 weight 1.000 item device5 weight 1.000 } host host1 { id -2 # do not change unnecessarily # weight 6.000 alg straw hash 0 # rjenkins1 item device6 weight 1.000 item device7 weight 1.000 item device8 weight 1.000 item device9 weight 1.000 item device10 weight 1.000 item device11 weight 1.000 } root root { id -3 # do not change unnecessarily # weight 6.000 alg straw hash 0 # rjenkins1 item host0 weight 1.000 item host1 weight 1.000 } # rules rule data { ruleset 0 type replicated min_size 1 max_size 10 step take root step chooseleaf firstn 0 type host step emit } rule metadata { ruleset 1 type replicated min_size 1 max_size 10 step take root step choose firstn 0 type host step emit } rule rbd { ruleset 2 type replicated min_size 1 max_size 10 step take root step choose firstn 0 type host step emit } # end crush map The end of "ceph pg dump -o -" shows the following which doesn't look correct. osdstat kbused kbavail kb hb in hb out 0 0 0 0 [] [] 1 393884 2927750484 2930265540 [0,2,3,4,5] [0,2,3,4,5] 2 304580 2927838884 2930265540 [0,1,3,4,5] [0,1,3,4,5] 3 0 0 0 [] [] 4 0 0 0 [] [] 5 158908 2927983588 2930265540 [0,1,2,3,4] [0,1,2,3,4] 6 496 2892788952 2894914980 [] [] 7 504 2928139544 2930265540 [] [] 8 504 2928139544 2930265540 [] [] 9 504 2928139544 2930265540 [] [] 10 504 2928139544 2930265540 [] [] 11 504 2928139544 2930265540 [] [] sum 860388 26317059628 26337039300 Which brings me to a couple of questions. 1. what is "hb in" and "hb out"? 2. The original crush map examples shows "type device". The new versions of ceph show the first type as osds? I changed my back to device, but how do you define an osd or this automatically done for you. I mean by "define" is the section in the crush map that shows all the devices, device 0 device0. 3. In the new default crushmap, there is a domain bucket. What is that intended for? Host? Thank you. This transmission and any attached files are privileged, confidential or otherwise the exclusive property of the intended recipient or Netelligent Corporation. If you are not the intended recipient, any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is strictly prohibited. If you have received this transmission in error, please contact us immediately by responding to this message or by telephone (314-392-6900) and promptly destroy the original transmission and its attachments. -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html