Hello,
I had a ceph cluster up and running for a few months now and all has been well and good except for today where I updated two osd nodes and well still well, these two nodes are designated within a rack and the rack is the failure domain so they are essentially mirrors of each. The issue came when I updated and rebooted the third node which has internal and external disks in a shelf and the failure domain is at the actual osd level as these are normal off the shelf disks for low priority storage that is not mission critical. The issue is that before the reboot the crush map look and behaved correctly but after the reboot the crush map was changed and had to be rebuilt to get the storage back online, all was well after the reassignment by I need to track down why it lots it configuration. The main differences here is that the first four disks (34-37) are supposed to be assigned to the chassis ceph-osd3-internal (like the before) and 21-31 assigned to chassis chassis-ceph-osd3-shelf1
(again like the before). After the reboot everything (34-37 and 21-31) was reassigned to the host ceph-osd3. Update was from 12.2.4 to 12.2.5. Any thoughts?
Jeremy
Before
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-58 0 root osd3-internal
-54 0 chassis ceph-osd3-internal
34 hdd 0.42899 osd.34 up 1.00000 1.00000
35 hdd 0.42899 osd.35 up 1.00000 1.00000
36 hdd 0.42899 osd.36 up 1.00000 1.00000
37 hdd 0.42899 osd.37 up 1.00000 1.00000
-50 0 root osd3-shelf1
-56 0 chassis ceph-osd3-shelf1
21 hdd 1.81898 osd.21 up 1.00000 1.00000
22 hdd 1.81898 osd.22 up 1.00000 1.00000
23 hdd 1.81898 osd.23 up 1.00000 1.00000
24 hdd 1.81898 osd.24 up 1.00000 1.00000
25 hdd 1.81898 osd.25 up 1.00000 1.00000
26 hdd 1.81898 osd.26 up 1.00000 1.00000
27 hdd 1.81898 osd.27 up 1.00000 1.00000
28 hdd 1.81898 osd.28 up 1.00000 1.00000
29 hdd 1.81898 osd.29 up 1.00000 1.00000
30 hdd 1.81898 osd.30 up 1.00000 1.00000
31 hdd 1.81898 osd.31 up 1.00000 1.00000
-7 0 host ceph-osd3
-1 47.21199 root default
-40 23.59000 rack mainehall
-3 23.59000 host ceph-osd1
0 hdd 1.81898 osd.0 up 1.00000 1.00000
Additional osd's left off for brevity
-5 23.62199 host ceph-osd2
11 hdd 1.81898 osd.11 up 1.00000 1.00000
Additional osd's left off for brevity
After
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-58 0 root osd3-internal
-54 0 chassis ceph-osd3-internal
-50 0 root osd3-shelf1
-56 0 chassis ceph-osd3-shelf1
-7 0 host ceph-osd3
21 hdd 1.81898 osd.21 up 1.00000 1.00000
22 hdd 1.81898 osd.22 up 1.00000 1.00000
23 hdd 1.81898 osd.23 up 1.00000 1.00000
24 hdd 1.81898 osd.24 up 1.00000 1.00000
25 hdd 1.81898 osd.25 up 1.00000 1.00000
26 hdd 1.81898 osd.26 up 1.00000 1.00000
27 hdd 1.81898 osd.27 up 1.00000 1.00000
28 hdd 1.81898 osd.28 up 1.00000 1.00000
29 hdd 1.81898 osd.29 up 1.00000 1.00000
30 hdd 1.81898 osd.30 up 1.00000 1.00000
31 hdd 1.81898 osd.31 up 1.00000 1.00000
34 hdd 0.42899 osd.34 up 1.00000 1.00000
35 hdd 0.42899 osd.35 up 1.00000 1.00000
36 hdd 0.42899 osd.36 up 1.00000 1.00000
37 hdd 0.42899 osd.37 up 1.00000 1.00000
-1 47.21199 root default
-40 23.59000 rack mainehall
-3 23.59000 host ceph-osd1
0 hdd 1.81898 osd.0 up 1.00000 1.00000
Additional osd's left off for brevity
-42 23.62199 rack rangleyhall
-5 23.62199 host ceph-osd2
11 hdd 1.81898 osd.11 up 1.00000 1.00000
Additional osd's left off for brevity
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
