Fwir, you could also put this into your ceph.conf to explicitly put an osd into the correct chassis at start if you have other osd's which you still want the crush_update_on_start setting set to true for:
[osd.34]
osd crush location = "chassis=ceph-osd3-internal"
[osd.35]
osd crush location = "chassis=ceph-osd3-internal"
[osd.34]
osd crush location = "chassis=ceph-osd3-internal"
[osd.35]
osd crush location = "chassis=ceph-osd3-internal"
etc..
Kind regards,
Kind regards,
Caspar
2018-05-22 3:03 GMT+02:00 David Turner <drakonstein@xxxxxxxxx>:
Your problem sounds like osd_crush_update_on_start. While set to the default of true, when an osd starts it tells the mons which server it is on and the mons will update the crush map to reflect it. While these osds running on the host, but placed in a custom host in the crush map... when they start they will update to show which host they are running on. You probably want to disable that in the config so that your custom crush placement is not altered.On Mon, May 21, 2018, 6:29 PM Martin, Jeremy <jmartin@xxxxxxxx> wrote:Hello,
I had a ceph cluster up and running for a few months now and all has been well and good except for today where I updated two osd nodes and well still well, these two nodes are designated within a rack and the rack is the failure domain so they are essentially mirrors of each. The issue came when I updated and rebooted the third node which has internal and external disks in a shelf and the failure domain is at the actual osd level as these are normal off the shelf disks for low priority storage that is not mission critical. The issue is that before the reboot the crush map look and behaved correctly but after the reboot the crush map was changed and had to be rebuilt to get the storage back online, all was well after the reassignment by I need to track down why it lots it configuration. The main differences here is that the first four disks (34-37) are supposed to be assigned to the chassis ceph-osd3-internal (like the before) and 21-31 assigned to chassis chassis-ceph-osd3-shelf1
(again like the before). After the reboot everything (34-37 and 21-31) was reassigned to the host ceph-osd3. Update was from 12.2.4 to 12.2.5. Any thoughts?
Jeremy
Before
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-58 0 root osd3-internal
-54 0 chassis ceph-osd3-internal
34 hdd 0.42899 osd.34 up 1.00000 1.00000
35 hdd 0.42899 osd.35 up 1.00000 1.00000
36 hdd 0.42899 osd.36 up 1.00000 1.00000
37 hdd 0.42899 osd.37 up 1.00000 1.00000
-50 0 root osd3-shelf1
-56 0 chassis ceph-osd3-shelf1
21 hdd 1.81898 osd.21 up 1.00000 1.00000
22 hdd 1.81898 osd.22 up 1.00000 1.00000
23 hdd 1.81898 osd.23 up 1.00000 1.00000
24 hdd 1.81898 osd.24 up 1.00000 1.00000
25 hdd 1.81898 osd.25 up 1.00000 1.00000
26 hdd 1.81898 osd.26 up 1.00000 1.00000
27 hdd 1.81898 osd.27 up 1.00000 1.00000
28 hdd 1.81898 osd.28 up 1.00000 1.00000
29 hdd 1.81898 osd.29 up 1.00000 1.00000
30 hdd 1.81898 osd.30 up 1.00000 1.00000
31 hdd 1.81898 osd.31 up 1.00000 1.00000
-7 0 host ceph-osd3
-1 47.21199 root default
-40 23.59000 rack mainehall
-3 23.59000 host ceph-osd1
0 hdd 1.81898 osd.0 up 1.00000 1.00000
Additional osd's left off for brevity
-5 23.62199 host ceph-osd2
11 hdd 1.81898 osd.11 up 1.00000 1.00000
Additional osd's left off for brevity
After
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-58 0 root osd3-internal
-54 0 chassis ceph-osd3-internal
-50 0 root osd3-shelf1
-56 0 chassis ceph-osd3-shelf1
-7 0 host ceph-osd3
21 hdd 1.81898 osd.21 up 1.00000 1.00000
22 hdd 1.81898 osd.22 up 1.00000 1.00000
23 hdd 1.81898 osd.23 up 1.00000 1.00000
24 hdd 1.81898 osd.24 up 1.00000 1.00000
25 hdd 1.81898 osd.25 up 1.00000 1.00000
26 hdd 1.81898 osd.26 up 1.00000 1.00000
27 hdd 1.81898 osd.27 up 1.00000 1.00000
28 hdd 1.81898 osd.28 up 1.00000 1.00000
29 hdd 1.81898 osd.29 up 1.00000 1.00000
30 hdd 1.81898 osd.30 up 1.00000 1.00000
31 hdd 1.81898 osd.31 up 1.00000 1.00000
34 hdd 0.42899 osd.34 up 1.00000 1.00000
35 hdd 0.42899 osd.35 up 1.00000 1.00000
36 hdd 0.42899 osd.36 up 1.00000 1.00000
37 hdd 0.42899 osd.37 up 1.00000 1.00000
-1 47.21199 root default
-40 23.59000 rack mainehall
-3 23.59000 host ceph-osd1
0 hdd 1.81898 osd.0 up 1.00000 1.00000
Additional osd's left off for brevity
-42 23.62199 rack rangleyhall
-5 23.62199 host ceph-osd2
11 hdd 1.81898 osd.11 up 1.00000 1.00000
Additional osd's left off for brevity
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
