Seems you are using customized crush rules.
There is an 'osd_crush_update_on_start' option that will automatically update osd's crush location on startup.
You might need to turn it off but keep in mind that will also prevent any newly created osds from automatically joining the host...
(or you can manually do it of course)
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
My Tree is supposed to look like below but it keeps changing to the map further below. Notice the drives moving from chassis osd3-shelf1 chassis to host ceph-osd3. Does anyone know why this may happen?
I wrote a script to monitor for this and to place the osds back where they belong if they notice the change, but this should obviously not be necessary. I would appreciate any help with this.
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-62 14.55199 root osd3-shelf2
-60 14.55199 chassis ceph-osd3-shelf2
3 hdd 1.81898 osd.3 down 1.00000 1.00000
40 hdd 1.81898 osd.40 down 1.00000 1.00000
41 hdd 1.81898 osd.41 down 1.00000 1.00000
42 hdd 1.81898 osd.42 down 1.00000 1.00000
43 hdd 1.81898 osd.43 down 1.00000 1.00000
44 hdd 1.81898 osd.44 down 1.00000 1.00000
45 hdd 1.81898 osd.45 down 1.00000 1.00000
46 hdd 1.81898 osd.46 down 1.00000 1.00000
-58 1.71599 root osd3-internal
-54 1.71599 chassis ceph-osd3-internal
34 hdd 0.42899 osd.34 down 1.00000 1.00000
35 hdd 0.42899 osd.35 down 1.00000 1.00000
36 hdd 0.42899 osd.36 down 1.00000 1.00000
37 hdd 0.42899 osd.37 down 1.00000 1.00000
-50 14.55199 root osd3-shelf1
-56 14.55199 chassis ceph-osd3-shelf1
21 hdd 1.81898 osd.21 down 1.00000 1.00000
22 hdd 1.81898 osd.22 down 1.00000 1.00000
23 hdd 1.81898 osd.23 down 1.00000 1.00000
24 hdd 1.81898 osd.24 down 1.00000 1.00000
25 hdd 1.81898 osd.25 down 1.00000 1.00000
28 hdd 1.81898 osd.28 down 1.00000 1.00000
29 hdd 1.81898 osd.29 down 1.00000 1.00000
31 hdd 1.81898 osd.31 down 1.00000 1.00000
-7 5.45695 host ceph-osd3
26 hdd 1.81898 osd.26 down 0 1.00000
27 hdd 1.81898 osd.27 down 0 1.00000
30 hdd 1.81898 osd.30 down 0 1.00000
-1 47.21199 root default
-40 23.59000 rack mainehall
-3 23.59000 host ceph-osd1
0 hdd 1.81898 osd.0 up 1.00000 1.00000
1 hdd 1.81898 osd.1 up 1.00000 1.00000
2 hdd 1.81898 osd.2 up 1.00000 1.00000
4 hdd 1.81898 osd.4 up 1.00000 1.00000
5 hdd 1.81898 osd.5 up 0.90002 1.00000
6 hdd 1.81898 osd.6 up 1.00000 1.00000
7 hdd 1.81898 osd.7 up 1.00000 1.00000
8 hdd 1.81898 osd.8 up 1.00000 1.00000
9 hdd 1.81898 osd.9 up 1.00000 1.00000
10 hdd 1.81898 osd.10 up 0.95001 1.00000
33 hdd 1.76099 osd.33 up 1.00000 1.00000
38 hdd 3.63899 osd.38 up 1.00000 1.00000
-42 23.62199 rack rangleyhall
-5 23.62199 host ceph-osd2
11 hdd 1.81898 osd.11 up 1.00000 1.00000
12 hdd 1.81898 osd.12 up 0.90002 1.00000
13 hdd 1.81898 osd.13 up 1.00000 1.00000
14 hdd 1.81898 osd.14 up 1.00000 1.00000
15 hdd 1.81898 osd.15 up 1.00000 1.00000
16 hdd 1.81898 osd.16 up 1.00000 1.00000
17 hdd 1.81898 osd.17 up 1.00000 1.00000
18 hdd 1.81898 osd.18 up 0.90002 1.00000
19 hdd 1.81898 osd.19 up 1.00000 1.00000
20 hdd 1.81898 osd.20 up 1.00000 1.00000
32 hdd 1.79300 osd.32 up 0.95001 1.00000
39 hdd 3.63899 osd.39 up 1.00000 1.00000
But it keeps changing to this:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-62 14.55199 root osd3-shelf2
-60 14.55199 chassis ceph-osd3-shelf2
3 hdd 1.81898 osd.3 down 1.00000 1.00000
40 hdd 1.81898 osd.40 down 1.00000 1.00000
41 hdd 1.81898 osd.41 down 1.00000 1.00000
42 hdd 1.81898 osd.42 down 1.00000 1.00000
43 hdd 1.81898 osd.43 down 1.00000 1.00000
44 hdd 1.81898 osd.44 down 1.00000 1.00000
45 hdd 1.81898 osd.45 down 1.00000 1.00000
46 hdd 1.81898 osd.46 down 1.00000 1.00000
-58 1.71599 root osd3-internal
-54 1.71599 chassis ceph-osd3-internal
34 hdd 0.42899 osd.34 down 1.00000 1.00000
35 hdd 0.42899 osd.35 down 1.00000 1.00000
36 hdd 0.42899 osd.36 down 1.00000 1.00000
37 hdd 0.42899 osd.37 down 1.00000 1.00000
-50 14.55199 root osd3-shelf1
-56 14.55199 chassis ceph-osd3-shelf1
21 hdd 1.81898 osd.21 down 1.00000 1.00000
22 hdd 1.81898 osd.22 down 1.00000 1.00000
23 hdd 1.81898 osd.23 down 1.00000 1.00000
24 hdd 1.81898 osd.24 down 1.00000 1.00000
25 hdd 1.81898 osd.25 down 1.00000 1.00000
-7 5.45695 host ceph-osd3
26 hdd 1.81898 osd.26 down 0 1.00000
27 hdd 1.81898 osd.27 down 0 1.00000
30 hdd 1.81898 osd.30 down 0 1.00000
28 hdd 1.81898 osd.28 down 1.00000 1.00000
29 hdd 1.81898 osd.29 down 1.00000 1.00000
31 hdd 1.81898 osd.31 down 1.00000 1.00000
-1 47.21199 root default
-40 23.59000 rack mainehall
-3 23.59000 host ceph-osd1
0 hdd 1.81898 osd.0 up 1.00000 1.00000
1 hdd 1.81898 osd.1 up 1.00000 1.00000
2 hdd 1.81898 osd.2 up 1.00000 1.00000
4 hdd 1.81898 osd.4 up 1.00000 1.00000
5 hdd 1.81898 osd.5 up 0.90002 1.00000
6 hdd 1.81898 osd.6 up 1.00000 1.00000
7 hdd 1.81898 osd.7 up 1.00000 1.00000
8 hdd 1.81898 osd.8 up 1.00000 1.00000
9 hdd 1.81898 osd.9 up 1.00000 1.00000
10 hdd 1.81898 osd.10 up 0.95001 1.00000
33 hdd 1.76099 osd.33 up 1.00000 1.00000
38 hdd 3.63899 osd.38 up 1.00000 1.00000
-42 23.62199 rack rangleyhall
-5 23.62199 host ceph-osd2
11 hdd 1.81898 osd.11 up 1.00000 1.00000
12 hdd 1.81898 osd.12 up 0.90002 1.00000
13 hdd 1.81898 osd.13 up 1.00000 1.00000
14 hdd 1.81898 osd.14 up 1.00000 1.00000
15 hdd 1.81898 osd.15 up 1.00000 1.00000
16 hdd 1.81898 osd.16 up 1.00000 1.00000
17 hdd 1.81898 osd.17 up 1.00000 1.00000
18 hdd 1.81898 osd.18 up 0.90002 1.00000
19 hdd 1.81898 osd.19 up 1.00000 1.00000
20 hdd 1.81898 osd.20 up 1.00000 1.00000
32 hdd 1.79300 osd.32 up 0.95001 1.00000
39 hdd 3.63899 osd.39 up 1.00000 1.00000
Robert Pelletier, IT and Security Specialist
Eastern Maine Community College
(207) 974-4782 | 354 Hogan Rd., Bangor, ME 04401
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
