-7 5.45695 host ceph-osd3
26 hdd 1.81898 osd.26 down 0 1.00000
27 hdd 1.81898 osd.27 down 0 1.00000
30 hdd 1.81898 osd.30 down 0 1.00000
Feng
_______________________________________________Here is a fuller picture. I inherited this ceph cluster from a previous admin whom has left the company. Although I am a linux administrator, I have very little experience with ceph and have had to learn; definitely still a lot to learn. I do know this crush map was made manually. To me it does not look right and would like to reorganize it, but I am concerned about what effects that would have with a cluster that has data on it.
I would like to remove both of the osd3-shelf1 and osd3-shelf2 chassis buckets and move them to host ceph-osd3 (I don’t see a need from separate buckets here). The “chassis” are actually two SAS disk shelves connected to ceph-osd3 host.
However, just moving one osd causes ceph to go unhealthy with OBJECT_MISPLACED messages and takes a while to go back into a healthy state. I am not too sure this is a big concern, but I am wondering if there is a recommended procedure for doing this. As I am just learning, I don’t want to do anything that will cause data loss.
My Tree is supposed to look like below but it keeps changing to the map further below. Notice the drives moving from chassis osd3-shelf1 chassis to host ceph-osd3. Does anyone know why this may happen?
I wrote a script to monitor for this and to place the osds back where they belong if they notice the change, but this should obviously not be necessary. I would appreciate any help with this.
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-62 14.55199 root osd3-shelf2
-60 14.55199 chassis ceph-osd3-shelf2
3 hdd 1.81898 osd.3 down 1.00000 1.00000
40 hdd 1.81898 osd.40 down 1.00000 1.00000
41 hdd 1.81898 osd.41 down 1.00000 1.00000
42 hdd 1.81898 osd.42 down 1.00000 1.00000
43 hdd 1.81898 osd.43 down 1.00000 1.00000
44 hdd 1.81898 osd.44 down 1.00000 1.00000
45 hdd 1.81898 osd.45 down 1.00000 1.00000
46 hdd 1.81898 osd.46 down 1.00000 1.00000
-58 1.71599 root osd3-internal
-54 1.71599 chassis ceph-osd3-internal
34 hdd 0.42899 osd.34 down 1.00000 1.00000
35 hdd 0.42899 osd.35 down 1.00000 1.00000
36 hdd 0.42899 osd.36 down 1.00000 1.00000
37 hdd 0.42899 osd.37 down 1.00000 1.00000
-50 14.55199 root osd3-shelf1
-56 14.55199 chassis ceph-osd3-shelf1
21 hdd 1.81898 osd.21 down 1.00000 1.00000
22 hdd 1.81898 osd.22 down 1.00000 1.00000
23 hdd 1.81898 osd.23 down 1.00000 1.00000
24 hdd 1.81898 osd.24 down 1.00000 1.00000
25 hdd 1.81898 osd.25 down 1.00000 1.00000
28 hdd 1.81898 osd.28 down 1.00000 1.00000
29 hdd 1.81898 osd.29 down 1.00000 1.00000
31 hdd 1.81898 osd.31 down 1.00000 1.00000
-7 5.45695 host ceph-osd3
26 hdd 1.81898 osd.26 down 0 1.00000
27 hdd 1.81898 osd.27 down 0 1.00000
30 hdd 1.81898 osd.30 down 0 1.00000
-1 47.21199 root default
-40 23.59000 rack mainehall
-3 23.59000 host ceph-osd1
0 hdd 1.81898 osd.0 up 1.00000 1.00000
1 hdd 1.81898 osd.1 up 1.00000 1.00000
2 hdd 1.81898 osd.2 up 1.00000 1.00000
4 hdd 1.81898 osd.4 up 1.00000 1.00000
5 hdd 1.81898 osd.5 up 0.90002 1.00000
6 hdd 1.81898 osd.6 up 1.00000 1.00000
7 hdd 1.81898 osd.7 up 1.00000 1.00000
8 hdd 1.81898 osd.8 up 1.00000 1.00000
9 hdd 1.81898 osd.9 up 1.00000 1.00000
10 hdd 1.81898 osd.10 up 0.95001 1.00000
33 hdd 1.76099 osd.33 up 1.00000 1.00000
38 hdd 3.63899 osd.38 up 1.00000 1.00000
-42 23.62199 rack rangleyhall
-5 23.62199 host ceph-osd2
11 hdd 1.81898 osd.11 up 1.00000 1.00000
12 hdd 1.81898 osd.12 up 0.90002 1.00000
13 hdd 1.81898 osd.13 up 1.00000 1.00000
14 hdd 1.81898 osd.14 up 1.00000 1.00000
15 hdd 1.81898 osd.15 up 1.00000 1.00000
16 hdd 1.81898 osd.16 up 1.00000 1.00000
17 hdd 1.81898 osd.17 up 1.00000 1.00000
18 hdd 1.81898 osd.18 up 0.90002 1.00000
19 hdd 1.81898 osd.19 up 1.00000 1.00000
20 hdd 1.81898 osd.20 up 1.00000 1.00000
32 hdd 1.79300 osd.32 up 0.95001 1.00000
39 hdd 3.63899 osd.39 up 1.00000 1.00000
But it keeps changing to this:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-62 14.55199 root osd3-shelf2
-60 14.55199 chassis ceph-osd3-shelf2
3 hdd 1.81898 osd.3 down 1.00000 1.00000
40 hdd 1.81898 osd.40 down 1.00000 1.00000
41 hdd 1.81898 osd.41 down 1.00000 1.00000
42 hdd 1.81898 osd.42 down 1.00000 1.00000
43 hdd 1.81898 osd.43 down 1.00000 1.00000
44 hdd 1.81898 osd.44 down 1.00000 1.00000
45 hdd 1.81898 osd.45 down 1.00000 1.00000
46 hdd 1.81898 osd.46 down 1.00000 1.00000
-58 1.71599 root osd3-internal
-54 1.71599 chassis ceph-osd3-internal
34 hdd 0.42899 osd.34 down 1.00000 1.00000
35 hdd 0.42899 osd.35 down 1.00000 1.00000
36 hdd 0.42899 osd.36 down 1.00000 1.00000
37 hdd 0.42899 osd.37 down 1.00000 1.00000
-50 14.55199 root osd3-shelf1
-56 14.55199 chassis ceph-osd3-shelf1
21 hdd 1.81898 osd.21 down 1.00000 1.00000
22 hdd 1.81898 osd.22 down 1.00000 1.00000
23 hdd 1.81898 osd.23 down 1.00000 1.00000
24 hdd 1.81898 osd.24 down 1.00000 1.00000
25 hdd 1.81898 osd.25 down 1.00000 1.00000
-7 5.45695 host ceph-osd3
26 hdd 1.81898 osd.26 down 0 1.00000
27 hdd 1.81898 osd.27 down 0 1.00000
30 hdd 1.81898 osd.30 down 0 1.00000
28 hdd 1.81898 osd.28 down 1.00000 1.00000
29 hdd 1.81898 osd.29 down 1.00000 1.00000
31 hdd 1.81898 osd.31 down 1.00000 1.00000
-1 47.21199 root default
-40 23.59000 rack mainehall
-3 23.59000 host ceph-osd1
0 hdd 1.81898 osd.0 up 1.00000 1.00000
1 hdd 1.81898 osd.1 up 1.00000 1.00000
2 hdd 1.81898 osd.2 up 1.00000 1.00000
4 hdd 1.81898 osd.4 up 1.00000 1.00000
5 hdd 1.81898 osd.5 up 0.90002 1.00000
6 hdd 1.81898 osd.6 up 1.00000 1.00000
7 hdd 1.81898 osd.7 up 1.00000 1.00000
8 hdd 1.81898 osd.8 up 1.00000 1.00000
9 hdd 1.81898 osd.9 up 1.00000 1.00000
10 hdd 1.81898 osd.10 up 0.95001 1.00000
33 hdd 1.76099 osd.33 up 1.00000 1.00000
38 hdd 3.63899 osd.38 up 1.00000 1.00000
-42 23.62199 rack rangleyhall
-5 23.62199 host ceph-osd2
11 hdd 1.81898 osd.11 up 1.00000 1.00000
12 hdd 1.81898 osd.12 up 0.90002 1.00000
13 hdd 1.81898 osd.13 up 1.00000 1.00000
14 hdd 1.81898 osd.14 up 1.00000 1.00000
15 hdd 1.81898 osd.15 up 1.00000 1.00000
16 hdd 1.81898 osd.16 up 1.00000 1.00000
17 hdd 1.81898 osd.17 up 1.00000 1.00000
18 hdd 1.81898 osd.18 up 0.90002 1.00000
19 hdd 1.81898 osd.19 up 1.00000 1.00000
20 hdd 1.81898 osd.20 up 1.00000 1.00000
32 hdd 1.79300 osd.32 up 0.95001 1.00000
39 hdd 3.63899 osd.39 up 1.00000 1.00000
Robert Pelletier, IT and Security Specialist
Eastern Maine Community College
(207) 974-4782 | 354 Hogan Rd., Bangor, ME 04401
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
