Mandi! Paweł Sadowsk
In chel di` si favelave...
From your osd tree it looks like you used 'ceph osd reweight'.
Yes, and i supposed also to do the right things!
Now, i've tried to lower the to-dimissi OSD, using:
ceph osd reweight 2 0.95
leading to an osd map tree like:
root@blackpanther:~# ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 21.83984 root default
-2 5.45996 host capitanamerica
0 1.81999 osd.0 up 1.00000 1.00000
1 1.81999 osd.1 up 1.00000 1.00000
10 0.90999 osd.10 up 1.00000 1.00000
11 0.90999 osd.11 up 1.00000 1.00000
-3 5.45996 host vedovanera
2 1.81999 osd.2 up 0.95000 1.00000
3 1.81999 osd.3 up 1.00000 1.00000
4 0.90999 osd.4 up 1.00000 1.00000
5 0.90999 osd.5 up 1.00000 1.00000
-4 5.45996 host deadpool
6 1.81999 osd.6 up 1.00000 1.00000
7 1.81999 osd.7 up 1.00000 1.00000
8 0.90999 osd.8 up 1.00000 1.00000
9 0.90999 osd.9 up 1.00000 1.00000
-5 5.45996 host blackpanther
12 1.81999 osd.12 up 0.04999 1.00000
13 1.81999 osd.13 up 0.04999 1.00000
14 0.90999 osd.14 up 0.04999 1.00000
15 0.90999 osd.15 up 0.04999 1.00000
and, after rebalancing, to:
root@blackpanther:~# ceph -s
cluster 8794c124-c2ec-4e81-8631-742992159bd6
health HEALTH_WARN
6 pgs stuck unclean
recovery 4/2550363 objects degraded (0.000%)
recovery 11282/2550363 objects misplaced (0.442%)
monmap e6: 6 mons at {0=10.27.251.7:6789/0,1=10.27.251.8:6789/0,2=10.27.251.11:6789/0,3=10.27.251.12:6789/0,4=10.27.251.9:6789/0,blackpanther=10.27.251.2:6789/0}
election epoch 2750, quorum 0,1,2,3,4,5 blackpanther,0,1,4,2,3
osdmap e7300: 16 osds: 16 up, 16 in; 6 remapped pgs
pgmap v54737590: 768 pgs, 3 pools, 3299 GB data, 830 kobjects
9870 GB used, 12474 GB / 22344 GB avail
4/2550363 objects degraded (0.000%)
11282/2550363 objects misplaced (0.442%)
761 active+clean
6 active+remapped
1 active+clean+scrubbing
client io 13476 B/s rd, 654 kB/s wr, 95 op/s
Why pgs that are in state 'stuck unclean'?
This is most probably due to big difference in weights between
your hosts (the new one has 20x lower weight than the old ones)
which in combination with straw algorithm is a 'known' issue. You
could try to increase choose_total_tries in your crush map
from 50 to some bigger number. The best IMO would be to use straw2
(which will cause some rebalance) and then use 'ceph osd crush
reweight' (instead of 'ceph osd reweight') with small steps to
slowly rebalance data onto new OSDs.
