TL;DR
bobtail Ceph cluster unable to finish rebalance after drive failure, usage increasing even with no clients connected.....
Waking up in the morning I found 2 OSDs were 100% full and two more were almost full. To get out of the situation I decreased the replication size from 3 to 2, and then also carefully (I believe carefully enough) remove some PGs in order to start things up again.
I got things going again and things appeared to be rebalancing correctly; however it got to the point were it stopped at 1420 PGs active+clean and the rest were stuck backfilling.
Looking at the PG dump, all of the PGs that were having issues were on osd.1. So I stopped it, verified things were continuing to rebalance after it was down/out and then formated osd.1's disk and put it back in.
Since then I've not been able to get the cluster back to HEALTHY, due to a combination of OSDs dying while recovering (not due to disk failure, just crashes) as well as the used space in the cluster increasing abnormally.
Right now I have all the clients disconnected and just the cluster rebalancing and the usage is increasing to the point where I have 12TB used when I have only < 3TB in cephfs and 2TB in a single RBD image (replication size 2). I've since shutdown the cluster so I don't fill it up.
My crushmap is the default, here is the usual suspects. I'm happy to provide additional information.
pg dump: http://pastebin.com/LUyu6Z09
ceph osd tree:
osd.8 is the failed drive (I will be replacing tonight), weight on osd.1 and osd.6 was done via reweight-by-utilization
# id weight type name up/down reweight
-1 19.5 root default
-3 19.5 rack unknownrack
-2 19.5 host ceph-test
0 1.5 osd.0 up 1
1 1.5 osd.1 up 0.6027
2 1.5 osd.2 up 1
3 1.5 osd.3 up 1
4 1.5 osd.4 up 1
5 2 osd.5 up 1
6 2 osd.6 up 0.6676
7 2 osd.7 up 1
8 2 osd.8 down 0
9 2 osd.9 up 1
10 2 osd.10 up 1
ceph -s:
health HEALTH_WARN 24 pgs backfill; 85 pgs backfill_toofull; 29 pgs backfilling; 40 pgs degraded; 1 pgs recovery_wait; 121 pgs stuck unclean; recovery 109306/2091318 degraded (5.227%); recovering 3 o/s, 43344KB/s; 2 near full osd(s); noout flag(s) set
monmap e2: 1 mons at {a=10.200.200.21:6789/0}, election epoch 1, quorum 0 a
osdmap e16251: 11 osds: 10 up, 10 in
pgmap v3145187: 1536 pgs: 1414 active+clean, 6 active+remapped+wait_backfill, 10 active+remapped+wait_backfill+backfill_toofull, 4 active+degraded+wait_backfill+backfill_toofull, 22 active+remapped+backfilling, 42 active+remapped+backfill_toofull, 7 active+degraded+backfilling, 17 active+degraded+backfill_toofull, 1 active+recovery_wait+remapped, 4 active+degraded+remapped+wait_backfill+backfill_toofull, 8 active+degraded+remapped+backfill_toofull, 1 active+clean+scrubbing+deep; 31607 GB data, 12251 GB used, 4042 GB / 16293 GB avail; 109306/2091318 degraded (5.227%); recovering 3 o/s, 43344KB/s
mdsmap e3363: 1/1/1 up {0=a=up:active}
rep size:
pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 384 pgp_num 384 last_change 897 owner 0 crash_replay_interval 45
pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins pg_num 384 pgp_num 384 last_change 13364 owner 0
pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num 384 pgp_num 384 last_change 13208 owner 0
pool 4 'media_video' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 384 pgp_num 384 last_change 890 owner 0
ceph.conf:
[global]
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd pool default size = 3
osd pool default min size = 1
osd pool default pg num = 366
osd pool default pgp num = 366
[osd]
osd journal size = 1000
journal_aio = true
#osd recovery max active = 10
osd mkfs type = xfs
osd mkfs options xfs = -f -i size=2048
osd mount options xfs = inode64,noatime
[mon.a]
host = ceph01
mon addr = 10.200.200.21:6789
[osd.0]
# 1.5 TB SATA
host = ceph01
devs = /dev/sdc
weight = 1.5
[osd.1]
# 1.5 TB SATA
host = ceph01
devs = /dev/sdd
weight = 1.5
[osd.2]
# 1.5 TB SATA
host = ceph01
devs = /dev/sdg
weight = 1.5
[osd.3]
# 1.5 TB SATA
host = ceph01
devs = /dev/sdj
weight = 1.5
[osd.4]
# 1.5 TB SATA
host = ceph01
devs = /dev/sdk
weight = 1.5
[osd.5]
# 2 TB SAS
host = ceph01
devs = /dev/sdf
weight = 2
[osd.6]
# 2 TB SAS
host = ceph01
devs = /dev/sdh
weight = 2
[osd.7]
# 2 TB SAS
host = ceph01
devs = /dev/sda
weight = 2
[osd.8]
# 2 TB SAS
host = ceph01
devs = /dev/sdb
weight = 2
[osd.9]
# 2 TB SAS
host = ceph01
devs = /dev/sdi
weight = 2
[osd.10]
# 2 TB SAS
host = ceph01
devs = /dev/sde
weight = 2
[mds.a]
host = ceph01
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
