Nick
-----Original Message-----
From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of
Jake Grimmett
Sent: 29 January 2018 12:46
To: ceph-users@xxxxxxxxxxxxxx
Subject: pgs down after adding 260 OSDs & increasing PGs
Dear All,
Our ceph luminous (12.2.2) cluster has just broken, due to either adding
260 OSDs drives in one go, or to increasing the PG number from 1024 to
4096 in one go, or a combination of both...
Prior to the upgrade, the cluster consisted of 10 dual v4 Xeon nodes
running
SL7.4, each node had 19 bluestore OSDs (8TB Seagate Ironwolf) & 64GB ram.
The cluster has just two pools;
1) 1024 pg/pgs 8+2 EC pool on 190 x hdd.
2) 4 nodes have 1 NVMe SSD's used for a 3x replicated MDS pool.
Cluster provides 500TB CephFS used for scratch space, four snapshots taken
daily and kept for one week only.
Everything was working perfectly, until 26 OSD's were added to each node,
bringing the total hdd OSD count to 450. (all 8TB Ironwolf)
After adding all 260 OSD's with ceph-deploy, ceph health shows
HEALTH_WARN noout flag(s) set;
732950716/1219068139 objects misplaced (60.124%); Degraded data
redundancy: 1024 pgs unclean; too few PGs per OSD (23 < min 30)
So far so good, I'd expected to see the cluster rebalancing, the complaint
about
too few pgs per OSD seemed reasonable.
Without waiting for the cluster to rebalance, I increased the pg/pgs to
4096. At
this point, ceph health showed this:
HEALTH_ERR 135858073/1219068139 objects misplaced (11.144%); Reduced
data availability: 3119 pgs inactive; Degraded data redundancy:
210609/1219068139 objects degraded (0.017%),
4088 pgs unclean, 1002 pgs degraded,
1002 pgs undersized; 5 stuck requests are blocked > 4096 sec
We then left the cluster to rebalance.
Next morning, two ceph nodes were down, and I could see lots of oom-killer
messages in the logs.
Each node only has 64GB for 45 OSD's which is probably the cause of this.
as a short term fix, we limited RAM usage by adding this to ceph.conf
bluestore_cache_size = 104857600 bluestore_cache_kv_max = 67108864
This appears to stop the oom problems, so we waited while the cluster
rebalanced, until it said it stopped saying "objects misplaced"
This took a couple of days...
The problem now, is that although all of the OSD's are up, lots of pgs are
down,
degraded, unclean, and it is not clear how to fix this.
I have tried issuing osd scrub, and pg repair commands, but these do not
appear
to do anything.
cephfs will mount, but when locks up when it hits a pg that is down.
I have tried sequentially restarting all OSD's on each node, slowly
walking
through the cluster several times, but this does not fix things.
Current Status:
# ceph health
HEALTH_ERR Reduced data availability:
3021 pgs inactive, 23 pgs down, 23 pgs stale; Degraded data redundancy:
3021
pgs unclean, 1879 pgs degraded,
1879 pgs undersized; 1 stuck requests are blocked > 4096 sec
ceph health detail (see http://p.ip.fi/Pwdb ) contains many lines such as:
pg 4.ffe is stuck unclean for 470551.768849, current state
activating+remapped, last acting [156,175,33,169,135,85,165,55,148,178]
pg 4.fff is stuck undersized for 49509.580577, current state
activating+undersized+degraded+remapped, last acting
[44,12,185,125,69,29,119,102,81,2147483647]
(Presumably the OSD number "2147483647" is due to Erasure Encoding,
as per
<http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-
May/001660.html>
?)
Tailing the stuck osd log with debug osd = 20 shows this:
2018-01-29 11:56:35.204391 7f0dab4fd700 20 osd.46 15482 share_map_peer
0x5647cb336800 already has epoch 15482
2018-01-29 11:56:35.213226 7f0da7537700 10 osd.46 15482
tick_without_osd_lock
2018-01-29 11:56:35.213252 7f0da7537700 20 osd.46 15482
scrub_random_backoff lost coin flip, randomly backing off
2018-01-29 11:56:35.213257 7f0da7537700 10 osd.46 15482
promote_throttle_recalibrate 0 attempts, promoted 0 objects and 0
bytes; target 25 obj/sec or 5120 k bytes/sec
2018-01-29 11:56:35.213263 7f0da7537700 20 osd.46 15482
promote_throttle_recalibrate new_prob 1000
2018-01-29 11:56:35.213266 7f0da7537700 10 osd.46 15482
promote_throttle_recalibrate actual 0, actual/prob ratio 1, adjusted
new_prob 1000, prob 1000 -> 1000
2018-01-29 11:56:35.232884 7f0dab4fd700 20 osd.46 15482 share_map_peer
0x5647cabf3800 already has epoch 15482
Currently this cluster is just storing scratch data, so could be wiped,
however we would be more confident about using ceph widely if we can fix
errors like this...
thanks for reading, any advice appreciated,
Jake
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
