Hi all,
We have 5 node ceph cluster (Luminous 12.2.1) installed via
ceph-ansible.
All servers have 16X1.5TB SSD disks.
3 of these servers are also acting as MON+MGRs.
We don't have separated network for cluster and public,
each node has 4 NICs bonded together (40G) and serves
cluster+public communication (We know it's not ideal and
planning to change it).
Last week we added another node to cluster (another
16*1.5TB ssd).
We used ceph-ansible latest stable release.
After OSD activation cluster started rebalancing and
problems began:
1. Cluster entered HEALTH_ERROR state
2. 67 pgs stuck at activating+remapped
3. A lot of blocked slow requests.
This cluster serves OpenStack volumes and almost all
OpenStack instances got 100% disk utilization and hanged,
eventually, cinder-volume has crushed.
Eventually, after restarting several OSDs, problem solved
and cluster got back to HEALTH_OK
Our configuration already has:
osd max backfills = 1
osd max scrubs = 1
osd recovery max active = 1
osd recovery op priority = 1
In addition, we see a lot of bad mappings:
for example: bad mapping rule 0 x 52 num_rep 8 result
[32,5,78,25,96,59,80]
What can be the cause and what can I do in order to avoid
this situation? we need to add another 9 osd servers and can't
afford downtime.
Any help would be appreciated. Thank you very much
Our ceph configuration:
[mgr]
mgr_modules = dashboard zabbix
[global]
cluster network = *removed for security resons*
fsid = *removed for security resons*
mon host = *removed for security resons*
mon initial members = *removed for security resons*
mon osd down out interval = 900
osd pool default size = 3
public network = *removed for security resons*
[client.libvirt]
admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok
# must be writable by QEMU and allowed by SELinux or
AppArmor
log file = /var/log/ceph/qemu-guest-$pid.log # must
be writable by QEMU and allowed by SELinux or AppArmor
[osd]
osd backfill scan max = 16
osd backfill scan min = 4
osd bluestore cache size = 104857600 **Due to 12.2.1
bluestore memory leak bug**
osd max backfills = 1
osd max scrubs = 1
osd recovery max active = 1
osd recovery max single start = 1
osd recovery op priority = 1
osd recovery threads = 1
--
Tzachi
Strul
Storage
DevOps // Kenshoo
