Hi Christian,
Yes, I very clearly know what is spillover, read that github leveled document in the last couple of days every day multiple time. (Answers for your questions are after the cluster background information).
About the cluster:
- users are doing continuously put/head/delete operations
- cluster iops: 10-50k read, 5000 write iops
- throughput: 142MiB/s write and 662 MiB/s read
- Not containerized deployment, 3 cluster in multisite
- 3x mon/mgr/rgw (5 rgw in each mon, altogether 15 behind haproxy vip)
7 nodes and in each node the following config:
- 1x 1.92TB nvme for index pool
- 6x 15.3 TB osd SAS SSD (hpe VO015360JWZJN read intensive ssd, SKU P19911-B21 in this document:https://h20195.www2.hpe.com/v2/getpdf.aspx/a00001288enw.pdf)
- 2x 1.92TB nvme block.db for the 6 ssd (model: HPE KCD6XLUL1T92 SKU: P20131-B21 in this documenthttps://h20195.www2.hpe.com/v2/getpdf.aspx/a00001288enw.pdf)
- osd deployed with dmcrypt
- data pool is on ec 4:2 other pools are on the ssds with 3 replica
Config file that we have on all nodes + on the mon nodes has the rgw definition also:
[global]
cluster network = 192.168.199.0/24
fsid = 5a07ec50-4eee-4336-aa11-46ca76edcc24
mon host = [v2:10.118.199.1:3300,v1:10.118.199.1:6789],[v2:10.118.199.2:3300,v1:10.118.199.2:6789],[v2:10.118.199.3:3300,v1:10.118.199.3:6789]
mon initial members = mon-2s01,mon-2s02,mon-2s03
osd pool default crush rule = -1
public network = 10.118.199.0/24
rgw_relaxed_s3_bucket_names = true
rgw_dynamic_resharding = false
rgw_enable_apis = s3, s3website, swift, swift_auth, admin, sts, iam, pubsub, notifications
#rgw_bucket_default_quota_max_objects = 1126400
[mon]
mon_allow_pool_delete = true
mon_pg_warn_max_object_skew = 0
mon_osd_nearfull_ratio = 70
[osd]
osd_max_backfills = 1
osd_recovery_max_active = 1
osd_recovery_op_priority = 1
osd_memory_target = 31490694621
# due to osd reboots, the below configs has been added to survive the suicide timeout
osd_scrub_during_recovery = true
osd_op_thread_suicide_timeout=3000
osd_op_thread_timeout=120
Stability issue that I mean:
- Pg increase still in progress, hasn’t been finished from 32-128 on the erasure coded data pool. 103 currently and the degraded objects are always stuck almost when finished, but at the end osd dies and start again the recovery process.
- compaction is happening all the time, so all the nvme drives are generating iowait continuously because it is 100% utilized (iowait is around 1-3). If I try to compact with ceph tell osd.x compact that is impossible, it will never finish, only with ctrl+c.
- At the beginning when we didn't have so much spilledover disks, I didn't mind it actually I was happy of the spillover because the underlaying ssd can take some load from the nvme, but after the osds started to reboot and I'd say started to collapse 1 by 1. When I monitor which osds are collapsing, it was always the one which was spillovered. This op thread and suicide timeout can keep a bit longer the osds up.
- Now ALL rgw started to die once 1 specific osd goes down, and this make total outage. In the logs there isn't anything about this, neither message, nor rgw log just like timeout the connections. This is unacceptable from the user's perspective that thay need to wait 1.5 hour until my manual compaction finished and I can start the osd.
Current cluster state ceph -s:
health: HEALTH_ERR
12 OSD(s) experiencing BlueFS spillover
4/1055038256 objects unfound (0.000%)
noout flag(s) set
Possible data damage: 2 pgs recovery_unfound
Degraded data redundancy: 12341016/6328900227 objects degraded (0.195%), 16 pgs degraded, 21 pgs u
ndersized
4 pgs not deep-scrubbed in time
services:
mon: 3 daemons, quorum mon-2s01,mon-2s02,mon-2s03 (age 2M)
mgr: mon-2s01(active, since 2M), standbys: mon-2s03, mon-2s02
osd: 49 osds: 49 up (since 101m), 49 in (since 4d); 23 remapped pgs
flags noout
rgw: 15 daemons active (mon-2s01.rgw0, mon-2s01.rgw1, mon-2s01.rgw2, mon-2s01.rgw3, mon-2s01.rgw4, mon-2s02.rgw0, mon-2s02.rgw1, mon-2s02.rgw2, mon-2s02.rgw3, mon-2s02.rgw4, mon-2s03.rgw0, mon-2s03.rgw1, mon-2s03.rgw2, mon-2s03.rgw3, mon-2s03.rgw4)
task status:
data:
pools: 9 pools, 425 pgs
objects: 1.06G objects, 67 TiB
usage: 159 TiB used, 465 TiB / 623 TiB avail
pgs: 12032346/6328762449 objects degraded (0.190%)
68127707/6328762449 objects misplaced (1.076%)
4/1055015441 objects unfound (0.000%)
397 active+clean
13 active+undersized+degraded+remapped+backfill_wait
4 active+undersized+remapped+backfill_wait
4 active+clean+scrubbing+deep
2 active+recovery_unfound+undersized+degraded+remapped
2 active+remapped+backfill_wait
1 active+clean+scrubbing
1 active+undersized+remapped+backfilling
1 active+undersized+degraded+remapped+backfilling
io:
client: 256 MiB/s rd, 94 MiB/s wr, 17.70k op/s rd, 2.75k op/s wr
recovery: 16 MiB/s, 223 objects/s
Ty
-----Original Message-----
From: Christian Wuerdig<christian.wuerdig@xxxxxxxxx>
Sent: Thursday, September 30, 2021 1:01 PM
To: Szabo, Istvan (Agoda)<Istvan.Szabo@xxxxxxxxx>
Cc: Ceph Users<ceph-users@xxxxxxx>
Subject: Re: osd_memory_target=level0 ?
Email received from the internet. If in doubt, don't click any link nor open any attachment !
________________________________
Bluestore memory targets have nothing to do with spillover. It's already been said several times: The spillover warning is simply telling you that instead of writing data to your supposedly fast wal/blockdb device it's now hitting your slow device.
You've stated previously that your fast device is nvme and your slow device is SSD. So the spill-over is probably less of a problem than you think. It's currently unclear what your actual problem is and why you think it's to do with spill-over.
What model are your NVMEs and SSDs - what IOPS can each sustain (4k random write direct IO), what's their current load? What are the actual problems that you are observing, i.e. what does "stability problems" actually mean?
On Thu, 30 Sept 2021 at 18:33, Szabo, Istvan (Agoda)<Istvan.Szabo@xxxxxxxxx> wrote:
Hi,
Still suffering with the spilledover disks and stability issue in 3 of my cluster after uploaded 6-900 millions objects to the cluster. (Octopus 15.2.10).
I’ve set memory target around 31-32GB so could that be that the spilledover issue is coming from here?
So have mem target 31GB, next level would be 310 and after go to the underlaying ssd disk. So the 4 level doesn’t have space on the nvme.
Let’s say set to default 4GB, it would be 444GB the level0-3 so it
should fit in on the 600GB lvm assigned on the nvme for db with wal.
This is how it looks like, eg. Osd 27 even after 2 times manual
compact still spilled over :(
osd.1 spilled over 198 GiB metadata from 'db' device (303 GiB used of 596 GiB) to slow device
osd.5 spilled over 251 GiB metadata from 'db' device (163 GiB used of 596 GiB) to slow device
osd.8 spilled over 61 GiB metadata from 'db' device (264 GiB used of 596 GiB) to slow device
osd.11 spilled over 260 GiB metadata from 'db' device (242 GiB used of 596 GiB) to slow device
osd.12 spilled over 149 GiB metadata from 'db' device (238 GiB used of 596 GiB) to slow device
osd.15 spilled over 259 GiB metadata from 'db' device (195 GiB used of 596 GiB) to slow device
osd.17 spilled over 10 GiB metadata from 'db' device (314 GiB used of 596 GiB) to slow device
osd.21 spilled over 324 MiB metadata from 'db' device (346 GiB used of 596 GiB) to slow device
osd.27 spilled over 12 GiB metadata from 'db' device (486 GiB used of 596 GiB) to slow device
osd.29 spilled over 61 GiB metadata from 'db' device (306 GiB used of 596 GiB) to slow device
osd.31 spilled over 59 GiB metadata from 'db' device (308 GiB used of 596 GiB) to slow device
osd.46 spilled over 69 GiB metadata from 'db' device (308 GiB
used of 596 GiB) to slow device
Also is there a way to fasten compaction? It takes 1-1.5 hours /osd to compact.
Thank you
_______________________________________________
ceph-users mailing list --ceph-users@xxxxxxx To unsubscribe send an
email toceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list --ceph-users@xxxxxxx
To unsubscribe send an email toceph-users-leave@xxxxxxx
