On Fri, Oct 06, 2023 at 02:55:22PM +1100, Chris Dunlop wrote:
Hi,
tl;dr why are my osds still spilling?
I've recently upgraded to 16.2.14 from 16.2.9 and started receiving
bluefs spillover warnings (due to the "fix spillover alert" per the
16.2.14 release notes). E.g. from 'ceph health detail', the warning on
one of these (there are a few):
osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device
Sigh. After battling this for days, I found the solution shortly after
sending this plea for help.
Via the hint from:
https://tracker.ceph.com/issues/44509
using ceph-bluestore-tool bluefs-bdev-new-db results in 128k or 64k leftover on slow device
The answer being, after expanding the db LV, and running the
"bluefs-bdev-expand", you need to run "bluefs-bdev-migrate", e.g. all
together:
{
lvextend -l ${size} ${lv}/${db} ${ssd_dev}
cephadm unit --fsid $cid --name osd.${osd} stop
cephadm shell --fsid $cid --name osd.${osd} -- ceph-bluestore-tool \
bluefs-bdev-expand \
--path /var/lib/ceph/osd/ceph-${osd}
cephadm shell --fsid $cid --name osd.${osd} -- ceph-bluestore-tool \
bluefs-bdev-migrate \
--path /var/lib/ceph/osd/ceph-${osd} \
--devs-source /var/lib/ceph/osd/ceph-${osd}/block \
--dev-target /var/lib/ceph/osd/ceph-${osd}/block.db \
cephadm unit --fsid $cid --name osd.${osd} start
}
After the "bluefs-bdev-migrate", the "128 KiB" that was previously in the
DB/SLOW location has reduced to zero:
# ceph tell osd.76 bluefs stats
1 : device size 0x7747ffe000 : using 0xe2e500000(57 GiB)
2 : device size 0xe8d7fc00000 : using 0x65598084000(6.3 TiB)
RocksDBBlueFSVolumeSelector Usage Matrix:
DEV/LEV WAL DB SLOW * * REAL FILES
LOG 0 B 6 MiB 0 B 0 B 0 B 2.0 MiB 1
WAL 0 B 282 MiB 0 B 0 B 0 B 1.3 MiB 1
DB 0 B 56 GiB 0 B 0 B 0 B 50 GiB 841
SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0
TOTAL 0 B 57 GiB 0 B 0 B 0 B 0 B 843
MAXIMUMS:
LOG 0 B 6 MiB 0 B 0 B 0 B 2.0 MiB
WAL 0 B 564 MiB 0 B 0 B 0 B 56 MiB
DB 0 B 56 GiB 0 B 0 B 0 B 50 GiB
SLOW 0 B 0 B 0 B 0 B 0 B 0 B
TOTAL 0 B 57 GiB 0 B 0 B 0 B 0 B
SIZE << 0 B 453 GiB 14 TiB
...and the "osd.xx spilled over" has disappeared from my "ceph health detail".
Now to do the others...
Cheers,
Chris
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
