Hi Andrej,
you might want to set debug_bluestore and debug_bluefs to 10 and check
what's happening during the startup...
Alternatively you might try to compact slow OSD's DB using
ceph_kvstore_tool and check if it helps to speedup the startup...
Just in case - is bluefs_buffered_io set to true for these OSDs?
Thanks,
Igor
On 2/11/2022 11:14 AM, Andrej Filipcic wrote:
Hi,
with 16.2.7, some OSDs are very slow to start, eg it takes ~30min for
an hdd (12TB, 5TB used) to become active. After initialization, there
is 20-40min of extreme reading at ~150MB/s from the OSD, just after
-------
Uptime(secs): 602.2 total, 0.0 interval
Flush(GB): cumulative 0.101, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.72 GB write, 1.22 MB/s write, 0.71 GB read,
1.21 MB/s read, 33.1 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read,
0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0
level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for
pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0
memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [default] **
-------
and the log (no special debug conf) is not updated during that time.
These slow OSDs were created with nautilus, and I am not quite sure
when this slow startup appeared. the newer OSDs are fast to start. The
minimum OSD release is set to pacific.
Any hints before I start debuging the problem? The history of installs:
2020-09-15T09:42:29Z SUBDEBUG Installed: ceph-base-2:15.2.4-0.el8.x86_64
2020-12-23T11:02:29Z SUBDEBUG Upgrade: ceph-base-2:15.2.8-0.el8.x86_64
2021-08-31T08:46:29Z SUBDEBUG Upgrade: ceph-base-2:16.2.5-0.el8.x86_64
2022-02-09T09:38:42+0100 SUBDEBUG Upgrade:
ceph-base-2:16.2.7-0.el8.x86_64
Best regards,
Andrej
--
Igor Fedotov
Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
