Re: slow pacific osd startup

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Andrej,

you might want to set debug_bluestore and debug_bluefs to 10 and check what's happening during the startup...

Alternatively you might try to compact slow OSD's DB using ceph_kvstore_tool and check if it helps to speedup the startup...


Just in case - is bluefs_buffered_io set to true for these OSDs?


Thanks,

Igor

On 2/11/2022 11:14 AM, Andrej Filipcic wrote:

Hi,

with 16.2.7, some OSDs are very slow to start, eg it takes ~30min for an hdd (12TB, 5TB used) to become active. After initialization, there is 20-40min of extreme reading at ~150MB/s from the OSD, just after
-------
Uptime(secs): 602.2 total, 0.0 interval
Flush(GB): cumulative 0.101, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.72 GB write, 1.22 MB/s write, 0.71 GB read, 1.21 MB/s read, 33.1 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

** File Read Latency Histogram By Level [default] **
-------
and the log (no special debug conf) is not updated during that time.

These slow OSDs were created with nautilus, and I am not quite sure when this slow startup appeared. the newer OSDs are fast to start. The minimum OSD release is set to pacific.

Any hints before I start debuging the problem? The history of installs:
2020-09-15T09:42:29Z SUBDEBUG Installed: ceph-base-2:15.2.4-0.el8.x86_64
2020-12-23T11:02:29Z SUBDEBUG Upgrade: ceph-base-2:15.2.8-0.el8.x86_64
2021-08-31T08:46:29Z SUBDEBUG Upgrade: ceph-base-2:16.2.5-0.el8.x86_64
2022-02-09T09:38:42+0100 SUBDEBUG Upgrade: ceph-base-2:16.2.7-0.el8.x86_64

Best regards,
Andrej

--
Igor Fedotov
Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux