In particular, do you have bluestore_fsck_quick_fix_on_mount set to true? Josh On Fri, Feb 11, 2022 at 2:08 AM Eugen Block <eblock@xxxxxx> wrote: > > Hi, > > is there a difference in PG size on new and old OSDs or are they all > similar in size? Is there some fsck enabled during OSD startup? > > > Zitat von Andrej Filipcic <andrej.filipcic@xxxxxx>: > > > Hi, > > > > with 16.2.7, some OSDs are very slow to start, eg it takes ~30min > > for an hdd (12TB, 5TB used) to become active. After initialization, > > there is 20-40min of extreme reading at ~150MB/s from the OSD, just > > after > > ------- > > Uptime(secs): 602.2 total, 0.0 interval > > Flush(GB): cumulative 0.101, interval 0.000 > > AddFile(GB): cumulative 0.000, interval 0.000 > > AddFile(Total Files): cumulative 0, interval 0 > > AddFile(L0 Files): cumulative 0, interval 0 > > AddFile(Keys): cumulative 0, interval 0 > > Cumulative compaction: 0.72 GB write, 1.22 MB/s write, 0.71 GB read, > > 1.21 MB/s read, 33.1 seconds > > Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, > > 0.00 MB/s read, 0.0 seconds > > Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, > > 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for > > pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 > > memtable_compaction, 0 memtable_slowdown, interval 0 total count > > > > ** File Read Latency Histogram By Level [default] ** > > ------- > > and the log (no special debug conf) is not updated during that time. > > > > These slow OSDs were created with nautilus, and I am not quite sure > > when this slow startup appeared. the newer OSDs are fast to start. > > The minimum OSD release is set to pacific. > > > > Any hints before I start debuging the problem? The history of installs: > > 2020-09-15T09:42:29Z SUBDEBUG Installed: ceph-base-2:15.2.4-0.el8.x86_64 > > 2020-12-23T11:02:29Z SUBDEBUG Upgrade: ceph-base-2:15.2.8-0.el8.x86_64 > > 2021-08-31T08:46:29Z SUBDEBUG Upgrade: ceph-base-2:16.2.5-0.el8.x86_64 > > 2022-02-09T09:38:42+0100 SUBDEBUG Upgrade: ceph-base-2:16.2.7-0.el8.x86_64 > > > > Best regards, > > Andrej > > > > -- > > _____________________________________________________________ > > prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic@xxxxxx > > Department of Experimental High Energy Physics - F9 > > Jozef Stefan Institute, Jamova 39, P.o.Box 3000 > > SI-1001 Ljubljana, Slovenia > > Tel.: +386-1-477-3674 Fax: +386-1-425-7074 > > ------------------------------------------------------------- > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |