Hi Philipp,
does the effect persist if you perform OSD restart once again shortly
after the first one?
Do you have any custom rocksdb/bluestore settings? "ceph config dump"
From the log you attached I can see 30 min gap during the load:
2022-04-10T11:42:51.607+0200 7fe6e8ba2f00 0
bluestore(/var/lib/ceph/osd/ceph-12) _open_db_and_around read-only:0
repair:0
2022-04-10T12:12:36.924+0200 7fe6e8ba2f00 0 _get_class not permitted to
load kvs
but you mentioned that it took more than an hour for OSD to load. Is
there any other long-running stuff on startup? Or you just overestimated
loading time? ;)
Thanks,
Igor
On 4/10/2022 10:18 PM, VELARTIS GmbH | Philipp Dürhammer wrote:
Here is the full log after the osd was up successfully (but looong time reading 100MB/sec)
v2022-04-10T11:42:44.219+0200 7f68961f8700 -1 received signal: Terminated from /sbin/init noibrs noibpb nopti nospectre_v2 nospec_store_bypass_disable no_stf_barrier (PID: 1) UID: 0
2022-04-10T11:42:44.219+0200 7f68961f8700 -1 osd.12 15550 *** Got signal Terminated ***
2022-04-10T11:42:44.219+0200 7f68961f8700 -1 osd.12 15550 *** Immediate shutdown (osd_fast_shutdown=true) ***
2022-04-10T11:42:47.007+0200 7fe6e8ba2f00 0 set uid:gid to 64045:64045 (ceph:ceph)
2022-04-10T11:42:47.007+0200 7fe6e8ba2f00 0 ceph version 16.2.7 (f9aa029788115b5df5eeee328f584156565ee5b7) pacific (stable), process ceph-osd, pid 1347448
2022-04-10T11:42:47.007+0200 7fe6e8ba2f00 0 pidfile_write: ignore empty --pid-file
2022-04-10T11:42:47.575+0200 7fe6e8ba2f00 0 starting osd.12 osd_data /var/lib/ceph/osd/ceph-12 /var/lib/ceph/osd/ceph-12/journal
2022-04-10T11:42:47.603+0200 7fe6e8ba2f00 0 load: jerasure load: lrc load: isa
2022-04-10T11:42:48.247+0200 7fe6e8ba2f00 0 osd.12:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-04-10T11:42:48.555+0200 7fe6e8ba2f00 0 osd.12:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-04-10T11:42:48.863+0200 7fe6e8ba2f00 0 osd.12:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-04-10T11:42:49.167+0200 7fe6e8ba2f00 0 osd.12:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-04-10T11:42:49.463+0200 7fe6e8ba2f00 0 osd.12:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-04-10T11:42:50.191+0200 7fe6e8ba2f00 0 osd.12:5.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-04-10T11:42:50.931+0200 7fe6e8ba2f00 0 osd.12:6.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-04-10T11:42:51.607+0200 7fe6e8ba2f00 0 osd.12:7.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-04-10T11:42:51.607+0200 7fe6e8ba2f00 0 bluestore(/var/lib/ceph/osd/ceph-12) _open_db_and_around read-only:0 repair:0
2022-04-10T12:12:36.924+0200 7fe6e8ba2f00 0 _get_class not permitted to load kvs
2022-04-10T12:12:36.924+0200 7fe6e8ba2f00 0 <cls> ./src/cls/cephfs/cls_cephfs.cc:201: loading cephfs
2022-04-10T12:12:36.924+0200 7fe6e8ba2f00 0 _get_class not permitted to load sdk
2022-04-10T12:12:36.928+0200 7fe6e8ba2f00 0 _get_class not permitted to load lua
2022-04-10T12:12:36.940+0200 7fe6e8ba2f00 0 <cls> ./src/cls/hello/cls_hello.cc:316: loading cls_hello
2022-04-10T12:12:36.940+0200 7fe6e8ba2f00 0 osd.12 15550 crush map has features 432629239337189376, adjusting msgr requires for clients
2022-04-10T12:12:36.940+0200 7fe6e8ba2f00 0 osd.12 15550 crush map has features 432629239337189376 was 8705, adjusting msgr requires for mons
2022-04-10T12:12:36.940+0200 7fe6e8ba2f00 0 osd.12 15550 crush map has features 3314933000854323200, adjusting msgr requires for osds
-----Ursprüngliche Nachricht-----
Von: VELARTIS GmbH | Philipp Dürhammer <p.duerhammer@xxxxxxxxxxx>
Gesendet: Sonntag, 10. April 2022 11:49
An: ceph-users@xxxxxxx
Betreff: osd needs moer then one hour to start with heavy reads
Hi,
allways when i try the start / restart an osd it takes more then one hour to start up. Meanwhile I see more then 100Mb read on the osd the whole time. (it's an enterprise ssd) logs hows:
2022-04-10T11:42:44.219+0200 7f68961f8700 -1 osd.12 15550 *** Immediate shutdown (osd_fast_shutdown=true) ***
2022-04-10T11:42:47.007+0200 7fe6e8ba2f00 0 set uid:gid to 64045:64045 (ceph:ceph)
2022-04-10T11:42:47.007+0200 7fe6e8ba2f00 0 ceph version 16.2.7 (f9aa029788115b5df5eeee328f584156565ee5b7) pacific (stable), process ceph-osd, pid 1347448
2022-04-10T11:42:47.007+0200 7fe6e8ba2f00 0 pidfile_write: ignore empty --pid-file
2022-04-10T11:42:47.575+0200 7fe6e8ba2f00 0 starting osd.12 osd_data /var/lib/ceph/osd/ceph-12 /var/lib/ceph/osd/ceph-12/journal
2022-04-10T11:42:47.603+0200 7fe6e8ba2f00 0 load: jerasure load: lrc load: isa
2022-04-10T11:42:48.247+0200 7fe6e8ba2f00 0 osd.12:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-04-10T11:42:48.555+0200 7fe6e8ba2f00 0 osd.12:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-04-10T11:42:48.863+0200 7fe6e8ba2f00 0 osd.12:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-04-10T11:42:49.167+0200 7fe6e8ba2f00 0 osd.12:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-04-10T11:42:49.463+0200 7fe6e8ba2f00 0 osd.12:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-04-10T11:42:50.191+0200 7fe6e8ba2f00 0 osd.12:5.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-04-10T11:42:50.931+0200 7fe6e8ba2f00 0 osd.12:6.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-04-10T11:42:51.607+0200 7fe6e8ba2f00 0 osd.12:7.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-04-10T11:42:51.607+0200 7fe6e8ba2f00 0 bluestore(/var/lib/ceph/osd/ceph-12) _open_db_and_around read-only:0 repair:0
(END)
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
--
Igor Fedotov
Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
