Re: osd needs moer then one hour to start with heavy reads

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

HI Igor,

This time it was maye only ~30 minutes. But it can take longer.
After one day running the restart took now 1:30 minutes

2022-04-11T13:43:59.832+0200 7f02feb14f00  0 bluestore(/var/lib/ceph/osd/ceph-24) _open_db_and_around read-only:0 repair:0
2022-04-11T13:45:34.688+0200 7f02feb14f00  0 _get_class not permitted to load kvs

directly again only some seconds:

2022-04-11T13:47:23.136+0200 7f4ad07f9f00  0 bluestore(/var/lib/ceph/osd/ceph-24) _open_db_and_around read-only:0 repair:0
2022-04-11T13:47:32.036+0200 7f4ad07f9f00  0 _get_class not permitted to load kvs

Should be standard config (I deployed from Proxmox)

ceph config dump
WHO    MASK  LEVEL     OPTION                                 VALUE           RO
  mon        advanced  auth_allow_insecure_global_id_reclaim  false
  mgr        advanced  mgr/pg_autoscaler/autoscale_profile    scale-down
  mgr        advanced  mgr/zabbix/identifier                  ceph1-cluster   *
  mgr        advanced  mgr/zabbix/zabbix_host                 192.168.11.101  *

thanks,
philipp



-----Ursprüngliche Nachricht-----
Von: Igor Fedotov <igor.fedotov@xxxxxxxx> 
Gesendet: Montag, 11. April 2022 12:13
An: VELARTIS GmbH | Philipp Dürhammer <p.duerhammer@xxxxxxxxxxx>; ceph-users@xxxxxxx
Betreff: Re:  Re: osd needs moer then one hour to start with heavy reads

Hi Philipp,

does the effect persist if you perform OSD restart once again shortly after the first one?

Do you have any custom rocksdb/bluestore settings? "ceph config dump"

 From the log you attached I can see 30 min gap during the load:

2022-04-10T11:42:51.607+0200 7fe6e8ba2f00  0
bluestore(/var/lib/ceph/osd/ceph-12) _open_db_and_around read-only:0
repair:0
2022-04-10T12:12:36.924+0200 7fe6e8ba2f00  0 _get_class not permitted to load kvs


but you mentioned that it took more than an hour for OSD to load. Is there any other long-running stuff on startup? Or you just overestimated loading time? ;)


Thanks,

Igor


On 4/10/2022 10:18 PM, VELARTIS GmbH | Philipp Dürhammer wrote:
> Here is the full log after the osd was up successfully (but looong 
> time reading 100MB/sec)
>
> v2022-04-10T11:42:44.219+0200 7f68961f8700 -1 received  signal: 
> Terminated from /sbin/init noibrs noibpb nopti nospectre_v2 
> nospec_store_bypass_disable no_stf_barrier  (PID: 1) UID: 0
> 2022-04-10T11:42:44.219+0200 7f68961f8700 -1 osd.12 15550 *** Got 
> signal Terminated ***
> 2022-04-10T11:42:44.219+0200 7f68961f8700 -1 osd.12 15550 *** 
> Immediate shutdown (osd_fast_shutdown=true) ***
> 2022-04-10T11:42:47.007+0200 7fe6e8ba2f00  0 set uid:gid to 
> 64045:64045 (ceph:ceph)
> 2022-04-10T11:42:47.007+0200 7fe6e8ba2f00  0 ceph version 16.2.7 
> (f9aa029788115b5df5eeee328f584156565ee5b7) pacific (stable), process 
> ceph-osd, pid 1347448
> 2022-04-10T11:42:47.007+0200 7fe6e8ba2f00  0 pidfile_write: ignore 
> empty --pid-file
> 2022-04-10T11:42:47.575+0200 7fe6e8ba2f00  0 starting osd.12 osd_data 
> /var/lib/ceph/osd/ceph-12 /var/lib/ceph/osd/ceph-12/journal
> 2022-04-10T11:42:47.603+0200 7fe6e8ba2f00  0 load: jerasure load: lrc 
> load: isa
> 2022-04-10T11:42:48.247+0200 7fe6e8ba2f00  0 osd.12:0.OSDShard using 
> op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, 
> cutoff=196)
> 2022-04-10T11:42:48.555+0200 7fe6e8ba2f00  0 osd.12:1.OSDShard using 
> op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, 
> cutoff=196)
> 2022-04-10T11:42:48.863+0200 7fe6e8ba2f00  0 osd.12:2.OSDShard using 
> op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, 
> cutoff=196)
> 2022-04-10T11:42:49.167+0200 7fe6e8ba2f00  0 osd.12:3.OSDShard using 
> op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, 
> cutoff=196)
> 2022-04-10T11:42:49.463+0200 7fe6e8ba2f00  0 osd.12:4.OSDShard using 
> op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, 
> cutoff=196)
> 2022-04-10T11:42:50.191+0200 7fe6e8ba2f00  0 osd.12:5.OSDShard using 
> op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, 
> cutoff=196)
> 2022-04-10T11:42:50.931+0200 7fe6e8ba2f00  0 osd.12:6.OSDShard using 
> op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, 
> cutoff=196)
> 2022-04-10T11:42:51.607+0200 7fe6e8ba2f00  0 osd.12:7.OSDShard using 
> op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, 
> cutoff=196)
> 2022-04-10T11:42:51.607+0200 7fe6e8ba2f00  0 
> bluestore(/var/lib/ceph/osd/ceph-12) _open_db_and_around read-only:0 
> repair:0
> 2022-04-10T12:12:36.924+0200 7fe6e8ba2f00  0 _get_class not permitted 
> to load kvs
> 2022-04-10T12:12:36.924+0200 7fe6e8ba2f00  0 <cls> 
> ./src/cls/cephfs/cls_cephfs.cc:201: loading cephfs
> 2022-04-10T12:12:36.924+0200 7fe6e8ba2f00  0 _get_class not permitted 
> to load sdk
> 2022-04-10T12:12:36.928+0200 7fe6e8ba2f00  0 _get_class not permitted 
> to load lua
> 2022-04-10T12:12:36.940+0200 7fe6e8ba2f00  0 <cls> 
> ./src/cls/hello/cls_hello.cc:316: loading cls_hello
> 2022-04-10T12:12:36.940+0200 7fe6e8ba2f00  0 osd.12 15550 crush map 
> has features 432629239337189376, adjusting msgr requires for clients
> 2022-04-10T12:12:36.940+0200 7fe6e8ba2f00  0 osd.12 15550 crush map 
> has features 432629239337189376 was 8705, adjusting msgr requires for 
> mons
> 2022-04-10T12:12:36.940+0200 7fe6e8ba2f00  0 osd.12 15550 crush map 
> has features 3314933000854323200, adjusting msgr requires for osds
>
> -----Ursprüngliche Nachricht-----
> Von: VELARTIS GmbH | Philipp Dürhammer <p.duerhammer@xxxxxxxxxxx>
> Gesendet: Sonntag, 10. April 2022 11:49
> An: ceph-users@xxxxxxx
> Betreff:  osd needs moer then one hour to start with heavy 
> reads
>
> Hi,
>
> allways when i try the start / restart an osd it takes more then one hour to start up. Meanwhile I see more then 100Mb read on the osd the whole time. (it's an enterprise ssd) logs hows:
> 2022-04-10T11:42:44.219+0200 7f68961f8700 -1 osd.12 15550 *** 
> Immediate shutdown (osd_fast_shutdown=true) ***
> 2022-04-10T11:42:47.007+0200 7fe6e8ba2f00  0 set uid:gid to 
> 64045:64045 (ceph:ceph)
> 2022-04-10T11:42:47.007+0200 7fe6e8ba2f00  0 ceph version 16.2.7 
> (f9aa029788115b5df5eeee328f584156565ee5b7) pacific (stable), process 
> ceph-osd, pid 1347448
> 2022-04-10T11:42:47.007+0200 7fe6e8ba2f00  0 pidfile_write: ignore 
> empty --pid-file
> 2022-04-10T11:42:47.575+0200 7fe6e8ba2f00  0 starting osd.12 osd_data 
> /var/lib/ceph/osd/ceph-12 /var/lib/ceph/osd/ceph-12/journal
> 2022-04-10T11:42:47.603+0200 7fe6e8ba2f00  0 load: jerasure load: lrc 
> load: isa
> 2022-04-10T11:42:48.247+0200 7fe6e8ba2f00  0 osd.12:0.OSDShard using 
> op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, 
> cutoff=196)
> 2022-04-10T11:42:48.555+0200 7fe6e8ba2f00  0 osd.12:1.OSDShard using 
> op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, 
> cutoff=196)
> 2022-04-10T11:42:48.863+0200 7fe6e8ba2f00  0 osd.12:2.OSDShard using 
> op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, 
> cutoff=196)
> 2022-04-10T11:42:49.167+0200 7fe6e8ba2f00  0 osd.12:3.OSDShard using 
> op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, 
> cutoff=196)
> 2022-04-10T11:42:49.463+0200 7fe6e8ba2f00  0 osd.12:4.OSDShard using 
> op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, 
> cutoff=196)
> 2022-04-10T11:42:50.191+0200 7fe6e8ba2f00  0 osd.12:5.OSDShard using 
> op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, 
> cutoff=196)
> 2022-04-10T11:42:50.931+0200 7fe6e8ba2f00  0 osd.12:6.OSDShard using 
> op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, 
> cutoff=196)
> 2022-04-10T11:42:51.607+0200 7fe6e8ba2f00  0 osd.12:7.OSDShard using 
> op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, 
> cutoff=196)
> 2022-04-10T11:42:51.607+0200 7fe6e8ba2f00  0 
> bluestore(/var/lib/ceph/osd/ceph-12) _open_db_and_around read-only:0 
> repair:0
> (END)
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an 
> email to ceph-users-leave@xxxxxxx
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an 
> email to ceph-users-leave@xxxxxxx

--
Igor Fedotov
Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx


_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux