Hi All
We had a cluster (v13.2.4) with 32 osds in total. At first, an osd (osd.18)
in cluster was down. So, we tried to remove the osd and added a new one
(osd.32) with new ID. We unplugged the disk (osd.18) and plugged in a new
disk in the same slot and add osd.32 into cluster. Then, osd.32 was
booting, but, we found it takes much time (around 18 mins) for the osd to
change to up state. Diving into osd.32 logs, we see that there is much
rocksdb activity before osd.32 change to up state. Can anyone explain why
this happened or give me any advice about how to prevent from this. Thanks.
[osd.32 log]
2020-08-03 15:36:58.852 7f88021fa1c0 0 osd.32 0 done with init, starting
boot process
2020-08-03 15:36:58.852 7f88021fa1c0 1 osd.32 0 start_boot
2020-08-03 15:36:58.854 7f87db02b700 -1 osd.32 0 waiting for initial osdmap
2020-08-03 15:36:58.855 7f87e4ba0700 -1 osd.32 0 failed to load OSD map for
epoch 22010, got 0 bytes
2020-08-03 15:36:58.955 7f87e0836700 0 osd.32 22011 crush map has features
283675107524608, adjusting msgr requires for clients
2020-08-03 15:36:58.955 7f87e0836700 0 osd.32 22011 crush map has features
283675107524608 was 288232575208792577, adjusting msgr requires for mons
*2020-08-03 15:36:58.955* 7f87e0836700 0 osd.32 22011 crush map has
features 720859615486820352, adjusting msgr requires for osds
2020-08-03 15:37:31.182 7f87e1037700 4 rocksdb:
[/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/db_impl_write.cc:1346]
[default] New memtable created with log file: #16. Immutable memtables: 0.
2020-08-03 15:37:31.285 7f87e8045700 4 rocksdb: (Original Log Time
2020/08/03-15:37:31.183995)
[/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/db_impl_compaction_flush.cc:1396]
Calling FlushMemTableToOutputFile with column family [default], flush slots
available 1, compaction slots available 1, flush slots scheduled 1,
compaction slots scheduled 0
2020-08-03 15:37:31.285 7f87e8045700 4 rocksdb:
[/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/flush_job.cc:300]
[default] [JOB 3] Flushing memtable with next log file: 16
-------- lots of rocksdb activity---------
2020-08-03 15:54:21.704 7f87e8045700 4 rocksdb: (Original Log Time
2020/08/03-15:54:21.705680)
[/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/memtable_list.cc:397]
[default] Level-0 commit table #112: memtable #1 done
2020-08-03 15:54:21.704 7f87e8045700 4 rocksdb: (Original Log Time
2020/08/03-15:54:21.705704) EVENT_LOG_v1 {"time_micros": 1596441261705697,
"job": 51, "event": "flush_finished", "output_compression":
"NoCompression", "lsm_state": [1, 3, 0, 0, 0, 0, 0], "immutable_memtables":
0}
2020-08-03 15:54:21.704 7f87e8045700 4 rocksdb: (Original Log Time
2020/08/03-15:54:21.705721)
[/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/db_impl_compaction_flush.cc:172]
[default] Level summary: base level 1 max bytes base 268435456 files[1 3 0
0 0 0 0] max score 0.75
*2020-08-03 15:54:38.567* 7f87e0836700 1 osd.32 502096 state: booting ->
active
2020-08-03 15:54:38.567 7f87d5820700 1 osd.32 pg_epoch: 502096 pg[1.17e(
empty local-lis/les=0/0 n=0 ec=11627/16 lis/c 501703/501703 les/c/f
501704/501704/0 502096/502096/502096) [32,26,28] r=0 lpr=502096
pi=[501703,502096)/1 crt=0'0 mlcod 0'0 unknown mbc={}] state<Start>:
transitioning to Primary
Best
Jerry
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
