Thanks for the responses so far.
Putting the OSD in is not a problem, but it will not report in the mgr as
being online. Podman ps does show the container for osd.13 as online
though. I've tried killing it and it relaunches, but the OSD does not
report as up in mgr. ceph orch restart osd.all-available-devices likewise
restarts all OSDs (120 so far) but this one never comes up. Same with
rebooting.
Here is a snippet of the log requested.
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.write_buffer_si>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.max_write_buffer_numb>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.compression: >
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.botto>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.prefix_extractor>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.memtable_insert_with>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.num_levels>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.min_write_buffe>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.max_write_buffer_n>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.max_write_buffer_s>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.bottommost_>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.botto>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.bottommo>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.bottommost_com>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.bottommost_com>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.botto>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.compression>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.compr>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.compress>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.compression_op>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.compression_op>
Oct 26 19:31:59 <HOST> conmon[780974]: rain_bytes: 0
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.compr>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.level0_file_num_c>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.level0_slowdo>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.level0_st>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.targ>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.target_fil>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.max_byt>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.level_compaction_dynam>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.max_bytes_for>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.max_bytes_for_level_mu>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.max_bytes_for_level_mu>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.max_bytes_for_level_mu>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.max_bytes_for_level_mu>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.max_bytes_for_level_mu>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.max_bytes_for_level_mu>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.max_bytes_for_level_mu>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.max_sequential_s>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.max>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.soft_pending_compact>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.hard_pending_compact>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.rate_limit_delay>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.disable>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Optio>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.compaction_options_uni>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.compaction_options_uni>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.compaction_options_uni>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.compaction_options_uni>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.compaction_options_uni>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.compaction_options_uni>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.compaction_options_fif>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.compaction_options_fif>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.tabl>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.inpl>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.inplac>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.memtable>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.memtable>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.memtable_huge_page_s>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Opti>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.max>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.optimiz>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.paranoi>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.force_c>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.report_>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: >
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: Options.periodic_comp>
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: [column_family.cc:555] >
Oct 26 19:31:59 <HOST> conmon[780974]:
Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
7f08e65a4080 4 rocksdb: [column_family.cc:555] >
Oct 26 19:31:59 <HOST> conmon[780974]:
Oct 26 19:32:09 <HOST> conmon[780974]: debug 2021-10-26T19:32:09.795+0000
7f08d0cb7700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:32:10 <HOST> conmon[780974]: debug 2021-10-26T19:32:10.771+0000
7f08d04b6700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:32:16 <HOST> conmon[780974]: debug 2021-10-26T19:32:16.183+0000
7f08cfcb5700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:32:18 <HOST> conmon[780974]: debug 2021-10-26T19:32:18.915+0000
7f08d04b6700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:32:21 <HOST> conmon[780974]: debug 2021-10-26T19:32:21.659+0000
7f08cfcb5700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:32:24 <HOST> conmon[780974]: debug 2021-10-26T19:32:24.387+0000
7f08d04b6700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:32:29 <HOST> conmon[780974]: debug 2021-10-26T19:32:29.547+0000
7f08ce4b2700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:32:32 <HOST> conmon[780974]: debug 2021-10-26T19:32:32.199+0000
7f08cecb3700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:32:34 <HOST> conmon[780974]: debug 2021-10-26T19:32:34.987+0000
7f08cdcb1700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:32:37 <HOST> conmon[780974]: debug 2021-10-26T19:32:37.423+0000
7f08ce4b2700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:32:40 <HOST> conmon[780974]: debug 2021-10-26T19:32:40.119+0000
7f08cdcb1700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:32:49 <HOST> conmon[780974]: debug 2021-10-26T19:32:49.075+0000
7f08cfcb5700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:02 <HOST> conmon[780974]: debug 2021-10-26T19:33:02.859+0000
7f08cf4b4700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:03 <HOST> conmon[780974]: debug 2021-10-26T19:33:03.983+0000
7f08cdcb1700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:06 <HOST> conmon[780974]: debug 2021-10-26T19:33:06.703+0000
7f08ce4b2700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:17 <HOST> conmon[780974]: debug 2021-10-26T19:33:17.911+0000
7f08d14b8700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:20 <HOST> conmon[780974]: debug 2021-10-26T19:33:20.579+0000
7f08cfcb5700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:23 <HOST> conmon[780974]: debug 2021-10-26T19:33:23.287+0000
7f08d04b6700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:28 <HOST> conmon[780974]: debug 2021-10-26T19:33:28.467+0000
7f08d04b6700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:33 <HOST> conmon[780974]: debug 2021-10-26T19:33:33.575+0000
7f08cdcb1700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:34 <HOST> conmon[780974]: debug 2021-10-26T19:33:34.651+0000
7f08cecb3700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:37 <HOST> conmon[780974]: debug 2021-10-26T19:33:37.343+0000
7f08cf4b4700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:38 <HOST> conmon[780974]: debug 2021-10-26T19:33:38.275+0000
7f08d4cbf700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:41 <HOST> conmon[780974]: debug 2021-10-26T19:33:41.019+0000
7f08ce4b2700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:41 <HOST> conmon[780974]: debug 2021-10-26T19:33:41.959+0000
7f08cdcb1700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:44 <HOST> conmon[780974]: debug 2021-10-26T19:33:44.667+0000
7f08cecb3700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:55 <HOST> conmon[780974]: debug 2021-10-26T19:33:55.724+0000
7f08cecb3700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:58 <HOST> conmon[780974]: debug 2021-10-26T19:33:58.580+0000
7f08ce4b2700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:33:59 <HOST> conmon[780974]: debug 2021-10-26T19:33:59.520+0000
7f08cdcb1700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08cdcb1700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4568] Recovere>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:10 <HOST> conmon[780974]: debug 2021-10-26T19:34:10.396+0000
7f08e65a4080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1>
Oct 26 19:34:10 <HOST> conmon[780974]: debug 2021-10-26T19:34:10.396+0000
7f08e65a4080 4 rocksdb: [db_impl/db_impl_open.cc:760] >
Oct 26 19:34:10 <HOST> conmon[780974]:
Oct 26 19:36:07 <HOST> conmon[780974]: debug 2021-10-26T19:36:07.436+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:36:38 <HOST> conmon[780974]: debug 2021-10-26T19:36:38.132+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:37:10 <HOST> conmon[780974]: debug 2021-10-26T19:37:10.408+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:37:42 <HOST> conmon[780974]: debug 2021-10-26T19:37:42.224+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:38:01 <HOST> conmon[780974]: debug 2021-10-26T19:38:01.256+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:38:23 <HOST> conmon[780974]: debug 2021-10-26T19:38:23.056+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:39:40 <HOST> conmon[780974]: debug 2021-10-26T19:39:40.160+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:40:48 <HOST> conmon[780974]: debug 2021-10-26T19:40:48.769+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:41:52 <HOST> conmon[780974]: debug 2021-10-26T19:41:52.509+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:42:05 <HOST> conmon[780974]: debug 2021-10-26T19:42:05.433+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:42:49 <HOST> conmon[780974]: debug 2021-10-26T19:42:49.985+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:43:09 <HOST> conmon[780974]: debug 2021-10-26T19:43:09.069+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:43:43 <HOST> conmon[780974]: debug 2021-10-26T19:43:43.677+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:43:55 <HOST> conmon[780974]: debug 2021-10-26T19:43:55.237+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:45:02 <HOST> conmon[780974]: debug 2021-10-26T19:45:02.113+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:45:46 <HOST> conmon[780974]: debug 2021-10-26T19:45:46.021+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:46:02 <HOST> conmon[780974]: debug 2021-10-26T19:46:02.757+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:46:24 <HOST> conmon[780974]: debug 2021-10-26T19:46:24.257+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:47:11 <HOST> conmon[780974]: debug 2021-10-26T19:47:11.394+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:47:30 <HOST> conmon[780974]: debug 2021-10-26T19:47:30.502+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:47:35 <HOST> conmon[780974]: debug 2021-10-26T19:47:35.914+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:48:39 <HOST> conmon[780974]: debug 2021-10-26T19:48:39.110+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:50:44 <HOST> conmon[780974]: debug 2021-10-26T19:50:44.506+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
Oct 26 19:51:41 <HOST> conmon[780974]: debug 2021-10-26T19:51:41.838+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
lines 94670-94720/94720 (END)
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [default] (I>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-0] (ID 1)>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-1] (ID 2)>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-2] (ID 3)>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-0] (ID 4)>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-1] (ID 5)>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-2] (ID 6)>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-0] (ID 7)>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-1] (ID 8)>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-2] (ID 9)>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [L] (ID 10),>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [P] (ID 11),>
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:10 <HOST> conmon[780974]: debug 2021-10-26T19:34:10.396+0000
7f08e65a4080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1635276850400811, ">
Oct 26 19:34:10 <HOST> conmon[780974]: debug 2021-10-26T19:34:10.396+0000
7f08e65a4080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #12>
Oct 26 19:34:10 <HOST> conmon[780974]:
Oct 26 19:36:07 <HOST> conmon[780974]: debug 2021-10-26T19:36:07.436+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:36:38 <HOST> conmon[780974]: debug 2021-10-26T19:36:38.132+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:37:10 <HOST> conmon[780974]: debug 2021-10-26T19:37:10.408+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:37:42 <HOST> conmon[780974]: debug 2021-10-26T19:37:42.224+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:38:01 <HOST> conmon[780974]: debug 2021-10-26T19:38:01.256+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:38:23 <HOST> conmon[780974]: debug 2021-10-26T19:38:23.056+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:39:40 <HOST> conmon[780974]: debug 2021-10-26T19:39:40.160+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:40:48 <HOST> conmon[780974]: debug 2021-10-26T19:40:48.769+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:41:52 <HOST> conmon[780974]: debug 2021-10-26T19:41:52.509+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:42:05 <HOST> conmon[780974]: debug 2021-10-26T19:42:05.433+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:42:49 <HOST> conmon[780974]: debug 2021-10-26T19:42:49.985+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:43:09 <HOST> conmon[780974]: debug 2021-10-26T19:43:09.069+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:43:43 <HOST> conmon[780974]: debug 2021-10-26T19:43:43.677+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:43:55 <HOST> conmon[780974]: debug 2021-10-26T19:43:55.237+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:45:02 <HOST> conmon[780974]: debug 2021-10-26T19:45:02.113+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:45:46 <HOST> conmon[780974]: debug 2021-10-26T19:45:46.021+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:46:02 <HOST> conmon[780974]: debug 2021-10-26T19:46:02.757+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:46:24 <HOST> conmon[780974]: debug 2021-10-26T19:46:24.257+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:47:11 <HOST> conmon[780974]: debug 2021-10-26T19:47:11.394+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:47:30 <HOST> conmon[780974]: debug 2021-10-26T19:47:30.502+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:47:35 <HOST> conmon[780974]: debug 2021-10-26T19:47:35.914+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:48:39 <HOST> conmon[780974]: debug 2021-10-26T19:48:39.110+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:50:44 <HOST> conmon[780974]: debug 2021-10-26T19:50:44.506+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
Oct 26 19:51:41 <HOST> conmon[780974]: debug 2021-10-26T19:51:41.838+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
lines 94670-94720/94720 (END)
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [default] (ID
0), log number is 127
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-0] (ID 1),
log number is 31
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-1] (ID 2),
log number is 31
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-2] (ID 3),
log number is 31
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-0] (ID 4),
log number is 127
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-1] (ID 5),
log number is 127
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-2] (ID 6),
log number is 127
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-0] (ID 7),
log number is 127
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-1] (ID 8),
log number is 127
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-2] (ID 9),
log number is 127
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [L] (ID 10),
log number is 137
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [P] (ID 11),
log number is 127
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:10 <HOST> conmon[780974]: debug 2021-10-26T19:34:10.396+0000
7f08e65a4080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1635276850400811,
"job": 1, "event": "recovery_st>
Oct 26 19:34:10 <HOST> conmon[780974]: debug 2021-10-26T19:34:10.396+0000
7f08e65a4080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #127
mode 2
Oct 26 19:34:10 <HOST> conmon[780974]:
Oct 26 19:36:07 <HOST> conmon[780974]: debug 2021-10-26T19:36:07.436+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3f8bd30000~10>
Oct 26 19:36:38 <HOST> conmon[780974]: debug 2021-10-26T19:36:38.132+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3f8e770000~10>
Oct 26 19:37:10 <HOST> conmon[780974]: debug 2021-10-26T19:37:10.408+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3f941f0000~10>
Oct 26 19:37:42 <HOST> conmon[780974]: debug 2021-10-26T19:37:42.224+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3f9a9d0000~50>
Oct 26 19:38:01 <HOST> conmon[780974]: debug 2021-10-26T19:38:01.256+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3f9e6c0000~10>
Oct 26 19:38:23 <HOST> conmon[780974]: debug 2021-10-26T19:38:23.056+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x2859fe0000~20>
Oct 26 19:39:40 <HOST> conmon[780974]: debug 2021-10-26T19:39:40.160+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3fb6c00000~10>
Oct 26 19:40:48 <HOST> conmon[780974]: debug 2021-10-26T19:40:48.769+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3fcd450000~10>
Oct 26 19:41:52 <HOST> conmon[780974]: debug 2021-10-26T19:41:52.509+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3fe0920000~40>
Oct 26 19:42:05 <HOST> conmon[780974]: debug 2021-10-26T19:42:05.433+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3fe24b0000~10>
Oct 26 19:42:49 <HOST> conmon[780974]: debug 2021-10-26T19:42:49.985+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x2871b70000~20>
Oct 26 19:43:09 <HOST> conmon[780974]: debug 2021-10-26T19:43:09.069+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3ff3730000~10>
Oct 26 19:43:43 <HOST> conmon[780974]: debug 2021-10-26T19:43:43.677+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3ff99a0000~10>
Oct 26 19:43:55 <HOST> conmon[780974]: debug 2021-10-26T19:43:55.237+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3ffcfd0000~20>
Oct 26 19:45:02 <HOST> conmon[780974]: debug 2021-10-26T19:45:02.113+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x28601b0000~40>
Oct 26 19:45:46 <HOST> conmon[780974]: debug 2021-10-26T19:45:46.021+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x4014b60000~20>
Oct 26 19:46:02 <HOST> conmon[780974]: debug 2021-10-26T19:46:02.757+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x4015e70000~10>
Oct 26 19:46:24 <HOST> conmon[780974]: debug 2021-10-26T19:46:24.257+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x401a520000~10>
Oct 26 19:47:11 <HOST> conmon[780974]: debug 2021-10-26T19:47:11.394+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x40303e0000~10>
Oct 26 19:47:30 <HOST> conmon[780974]: debug 2021-10-26T19:47:30.502+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x40408c0000~20>
Oct 26 19:47:35 <HOST> conmon[780974]: debug 2021-10-26T19:47:35.914+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x40409b0000~10>
Oct 26 19:48:39 <HOST> conmon[780974]: debug 2021-10-26T19:48:39.110+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x4069060000~10>
Oct 26 19:50:44 <HOST> conmon[780974]: debug 2021-10-26T19:50:44.506+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x407e7b0000~10>
Oct 26 19:51:41 <HOST> conmon[780974]: debug 2021-10-26T19:51:41.838+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x4092c50000~10>
lines 94670-94720/94720 (END)
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [default] (ID
0), log number is 127
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-0] (ID 1),
log number is 31
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-1] (ID 2),
log number is 31
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-2] (ID 3),
log number is 31
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-0] (ID 4),
log number is 127
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-1] (ID 5),
log number is 127
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-2] (ID 6),
log number is 127
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-0] (ID 7),
log number is 127
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-1] (ID 8),
log number is 127
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-2] (ID 9),
log number is 127
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [L] (ID 10),
log number is 137
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [P] (ID 11),
log number is 127
Oct 26 19:34:09 <HOST> conmon[780974]:
Oct 26 19:34:10 <HOST> conmon[780974]: debug 2021-10-26T19:34:10.396+0000
7f08e65a4080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1635276850400811,
"job": 1, "event": "recovery_star>
Oct 26 19:34:10 <HOST> conmon[780974]: debug 2021-10-26T19:34:10.396+0000
7f08e65a4080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #127
mode 2
Oct 26 19:34:10 <HOST> conmon[780974]:
Oct 26 19:36:07 <HOST> conmon[780974]: debug 2021-10-26T19:36:07.436+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3f8bd30000~1000>
Oct 26 19:36:38 <HOST> conmon[780974]: debug 2021-10-26T19:36:38.132+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3f8e770000~1000>
Oct 26 19:37:10 <HOST> conmon[780974]: debug 2021-10-26T19:37:10.408+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3f941f0000~1000>
Oct 26 19:37:42 <HOST> conmon[780974]: debug 2021-10-26T19:37:42.224+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3f9a9d0000~5000>
Oct 26 19:38:01 <HOST> conmon[780974]: debug 2021-10-26T19:38:01.256+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3f9e6c0000~1000>
Oct 26 19:38:23 <HOST> conmon[780974]: debug 2021-10-26T19:38:23.056+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x2859fe0000~2000>
Oct 26 19:39:40 <HOST> conmon[780974]: debug 2021-10-26T19:39:40.160+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3fb6c00000~1000>
Oct 26 19:40:48 <HOST> conmon[780974]: debug 2021-10-26T19:40:48.769+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3fcd450000~1000>
Oct 26 19:41:52 <HOST> conmon[780974]: debug 2021-10-26T19:41:52.509+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3fe0920000~4000>
Oct 26 19:42:05 <HOST> conmon[780974]: debug 2021-10-26T19:42:05.433+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3fe24b0000~1000>
Oct 26 19:42:49 <HOST> conmon[780974]: debug 2021-10-26T19:42:49.985+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x2871b70000~2000>
Oct 26 19:43:09 <HOST> conmon[780974]: debug 2021-10-26T19:43:09.069+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3ff3730000~1000>
Oct 26 19:43:43 <HOST> conmon[780974]: debug 2021-10-26T19:43:43.677+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3ff99a0000~1000>
Oct 26 19:43:55 <HOST> conmon[780974]: debug 2021-10-26T19:43:55.237+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x3ffcfd0000~2000>
Oct 26 19:45:02 <HOST> conmon[780974]: debug 2021-10-26T19:45:02.113+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x28601b0000~4000>
Oct 26 19:45:46 <HOST> conmon[780974]: debug 2021-10-26T19:45:46.021+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x4014b60000~2000>
Oct 26 19:46:02 <HOST> conmon[780974]: debug 2021-10-26T19:46:02.757+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x4015e70000~1000>
Oct 26 19:46:24 <HOST> conmon[780974]: debug 2021-10-26T19:46:24.257+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x401a520000~1000>
Oct 26 19:47:11 <HOST> conmon[780974]: debug 2021-10-26T19:47:11.394+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x40303e0000~1000>
Oct 26 19:47:30 <HOST> conmon[780974]: debug 2021-10-26T19:47:30.502+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x40408c0000~2000>
Oct 26 19:47:35 <HOST> conmon[780974]: debug 2021-10-26T19:47:35.914+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x40409b0000~1000>
Oct 26 19:48:39 <HOST> conmon[780974]: debug 2021-10-26T19:48:39.110+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x4069060000~1000>
Oct 26 19:50:44 <HOST> conmon[780974]: debug 2021-10-26T19:50:44.506+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x407e7b0000~1000>
Oct 26 19:51:41 <HOST> conmon[780974]: debug 2021-10-26T19:51:41.838+0000
7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
stalled read 0x4092c50000~1000>
lines 94670-94720/94720 (END)
On Tue, Oct 26, 2021 at 1:39 AM 胡 玮文 <huww98@xxxxxxxxxxx> wrote:
> Could you post the logs of the problematic OSDs? E.g.:
>
>
>
> cephadm logs --name osd.0
>
>
>
> *发件人: *Marco Pizzolo <marcopizzolo@xxxxxxxxx>
> *发送时间: *2021年10月26日 7:15
> *收件人: *ceph-users <ceph-users@xxxxxxx>
> *主题: * 16.2.6 OSD down, out but container running....
>
>
>
> Hello Everyone,
>
> I'm seeing an issue where the podman container is running, but the osd is
> being reported as down and out. restarting service doesn't help, neither
> does rebooting the host.
>
> What am I missing?
>
> Thanks,
> Marco
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
