Is there any command or log I can provide a sample from that would help to
pinpoint the issue? The 119 of 120 OSDs are working correctly by all
accounts, but I am just unable to have the bring the last one fully online.
Thank you,
On Tue, Oct 26, 2021 at 3:59 PM Marco Pizzolo <marcopizzolo@xxxxxxxxx>
wrote:
> Thanks for the responses so far.
>
> Putting the OSD in is not a problem, but it will not report in the mgr as
> being online. Podman ps does show the container for osd.13 as online
> though. I've tried killing it and it relaunches, but the OSD does not
> report as up in mgr. ceph orch restart osd.all-available-devices likewise
> restarts all OSDs (120 so far) but this one never comes up. Same with
> rebooting.
>
> Here is a snippet of the log requested.
>
>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.write_buffer_si>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.max_write_buffer_numb>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.compression: >
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.botto>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.prefix_extractor>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.memtable_insert_with>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.num_levels>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.min_write_buffe>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.max_write_buffer_n>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.max_write_buffer_s>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.bottommost_>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.botto>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.bottommo>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.bottommost_com>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.bottommost_com>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.botto>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.compression>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.compr>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.compress>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.compression_op>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.compression_op>
> Oct 26 19:31:59 <HOST> conmon[780974]: rain_bytes: 0
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.compr>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.level0_file_num_c>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.level0_slowdo>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.level0_st>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.targ>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.target_fil>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.max_byt>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.level_compaction_dynam>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.max_bytes_for>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.max_bytes_for_level_mu>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.max_bytes_for_level_mu>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.max_bytes_for_level_mu>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.max_bytes_for_level_mu>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.max_bytes_for_level_mu>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.max_bytes_for_level_mu>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.max_bytes_for_level_mu>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.max_sequential_s>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.max>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.soft_pending_compact>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.hard_pending_compact>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.rate_limit_delay>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.disable>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Optio>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.compaction_options_uni>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.compaction_options_uni>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.compaction_options_uni>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.compaction_options_uni>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.compaction_options_uni>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.compaction_options_uni>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.compaction_options_fif>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.compaction_options_fif>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.tabl>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.inpl>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.inplac>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.memtable>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.memtable>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.memtable_huge_page_s>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Opti>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.max>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.optimiz>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.paranoi>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.force_c>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.report_>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: >
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: Options.periodic_comp>
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: [column_family.cc:555] >
> Oct 26 19:31:59 <HOST> conmon[780974]:
> Oct 26 19:31:59 <HOST> conmon[780974]: debug 2021-10-26T19:31:59.827+0000
> 7f08e65a4080 4 rocksdb: [column_family.cc:555] >
> Oct 26 19:31:59 <HOST> conmon[780974]:
> Oct 26 19:32:09 <HOST> conmon[780974]: debug 2021-10-26T19:32:09.795+0000
> 7f08d0cb7700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:32:10 <HOST> conmon[780974]: debug 2021-10-26T19:32:10.771+0000
> 7f08d04b6700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:32:16 <HOST> conmon[780974]: debug 2021-10-26T19:32:16.183+0000
> 7f08cfcb5700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:32:18 <HOST> conmon[780974]: debug 2021-10-26T19:32:18.915+0000
> 7f08d04b6700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:32:21 <HOST> conmon[780974]: debug 2021-10-26T19:32:21.659+0000
> 7f08cfcb5700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:32:24 <HOST> conmon[780974]: debug 2021-10-26T19:32:24.387+0000
> 7f08d04b6700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:32:29 <HOST> conmon[780974]: debug 2021-10-26T19:32:29.547+0000
> 7f08ce4b2700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:32:32 <HOST> conmon[780974]: debug 2021-10-26T19:32:32.199+0000
> 7f08cecb3700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:32:34 <HOST> conmon[780974]: debug 2021-10-26T19:32:34.987+0000
> 7f08cdcb1700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:32:37 <HOST> conmon[780974]: debug 2021-10-26T19:32:37.423+0000
> 7f08ce4b2700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:32:40 <HOST> conmon[780974]: debug 2021-10-26T19:32:40.119+0000
> 7f08cdcb1700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:32:49 <HOST> conmon[780974]: debug 2021-10-26T19:32:49.075+0000
> 7f08cfcb5700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:02 <HOST> conmon[780974]: debug 2021-10-26T19:33:02.859+0000
> 7f08cf4b4700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:03 <HOST> conmon[780974]: debug 2021-10-26T19:33:03.983+0000
> 7f08cdcb1700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:06 <HOST> conmon[780974]: debug 2021-10-26T19:33:06.703+0000
> 7f08ce4b2700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:17 <HOST> conmon[780974]: debug 2021-10-26T19:33:17.911+0000
> 7f08d14b8700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:20 <HOST> conmon[780974]: debug 2021-10-26T19:33:20.579+0000
> 7f08cfcb5700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:23 <HOST> conmon[780974]: debug 2021-10-26T19:33:23.287+0000
> 7f08d04b6700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:28 <HOST> conmon[780974]: debug 2021-10-26T19:33:28.467+0000
> 7f08d04b6700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:33 <HOST> conmon[780974]: debug 2021-10-26T19:33:33.575+0000
> 7f08cdcb1700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:34 <HOST> conmon[780974]: debug 2021-10-26T19:33:34.651+0000
> 7f08cecb3700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:37 <HOST> conmon[780974]: debug 2021-10-26T19:33:37.343+0000
> 7f08cf4b4700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:38 <HOST> conmon[780974]: debug 2021-10-26T19:33:38.275+0000
> 7f08d4cbf700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:41 <HOST> conmon[780974]: debug 2021-10-26T19:33:41.019+0000
> 7f08ce4b2700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:41 <HOST> conmon[780974]: debug 2021-10-26T19:33:41.959+0000
> 7f08cdcb1700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:44 <HOST> conmon[780974]: debug 2021-10-26T19:33:44.667+0000
> 7f08cecb3700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:55 <HOST> conmon[780974]: debug 2021-10-26T19:33:55.724+0000
> 7f08cecb3700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:58 <HOST> conmon[780974]: debug 2021-10-26T19:33:58.580+0000
> 7f08ce4b2700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:33:59 <HOST> conmon[780974]: debug 2021-10-26T19:33:59.520+0000
> 7f08cdcb1700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08cdcb1700 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4568] Recovere>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column f>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:10 <HOST> conmon[780974]: debug 2021-10-26T19:34:10.396+0000
> 7f08e65a4080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1>
> Oct 26 19:34:10 <HOST> conmon[780974]: debug 2021-10-26T19:34:10.396+0000
> 7f08e65a4080 4 rocksdb: [db_impl/db_impl_open.cc:760] >
> Oct 26 19:34:10 <HOST> conmon[780974]:
> Oct 26 19:36:07 <HOST> conmon[780974]: debug 2021-10-26T19:36:07.436+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:36:38 <HOST> conmon[780974]: debug 2021-10-26T19:36:38.132+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:37:10 <HOST> conmon[780974]: debug 2021-10-26T19:37:10.408+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:37:42 <HOST> conmon[780974]: debug 2021-10-26T19:37:42.224+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:38:01 <HOST> conmon[780974]: debug 2021-10-26T19:38:01.256+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:38:23 <HOST> conmon[780974]: debug 2021-10-26T19:38:23.056+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:39:40 <HOST> conmon[780974]: debug 2021-10-26T19:39:40.160+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:40:48 <HOST> conmon[780974]: debug 2021-10-26T19:40:48.769+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:41:52 <HOST> conmon[780974]: debug 2021-10-26T19:41:52.509+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:42:05 <HOST> conmon[780974]: debug 2021-10-26T19:42:05.433+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:42:49 <HOST> conmon[780974]: debug 2021-10-26T19:42:49.985+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:43:09 <HOST> conmon[780974]: debug 2021-10-26T19:43:09.069+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:43:43 <HOST> conmon[780974]: debug 2021-10-26T19:43:43.677+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:43:55 <HOST> conmon[780974]: debug 2021-10-26T19:43:55.237+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:45:02 <HOST> conmon[780974]: debug 2021-10-26T19:45:02.113+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:45:46 <HOST> conmon[780974]: debug 2021-10-26T19:45:46.021+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:46:02 <HOST> conmon[780974]: debug 2021-10-26T19:46:02.757+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:46:24 <HOST> conmon[780974]: debug 2021-10-26T19:46:24.257+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:47:11 <HOST> conmon[780974]: debug 2021-10-26T19:47:11.394+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:47:30 <HOST> conmon[780974]: debug 2021-10-26T19:47:30.502+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:47:35 <HOST> conmon[780974]: debug 2021-10-26T19:47:35.914+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:48:39 <HOST> conmon[780974]: debug 2021-10-26T19:48:39.110+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:50:44 <HOST> conmon[780974]: debug 2021-10-26T19:50:44.506+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> Oct 26 19:51:41 <HOST> conmon[780974]: debug 2021-10-26T19:51:41.838+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/c>
> lines 94670-94720/94720 (END)
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [default] (I>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-0] (ID 1)>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-1] (ID 2)>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-2] (ID 3)>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-0] (ID 4)>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-1] (ID 5)>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-2] (ID 6)>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-0] (ID 7)>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-1] (ID 8)>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-2] (ID 9)>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [L] (ID 10),>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [P] (ID 11),>
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:10 <HOST> conmon[780974]: debug 2021-10-26T19:34:10.396+0000
> 7f08e65a4080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1635276850400811, ">
> Oct 26 19:34:10 <HOST> conmon[780974]: debug 2021-10-26T19:34:10.396+0000
> 7f08e65a4080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #12>
> Oct 26 19:34:10 <HOST> conmon[780974]:
> Oct 26 19:36:07 <HOST> conmon[780974]: debug 2021-10-26T19:36:07.436+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:36:38 <HOST> conmon[780974]: debug 2021-10-26T19:36:38.132+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:37:10 <HOST> conmon[780974]: debug 2021-10-26T19:37:10.408+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:37:42 <HOST> conmon[780974]: debug 2021-10-26T19:37:42.224+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:38:01 <HOST> conmon[780974]: debug 2021-10-26T19:38:01.256+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:38:23 <HOST> conmon[780974]: debug 2021-10-26T19:38:23.056+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:39:40 <HOST> conmon[780974]: debug 2021-10-26T19:39:40.160+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:40:48 <HOST> conmon[780974]: debug 2021-10-26T19:40:48.769+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:41:52 <HOST> conmon[780974]: debug 2021-10-26T19:41:52.509+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:42:05 <HOST> conmon[780974]: debug 2021-10-26T19:42:05.433+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:42:49 <HOST> conmon[780974]: debug 2021-10-26T19:42:49.985+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:43:09 <HOST> conmon[780974]: debug 2021-10-26T19:43:09.069+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:43:43 <HOST> conmon[780974]: debug 2021-10-26T19:43:43.677+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:43:55 <HOST> conmon[780974]: debug 2021-10-26T19:43:55.237+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:45:02 <HOST> conmon[780974]: debug 2021-10-26T19:45:02.113+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:45:46 <HOST> conmon[780974]: debug 2021-10-26T19:45:46.021+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:46:02 <HOST> conmon[780974]: debug 2021-10-26T19:46:02.757+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:46:24 <HOST> conmon[780974]: debug 2021-10-26T19:46:24.257+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:47:11 <HOST> conmon[780974]: debug 2021-10-26T19:47:11.394+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:47:30 <HOST> conmon[780974]: debug 2021-10-26T19:47:30.502+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:47:35 <HOST> conmon[780974]: debug 2021-10-26T19:47:35.914+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:48:39 <HOST> conmon[780974]: debug 2021-10-26T19:48:39.110+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:50:44 <HOST> conmon[780974]: debug 2021-10-26T19:50:44.506+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> Oct 26 19:51:41 <HOST> conmon[780974]: debug 2021-10-26T19:51:41.838+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read>
> lines 94670-94720/94720 (END)
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [default] (ID
> 0), log number is 127
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-0] (ID 1),
> log number is 31
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-1] (ID 2),
> log number is 31
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-2] (ID 3),
> log number is 31
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-0] (ID 4),
> log number is 127
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-1] (ID 5),
> log number is 127
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-2] (ID 6),
> log number is 127
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-0] (ID 7),
> log number is 127
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-1] (ID 8),
> log number is 127
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-2] (ID 9),
> log number is 127
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [L] (ID 10),
> log number is 137
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [P] (ID 11),
> log number is 127
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:10 <HOST> conmon[780974]: debug 2021-10-26T19:34:10.396+0000
> 7f08e65a4080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1635276850400811,
> "job": 1, "event": "recovery_st>
> Oct 26 19:34:10 <HOST> conmon[780974]: debug 2021-10-26T19:34:10.396+0000
> 7f08e65a4080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #127
> mode 2
> Oct 26 19:34:10 <HOST> conmon[780974]:
> Oct 26 19:36:07 <HOST> conmon[780974]: debug 2021-10-26T19:36:07.436+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3f8bd30000~10>
> Oct 26 19:36:38 <HOST> conmon[780974]: debug 2021-10-26T19:36:38.132+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3f8e770000~10>
> Oct 26 19:37:10 <HOST> conmon[780974]: debug 2021-10-26T19:37:10.408+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3f941f0000~10>
> Oct 26 19:37:42 <HOST> conmon[780974]: debug 2021-10-26T19:37:42.224+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3f9a9d0000~50>
> Oct 26 19:38:01 <HOST> conmon[780974]: debug 2021-10-26T19:38:01.256+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3f9e6c0000~10>
> Oct 26 19:38:23 <HOST> conmon[780974]: debug 2021-10-26T19:38:23.056+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x2859fe0000~20>
> Oct 26 19:39:40 <HOST> conmon[780974]: debug 2021-10-26T19:39:40.160+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3fb6c00000~10>
> Oct 26 19:40:48 <HOST> conmon[780974]: debug 2021-10-26T19:40:48.769+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3fcd450000~10>
> Oct 26 19:41:52 <HOST> conmon[780974]: debug 2021-10-26T19:41:52.509+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3fe0920000~40>
> Oct 26 19:42:05 <HOST> conmon[780974]: debug 2021-10-26T19:42:05.433+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3fe24b0000~10>
> Oct 26 19:42:49 <HOST> conmon[780974]: debug 2021-10-26T19:42:49.985+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x2871b70000~20>
> Oct 26 19:43:09 <HOST> conmon[780974]: debug 2021-10-26T19:43:09.069+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3ff3730000~10>
> Oct 26 19:43:43 <HOST> conmon[780974]: debug 2021-10-26T19:43:43.677+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3ff99a0000~10>
> Oct 26 19:43:55 <HOST> conmon[780974]: debug 2021-10-26T19:43:55.237+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3ffcfd0000~20>
> Oct 26 19:45:02 <HOST> conmon[780974]: debug 2021-10-26T19:45:02.113+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x28601b0000~40>
> Oct 26 19:45:46 <HOST> conmon[780974]: debug 2021-10-26T19:45:46.021+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x4014b60000~20>
> Oct 26 19:46:02 <HOST> conmon[780974]: debug 2021-10-26T19:46:02.757+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x4015e70000~10>
> Oct 26 19:46:24 <HOST> conmon[780974]: debug 2021-10-26T19:46:24.257+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x401a520000~10>
> Oct 26 19:47:11 <HOST> conmon[780974]: debug 2021-10-26T19:47:11.394+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x40303e0000~10>
> Oct 26 19:47:30 <HOST> conmon[780974]: debug 2021-10-26T19:47:30.502+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x40408c0000~20>
> Oct 26 19:47:35 <HOST> conmon[780974]: debug 2021-10-26T19:47:35.914+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x40409b0000~10>
> Oct 26 19:48:39 <HOST> conmon[780974]: debug 2021-10-26T19:48:39.110+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x4069060000~10>
> Oct 26 19:50:44 <HOST> conmon[780974]: debug 2021-10-26T19:50:44.506+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x407e7b0000~10>
> Oct 26 19:51:41 <HOST> conmon[780974]: debug 2021-10-26T19:51:41.838+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x4092c50000~10>
> lines 94670-94720/94720 (END)
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [default] (ID
> 0), log number is 127
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-0] (ID 1),
> log number is 31
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-1] (ID 2),
> log number is 31
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [m-2] (ID 3),
> log number is 31
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-0] (ID 4),
> log number is 127
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-1] (ID 5),
> log number is 127
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [p-2] (ID 6),
> log number is 127
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-0] (ID 7),
> log number is 127
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-1] (ID 8),
> log number is 127
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [O-2] (ID 9),
> log number is 127
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [L] (ID 10),
> log number is 137
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:09 <HOST> conmon[780974]: debug 2021-10-26T19:34:09.476+0000
> 7f08e65a4080 4 rocksdb: [version_set.cc:4577] Column family [P] (ID 11),
> log number is 127
> Oct 26 19:34:09 <HOST> conmon[780974]:
> Oct 26 19:34:10 <HOST> conmon[780974]: debug 2021-10-26T19:34:10.396+0000
> 7f08e65a4080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1635276850400811,
> "job": 1, "event": "recovery_star>
> Oct 26 19:34:10 <HOST> conmon[780974]: debug 2021-10-26T19:34:10.396+0000
> 7f08e65a4080 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #127
> mode 2
> Oct 26 19:34:10 <HOST> conmon[780974]:
> Oct 26 19:36:07 <HOST> conmon[780974]: debug 2021-10-26T19:36:07.436+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3f8bd30000~1000>
> Oct 26 19:36:38 <HOST> conmon[780974]: debug 2021-10-26T19:36:38.132+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3f8e770000~1000>
> Oct 26 19:37:10 <HOST> conmon[780974]: debug 2021-10-26T19:37:10.408+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3f941f0000~1000>
> Oct 26 19:37:42 <HOST> conmon[780974]: debug 2021-10-26T19:37:42.224+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3f9a9d0000~5000>
> Oct 26 19:38:01 <HOST> conmon[780974]: debug 2021-10-26T19:38:01.256+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3f9e6c0000~1000>
> Oct 26 19:38:23 <HOST> conmon[780974]: debug 2021-10-26T19:38:23.056+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x2859fe0000~2000>
> Oct 26 19:39:40 <HOST> conmon[780974]: debug 2021-10-26T19:39:40.160+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3fb6c00000~1000>
> Oct 26 19:40:48 <HOST> conmon[780974]: debug 2021-10-26T19:40:48.769+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3fcd450000~1000>
> Oct 26 19:41:52 <HOST> conmon[780974]: debug 2021-10-26T19:41:52.509+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3fe0920000~4000>
> Oct 26 19:42:05 <HOST> conmon[780974]: debug 2021-10-26T19:42:05.433+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3fe24b0000~1000>
> Oct 26 19:42:49 <HOST> conmon[780974]: debug 2021-10-26T19:42:49.985+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x2871b70000~2000>
> Oct 26 19:43:09 <HOST> conmon[780974]: debug 2021-10-26T19:43:09.069+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3ff3730000~1000>
> Oct 26 19:43:43 <HOST> conmon[780974]: debug 2021-10-26T19:43:43.677+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3ff99a0000~1000>
> Oct 26 19:43:55 <HOST> conmon[780974]: debug 2021-10-26T19:43:55.237+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x3ffcfd0000~2000>
> Oct 26 19:45:02 <HOST> conmon[780974]: debug 2021-10-26T19:45:02.113+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x28601b0000~4000>
> Oct 26 19:45:46 <HOST> conmon[780974]: debug 2021-10-26T19:45:46.021+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x4014b60000~2000>
> Oct 26 19:46:02 <HOST> conmon[780974]: debug 2021-10-26T19:46:02.757+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x4015e70000~1000>
> Oct 26 19:46:24 <HOST> conmon[780974]: debug 2021-10-26T19:46:24.257+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x401a520000~1000>
> Oct 26 19:47:11 <HOST> conmon[780974]: debug 2021-10-26T19:47:11.394+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x40303e0000~1000>
> Oct 26 19:47:30 <HOST> conmon[780974]: debug 2021-10-26T19:47:30.502+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x40408c0000~2000>
> Oct 26 19:47:35 <HOST> conmon[780974]: debug 2021-10-26T19:47:35.914+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x40409b0000~1000>
> Oct 26 19:48:39 <HOST> conmon[780974]: debug 2021-10-26T19:48:39.110+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x4069060000~1000>
> Oct 26 19:50:44 <HOST> conmon[780974]: debug 2021-10-26T19:50:44.506+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x407e7b0000~1000>
> Oct 26 19:51:41 <HOST> conmon[780974]: debug 2021-10-26T19:51:41.838+0000
> 7f08e65a4080 -1 bdev(0x56377e188800 /var/lib/ceph/osd/ceph-13/block) read
> stalled read 0x4092c50000~1000>
> lines 94670-94720/94720 (END)
>
> On Tue, Oct 26, 2021 at 1:39 AM 胡 玮文 <huww98@xxxxxxxxxxx> wrote:
>
>> Could you post the logs of the problematic OSDs? E.g.:
>>
>>
>>
>> cephadm logs --name osd.0
>>
>>
>>
>> *发件人: *Marco Pizzolo <marcopizzolo@xxxxxxxxx>
>> *发送时间: *2021年10月26日 7:15
>> *收件人: *ceph-users <ceph-users@xxxxxxx>
>> *主题: * 16.2.6 OSD down, out but container running....
>>
>>
>>
>> Hello Everyone,
>>
>> I'm seeing an issue where the podman container is running, but the osd is
>> being reported as down and out. restarting service doesn't help, neither
>> does rebooting the host.
>>
>> What am I missing?
>>
>> Thanks,
>> Marco
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@xxxxxxx
>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
>>
>>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
