Hello, I've problem: my ceph cluster (3x mon nodes, 6x osd nodes, every osd node have 12 rotational disk and one NVMe device for bluestore DB). CEPH is installed by ceph orchestrator and have bluefs storage on osd. I've started process upgrade from version 17.2.6 to 18.2.1 by invocating: ceph orch upgrade start --ceph-version 18.2.1 After upgrade of mon and mgr processes orchestrator tried to upgrade the first OSD node, but they are falling down. I've stop the process of upgrade, but I have 1 osd node completely down. After upgrade I've got some error messages and I've found /var/lib/ceph/crashxxxx directories, I attach to this message files, which I've found here. Please, can you advice, what now I can do? It seems, that rocksdb is even non-compatible or corrupted :-( Thanks in advance. Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
--- begin dump of recent events ---
-250> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command assert hook 0x555d1f85cb30
-249> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command abort hook 0x555d1f85cb30
-248> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command leak_some_memory hook 0x555d1f85cb30
-247> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command perfcounters_dump hook 0x555d1f85cb30
-246> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command 1 hook 0x555d1f85cb30
-245> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command perf dump hook 0x555d1f85cb30
-244> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command perfcounters_schema hook 0x555d1f85cb30
-243> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command perf histogram dump hook 0x555d1f85cb30
-242> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command 2 hook 0x555d1f85cb30
-241> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command perf schema hook 0x555d1f85cb30
-240> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command counter dump hook 0x555d1f85cb30
-239> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command counter schema hook 0x555d1f85cb30
-238> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command perf histogram schema hook 0x555d1f85cb30
-237> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command perf reset hook 0x555d1f85cb30
-236> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command config show hook 0x555d1f85cb30
-235> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command config help hook 0x555d1f85cb30
-234> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command config set hook 0x555d1f85cb30
-233> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command config unset hook 0x555d1f85cb30
-232> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command config get hook 0x555d1f85cb30
-231> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command config diff hook 0x555d1f85cb30
-230> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command config diff get hook 0x555d1f85cb30
-229> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command injectargs hook 0x555d1f85cb30
-228> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command log flush hook 0x555d1f85cb30
-227> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command log dump hook 0x555d1f85cb30
-226> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command log reopen hook 0x555d1f85cb30
-225> 2023-12-22T21:26:57.856+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command dump_mempools hook 0x555d20602068
-224> 2023-12-22T21:26:57.860+0100 7f5158282a00 5 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _open_db_and_around::NCB::read_only=0, to_repair=1
-223> 2023-12-22T21:26:57.864+0100 7f5158282a00 1 bdev(0x555d1f8c2000 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block) open path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block
-222> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 bdev(0x555d1f8c2000 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block) open size 8001561821184 (0x74702400000, 7.3 TiB) block_size 4096 (4 KiB) rotational device, discard not supported
-221> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06
-220> 2023-12-22T21:26:57.868+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore bluefs device info hook 0x555d1f85cb00
-219> 2023-12-22T21:26:57.868+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluefs stats hook 0x555d1f85cb00
-218> 2023-12-22T21:26:57.868+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluefs files list hook 0x555d1f85cb00
-217> 2023-12-22T21:26:57.868+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluefs debug_inject_read_zeros hook 0x555d1f85cb00
-216> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 bdev(0x555d1f8c2700 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db) open path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db
-215> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 bdev(0x555d1f8c2700 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db) open size 68719476736 (0x1000000000, 64 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported
-214> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 bluefs add_block_device bdev 1 path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db size 64 GiB
-213> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 bdev(0x555d1f8c2e00 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block) open path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block
-212> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 bdev(0x555d1f8c2e00 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block) open size 8001561821184 (0x74702400000, 7.3 TiB) block_size 4096 (4 KiB) rotational device, discard not supported
-211> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 bluefs add_block_device bdev 2 path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block size 7.3 TiB
-210> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 set rocksdb option writable_file_max_buffer_size = 0
-209> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_base = 1073741824
-208> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 set rocksdb option level0_file_num_compaction_trigger = 8
-207> 2023-12-22T21:26:57.868+0100 7f5158282a00 1 set rocksdb option max_background_jobs = 4
-206> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 set rocksdb option write_buffer_size = 16777216
-205> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 set rocksdb option compaction_style = kCompactionStyleLevel
-204> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 set rocksdb option compaction_readahead_size = 2MB
-203> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_multiplier = 8
-202> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 set rocksdb option max_write_buffer_number = 64
-201> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 set rocksdb option max_total_wal_size = 1073741824
-200> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 set rocksdb option min_write_buffer_number_to_merge = 6
-199> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 set rocksdb option compression = kNoCompression
-198> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 bluefs mount
-197> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 bluefs _init_alloc new, id 1, allocator name bluefs-db, allocator type hybrid, capacity 0x1000000000, block size 0x100000
-196> 2023-12-22T21:26:57.872+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator dump bluefs-db hook 0x555d1f970db0
-195> 2023-12-22T21:26:57.872+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator score bluefs-db hook 0x555d1f970db0
-194> 2023-12-22T21:26:57.872+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator fragmentation bluefs-db hook 0x555d1f970db0
-193> 2023-12-22T21:26:57.872+0100 7f5158282a00 1 bluefs _init_alloc shared, id 2, capacity 0x74702400000, block size 0x10000
-192> 2023-12-22T21:26:58.020+0100 7f5158282a00 1 bluefs mount shared_bdev_used = 0
-191> 2023-12-22T21:26:58.020+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _prepare_db_environment set db_paths to db,65283502899 db.slow,7601483730124
-190> 2023-12-22T21:26:58.020+0100 7f5158282a00 1 set rocksdb option writable_file_max_buffer_size = 0
-189> 2023-12-22T21:26:58.020+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_base = 1073741824
-188> 2023-12-22T21:26:58.020+0100 7f5158282a00 1 set rocksdb option level0_file_num_compaction_trigger = 8
-187> 2023-12-22T21:26:58.020+0100 7f5158282a00 1 set rocksdb option max_background_jobs = 4
-186> 2023-12-22T21:26:58.020+0100 7f5158282a00 1 set rocksdb option write_buffer_size = 16777216
-185> 2023-12-22T21:26:58.020+0100 7f5158282a00 1 set rocksdb option compaction_style = kCompactionStyleLevel
-184> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option compaction_readahead_size = 2MB
-183> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_multiplier = 8
-182> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option max_write_buffer_number = 64
-181> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option max_total_wal_size = 1073741824
-180> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option min_write_buffer_number_to_merge = 6
-179> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option compression = kNoCompression
-178> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option writable_file_max_buffer_size = 0
-177> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_base = 1073741824
-176> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option level0_file_num_compaction_trigger = 8
-175> 2023-12-22T21:26:58.024+0100 7f5158282a00 1 set rocksdb option max_background_jobs = 4
-174> 2023-12-22T21:26:58.028+0100 7f5158282a00 1 set rocksdb option write_buffer_size = 16777216
-173> 2023-12-22T21:26:58.028+0100 7f5158282a00 1 set rocksdb option compaction_style = kCompactionStyleLevel
-172> 2023-12-22T21:26:58.028+0100 7f5158282a00 1 set rocksdb option compaction_readahead_size = 2MB
-171> 2023-12-22T21:26:58.028+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_multiplier = 8
-170> 2023-12-22T21:26:58.028+0100 7f5158282a00 1 set rocksdb option max_write_buffer_number = 64
-169> 2023-12-22T21:26:58.028+0100 7f5158282a00 1 set rocksdb option max_total_wal_size = 1073741824
-168> 2023-12-22T21:26:58.028+0100 7f5158282a00 1 set rocksdb option min_write_buffer_number_to_merge = 6
-167> 2023-12-22T21:26:58.028+0100 7f5158282a00 1 set rocksdb option compression = kNoCompression
-166> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
-165> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
-164> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
-163> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
-162> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
-161> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
-160> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
-159> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
-158> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
-157> 2023-12-22T21:26:58.032+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
-156> 2023-12-22T21:26:58.036+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
-155> 2023-12-22T21:26:58.036+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
-154> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0
-153> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _open_super_meta old nid_max 2302905
-152> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _open_super_meta old blobid_max 430253
-151> 2023-12-22T21:26:58.136+0100 7f5158282a00 5 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _open_super_meta::NCB::freelist_type=null
-150> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _open_super_meta ondisk_format 4 compat_ondisk_format 3
-149> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _open_super_meta min_alloc_size 0x1000
-148> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 get compressor snappy = 0
-147> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 load compressor snappy
-146> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 add compressor snappy 0x555d1fd40330
-145> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 get compressor snappy = 0x555d1fd40330
-144> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 load: compressor snappy loaded and registered
-143> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 get compressor snappy = 0x555d1fd40330
-142> 2023-12-22T21:26:58.136+0100 7f5158282a00 5 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _open_fm::NCB::freelist_type=null
-141> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 freelist init
-140> 2023-12-22T21:26:58.136+0100 7f5158282a00 1 freelist _read_cfg
-139> 2023-12-22T21:26:58.136+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator dump block hook 0x555d1fd418c0
-138> 2023-12-22T21:26:58.136+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator score block hook 0x555d1fd418c0
-137> 2023-12-22T21:26:58.136+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator fragmentation block hook 0x555d1fd418c0
-136> 2023-12-22T21:26:58.136+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator dump recovery hook 0x555d1fd41ce0
-135> 2023-12-22T21:26:58.136+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator score recovery hook 0x555d1fd41ce0
-134> 2023-12-22T21:26:58.136+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator fragmentation recovery hook 0x555d1fd41ce0
-133> 2023-12-22T21:26:58.168+0100 7f5158282a00 5 bluestore::NCB::__restore_allocator::file_size=0,sizeof(extent_t)=16
-132> 2023-12-22T21:26:58.168+0100 7f5158282a00 1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file)
-131> 2023-12-22T21:26:58.168+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluestore allocator dump recovery
-130> 2023-12-22T21:26:58.168+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluestore allocator fragmentation recovery
-129> 2023-12-22T21:26:58.168+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluestore allocator score recovery
-128> 2023-12-22T21:26:58.168+0100 7f5158282a00 0 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ...
-127> 2023-12-22T21:27:10.868+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes processed objects count = 1000000
-126> 2023-12-22T21:27:23.400+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes processed objects count = 2000000
-125> 2023-12-22T21:27:35.828+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes processed objects count = 3000000
-124> 2023-12-22T21:27:48.348+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes processed objects count = 4000000
-123> 2023-12-22T21:28:00.904+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes processed objects count = 5000000
-122> 2023-12-22T21:28:13.621+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes processed objects count = 6000000
-121> 2023-12-22T21:28:26.037+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes processed objects count = 7000000
-120> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 1->store_statfs(0x0/0x0/0x0, data 0x1b9768318a4/0x1b977f9b000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1
-119> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 3->store_statfs(0x0/0x0/0x0, data 0x19400000/0x19400000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1
-118> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 4->store_statfs(0x0/0x0/0x0, data 0x95c6800/0x95c7000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1
-117> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 5->store_statfs(0x0/0x0/0x0, data 0x3c00000/0x3c00000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1
-116> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 6->store_statfs(0x0/0x0/0x0, data 0x6761298925/0x6782476000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1
-115> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 7->store_statfs(0x0/0x0/0x0, data 0x17b62e4/0x17b9000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1
-114> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 8->store_statfs(0x0/0x0/0x0, data 0x5a033600/0x5a036000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1
-113> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 9->store_statfs(0x0/0x0/0x0, data 0x2822d21000/0x2822d25000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1
-112> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool a->store_statfs(0x0/0x0/0x0, data 0x0/0x0, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1
-111> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool b->store_statfs(0x0/0x0/0x0, data 0x1f80da012/0x1f80e3000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1
-110> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool c->store_statfs(0x0/0x0/0x0, data 0x18d7400/0x18da000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1
-109> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool d->store_statfs(0x0/0x0/0x0, data 0x0/0x0, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1
-108> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool e->store_statfs(0x0/0x0/0x0, data 0x14a/0x3000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1
-107> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool 10->store_statfs(0x0/0x0/0x0, data 0x0/0x0, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1
-106> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered pool ffffffffffffffff->store_statfs(0x0/0x0/0x0, data 0x57a83eb/0x6113000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0) per-pool:1
-105> 2023-12-22T21:28:28.797+0100 7f5158282a00 5 bluestore::NCB::read_allocation_from_onodes::read_allocation_from_onodes recovered store_statfs(0x0/0x0/0x0, data 0x24b7b6f4bf4/0x24b9e9bf000, compress 0x0/0x0/0x0, omap 0x0, meta 0x0)
-104> 2023-12-22T21:28:29.205+0100 7f5158282a00 1 HybridAllocator _spillover_range constructing fallback allocator
-103> 2023-12-22T21:28:29.205+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator dump block.fallback hook 0x555d1fd6b0e0
-102> 2023-12-22T21:28:29.205+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator score block.fallback hook 0x555d1fd6b0e0
-101> 2023-12-22T21:28:29.205+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator fragmentation block.fallback hook 0x555d1fd6b0e0
-100> 2023-12-22T21:28:30.741+0100 7f5158282a00 1 bluestore::NCB::read_allocation_from_drive_on_startup::::Allocation Recovery was completed in 92.570249 seconds, extent_count=38970352
-99> 2023-12-22T21:28:30.741+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _init_alloc loaded 0 B in 0 extents, allocator type hybrid, capacity 0x74702400000, block size 0x1000, free 0x4fb63a3f000, fragmentation 0.00727618
-98> 2023-12-22T21:28:30.817+0100 7f5158282a00 1 bluefs umount
-97> 2023-12-22T21:28:30.817+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluestore allocator dump bluefs-db
-96> 2023-12-22T21:28:30.817+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluestore allocator fragmentation bluefs-db
-95> 2023-12-22T21:28:30.817+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluestore allocator score bluefs-db
-94> 2023-12-22T21:28:30.817+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluefs debug_inject_read_zeros
-93> 2023-12-22T21:28:30.817+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluefs files list
-92> 2023-12-22T21:28:30.817+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluefs stats
-91> 2023-12-22T21:28:30.817+0100 7f5158282a00 5 asok(0x555d1fb54000) unregister_commands bluestore bluefs device info
-90> 2023-12-22T21:28:30.817+0100 7f5158282a00 1 bdev(0x555d1f8c2700 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db) close
-89> 2023-12-22T21:28:31.293+0100 7f5158282a00 1 bdev(0x555d1f8c2e00 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block) close
-88> 2023-12-22T21:28:31.449+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore bluefs device info hook 0x555d1f85ca90
-87> 2023-12-22T21:28:31.449+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluefs stats hook 0x555d1f85ca90
-86> 2023-12-22T21:28:31.449+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluefs files list hook 0x555d1f85ca90
-85> 2023-12-22T21:28:31.449+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluefs debug_inject_read_zeros hook 0x555d1f85ca90
-84> 2023-12-22T21:28:31.449+0100 7f5158282a00 1 bdev(0x555d1f8c2e00 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db) open path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db
-83> 2023-12-22T21:28:31.449+0100 7f5158282a00 1 bdev(0x555d1f8c2e00 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db) open size 68719476736 (0x1000000000, 64 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported
-82> 2023-12-22T21:28:31.449+0100 7f5158282a00 1 bluefs add_block_device bdev 1 path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block.db size 64 GiB
-81> 2023-12-22T21:28:31.449+0100 7f5158282a00 1 bdev(0x555d1f8c2700 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block) open path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block
-80> 2023-12-22T21:28:31.449+0100 7f5158282a00 1 bdev(0x555d1f8c2700 /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block) open size 8001561821184 (0x74702400000, 7.3 TiB) block_size 4096 (4 KiB) rotational device, discard not supported
-79> 2023-12-22T21:28:31.449+0100 7f5158282a00 1 bluefs add_block_device bdev 2 path /var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0/block size 7.3 TiB
-78> 2023-12-22T21:28:31.449+0100 7f5158282a00 1 set rocksdb option writable_file_max_buffer_size = 0
-77> 2023-12-22T21:28:31.453+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_base = 1073741824
-76> 2023-12-22T21:28:31.453+0100 7f5158282a00 1 set rocksdb option level0_file_num_compaction_trigger = 8
-75> 2023-12-22T21:28:31.453+0100 7f5158282a00 1 set rocksdb option max_background_jobs = 4
-74> 2023-12-22T21:28:31.453+0100 7f5158282a00 1 set rocksdb option write_buffer_size = 16777216
-73> 2023-12-22T21:28:31.453+0100 7f5158282a00 1 set rocksdb option compaction_style = kCompactionStyleLevel
-72> 2023-12-22T21:28:31.453+0100 7f5158282a00 1 set rocksdb option compaction_readahead_size = 2MB
-71> 2023-12-22T21:28:31.453+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_multiplier = 8
-70> 2023-12-22T21:28:31.453+0100 7f5158282a00 1 set rocksdb option max_write_buffer_number = 64
-69> 2023-12-22T21:28:31.457+0100 7f5158282a00 1 set rocksdb option max_total_wal_size = 1073741824
-68> 2023-12-22T21:28:31.457+0100 7f5158282a00 1 set rocksdb option min_write_buffer_number_to_merge = 6
-67> 2023-12-22T21:28:31.457+0100 7f5158282a00 1 set rocksdb option compression = kNoCompression
-66> 2023-12-22T21:28:31.457+0100 7f5158282a00 1 bluefs mount
-65> 2023-12-22T21:28:31.457+0100 7f5158282a00 1 bluefs _init_alloc new, id 1, allocator name bluefs-db, allocator type hybrid, capacity 0x1000000000, block size 0x100000
-64> 2023-12-22T21:28:31.457+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator dump bluefs-db hook 0x555d1f970fc0
-63> 2023-12-22T21:28:31.457+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator score bluefs-db hook 0x555d1f970fc0
-62> 2023-12-22T21:28:31.457+0100 7f5158282a00 5 asok(0x555d1fb54000) register_command bluestore allocator fragmentation bluefs-db hook 0x555d1f970fc0
-61> 2023-12-22T21:28:31.457+0100 7f5158282a00 1 bluefs _init_alloc shared, id 2, capacity 0x74702400000, block size 0x10000
-60> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 bluefs mount shared_bdev_used = 0
-59> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 bluestore(/var/lib/ceph/2c565e24-7850-47dc-a751-a6357cbbaf2a/osd.0) _prepare_db_environment set db_paths to db,65283502899 db.slow,7601483730124
-58> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 set rocksdb option writable_file_max_buffer_size = 0
-57> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_base = 1073741824
-56> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 set rocksdb option level0_file_num_compaction_trigger = 8
-55> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 set rocksdb option max_background_jobs = 4
-54> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 set rocksdb option write_buffer_size = 16777216
-53> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 set rocksdb option compaction_style = kCompactionStyleLevel
-52> 2023-12-22T21:28:31.517+0100 7f5158282a00 1 set rocksdb option compaction_readahead_size = 2MB
-51> 2023-12-22T21:28:31.521+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_multiplier = 8
-50> 2023-12-22T21:28:31.521+0100 7f5158282a00 1 set rocksdb option max_write_buffer_number = 64
-49> 2023-12-22T21:28:31.521+0100 7f5158282a00 1 set rocksdb option max_total_wal_size = 1073741824
-48> 2023-12-22T21:28:31.521+0100 7f5158282a00 1 set rocksdb option min_write_buffer_number_to_merge = 6
-47> 2023-12-22T21:28:31.521+0100 7f5158282a00 1 set rocksdb option compression = kNoCompression
-46> 2023-12-22T21:28:31.521+0100 7f5158282a00 5 bluestore::NCB::invalidate_allocation_file_on_bluefs::invalidate using bluefs->truncate(p_handle, 0)
-45> 2023-12-22T21:28:31.521+0100 7f5158282a00 1 set rocksdb option writable_file_max_buffer_size = 0
-44> 2023-12-22T21:28:31.521+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_base = 1073741824
-43> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option level0_file_num_compaction_trigger = 8
-42> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option max_background_jobs = 4
-41> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option write_buffer_size = 16777216
-40> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option compaction_style = kCompactionStyleLevel
-39> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option compaction_readahead_size = 2MB
-38> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_multiplier = 8
-37> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option max_write_buffer_number = 64
-36> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option max_total_wal_size = 1073741824
-35> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option min_write_buffer_number_to_merge = 6
-34> 2023-12-22T21:28:31.525+0100 7f5158282a00 1 set rocksdb option compression = kNoCompression
-33> 2023-12-22T21:28:31.529+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
-32> 2023-12-22T21:28:31.529+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
-31> 2023-12-22T21:28:31.533+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
-30> 2023-12-22T21:28:32.957+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
-29> 2023-12-22T21:28:35.705+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
-28> 2023-12-22T21:28:35.717+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
-27> 2023-12-22T21:28:44.209+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
-26> 2023-12-22T21:28:52.773+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
-25> 2023-12-22T21:29:01.313+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
-24> 2023-12-22T21:29:01.485+0100 7f5158282a00 1 set rocksdb option writable_file_max_buffer_size = 0
-23> 2023-12-22T21:29:01.485+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_base = 1073741824
-22> 2023-12-22T21:29:01.485+0100 7f5158282a00 1 set rocksdb option level0_file_num_compaction_trigger = 8
-21> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option max_background_jobs = 4
-20> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option write_buffer_size = 16777216
-19> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option compaction_style = kCompactionStyleLevel
-18> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option compaction_readahead_size = 2MB
-17> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option max_bytes_for_level_multiplier = 8
-16> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option max_write_buffer_number = 64
-15> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option max_total_wal_size = 1073741824
-14> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option min_write_buffer_number_to_merge = 6
-13> 2023-12-22T21:29:01.489+0100 7f5158282a00 1 set rocksdb option compression = kNoCompression
-12> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
-11> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
-10> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
-9> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
-8> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
-7> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
-6> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
-5> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
-4> 2023-12-22T21:29:01.493+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
-3> 2023-12-22T21:29:01.561+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 12, name: m-0)
-2> 2023-12-22T21:29:01.577+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 13, name: m-1)
-1> 2023-12-22T21:29:01.593+0100 7f5158282a00 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 14, name: m-2)
0> 2023-12-22T21:29:01.689+0100 7f5158282a00 -1 *** Caught signal (Segmentation fault) **
in thread 7f5158282a00 thread_name:ceph-kvstore-to
ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
1: /lib/x86_64-linux-gnu/libc.so.6(+0x3bfd0) [0x7f515956bfd0]
2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xb9) [0x555d1e4d2a19]
3: (RocksDBStore::get_iterator(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, KeyValueDB::IteratorBounds)+0x19f) [0x555d1e4d278f]
4: (BlueStore::_close_db()+0x14f) [0x555d1e39ee5f]
5: (BlueStore::_close_db_and_around()+0x13) [0x555d1e39fa03]
6: (BlueStore::umount()+0xdf) [0x555d1e3a548f]
7: (std::unique_ptr<KeyValueDB, StoreTool::Deleter>::~unique_ptr()+0x1e) [0x555d1e2f8d0e]
8: main()
9: /lib/x86_64-linux-gnu/libc.so.6(+0x271ca) [0x7f51595571ca]
10: __libc_start_main()
11: _start()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 rbd_pwl
0/ 5 journaler
0/ 5 objectcacher
0/ 5 immutable_obj_cache
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/ 5 rgw_datacache
1/ 5 rgw_access
1/ 5 rgw_dbstore
1/ 5 rgw_flight
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
2/ 2 rocksdb
4/ 5 leveldb
1/ 5 fuse
2/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
1/ 5 prioritycache
0/ 5 test
0/ 5 cephfs_mirror
0/ 5 cephsqlite
0/ 5 seastore
0/ 5 seastore_onode
0/ 5 seastore_odata
0/ 5 seastore_omap
0/ 5 seastore_tm
0/ 5 seastore_t
0/ 5 seastore_cleaner
0/ 5 seastore_epm
0/ 5 seastore_lba
0/ 5 seastore_fixedkv_tree
0/ 5 seastore_cache
0/ 5 seastore_journal
0/ 5 seastore_device
0/ 5 seastore_backref
0/ 5 alienstore
1/ 5 mclock
0/ 5 cyanstore
1/ 5 ceph_exporter
1/ 5 memstore
-2/-2 (syslog threshold)
99/99 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
7f5158282a00 / ceph-kvstore-to
max_recent 500
max_new 1000
log_file /var/lib/ceph/crash/2023-12-22T20:29:01.692109Z_6d62fc0a-38a3-4725-9c37-5cbf70d0ee5f/log
--- end dump of recent events ---
{
"crash_id": "2023-12-22T20:29:01.692109Z_6d62fc0a-38a3-4725-9c37-5cbf70d0ee5f",
"timestamp": "2023-12-22T20:29:01.692109Z",
"process_name": "ceph-kvstore-to",
"entity_name": "client.admin",
"ceph_version": "18.2.1",
"utsname_hostname": "osd1",
"utsname_sysname": "Linux",
"utsname_release": "5.10.0-26-amd64",
"utsname_version": "#1 SMP Debian 5.10.197-1 (2023-09-29)",
"utsname_machine": "x86_64",
"os_name": "Debian GNU/Linux 12 (bookworm)",
"os_id": "12",
"os_version_id": "12",
"os_version": "12 (bookworm)",
"backtrace": [
"/lib/x86_64-linux-gnu/libc.so.6(+0x3bfd0) [0x7f515956bfd0]",
"(RocksDBStore::get_wholespace_iterator(unsigned int)+0xb9) [0x555d1e4d2a19]",
"(RocksDBStore::get_iterator(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, KeyValueDB::IteratorBounds)+0x19f) [0x555d1e4d278f]",
"(BlueStore::_close_db()+0x14f) [0x555d1e39ee5f]",
"(BlueStore::_close_db_and_around()+0x13) [0x555d1e39fa03]",
"(BlueStore::umount()+0xdf) [0x555d1e3a548f]",
"(std::unique_ptr<KeyValueDB, StoreTool::Deleter>::~unique_ptr()+0x1e) [0x555d1e2f8d0e]",
"main()",
"/lib/x86_64-linux-gnu/libc.so.6(+0x271ca) [0x7f51595571ca]",
"__libc_start_main()",
"_start()"
]
}
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
