Hi,
We are experiencing a recurrent problem with some OSD that fails to
startup after a crash. Here it happened with an OSD during recovery and
this is a very annoying bug, because recovery takes time (small
increment of crush weight) and the way I found to fix this is to use
ceph-volume lvm zap /var/lib/ceph/osd/ceph-$i/block, remove the OSD, and
create it again. Thus needing to start the recovery all over again.
Is there another way to fix this ?
# /usr/bin/ceph-osd -d -i 40 --pid-file /var/run/ceph/osd.40.pid -c
/etc/ceph/ceph.conf --cluster ceph --setuser ceph --setgroup ceph
2018-10-09 00:33:23.025913 7f91c7ae2e00 0 set uid:gid to 64045:64045
(ceph:ceph)
2018-10-09 00:33:23.025934 7f91c7ae2e00 0 ceph version 12.2.8
(ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable), process
ceph-osd, pid 322330
starting osd.40 at - osd_data /var/lib/ceph/osd/ceph-40
/var/lib/ceph/osd/ceph-40/journal
2018-10-09 00:33:23.203930 7f91c7ae2e00 0 load: jerasure load: lrc
load: isa
2018-10-09 00:33:23.204022 7f91c7ae2e00 1 bdev create path
/var/lib/ceph/osd/ceph-40/block type kernel
2018-10-09 00:33:23.204032 7f91c7ae2e00 1 bdev(0x558d31b68b40
/var/lib/ceph/osd/ceph-40/block) open path /var/lib/ceph/osd/ceph-40/block
2018-10-09 00:33:23.204336 7f91c7ae2e00 1 bdev(0x558d31b68b40
/var/lib/ceph/osd/ceph-40/block) open size 1990292758528 (0x1cf66b16000,
1.81TiB) block_size 4096 (4KiB) rotational
2018-10-09 00:33:23.204766 7f91c7ae2e00 1
bluestore(/var/lib/ceph/osd/ceph-40) _set_cache_sizes max 0.5 < ratio 0.99
2018-10-09 00:33:23.204790 7f91c7ae2e00 1
bluestore(/var/lib/ceph/osd/ceph-40) _set_cache_sizes cache_size
1073741824 meta 0.5 kv 0.5 data 0
2018-10-09 00:33:23.204796 7f91c7ae2e00 1 bdev(0x558d31b68b40
/var/lib/ceph/osd/ceph-40/block) close
2018-10-09 00:33:23.499876 7f91c7ae2e00 1
bluestore(/var/lib/ceph/osd/ceph-40) _mount path /var/lib/ceph/osd/ceph-40
2018-10-09 00:33:23.500355 7f91c7ae2e00 1 bdev create path
/var/lib/ceph/osd/ceph-40/block type kernel
2018-10-09 00:33:23.500368 7f91c7ae2e00 1 bdev(0x558d31b68d80
/var/lib/ceph/osd/ceph-40/block) open path /var/lib/ceph/osd/ceph-40/block
2018-10-09 00:33:23.500583 7f91c7ae2e00 1 bdev(0x558d31b68d80
/var/lib/ceph/osd/ceph-40/block) open size 1990292758528 (0x1cf66b16000,
1.81TiB) block_size 4096 (4KiB) rotational
2018-10-09 00:33:23.500884 7f91c7ae2e00 1
bluestore(/var/lib/ceph/osd/ceph-40) _set_cache_sizes max 0.5 < ratio 0.99
2018-10-09 00:33:23.500900 7f91c7ae2e00 1
bluestore(/var/lib/ceph/osd/ceph-40) _set_cache_sizes cache_size
1073741824 meta 0.5 kv 0.5 data 0
2018-10-09 00:33:23.501051 7f91c7ae2e00 1 bdev create path
/var/lib/ceph/osd/ceph-40/block type kernel
2018-10-09 00:33:23.501061 7f91c7ae2e00 1 bdev(0x558d31b69200
/var/lib/ceph/osd/ceph-40/block) open path /var/lib/ceph/osd/ceph-40/block
2018-10-09 00:33:23.501263 7f91c7ae2e00 1 bdev(0x558d31b69200
/var/lib/ceph/osd/ceph-40/block) open size 1990292758528 (0x1cf66b16000,
1.81TiB) block_size 4096 (4KiB) rotational
2018-10-09 00:33:23.501277 7f91c7ae2e00 1 bluefs add_block_device bdev
1 path /var/lib/ceph/osd/ceph-40/block size 1.81TiB
2018-10-09 00:33:23.501310 7f91c7ae2e00 1 bluefs mount
2018-10-09 00:33:23.589465 7f91c7ae2e00 0 set rocksdb option
compaction_readahead_size = 2097152
2018-10-09 00:33:23.589482 7f91c7ae2e00 0 set rocksdb option
compression = kNoCompression
2018-10-09 00:33:23.589487 7f91c7ae2e00 0 set rocksdb option
max_write_buffer_number = 4
2018-10-09 00:33:23.589490 7f91c7ae2e00 0 set rocksdb option
min_write_buffer_number_to_merge = 1
2018-10-09 00:33:23.589494 7f91c7ae2e00 0 set rocksdb option
recycle_log_file_num = 4
2018-10-09 00:33:23.589496 7f91c7ae2e00 0 set rocksdb option
writable_file_max_buffer_size = 0
2018-10-09 00:33:23.589500 7f91c7ae2e00 0 set rocksdb option
write_buffer_size = 268435456
2018-10-09 00:33:23.589517 7f91c7ae2e00 0 set rocksdb option
compaction_readahead_size = 2097152
2018-10-09 00:33:23.589520 7f91c7ae2e00 0 set rocksdb option
compression = kNoCompression
2018-10-09 00:33:23.589523 7f91c7ae2e00 0 set rocksdb option
max_write_buffer_number = 4
2018-10-09 00:33:23.589526 7f91c7ae2e00 0 set rocksdb option
min_write_buffer_number_to_merge = 1
2018-10-09 00:33:23.589528 7f91c7ae2e00 0 set rocksdb option
recycle_log_file_num = 4
2018-10-09 00:33:23.589531 7f91c7ae2e00 0 set rocksdb option
writable_file_max_buffer_size = 0
2018-10-09 00:33:23.589533 7f91c7ae2e00 0 set rocksdb option
write_buffer_size = 268435456
2018-10-09 00:33:23.589656 7f91c7ae2e00 4 rocksdb: RocksDB version: 5.4.0
2018-10-09 00:33:23.589667 7f91c7ae2e00 4 rocksdb: Git sha
rocksdb_build_git_sha:@0@
2018-10-09 00:33:23.589670 7f91c7ae2e00 4 rocksdb: Compile date Aug 30 2018
2018-10-09 00:33:23.589671 7f91c7ae2e00 4 rocksdb: DB SUMMARY
2018-10-09 00:33:23.589706 7f91c7ae2e00 4 rocksdb: CURRENT file: CURRENT
2018-10-09 00:33:23.589707 7f91c7ae2e00 4 rocksdb: IDENTITY file: IDENTITY
2018-10-09 00:33:23.589709 7f91c7ae2e00 4 rocksdb: MANIFEST file:
MANIFEST-000011 size: 51827 Bytes
2018-10-09 00:33:23.589712 7f91c7ae2e00 4 rocksdb: SST files in db dir,
Total Num: 28, files: 000260.sst 000261.sst 000262.sst 000274.sst
000275.sst 000276.sst 000277.sst 000286.sst 000287.sst
2018-10-09 00:33:23.589714 7f91c7ae2e00 4 rocksdb: Write Ahead Log file
in db: 000310.log size: 258554009 ; 000312.log size: 256884429 ;
2018-10-09 00:33:23.589715 7f91c7ae2e00 4
rocksdb: Options.error_if_exists: 0
2018-10-09 00:33:23.589716 7f91c7ae2e00 4
rocksdb: Options.create_if_missing: 0
2018-10-09 00:33:23.589717 7f91c7ae2e00 4
rocksdb: Options.paranoid_checks: 1
2018-10-09 00:33:23.589718 7f91c7ae2e00 4
rocksdb: Options.env: 0x558d31d42280
2018-10-09 00:33:23.589719 7f91c7ae2e00 4
rocksdb: Options.info_log: 0x558d31d42a00
2018-10-09 00:33:23.589720 7f91c7ae2e00 4
rocksdb: Options.max_open_files: -1
2018-10-09 00:33:23.589721 7f91c7ae2e00 4 rocksdb:
Options.max_file_opening_threads: 16
2018-10-09 00:33:23.589722 7f91c7ae2e00 4
rocksdb: Options.use_fsync: 0
2018-10-09 00:33:23.589723 7f91c7ae2e00 4
rocksdb: Options.max_log_file_size: 0
2018-10-09 00:33:23.589724 7f91c7ae2e00 4 rocksdb:
Options.max_manifest_file_size: 18446744073709551615
2018-10-09 00:33:23.589725 7f91c7ae2e00 4 rocksdb:
Options.log_file_time_to_roll: 0
2018-10-09 00:33:23.589726 7f91c7ae2e00 4
rocksdb: Options.keep_log_file_num: 1000
2018-10-09 00:33:23.589726 7f91c7ae2e00 4 rocksdb:
Options.recycle_log_file_num: 4
2018-10-09 00:33:23.589727 7f91c7ae2e00 4
rocksdb: Options.allow_fallocate: 1
2018-10-09 00:33:23.589728 7f91c7ae2e00 4
rocksdb: Options.allow_mmap_reads: 0
2018-10-09 00:33:23.589728 7f91c7ae2e00 4
rocksdb: Options.allow_mmap_writes: 0
2018-10-09 00:33:23.589729 7f91c7ae2e00 4
rocksdb: Options.use_direct_reads: 0
2018-10-09 00:33:23.589730 7f91c7ae2e00 4
rocksdb:
Options.use_direct_io_for_flush_and_compaction: 0
2018-10-09 00:33:23.589731 7f91c7ae2e00 4 rocksdb:
Options.create_missing_column_families: 0
2018-10-09 00:33:23.589731 7f91c7ae2e00 4
rocksdb: Options.db_log_dir:
2018-10-09 00:33:23.589732 7f91c7ae2e00 4
rocksdb: Options.wal_dir: db
2018-10-09 00:33:23.589733 7f91c7ae2e00 4 rocksdb:
Options.table_cache_numshardbits: 6
2018-10-09 00:33:23.589734 7f91c7ae2e00 4 rocksdb:
Options.max_subcompactions: 1
2018-10-09 00:33:23.589735 7f91c7ae2e00 4 rocksdb:
Options.max_background_flushes: 1
2018-10-09 00:33:23.589736 7f91c7ae2e00 4
rocksdb: Options.WAL_ttl_seconds: 0
2018-10-09 00:33:23.589737 7f91c7ae2e00 4
rocksdb: Options.WAL_size_limit_MB: 0
2018-10-09 00:33:23.589738 7f91c7ae2e00 4 rocksdb:
Options.manifest_preallocation_size: 4194304
2018-10-09 00:33:23.589739 7f91c7ae2e00 4 rocksdb:
Options.is_fd_close_on_exec: 1
2018-10-09 00:33:23.589740 7f91c7ae2e00 4 rocksdb:
Options.advise_random_on_open: 1
2018-10-09 00:33:23.589741 7f91c7ae2e00 4 rocksdb:
Options.db_write_buffer_size: 0
2018-10-09 00:33:23.589741 7f91c7ae2e00 4 rocksdb:
Options.access_hint_on_compaction_start: 1
2018-10-09 00:33:23.589742 7f91c7ae2e00 4 rocksdb:
Options.new_table_reader_for_compaction_inputs: 1
2018-10-09 00:33:23.589743 7f91c7ae2e00 4 rocksdb:
Options.compaction_readahead_size: 2097152
2018-10-09 00:33:23.589744 7f91c7ae2e00 4 rocksdb:
Options.random_access_max_buffer_size: 1048576
2018-10-09 00:33:23.589745 7f91c7ae2e00 4 rocksdb:
Options.writable_file_max_buffer_size: 0
2018-10-09 00:33:23.589746 7f91c7ae2e00 4 rocksdb:
Options.use_adaptive_mutex: 0
2018-10-09 00:33:23.589746 7f91c7ae2e00 4
rocksdb: Options.rate_limiter: (nil)
2018-10-09 00:33:23.589747 7f91c7ae2e00 4 rocksdb:
Options.sst_file_manager.rate_bytes_per_sec: 0
2018-10-09 00:33:23.589748 7f91c7ae2e00 4
rocksdb: Options.bytes_per_sync: 0
2018-10-09 00:33:23.589749 7f91c7ae2e00 4 rocksdb:
Options.wal_bytes_per_sync: 0
2018-10-09 00:33:23.589749 7f91c7ae2e00 4
rocksdb: Options.wal_recovery_mode: 0
2018-10-09 00:33:23.589750 7f91c7ae2e00 4 rocksdb:
Options.enable_thread_tracking: 0
2018-10-09 00:33:23.589751 7f91c7ae2e00 4 rocksdb:
Options.allow_concurrent_memtable_write: 1
2018-10-09 00:33:23.589762 7f91c7ae2e00 4 rocksdb:
Options.enable_write_thread_adaptive_yield: 1
2018-10-09 00:33:23.589763 7f91c7ae2e00 4 rocksdb:
Options.write_thread_max_yield_usec: 100
2018-10-09 00:33:23.589764 7f91c7ae2e00 4 rocksdb:
Options.write_thread_slow_yield_usec: 3
2018-10-09 00:33:23.589765 7f91c7ae2e00 4
rocksdb: Options.row_cache: None
2018-10-09 00:33:23.589766 7f91c7ae2e00 4
rocksdb: Options.wal_filter: None
2018-10-09 00:33:23.589766 7f91c7ae2e00 4 rocksdb:
Options.avoid_flush_during_recovery: 0
2018-10-09 00:33:23.589767 7f91c7ae2e00 4 rocksdb:
Options.base_background_compactions: 1
2018-10-09 00:33:23.589768 7f91c7ae2e00 4 rocksdb:
Options.max_background_compactions: 1
2018-10-09 00:33:23.589768 7f91c7ae2e00 4 rocksdb:
Options.avoid_flush_during_shutdown: 0
2018-10-09 00:33:23.589769 7f91c7ae2e00 4 rocksdb:
Options.delayed_write_rate : 16777216
2018-10-09 00:33:23.589770 7f91c7ae2e00 4 rocksdb:
Options.max_total_wal_size: 0
2018-10-09 00:33:23.589770 7f91c7ae2e00 4 rocksdb:
Options.delete_obsolete_files_period_micros: 21600000000
2018-10-09 00:33:23.589771 7f91c7ae2e00 4 rocksdb:
Options.stats_dump_period_sec: 600
2018-10-09 00:33:23.589772 7f91c7ae2e00 4 rocksdb: Compression
algorithms supported:
2018-10-09 00:33:23.589773 7f91c7ae2e00 4 rocksdb: Snappy supported: 1
2018-10-09 00:33:23.589774 7f91c7ae2e00 4 rocksdb: Zlib supported: 0
2018-10-09 00:33:23.589774 7f91c7ae2e00 4 rocksdb: Bzip supported: 0
2018-10-09 00:33:23.589775 7f91c7ae2e00 4 rocksdb: LZ4 supported: 0
2018-10-09 00:33:23.589776 7f91c7ae2e00 4 rocksdb: ZSTD supported: 0
2018-10-09 00:33:23.589777 7f91c7ae2e00 4 rocksdb: Fast CRC32 supported: 1
2018-10-09 00:33:23.596185 7f91c7ae2e00 4 rocksdb:
[/build/ceph-12.2.8/src/rocksdb/db/version_set.cc:2609] Recovering from
manifest file: MANIFEST-000011
2018-10-09 00:33:23.596256 7f91c7ae2e00 4 rocksdb:
[/build/ceph-12.2.8/src/rocksdb/db/column_family.cc:407] ---------------
Options for column family [default]:
2018-10-09 00:33:23.596261 7f91c7ae2e00 4 rocksdb:
Options.comparator: leveldb.BytewiseComparator
2018-10-09 00:33:23.596267 7f91c7ae2e00 4 rocksdb:
Options.merge_operator: .T:int64_array.b:bitwise_xor
2018-10-09 00:33:23.596268 7f91c7ae2e00 4 rocksdb:
Options.compaction_filter: None
2018-10-09 00:33:23.596272 7f91c7ae2e00 4 rocksdb:
Options.compaction_filter_factory: None
2018-10-09 00:33:23.596274 7f91c7ae2e00 4 rocksdb:
Options.memtable_factory: SkipListFactory
2018-10-09 00:33:23.596275 7f91c7ae2e00 4 rocksdb:
Options.table_factory: BlockBasedTable
2018-10-09 00:33:23.596299 7f91c7ae2e00 4 rocksdb:
table_factory options: flush_block_policy_factory:
FlushBlockBySizePolicyFactory (0x558d31b641f0)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 1
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x558d31e6f0b0
block_cache_name: LRUCache
block_cache_options:
capacity : 536870912
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
format_version: 2
2018-10-09 00:33:23.596304 7f91c7ae2e00 4 rocksdb:
Options.write_buffer_size: 268435456
2018-10-09 00:33:23.596306 7f91c7ae2e00 4 rocksdb:
Options.max_write_buffer_number: 4
2018-10-09 00:33:23.596307 7f91c7ae2e00 4 rocksdb:
Options.compression: NoCompression
2018-10-09 00:33:23.596309 7f91c7ae2e00 4 rocksdb:
Options.bottommost_compression: Disabled
2018-10-09 00:33:23.596310 7f91c7ae2e00 4 rocksdb:
Options.prefix_extractor: nullptr
2018-10-09 00:33:23.596312 7f91c7ae2e00 4 rocksdb:
Options.memtable_insert_with_hint_prefix_extractor: nullptr
2018-10-09 00:33:23.596313 7f91c7ae2e00 4 rocksdb:
Options.num_levels: 7
2018-10-09 00:33:23.596314 7f91c7ae2e00 4 rocksdb:
Options.min_write_buffer_number_to_merge: 1
2018-10-09 00:33:23.596315 7f91c7ae2e00 4 rocksdb:
Options.max_write_buffer_number_to_maintain: 0
2018-10-09 00:33:23.596317 7f91c7ae2e00 4 rocksdb:
Options.compression_opts.window_bits: -14
2018-10-09 00:33:23.596318 7f91c7ae2e00 4 rocksdb:
Options.compression_opts.level: -1
2018-10-09 00:33:23.596319 7f91c7ae2e00 4 rocksdb:
Options.compression_opts.strategy: 0
2018-10-09 00:33:23.596321 7f91c7ae2e00 4 rocksdb:
Options.compression_opts.max_dict_bytes: 0
2018-10-09 00:33:23.596324 7f91c7ae2e00 4 rocksdb:
Options.level0_file_num_compaction_trigger: 4
2018-10-09 00:33:23.596325 7f91c7ae2e00 4 rocksdb:
Options.level0_slowdown_writes_trigger: 20
2018-10-09 00:33:23.596326 7f91c7ae2e00 4 rocksdb:
Options.level0_stop_writes_trigger: 36
2018-10-09 00:33:23.596328 7f91c7ae2e00 4 rocksdb:
Options.target_file_size_base: 67108864
2018-10-09 00:33:23.596330 7f91c7ae2e00 4 rocksdb:
Options.target_file_size_multiplier: 1
2018-10-09 00:33:23.596331 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_base: 268435456
2018-10-09 00:33:23.596332 7f91c7ae2e00 4 rocksdb:
Options.level_compaction_dynamic_level_bytes: 0
2018-10-09 00:33:23.596334 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_multiplier: 10.000000
2018-10-09 00:33:23.596337 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[0]: 1
2018-10-09 00:33:23.596339 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[1]: 1
2018-10-09 00:33:23.596341 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[2]: 1
2018-10-09 00:33:23.596342 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[3]: 1
2018-10-09 00:33:23.596343 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[4]: 1
2018-10-09 00:33:23.596344 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[5]: 1
2018-10-09 00:33:23.596345 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[6]: 1
2018-10-09 00:33:23.596346 7f91c7ae2e00 4 rocksdb:
Options.max_sequential_skip_in_iterations: 8
2018-10-09 00:33:23.596348 7f91c7ae2e00 4 rocksdb:
Options.max_compaction_bytes: 1677721600
2018-10-09 00:33:23.596349 7f91c7ae2e00 4
rocksdb: Options.arena_block_size: 33554432
2018-10-09 00:33:23.596351 7f91c7ae2e00 4 rocksdb:
Options.soft_pending_compaction_bytes_limit: 68719476736
2018-10-09 00:33:23.596352 7f91c7ae2e00 4 rocksdb:
Options.hard_pending_compaction_bytes_limit: 274877906944
2018-10-09 00:33:23.596353 7f91c7ae2e00 4 rocksdb:
Options.rate_limit_delay_max_milliseconds: 100
2018-10-09 00:33:23.596354 7f91c7ae2e00 4 rocksdb:
Options.disable_auto_compactions: 0
2018-10-09 00:33:23.596356 7f91c7ae2e00 4
rocksdb: Options.compaction_style:
kCompactionStyleLevel
2018-10-09 00:33:23.596357 7f91c7ae2e00 4
rocksdb: Options.compaction_pri:
kByCompensatedSize
2018-10-09 00:33:23.596359 7f91c7ae2e00 4 rocksdb:
Options.compaction_options_universal.size_ratio: 1
2018-10-09 00:33:23.596360 7f91c7ae2e00 4 rocksdb:
Options.compaction_options_universal.min_merge_width: 2
2018-10-09 00:33:23.596361 7f91c7ae2e00 4 rocksdb:
Options.compaction_options_universal.max_merge_width: 4294967295
2018-10-09 00:33:23.596362 7f91c7ae2e00 4 rocksdb:
Options.compaction_options_universal.max_size_amplification_percent: 200
2018-10-09 00:33:23.596363 7f91c7ae2e00 4 rocksdb:
Options.compaction_options_universal.compression_size_percent: -1
2018-10-09 00:33:23.596364 7f91c7ae2e00 4 rocksdb:
Options.compaction_options_fifo.max_table_files_size: 1073741824
2018-10-09 00:33:23.596366 7f91c7ae2e00 4 rocksdb:
Options.table_properties_collectors:
2018-10-09 00:33:23.596367 7f91c7ae2e00 4 rocksdb:
Options.inplace_update_support: 0
2018-10-09 00:33:23.596368 7f91c7ae2e00 4 rocksdb:
Options.inplace_update_num_locks: 10000
2018-10-09 00:33:23.596369 7f91c7ae2e00 4 rocksdb:
Options.memtable_prefix_bloom_size_ratio: 0.000000
2018-10-09 00:33:23.596371 7f91c7ae2e00 4 rocksdb:
Options.memtable_huge_page_size: 0
2018-10-09 00:33:23.596372 7f91c7ae2e00 4
rocksdb: Options.bloom_locality: 0
2018-10-09 00:33:23.596374 7f91c7ae2e00 4 rocksdb:
Options.max_successive_merges: 0
2018-10-09 00:33:23.596374 7f91c7ae2e00 4 rocksdb:
Options.optimize_filters_for_hits: 0
2018-10-09 00:33:23.596375 7f91c7ae2e00 4 rocksdb:
Options.paranoid_file_checks: 0
2018-10-09 00:33:23.596380 7f91c7ae2e00 4 rocksdb:
Options.force_consistency_checks: 0
2018-10-09 00:33:23.596381 7f91c7ae2e00 4 rocksdb:
Options.report_bg_io_stats: 0
/build/ceph-12.2.8/src/os/bluestore/BlueFS.cc: In function 'int
BlueFS::_read_random(BlueFS::FileReader*, uint64_t, size_t, char*)'
thread 7f91ba517700 time 2018-10-09 00:33:26.394736
/build/ceph-12.2.8/src/os/bluestore/BlueFS.cc: 916: FAILED assert(r == 0)
2018-10-09 00:33:26.394691 7f91ba517700 -1 bdev(0x558d31b69200
/var/lib/ceph/osd/ceph-40/block) direct_read_unaligned
0xe0cd46efad~1c053 error: (5) Input/output error
ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x558d27c98a72]
2: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned
long, char*)+0x7a5) [0x558d27bfac15]
3: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0x20) [0x558d27c25d50]
4: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0xbb7) [0x558d28078087]
5: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::ReadOptions const&,
rocksdb::BlockHandle const&, rocksdb::BlockContents*,
rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&,
rocksdb::PersistentCacheOptions const&)+0x569) [0x558d280475a9]
6: (()+0xe22646) [0x558d28037646]
7:
(rocksdb::BinarySearchIndexReader::Create(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::BlockHandle const&,
rocksdb::ImmutableCFOptions const&, rocksdb::Comparator const*,
rocksdb::BlockBasedTable::IndexReader**, rocksdb::PersistentCacheOptions
const&)+0xc0) [0x558d28044160]
8:
(rocksdb::BlockBasedTable::CreateIndexReader(rocksdb::BlockBasedTable::IndexReader**,
rocksdb::InternalIterator*, int)+0x439) [0x558d2803ab29]
9: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions
const&, rocksdb::BlockIter*,
rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x39c)
[0x558d2803b0bc]
10: (rocksdb::BlockBasedTable::Open(rocksdb::ImmutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&,
rocksdb::InternalKeyComparator const&,
std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, bool, int)+0xf23)
[0x558d28041833]
11:
(rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::TableReaderOptions
const&, std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool) const+0x4d)
[0x558d2803509d]
12: (rocksdb::TableCache::GetTableReader(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
bool, unsigned long, bool, rocksdb::HistogramImpl*,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, int, bool)+0x53c)
[0x558d28121a6c]
13: (rocksdb::TableCache::FindTable(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
rocksdb::Cache::Handle**, bool, bool, rocksdb::HistogramImpl*, bool,
int, bool)+0x333) [0x558d28122253]
14: (std::_Function_handler<void (),
rocksdb::VersionBuilder::Rep::LoadTableHandlers(rocksdb::InternalStats*,
int, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&)+0x80)
[0x558d28127780]
15: (()+0xb9e6f) [0x7f91c4bffe6f]
16: (()+0x7494) [0x7f91c52ed494]
17: (clone()+0x3f) [0x7f91c4374acf]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
2018-10-09 00:33:26.400883 7f91ba517700 -1
/build/ceph-12.2.8/src/os/bluestore/BlueFS.cc: In function 'int
BlueFS::_read_random(BlueFS::FileReader*, uint64_t, size_t, char*)'
thread 7f91ba517700 time 2018-10-09 00:33:26.394736
/build/ceph-12.2.8/src/os/bluestore/BlueFS.cc: 916: FAILED assert(r == 0)
ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x558d27c98a72]
2: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned
long, char*)+0x7a5) [0x558d27bfac15]
3: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0x20) [0x558d27c25d50]
4: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0xbb7) [0x558d28078087]
5: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::ReadOptions const&,
rocksdb::BlockHandle const&, rocksdb::BlockContents*,
rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&,
rocksdb::PersistentCacheOptions const&)+0x569) [0x558d280475a9]
6: (()+0xe22646) [0x558d28037646]
7:
(rocksdb::BinarySearchIndexReader::Create(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::BlockHandle const&,
rocksdb::ImmutableCFOptions const&, rocksdb::Comparator const*,
rocksdb::BlockBasedTable::IndexReader**, rocksdb::PersistentCacheOptions
const&)+0xc0) [0x558d28044160]
8:
(rocksdb::BlockBasedTable::CreateIndexReader(rocksdb::BlockBasedTable::IndexReader**,
rocksdb::InternalIterator*, int)+0x439) [0x558d2803ab29]
9: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions
const&, rocksdb::BlockIter*,
rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x39c)
[0x558d2803b0bc]
10: (rocksdb::BlockBasedTable::Open(rocksdb::ImmutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&,
rocksdb::InternalKeyComparator const&,
std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, bool, int)+0xf23)
[0x558d28041833]
11:
(rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::TableReaderOptions
const&, std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool) const+0x4d)
[0x558d2803509d]
12: (rocksdb::TableCache::GetTableReader(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
bool, unsigned long, bool, rocksdb::HistogramImpl*,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, int, bool)+0x53c)
[0x558d28121a6c]
13: (rocksdb::TableCache::FindTable(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
rocksdb::Cache::Handle**, bool, bool, rocksdb::HistogramImpl*, bool,
int, bool)+0x333) [0x558d28122253]
14: (std::_Function_handler<void (),
rocksdb::VersionBuilder::Rep::LoadTableHandlers(rocksdb::InternalStats*,
int, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&)+0x80)
[0x558d28127780]
15: (()+0xb9e6f) [0x7f91c4bffe6f]
16: (()+0x7494) [0x7f91c52ed494]
17: (clone()+0x3f) [0x7f91c4374acf]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- begin dump of recent events ---
-222> 2018-10-09 00:33:23.018426 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command perfcounters_dump hook 0x558d31b661c0
-221> 2018-10-09 00:33:23.018438 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command 1 hook 0x558d31b661c0
-220> 2018-10-09 00:33:23.018441 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command perf dump hook 0x558d31b661c0
-219> 2018-10-09 00:33:23.018443 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command perfcounters_schema hook 0x558d31b661c0
-218> 2018-10-09 00:33:23.018445 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command perf histogram dump hook 0x558d31b661c0
-217> 2018-10-09 00:33:23.018447 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command 2 hook 0x558d31b661c0
-216> 2018-10-09 00:33:23.018449 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command perf schema hook 0x558d31b661c0
-215> 2018-10-09 00:33:23.018451 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command perf histogram schema hook 0x558d31b661c0
-214> 2018-10-09 00:33:23.018454 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command perf reset hook 0x558d31b661c0
-213> 2018-10-09 00:33:23.018456 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command config show hook 0x558d31b661c0
-212> 2018-10-09 00:33:23.018458 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command config help hook 0x558d31b661c0
-211> 2018-10-09 00:33:23.018460 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command config set hook 0x558d31b661c0
-210> 2018-10-09 00:33:23.018462 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command config get hook 0x558d31b661c0
-209> 2018-10-09 00:33:23.018464 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command config diff hook 0x558d31b661c0
-208> 2018-10-09 00:33:23.018466 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command config diff get hook 0x558d31b661c0
-207> 2018-10-09 00:33:23.018469 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command log flush hook 0x558d31b661c0
-206> 2018-10-09 00:33:23.018471 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command log dump hook 0x558d31b661c0
-205> 2018-10-09 00:33:23.018473 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command log reopen hook 0x558d31b661c0
-204> 2018-10-09 00:33:23.018486 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command dump_mempools hook 0x558d31e175c8
-203> 2018-10-09 00:33:23.025913 7f91c7ae2e00 0 set uid:gid to
64045:64045 (ceph:ceph)
-202> 2018-10-09 00:33:23.025934 7f91c7ae2e00 0 ceph version 12.2.8
(ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable), process
ceph-osd, pid 322330
-201> 2018-10-09 00:33:23.025974 7f91c7ae2e00 5 object store type is
bluestore
-200> 2018-10-09 00:33:23.039921 7f91c1f4d700 2 Event(0x558d31b68500
nevent=5000 time_id=1).set_owner idx=0 owner=140264001033984
-199> 2018-10-09 00:33:23.039975 7f91c174c700 2 Event(0x558d31b68740
nevent=5000 time_id=1).set_owner idx=1 owner=140263992641280
-198> 2018-10-09 00:33:23.039980 7f91c0f4b700 2 Event(0x558d31b68980
nevent=5000 time_id=1).set_owner idx=2 owner=140263984248576
-197> 2018-10-09 00:33:23.040489 7f91c7ae2e00 1 --
[2a03:4980:0:6::2e]:0/0 learned_addr learned my addr [2a03:4980:0:6::2e]:0/0
-196> 2018-10-09 00:33:23.040510 7f91c7ae2e00 1 --
[2a03:4980:0:6::2e]:6800/322330 _finish_bind bind my_inst.addr is
[2a03:4980:0:6::2e]:6800/322330
-195> 2018-10-09 00:33:23.040805 7f91c7ae2e00 1 --
[2a03:4980:0:6::2e]:0/0 learned_addr learned my addr [2a03:4980:0:6::2e]:0/0
-194> 2018-10-09 00:33:23.040820 7f91c7ae2e00 1 --
[2a03:4980:0:6::2e]:6801/322330 _finish_bind bind my_inst.addr is
[2a03:4980:0:6::2e]:6801/322330
-193> 2018-10-09 00:33:23.041264 7f91c7ae2e00 1 --
[2a03:4980:0:6::2e]:0/0 learned_addr learned my addr [2a03:4980:0:6::2e]:0/0
-192> 2018-10-09 00:33:23.041279 7f91c7ae2e00 1 --
[2a03:4980:0:6::2e]:6802/322330 _finish_bind bind my_inst.addr is
[2a03:4980:0:6::2e]:6802/322330
-191> 2018-10-09 00:33:23.041565 7f91c7ae2e00 1 --
[2a03:4980:0:6::2e]:0/0 learned_addr learned my addr [2a03:4980:0:6::2e]:0/0
-190> 2018-10-09 00:33:23.041580 7f91c7ae2e00 1 --
[2a03:4980:0:6::2e]:6803/322330 _finish_bind bind my_inst.addr is
[2a03:4980:0:6::2e]:6803/322330
-189> 2018-10-09 00:33:23.044207 7f91c7ae2e00 5 asok(0x558d31bc74a0)
init /var/run/ceph/ceph-osd.40.asok
-188> 2018-10-09 00:33:23.044224 7f91c7ae2e00 5 asok(0x558d31bc74a0)
bind_and_listen /var/run/ceph/ceph-osd.40.asok
-187> 2018-10-09 00:33:23.044292 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command 0 hook 0x558d31b641a8
-186> 2018-10-09 00:33:23.044299 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command version hook 0x558d31b641a8
-185> 2018-10-09 00:33:23.044302 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command git_version hook 0x558d31b641a8
-184> 2018-10-09 00:33:23.044306 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command help hook 0x558d31b66620
-183> 2018-10-09 00:33:23.044310 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command get_command_descriptions hook 0x558d31b66630
-182> 2018-10-09 00:33:23.044391 7f91c7ae2e00 10 monclient:
build_initial_monmap
-181> 2018-10-09 00:33:23.044443 7f91bf77a700 5 asok(0x558d31bc74a0)
entry start
-180> 2018-10-09 00:33:23.203930 7f91c7ae2e00 0 load: jerasure load:
lrc load: isa
-179> 2018-10-09 00:33:23.203960 7f91c7ae2e00 5 adding auth protocol:
cephx
-178> 2018-10-09 00:33:23.203962 7f91c7ae2e00 5 adding auth protocol:
cephx
-177> 2018-10-09 00:33:23.204022 7f91c7ae2e00 1 bdev create path
/var/lib/ceph/osd/ceph-40/block type kernel
-176> 2018-10-09 00:33:23.204032 7f91c7ae2e00 1 bdev(0x558d31b68b40
/var/lib/ceph/osd/ceph-40/block) open path /var/lib/ceph/osd/ceph-40/block
-175> 2018-10-09 00:33:23.204336 7f91c7ae2e00 1 bdev(0x558d31b68b40
/var/lib/ceph/osd/ceph-40/block) open size 1990292758528 (0x1cf66b16000,
1.81TiB) block_size 4096 (4KiB) rotational
-174> 2018-10-09 00:33:23.204766 7f91c7ae2e00 1
bluestore(/var/lib/ceph/osd/ceph-40) _set_cache_sizes max 0.5 < ratio 0.99
-173> 2018-10-09 00:33:23.204790 7f91c7ae2e00 1
bluestore(/var/lib/ceph/osd/ceph-40) _set_cache_sizes cache_size
1073741824 meta 0.5 kv 0.5 data 0
-172> 2018-10-09 00:33:23.204796 7f91c7ae2e00 1 bdev(0x558d31b68b40
/var/lib/ceph/osd/ceph-40/block) close
-171> 2018-10-09 00:33:23.499587 7f91c7ae2e00 5 asok(0x558d31bc74a0)
register_command objecter_requests hook 0x558d31b666d0
-170> 2018-10-09 00:33:23.499679 7f91c7ae2e00 1 --
[2a03:4980:0:6::2e]:6800/322330 start start
-169> 2018-10-09 00:33:23.499693 7f91c7ae2e00 1 -- - start start
-168> 2018-10-09 00:33:23.499696 7f91c7ae2e00 1 -- - start start
-167> 2018-10-09 00:33:23.499697 7f91c7ae2e00 1 --
[2a03:4980:0:6::2e]:6803/322330 start start
-166> 2018-10-09 00:33:23.499701 7f91c7ae2e00 1 --
[2a03:4980:0:6::2e]:6802/322330 start start
-165> 2018-10-09 00:33:23.499705 7f91c7ae2e00 1 --
[2a03:4980:0:6::2e]:6801/322330 start start
-164> 2018-10-09 00:33:23.499708 7f91c7ae2e00 1 -- - start start
-163> 2018-10-09 00:33:23.499855 7f91c7ae2e00 2 osd.40 0 init
/var/lib/ceph/osd/ceph-40 (looks like hdd)
-162> 2018-10-09 00:33:23.499863 7f91c7ae2e00 2 osd.40 0 journal
/var/lib/ceph/osd/ceph-40/journal
-161> 2018-10-09 00:33:23.499876 7f91c7ae2e00 1
bluestore(/var/lib/ceph/osd/ceph-40) _mount path /var/lib/ceph/osd/ceph-40
-160> 2018-10-09 00:33:23.500355 7f91c7ae2e00 1 bdev create path
/var/lib/ceph/osd/ceph-40/block type kernel
-159> 2018-10-09 00:33:23.500368 7f91c7ae2e00 1 bdev(0x558d31b68d80
/var/lib/ceph/osd/ceph-40/block) open path /var/lib/ceph/osd/ceph-40/block
-158> 2018-10-09 00:33:23.500583 7f91c7ae2e00 1 bdev(0x558d31b68d80
/var/lib/ceph/osd/ceph-40/block) open size 1990292758528 (0x1cf66b16000,
1.81TiB) block_size 4096 (4KiB) rotational
-157> 2018-10-09 00:33:23.500884 7f91c7ae2e00 1
bluestore(/var/lib/ceph/osd/ceph-40) _set_cache_sizes max 0.5 < ratio 0.99
-156> 2018-10-09 00:33:23.500900 7f91c7ae2e00 1
bluestore(/var/lib/ceph/osd/ceph-40) _set_cache_sizes cache_size
1073741824 meta 0.5 kv 0.5 data 0
-155> 2018-10-09 00:33:23.501051 7f91c7ae2e00 1 bdev create path
/var/lib/ceph/osd/ceph-40/block type kernel
-154> 2018-10-09 00:33:23.501061 7f91c7ae2e00 1 bdev(0x558d31b69200
/var/lib/ceph/osd/ceph-40/block) open path /var/lib/ceph/osd/ceph-40/block
-153> 2018-10-09 00:33:23.501263 7f91c7ae2e00 1 bdev(0x558d31b69200
/var/lib/ceph/osd/ceph-40/block) open size 1990292758528 (0x1cf66b16000,
1.81TiB) block_size 4096 (4KiB) rotational
-152> 2018-10-09 00:33:23.501277 7f91c7ae2e00 1 bluefs
add_block_device bdev 1 path /var/lib/ceph/osd/ceph-40/block size 1.81TiB
-151> 2018-10-09 00:33:23.501310 7f91c7ae2e00 1 bluefs mount
-150> 2018-10-09 00:33:23.589465 7f91c7ae2e00 0 set rocksdb option
compaction_readahead_size = 2097152
-149> 2018-10-09 00:33:23.589482 7f91c7ae2e00 0 set rocksdb option
compression = kNoCompression
-148> 2018-10-09 00:33:23.589487 7f91c7ae2e00 0 set rocksdb option
max_write_buffer_number = 4
-147> 2018-10-09 00:33:23.589490 7f91c7ae2e00 0 set rocksdb option
min_write_buffer_number_to_merge = 1
-146> 2018-10-09 00:33:23.589494 7f91c7ae2e00 0 set rocksdb option
recycle_log_file_num = 4
-145> 2018-10-09 00:33:23.589496 7f91c7ae2e00 0 set rocksdb option
writable_file_max_buffer_size = 0
-144> 2018-10-09 00:33:23.589500 7f91c7ae2e00 0 set rocksdb option
write_buffer_size = 268435456
-143> 2018-10-09 00:33:23.589517 7f91c7ae2e00 0 set rocksdb option
compaction_readahead_size = 2097152
-142> 2018-10-09 00:33:23.589520 7f91c7ae2e00 0 set rocksdb option
compression = kNoCompression
-141> 2018-10-09 00:33:23.589523 7f91c7ae2e00 0 set rocksdb option
max_write_buffer_number = 4
-140> 2018-10-09 00:33:23.589526 7f91c7ae2e00 0 set rocksdb option
min_write_buffer_number_to_merge = 1
-139> 2018-10-09 00:33:23.589528 7f91c7ae2e00 0 set rocksdb option
recycle_log_file_num = 4
-138> 2018-10-09 00:33:23.589531 7f91c7ae2e00 0 set rocksdb option
writable_file_max_buffer_size = 0
-137> 2018-10-09 00:33:23.589533 7f91c7ae2e00 0 set rocksdb option
write_buffer_size = 268435456
-136> 2018-10-09 00:33:23.589656 7f91c7ae2e00 4 rocksdb: RocksDB
version: 5.4.0
-135> 2018-10-09 00:33:23.589667 7f91c7ae2e00 4 rocksdb: Git sha
rocksdb_build_git_sha:@0@
-134> 2018-10-09 00:33:23.589670 7f91c7ae2e00 4 rocksdb: Compile date
Aug 30 2018
-133> 2018-10-09 00:33:23.589671 7f91c7ae2e00 4 rocksdb: DB SUMMARY
-132> 2018-10-09 00:33:23.589706 7f91c7ae2e00 4 rocksdb: CURRENT
file: CURRENT
-131> 2018-10-09 00:33:23.589707 7f91c7ae2e00 4 rocksdb: IDENTITY
file: IDENTITY
-130> 2018-10-09 00:33:23.589709 7f91c7ae2e00 4 rocksdb: MANIFEST
file: MANIFEST-000011 size: 51827 Bytes
-129> 2018-10-09 00:33:23.589712 7f91c7ae2e00 4 rocksdb: SST files in
db dir, Total Num: 28, files: 000260.sst 000261.sst 000262.sst
000274.sst 000275.sst 000276.sst 000277.sst 000286.sst 000287.sst
-128> 2018-10-09 00:33:23.589714 7f91c7ae2e00 4 rocksdb: Write Ahead
Log file in db: 000310.log size: 258554009 ; 000312.log size: 256884429 ;
-127> 2018-10-09 00:33:23.589715 7f91c7ae2e00 4
rocksdb: Options.error_if_exists: 0
-126> 2018-10-09 00:33:23.589716 7f91c7ae2e00 4
rocksdb: Options.create_if_missing: 0
-125> 2018-10-09 00:33:23.589717 7f91c7ae2e00 4
rocksdb: Options.paranoid_checks: 1
-124> 2018-10-09 00:33:23.589718 7f91c7ae2e00 4
rocksdb: Options.env: 0x558d31d42280
-123> 2018-10-09 00:33:23.589719 7f91c7ae2e00 4
rocksdb: Options.info_log: 0x558d31d42a00
-122> 2018-10-09 00:33:23.589720 7f91c7ae2e00 4
rocksdb: Options.max_open_files: -1
-121> 2018-10-09 00:33:23.589721 7f91c7ae2e00 4
rocksdb: Options.max_file_opening_threads: 16
-120> 2018-10-09 00:33:23.589722 7f91c7ae2e00 4
rocksdb: Options.use_fsync: 0
-119> 2018-10-09 00:33:23.589723 7f91c7ae2e00 4
rocksdb: Options.max_log_file_size: 0
-118> 2018-10-09 00:33:23.589724 7f91c7ae2e00 4
rocksdb: Options.max_manifest_file_size:
18446744073709551615
-117> 2018-10-09 00:33:23.589725 7f91c7ae2e00 4
rocksdb: Options.log_file_time_to_roll: 0
-116> 2018-10-09 00:33:23.589726 7f91c7ae2e00 4
rocksdb: Options.keep_log_file_num: 1000
-115> 2018-10-09 00:33:23.589726 7f91c7ae2e00 4
rocksdb: Options.recycle_log_file_num: 4
-114> 2018-10-09 00:33:23.589727 7f91c7ae2e00 4
rocksdb: Options.allow_fallocate: 1
-113> 2018-10-09 00:33:23.589728 7f91c7ae2e00 4
rocksdb: Options.allow_mmap_reads: 0
-112> 2018-10-09 00:33:23.589728 7f91c7ae2e00 4
rocksdb: Options.allow_mmap_writes: 0
-111> 2018-10-09 00:33:23.589729 7f91c7ae2e00 4
rocksdb: Options.use_direct_reads: 0
-110> 2018-10-09 00:33:23.589730 7f91c7ae2e00 4
rocksdb:
Options.use_direct_io_for_flush_and_compaction: 0
-109> 2018-10-09 00:33:23.589731 7f91c7ae2e00 4 rocksdb:
Options.create_missing_column_families: 0
-108> 2018-10-09 00:33:23.589731 7f91c7ae2e00 4
rocksdb: Options.db_log_dir:
-107> 2018-10-09 00:33:23.589732 7f91c7ae2e00 4
rocksdb: Options.wal_dir: db
-106> 2018-10-09 00:33:23.589733 7f91c7ae2e00 4
rocksdb: Options.table_cache_numshardbits: 6
-105> 2018-10-09 00:33:23.589734 7f91c7ae2e00 4
rocksdb: Options.max_subcompactions: 1
-104> 2018-10-09 00:33:23.589735 7f91c7ae2e00 4
rocksdb: Options.max_background_flushes: 1
-103> 2018-10-09 00:33:23.589736 7f91c7ae2e00 4
rocksdb: Options.WAL_ttl_seconds: 0
-102> 2018-10-09 00:33:23.589737 7f91c7ae2e00 4
rocksdb: Options.WAL_size_limit_MB: 0
-101> 2018-10-09 00:33:23.589738 7f91c7ae2e00 4 rocksdb:
Options.manifest_preallocation_size: 4194304
-100> 2018-10-09 00:33:23.589739 7f91c7ae2e00 4
rocksdb: Options.is_fd_close_on_exec: 1
-99> 2018-10-09 00:33:23.589740 7f91c7ae2e00 4
rocksdb: Options.advise_random_on_open: 1
-98> 2018-10-09 00:33:23.589741 7f91c7ae2e00 4
rocksdb: Options.db_write_buffer_size: 0
-97> 2018-10-09 00:33:23.589741 7f91c7ae2e00 4 rocksdb:
Options.access_hint_on_compaction_start: 1
-96> 2018-10-09 00:33:23.589742 7f91c7ae2e00 4 rocksdb:
Options.new_table_reader_for_compaction_inputs: 1
-95> 2018-10-09 00:33:23.589743 7f91c7ae2e00 4
rocksdb: Options.compaction_readahead_size: 2097152
-94> 2018-10-09 00:33:23.589744 7f91c7ae2e00 4 rocksdb:
Options.random_access_max_buffer_size: 1048576
-93> 2018-10-09 00:33:23.589745 7f91c7ae2e00 4 rocksdb:
Options.writable_file_max_buffer_size: 0
-92> 2018-10-09 00:33:23.589746 7f91c7ae2e00 4
rocksdb: Options.use_adaptive_mutex: 0
-91> 2018-10-09 00:33:23.589746 7f91c7ae2e00 4
rocksdb: Options.rate_limiter: (nil)
-90> 2018-10-09 00:33:23.589747 7f91c7ae2e00 4 rocksdb:
Options.sst_file_manager.rate_bytes_per_sec: 0
-89> 2018-10-09 00:33:23.589748 7f91c7ae2e00 4
rocksdb: Options.bytes_per_sync: 0
-88> 2018-10-09 00:33:23.589749 7f91c7ae2e00 4
rocksdb: Options.wal_bytes_per_sync: 0
-87> 2018-10-09 00:33:23.589749 7f91c7ae2e00 4
rocksdb: Options.wal_recovery_mode: 0
-86> 2018-10-09 00:33:23.589750 7f91c7ae2e00 4
rocksdb: Options.enable_thread_tracking: 0
-85> 2018-10-09 00:33:23.589751 7f91c7ae2e00 4 rocksdb:
Options.allow_concurrent_memtable_write: 1
-84> 2018-10-09 00:33:23.589762 7f91c7ae2e00 4 rocksdb:
Options.enable_write_thread_adaptive_yield: 1
-83> 2018-10-09 00:33:23.589763 7f91c7ae2e00 4 rocksdb:
Options.write_thread_max_yield_usec: 100
-82> 2018-10-09 00:33:23.589764 7f91c7ae2e00 4 rocksdb:
Options.write_thread_slow_yield_usec: 3
-81> 2018-10-09 00:33:23.589765 7f91c7ae2e00 4
rocksdb: Options.row_cache: None
-80> 2018-10-09 00:33:23.589766 7f91c7ae2e00 4
rocksdb: Options.wal_filter: None
-79> 2018-10-09 00:33:23.589766 7f91c7ae2e00 4 rocksdb:
Options.avoid_flush_during_recovery: 0
-78> 2018-10-09 00:33:23.589767 7f91c7ae2e00 4 rocksdb:
Options.base_background_compactions: 1
-77> 2018-10-09 00:33:23.589768 7f91c7ae2e00 4 rocksdb:
Options.max_background_compactions: 1
-76> 2018-10-09 00:33:23.589768 7f91c7ae2e00 4 rocksdb:
Options.avoid_flush_during_shutdown: 0
-75> 2018-10-09 00:33:23.589769 7f91c7ae2e00 4 rocksdb:
Options.delayed_write_rate : 16777216
-74> 2018-10-09 00:33:23.589770 7f91c7ae2e00 4 rocksdb:
Options.max_total_wal_size: 0
-73> 2018-10-09 00:33:23.589770 7f91c7ae2e00 4 rocksdb:
Options.delete_obsolete_files_period_micros: 21600000000
-72> 2018-10-09 00:33:23.589771 7f91c7ae2e00 4
rocksdb: Options.stats_dump_period_sec: 600
-71> 2018-10-09 00:33:23.589772 7f91c7ae2e00 4 rocksdb: Compression
algorithms supported:
-70> 2018-10-09 00:33:23.589773 7f91c7ae2e00 4 rocksdb: Snappy
supported: 1
-69> 2018-10-09 00:33:23.589774 7f91c7ae2e00 4 rocksdb: Zlib
supported: 0
-68> 2018-10-09 00:33:23.589774 7f91c7ae2e00 4 rocksdb: Bzip
supported: 0
-67> 2018-10-09 00:33:23.589775 7f91c7ae2e00 4 rocksdb: LZ4
supported: 0
-66> 2018-10-09 00:33:23.589776 7f91c7ae2e00 4 rocksdb: ZSTD
supported: 0
-65> 2018-10-09 00:33:23.589777 7f91c7ae2e00 4 rocksdb: Fast CRC32
supported: 1
-64> 2018-10-09 00:33:23.596185 7f91c7ae2e00 4 rocksdb:
[/build/ceph-12.2.8/src/rocksdb/db/version_set.cc:2609] Recovering from
manifest file: MANIFEST-000011
-63> 2018-10-09 00:33:23.596256 7f91c7ae2e00 4 rocksdb:
[/build/ceph-12.2.8/src/rocksdb/db/column_family.cc:407] ---------------
Options for column family [default]:
-62> 2018-10-09 00:33:23.596261 7f91c7ae2e00 4
rocksdb: Options.comparator: leveldb.BytewiseComparator
-61> 2018-10-09 00:33:23.596267 7f91c7ae2e00 4 rocksdb:
Options.merge_operator: .T:int64_array.b:bitwise_xor
-60> 2018-10-09 00:33:23.596268 7f91c7ae2e00 4 rocksdb:
Options.compaction_filter: None
-59> 2018-10-09 00:33:23.596272 7f91c7ae2e00 4 rocksdb:
Options.compaction_filter_factory: None
-58> 2018-10-09 00:33:23.596274 7f91c7ae2e00 4 rocksdb:
Options.memtable_factory: SkipListFactory
-57> 2018-10-09 00:33:23.596275 7f91c7ae2e00 4 rocksdb:
Options.table_factory: BlockBasedTable
-56> 2018-10-09 00:33:23.596299 7f91c7ae2e00 4 rocksdb:
table_factory options: flush_block_policy_factory:
FlushBlockBySizePolicyFactory (0x558d31b641f0)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 1
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x558d31e6f0b0
block_cache_name: LRUCache
block_cache_options:
capacity : 536870912
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
format_version: 2
-55> 2018-10-09 00:33:23.596304 7f91c7ae2e00 4 rocksdb:
Options.write_buffer_size: 268435456
-54> 2018-10-09 00:33:23.596306 7f91c7ae2e00 4 rocksdb:
Options.max_write_buffer_number: 4
-53> 2018-10-09 00:33:23.596307 7f91c7ae2e00 4 rocksdb:
Options.compression: NoCompression
-52> 2018-10-09 00:33:23.596309 7f91c7ae2e00 4
rocksdb: Options.bottommost_compression: Disabled
-51> 2018-10-09 00:33:23.596310 7f91c7ae2e00 4 rocksdb:
Options.prefix_extractor: nullptr
-50> 2018-10-09 00:33:23.596312 7f91c7ae2e00 4 rocksdb:
Options.memtable_insert_with_hint_prefix_extractor: nullptr
-49> 2018-10-09 00:33:23.596313 7f91c7ae2e00 4 rocksdb:
Options.num_levels: 7
-48> 2018-10-09 00:33:23.596314 7f91c7ae2e00 4 rocksdb:
Options.min_write_buffer_number_to_merge: 1
-47> 2018-10-09 00:33:23.596315 7f91c7ae2e00 4 rocksdb:
Options.max_write_buffer_number_to_maintain: 0
-46> 2018-10-09 00:33:23.596317 7f91c7ae2e00 4 rocksdb:
Options.compression_opts.window_bits: -14
-45> 2018-10-09 00:33:23.596318 7f91c7ae2e00 4
rocksdb: Options.compression_opts.level: -1
-44> 2018-10-09 00:33:23.596319 7f91c7ae2e00 4
rocksdb: Options.compression_opts.strategy: 0
-43> 2018-10-09 00:33:23.596321 7f91c7ae2e00 4 rocksdb:
Options.compression_opts.max_dict_bytes: 0
-42> 2018-10-09 00:33:23.596324 7f91c7ae2e00 4 rocksdb:
Options.level0_file_num_compaction_trigger: 4
-41> 2018-10-09 00:33:23.596325 7f91c7ae2e00 4 rocksdb:
Options.level0_slowdown_writes_trigger: 20
-40> 2018-10-09 00:33:23.596326 7f91c7ae2e00 4 rocksdb:
Options.level0_stop_writes_trigger: 36
-39> 2018-10-09 00:33:23.596328 7f91c7ae2e00 4
rocksdb: Options.target_file_size_base: 67108864
-38> 2018-10-09 00:33:23.596330 7f91c7ae2e00 4 rocksdb:
Options.target_file_size_multiplier: 1
-37> 2018-10-09 00:33:23.596331 7f91c7ae2e00 4
rocksdb: Options.max_bytes_for_level_base: 268435456
-36> 2018-10-09 00:33:23.596332 7f91c7ae2e00 4 rocksdb:
Options.level_compaction_dynamic_level_bytes: 0
-35> 2018-10-09 00:33:23.596334 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_multiplier: 10.000000
-34> 2018-10-09 00:33:23.596337 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[0]: 1
-33> 2018-10-09 00:33:23.596339 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[1]: 1
-32> 2018-10-09 00:33:23.596341 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[2]: 1
-31> 2018-10-09 00:33:23.596342 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[3]: 1
-30> 2018-10-09 00:33:23.596343 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[4]: 1
-29> 2018-10-09 00:33:23.596344 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[5]: 1
-28> 2018-10-09 00:33:23.596345 7f91c7ae2e00 4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[6]: 1
-27> 2018-10-09 00:33:23.596346 7f91c7ae2e00 4 rocksdb:
Options.max_sequential_skip_in_iterations: 8
-26> 2018-10-09 00:33:23.596348 7f91c7ae2e00 4
rocksdb: Options.max_compaction_bytes: 1677721600
-25> 2018-10-09 00:33:23.596349 7f91c7ae2e00 4
rocksdb: Options.arena_block_size: 33554432
-24> 2018-10-09 00:33:23.596351 7f91c7ae2e00 4 rocksdb:
Options.soft_pending_compaction_bytes_limit: 68719476736
-23> 2018-10-09 00:33:23.596352 7f91c7ae2e00 4 rocksdb:
Options.hard_pending_compaction_bytes_limit: 274877906944
-22> 2018-10-09 00:33:23.596353 7f91c7ae2e00 4 rocksdb:
Options.rate_limit_delay_max_milliseconds: 100
-21> 2018-10-09 00:33:23.596354 7f91c7ae2e00 4
rocksdb: Options.disable_auto_compactions: 0
-20> 2018-10-09 00:33:23.596356 7f91c7ae2e00 4
rocksdb: Options.compaction_style:
kCompactionStyleLevel
-19> 2018-10-09 00:33:23.596357 7f91c7ae2e00 4
rocksdb: Options.compaction_pri:
kByCompensatedSize
-18> 2018-10-09 00:33:23.596359 7f91c7ae2e00 4 rocksdb:
Options.compaction_options_universal.size_ratio: 1
-17> 2018-10-09 00:33:23.596360 7f91c7ae2e00 4 rocksdb:
Options.compaction_options_universal.min_merge_width: 2
-16> 2018-10-09 00:33:23.596361 7f91c7ae2e00 4 rocksdb:
Options.compaction_options_universal.max_merge_width: 4294967295
-15> 2018-10-09 00:33:23.596362 7f91c7ae2e00 4 rocksdb:
Options.compaction_options_universal.max_size_amplification_percent: 200
-14> 2018-10-09 00:33:23.596363 7f91c7ae2e00 4 rocksdb:
Options.compaction_options_universal.compression_size_percent: -1
-13> 2018-10-09 00:33:23.596364 7f91c7ae2e00 4 rocksdb:
Options.compaction_options_fifo.max_table_files_size: 1073741824
-12> 2018-10-09 00:33:23.596366 7f91c7ae2e00 4
rocksdb: Options.table_properties_collectors:
-11> 2018-10-09 00:33:23.596367 7f91c7ae2e00 4
rocksdb: Options.inplace_update_support: 0
-10> 2018-10-09 00:33:23.596368 7f91c7ae2e00 4
rocksdb: Options.inplace_update_num_locks: 10000
-9> 2018-10-09 00:33:23.596369 7f91c7ae2e00 4
rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
-8> 2018-10-09 00:33:23.596371 7f91c7ae2e00 4 rocksdb:
Options.memtable_huge_page_size: 0
-7> 2018-10-09 00:33:23.596372 7f91c7ae2e00 4
rocksdb: Options.bloom_locality: 0
-6> 2018-10-09 00:33:23.596374 7f91c7ae2e00 4
rocksdb: Options.max_successive_merges: 0
-5> 2018-10-09 00:33:23.596374 7f91c7ae2e00 4
rocksdb: Options.optimize_filters_for_hits: 0
-4> 2018-10-09 00:33:23.596375 7f91c7ae2e00 4
rocksdb: Options.paranoid_file_checks: 0
-3> 2018-10-09 00:33:23.596380 7f91c7ae2e00 4
rocksdb: Options.force_consistency_checks: 0
-2> 2018-10-09 00:33:23.596381 7f91c7ae2e00 4
rocksdb: Options.report_bg_io_stats: 0
-1> 2018-10-09 00:33:26.394691 7f91ba517700 -1 bdev(0x558d31b69200
/var/lib/ceph/osd/ceph-40/block) direct_read_unaligned
0xe0cd46efad~1c053 error: (5) Input/output error
0> 2018-10-09 00:33:26.400883 7f91ba517700 -1
/build/ceph-12.2.8/src/os/bluestore/BlueFS.cc: In function 'int
BlueFS::_read_random(BlueFS::FileReader*, uint64_t, size_t, char*)'
thread 7f91ba517700 time 2018-10-09 00:33:26.394736
/build/ceph-12.2.8/src/os/bluestore/BlueFS.cc: 916: FAILED assert(r == 0)
ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x558d27c98a72]
2: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned
long, char*)+0x7a5) [0x558d27bfac15]
3: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0x20) [0x558d27c25d50]
4: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0xbb7) [0x558d28078087]
5: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::ReadOptions const&,
rocksdb::BlockHandle const&, rocksdb::BlockContents*,
rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&,
rocksdb::PersistentCacheOptions const&)+0x569) [0x558d280475a9]
6: (()+0xe22646) [0x558d28037646]
7:
(rocksdb::BinarySearchIndexReader::Create(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::BlockHandle const&,
rocksdb::ImmutableCFOptions const&, rocksdb::Comparator const*,
rocksdb::BlockBasedTable::IndexReader**, rocksdb::PersistentCacheOptions
const&)+0xc0) [0x558d28044160]
8:
(rocksdb::BlockBasedTable::CreateIndexReader(rocksdb::BlockBasedTable::IndexReader**,
rocksdb::InternalIterator*, int)+0x439) [0x558d2803ab29]
9: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions
const&, rocksdb::BlockIter*,
rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x39c)
[0x558d2803b0bc]
10: (rocksdb::BlockBasedTable::Open(rocksdb::ImmutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&,
rocksdb::InternalKeyComparator const&,
std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, bool, int)+0xf23)
[0x558d28041833]
11:
(rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::TableReaderOptions
const&, std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool) const+0x4d)
[0x558d2803509d]
12: (rocksdb::TableCache::GetTableReader(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
bool, unsigned long, bool, rocksdb::HistogramImpl*,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, int, bool)+0x53c)
[0x558d28121a6c]
13: (rocksdb::TableCache::FindTable(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
rocksdb::Cache::Handle**, bool, bool, rocksdb::HistogramImpl*, bool,
int, bool)+0x333) [0x558d28122253]
14: (std::_Function_handler<void (),
rocksdb::VersionBuilder::Rep::LoadTableHandlers(rocksdb::InternalStats*,
int, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&)+0x80)
[0x558d28127780]
15: (()+0xb9e6f) [0x7f91c4bffe6f]
16: (()+0x7494) [0x7f91c52ed494]
17: (clone()+0x3f) [0x7f91c4374acf]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-1/-1 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
*** Caught signal (Aborted) **
in thread 7f91ba517700 thread_name:ceph-osd
ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous
(stable)
1: (()+0xa3bba4) [0x558d27c50ba4]
2: (()+0x110c0) [0x7f91c52f70c0]
3: (gsignal()+0xcf) [0x7f91c42befff]
4: (abort()+0x16a) [0x7f91c42c042a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x28e) [0x558d27c98bfe]
6: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned
long, char*)+0x7a5) [0x558d27bfac15]
7: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0x20) [0x558d27c25d50]
8: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0xbb7) [0x558d28078087]
9: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::ReadOptions const&,
rocksdb::BlockHandle const&, rocksdb::BlockContents*,
rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&,
rocksdb::PersistentCacheOptions const&)+0x569) [0x558d280475a9]
10: (()+0xe22646) [0x558d28037646]
11:
(rocksdb::BinarySearchIndexReader::Create(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::BlockHandle const&,
rocksdb::ImmutableCFOptions const&, rocksdb::Comparator const*,
rocksdb::BlockBasedTable::IndexReader**, rocksdb::PersistentCacheOptions
const&)+0xc0) [0x558d28044160]
12:
(rocksdb::BlockBasedTable::CreateIndexReader(rocksdb::BlockBasedTable::IndexReader**,
rocksdb::InternalIterator*, int)+0x439) [0x558d2803ab29]
13: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions
const&, rocksdb::BlockIter*,
rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x39c)
[0x558d2803b0bc]
14: (rocksdb::BlockBasedTable::Open(rocksdb::ImmutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&,
rocksdb::InternalKeyComparator const&,
std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, bool, int)+0xf23)
[0x558d28041833]
15:
(rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::TableReaderOptions
const&, std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool) const+0x4d)
[0x558d2803509d]
16: (rocksdb::TableCache::GetTableReader(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
bool, unsigned long, bool, rocksdb::HistogramImpl*,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, int, bool)+0x53c)
[0x558d28121a6c]
17: (rocksdb::TableCache::FindTable(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
rocksdb::Cache::Handle**, bool, bool, rocksdb::HistogramImpl*, bool,
int, bool)+0x333) [0x558d28122253]
18: (std::_Function_handler<void (),
rocksdb::VersionBuilder::Rep::LoadTableHandlers(rocksdb::InternalStats*,
int, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&)+0x80)
[0x558d28127780]
19: (()+0xb9e6f) [0x7f91c4bffe6f]
20: (()+0x7494) [0x7f91c52ed494]
21: (clone()+0x3f) [0x7f91c4374acf]
2018-10-09 00:33:26.408732 7f91ba517700 -1 *** Caught signal (Aborted) **
in thread 7f91ba517700 thread_name:ceph-osd
ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous
(stable)
1: (()+0xa3bba4) [0x558d27c50ba4]
2: (()+0x110c0) [0x7f91c52f70c0]
3: (gsignal()+0xcf) [0x7f91c42befff]
4: (abort()+0x16a) [0x7f91c42c042a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x28e) [0x558d27c98bfe]
6: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned
long, char*)+0x7a5) [0x558d27bfac15]
7: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0x20) [0x558d27c25d50]
8: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0xbb7) [0x558d28078087]
9: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::ReadOptions const&,
rocksdb::BlockHandle const&, rocksdb::BlockContents*,
rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&,
rocksdb::PersistentCacheOptions const&)+0x569) [0x558d280475a9]
10: (()+0xe22646) [0x558d28037646]
11:
(rocksdb::BinarySearchIndexReader::Create(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::BlockHandle const&,
rocksdb::ImmutableCFOptions const&, rocksdb::Comparator const*,
rocksdb::BlockBasedTable::IndexReader**, rocksdb::PersistentCacheOptions
const&)+0xc0) [0x558d28044160]
12:
(rocksdb::BlockBasedTable::CreateIndexReader(rocksdb::BlockBasedTable::IndexReader**,
rocksdb::InternalIterator*, int)+0x439) [0x558d2803ab29]
13: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions
const&, rocksdb::BlockIter*,
rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x39c)
[0x558d2803b0bc]
14: (rocksdb::BlockBasedTable::Open(rocksdb::ImmutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&,
rocksdb::InternalKeyComparator const&,
std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, bool, int)+0xf23)
[0x558d28041833]
15:
(rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::TableReaderOptions
const&, std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool) const+0x4d)
[0x558d2803509d]
16: (rocksdb::TableCache::GetTableReader(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
bool, unsigned long, bool, rocksdb::HistogramImpl*,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, int, bool)+0x53c)
[0x558d28121a6c]
17: (rocksdb::TableCache::FindTable(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
rocksdb::Cache::Handle**, bool, bool, rocksdb::HistogramImpl*, bool,
int, bool)+0x333) [0x558d28122253]
18: (std::_Function_handler<void (),
rocksdb::VersionBuilder::Rep::LoadTableHandlers(rocksdb::InternalStats*,
int, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&)+0x80)
[0x558d28127780]
19: (()+0xb9e6f) [0x7f91c4bffe6f]
20: (()+0x7494) [0x7f91c52ed494]
21: (clone()+0x3f) [0x7f91c4374acf]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- begin dump of recent events ---
0> 2018-10-09 00:33:26.408732 7f91ba517700 -1 *** Caught signal
(Aborted) **
in thread 7f91ba517700 thread_name:ceph-osd
ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous
(stable)
1: (()+0xa3bba4) [0x558d27c50ba4]
2: (()+0x110c0) [0x7f91c52f70c0]
3: (gsignal()+0xcf) [0x7f91c42befff]
4: (abort()+0x16a) [0x7f91c42c042a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x28e) [0x558d27c98bfe]
6: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned
long, char*)+0x7a5) [0x558d27bfac15]
7: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0x20) [0x558d27c25d50]
8: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0xbb7) [0x558d28078087]
9: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::ReadOptions const&,
rocksdb::BlockHandle const&, rocksdb::BlockContents*,
rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&,
rocksdb::PersistentCacheOptions const&)+0x569) [0x558d280475a9]
10: (()+0xe22646) [0x558d28037646]
11:
(rocksdb::BinarySearchIndexReader::Create(rocksdb::RandomAccessFileReader*,
rocksdb::Footer const&, rocksdb::BlockHandle const&,
rocksdb::ImmutableCFOptions const&, rocksdb::Comparator const*,
rocksdb::BlockBasedTable::IndexReader**, rocksdb::PersistentCacheOptions
const&)+0xc0) [0x558d28044160]
12:
(rocksdb::BlockBasedTable::CreateIndexReader(rocksdb::BlockBasedTable::IndexReader**,
rocksdb::InternalIterator*, int)+0x439) [0x558d2803ab29]
13: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions
const&, rocksdb::BlockIter*,
rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x39c)
[0x558d2803b0bc]
14: (rocksdb::BlockBasedTable::Open(rocksdb::ImmutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&,
rocksdb::InternalKeyComparator const&,
std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, bool, int)+0xf23)
[0x558d28041833]
15:
(rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::TableReaderOptions
const&, std::unique_ptr<rocksdb::RandomAccessFileReader,
std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool) const+0x4d)
[0x558d2803509d]
16: (rocksdb::TableCache::GetTableReader(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
bool, unsigned long, bool, rocksdb::HistogramImpl*,
std::unique_ptr<rocksdb::TableReader,
std::default_delete<rocksdb::TableReader> >*, bool, int, bool)+0x53c)
[0x558d28121a6c]
17: (rocksdb::TableCache::FindTable(rocksdb::EnvOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
rocksdb::Cache::Handle**, bool, bool, rocksdb::HistogramImpl*, bool,
int, bool)+0x333) [0x558d28122253]
18: (std::_Function_handler<void (),
rocksdb::VersionBuilder::Rep::LoadTableHandlers(rocksdb::InternalStats*,
int, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&)+0x80)
[0x558d28127780]
19: (()+0xb9e6f) [0x7f91c4bffe6f]
20: (()+0x7494) [0x7f91c52ed494]
21: (clone()+0x3f) [0x7f91c4374acf]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-1/-1 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
Attachment:
pEpkey.asc
Description: application/pgp-keys
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
