Hello,
We use 3 node cluster with EC 8+2. Kraken 11.2.0
Cluster was installed with 11.1.1 and upgrade to 11.2.0
After a couple of days 1 osd is stop and fail to start.
this osd was recreated from scratch on ver11.2.0, but after some times it fail again.
2017-01-27 11:21:35.333547 7fef07711940 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2017-01-27 11:21:35.333562 7fef07711940 0 set uid:gid to 64045:64045 (ceph:ceph)
2017-01-27 11:21:35.333581 7fef07711940 0 ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7), process ceph-osd, pid 3701964
2017-01-27 11:21:35.333858 7fef07711940 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
2017-01-27 11:21:35.336332 7fef07711940 0 pidfile_write: ignore empty --pid-file
2017-01-27 11:21:35.339124 7fef07711940 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2017-01-27 11:21:35.352842 7fef07711940 0 load: jerasure load: lrc load: isa
2017-01-27 11:21:35.353651 7fef07711940 1 bluestore(/var/lib/ceph/osd/ceph-2) mount path /var/lib/ceph/osd/ceph-2
2017-01-27 11:21:35.353726 7fef07711940 1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
2017-01-27 11:21:35.354569 7fef07711940 1 bdev(/var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
2017-01-27 11:21:35.354861 7fef07711940 1 bdev(/var/lib/ceph/osd/ceph-2/block) open size 2000293007360 (0x1d1bac11000, 1862 GB) block_size 4096 (4096 B) rotational
2017-01-27 11:21:35.355413 7fef07711940 1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
2017-01-27 11:21:35.356189 7fef07711940 1 bdev(/var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
2017-01-27 11:21:35.356343 7fef07711940 1 bdev(/var/lib/ceph/osd/ceph-2/block) open size 2000293007360 (0x1d1bac11000, 1862 GB) block_size 4096 (4096 B) rotational
2017-01-27 11:21:35.356353 7fef07711940 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block size 1862 GB
2017-01-27 11:21:35.356425 7fef07711940 1 bluefs mount
2017-01-27 11:21:35.515297 7fef07711940 0 set rocksdb option compression = kNoCompression
2017-01-27 11:21:35.515325 7fef07711940 0 set rocksdb option max_write_buffer_number = 4
2017-01-27 11:21:35.515337 7fef07711940 0 set rocksdb option min_write_buffer_number_to_merge = 1
2017-01-27 11:21:35.515351 7fef07711940 0 set rocksdb option recycle_log_file_num = 4
2017-01-27 11:21:35.515361 7fef07711940 0 set rocksdb option write_buffer_size = 268435456
2017-01-27 11:21:35.515432 7fef07711940 0 set rocksdb option compression = kNoCompression
2017-01-27 11:21:35.515446 7fef07711940 0 set rocksdb option max_write_buffer_number = 4
2017-01-27 11:21:35.515457 7fef07711940 0 set rocksdb option min_write_buffer_number_to_merge = 1
2017-01-27 11:21:35.515466 7fef07711940 0 set rocksdb option recycle_log_file_num = 4
2017-01-27 11:21:35.515476 7fef07711940 0 set rocksdb option write_buffer_size = 268435456
2017-01-27 11:21:35.515747 7fef07711940 4 rocksdb: RocksDB version: 5.0.0
2017-01-27 11:21:35.515764 7fef07711940 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
2017-01-27 11:21:35.515767 7fef07711940 4 rocksdb: Compile date Jan 19 2017
2017-01-27 11:21:35.515772 7fef07711940 4 rocksdb: DB SUMMARY
2017-01-27 11:21:35.515862 7fef07711940 4 rocksdb: CURRENT file: CURRENT
2017-01-27 11:21:35.515870 7fef07711940 4 rocksdb: IDENTITY file: IDENTITY
2017-01-27 11:21:35.515875 7fef07711940 4 rocksdb: MANIFEST file: MANIFEST-000694 size: 21330 Bytes
2017-01-27 11:21:35.515903 7fef07711940 4 rocksdb: SST files in db dir, Total Num: 72, files: 000219.sst 000341.sst 000394.sst 000435.sst 000499.sst 000500.sst 000515.sst 000516.sst 000517.sst
2017-01-27 11:21:35.515910 7fef07711940 4 rocksdb: Write Ahead Log file in db: 000695.log size: 0 ;
2017-01-27 11:21:35.515917 7fef07711940 4 rocksdb: Options.error_if_exists: 0
2017-01-27 11:21:35.515920 7fef07711940 4 rocksdb: Options.create_if_missing: 0
2017-01-27 11:21:35.515923 7fef07711940 4 rocksdb: Options.paranoid_checks: 1
2017-01-27 11:21:35.515925 7fef07711940 4 rocksdb: Options.env: 0x56477b543020
2017-01-27 11:21:35.515928 7fef07711940 4 rocksdb: Options.info_log: 0x56477b544160
2017-01-27 11:21:35.515930 7fef07711940 4 rocksdb: Options.max_open_files: -1
2017-01-27 11:21:35.515932 7fef07711940 4 rocksdb: Options.max_file_opening_threads: 16
2017-01-27 11:21:35.515933 7fef07711940 4 rocksdb: Options.disableDataSync: 0
2017-01-27 11:21:35.515936 7fef07711940 4 rocksdb: Options.use_fsync: 0
2017-01-27 11:21:35.515939 7fef07711940 4 rocksdb: Options.max_log_file_size: 0
2017-01-27 11:21:35.515941 7fef07711940 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
2017-01-27 11:21:35.515944 7fef07711940 4 rocksdb: Options.log_file_time_to_roll: 0
2017-01-27 11:21:35.515946 7fef07711940 4 rocksdb: Options.keep_log_file_num: 1000
2017-01-27 11:21:35.515948 7fef07711940 4 rocksdb: Options.recycle_log_file_num: 4
2017-01-27 11:21:35.515951 7fef07711940 4 rocksdb: Options.allow_os_buffer: 1
2017-01-27 11:21:35.515954 7fef07711940 4 rocksdb: Options.allow_mmap_reads: 0
2017-01-27 11:21:35.515956 7fef07711940 4 rocksdb: Options.allow_fallocate: 1
2017-01-27 11:21:35.515958 7fef07711940 4 rocksdb: Options.allow_mmap_writes: 0
2017-01-27 11:21:35.515960 7fef07711940 4 rocksdb: Options.use_direct_reads: 0
2017-01-27 11:21:35.515962 7fef07711940 4 rocksdb: Options.create_missing_column_families: 0
2017-01-27 11:21:35.515964 7fef07711940 4 rocksdb: Options.db_log_dir:
2017-01-27 11:21:35.515967 7fef07711940 4 rocksdb: Options.wal_dir: db
2017-01-27 11:21:35.515969 7fef07711940 4 rocksdb: Options.table_cache_numshardbits: 6
2017-01-27 11:21:35.515970 7fef07711940 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
2017-01-27 11:21:35.515973 7fef07711940 4 rocksdb: Options.max_subcompactions: 1
2017-01-27 11:21:35.515974 7fef07711940 4 rocksdb: Options.max_background_flushes: 1
2017-01-27 11:21:35.515977 7fef07711940 4 rocksdb: Options.WAL_ttl_seconds: 0
2017-01-27 11:21:35.515979 7fef07711940 4 rocksdb: Options.WAL_size_limit_MB: 0
2017-01-27 11:21:35.515981 7fef07711940 4 rocksdb: Options.manifest_preallocation_size: 4194304
2017-01-27 11:21:35.515983 7fef07711940 4 rocksdb: Options.allow_os_buffer: 1
2017-01-27 11:21:35.515986 7fef07711940 4 rocksdb: Options.allow_mmap_reads: 0
2017-01-27 11:21:35.515988 7fef07711940 4 rocksdb: Options.allow_mmap_writes: 0
2017-01-27 11:21:35.515990 7fef07711940 4 rocksdb: Options.is_fd_close_on_exec: 1
2017-01-27 11:21:35.515992 7fef07711940 4 rocksdb: Options.stats_dump_period_sec: 600
2017-01-27 11:21:35.515995 7fef07711940 4 rocksdb: Options.advise_random_on_open: 1
2017-01-27 11:21:35.515997 7fef07711940 4 rocksdb: Options.db_write_buffer_size: 0
2017-01-27 11:21:35.515999 7fef07711940 4 rocksdb: Options.access_hint_on_compaction_start: 1
2017-01-27 11:21:35.516001 7fef07711940 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
2017-01-27 11:21:35.516003 7fef07711940 4 rocksdb: Options.compaction_readahead_size: 0
2017-01-27 11:21:35.516005 7fef07711940 4 rocksdb: Options.random_access_max_buffer_size: 1048576
2017-01-27 11:21:35.516007 7fef07711940 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
2017-01-27 11:21:35.516009 7fef07711940 4 rocksdb: Options.use_adaptive_mutex: 0
2017-01-27 11:21:35.516011 7fef07711940 4 rocksdb: Options.rate_limiter: (nil)
2017-01-27 11:21:35.516013 7fef07711940 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
2017-01-27 11:21:35.516016 7fef07711940 4 rocksdb: Options.bytes_per_sync: 0
2017-01-27 11:21:35.516018 7fef07711940 4 rocksdb: Options.wal_bytes_per_sync: 0
2017-01-27 11:21:35.516020 7fef07711940 4 rocksdb: Options.wal_recovery_mode: 0
2017-01-27 11:21:35.516022 7fef07711940 4 rocksdb: Options.enable_thread_tracking: 0
2017-01-27 11:21:35.516023 7fef07711940 4 rocksdb: Options.allow_concurrent_memtable_write: 1
2017-01-27 11:21:35.516025 7fef07711940 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
2017-01-27 11:21:35.516027 7fef07711940 4 rocksdb: Options.write_thread_max_yield_usec: 100
2017-01-27 11:21:35.516029 7fef07711940 4 rocksdb: Options.write_thread_slow_yield_usec: 3
2017-01-27 11:21:35.516032 7fef07711940 4 rocksdb: Options.row_cache: None
2017-01-27 11:21:35.516034 7fef07711940 4 rocksdb: Options.wal_filter: None
2017-01-27 11:21:35.516037 7fef07711940 4 rocksdb: Options.avoid_flush_during_recovery: 0
2017-01-27 11:21:35.516039 7fef07711940 4 rocksdb: Options.base_background_compactions: 1
2017-01-27 11:21:35.516041 7fef07711940 4 rocksdb: Options.max_background_compactions: 1
2017-01-27 11:21:35.516043 7fef07711940 4 rocksdb: Options.avoid_flush_during_shutdown: 0
2017-01-27 11:21:35.516045 7fef07711940 4 rocksdb: Options.delayed_write_rate : 2097152
2017-01-27 11:21:35.516047 7fef07711940 4 rocksdb: Options.max_total_wal_size: 0
2017-01-27 11:21:35.516050 7fef07711940 4 rocksdb: Compression algorithms supported:
2017-01-27 11:21:35.516052 7fef07711940 4 rocksdb: Snappy supported: 0
2017-01-27 11:21:35.516054 7fef07711940 4 rocksdb: Zlib supported: 0
2017-01-27 11:21:35.516056 7fef07711940 4 rocksdb: Bzip supported: 0
2017-01-27 11:21:35.516058 7fef07711940 4 rocksdb: LZ4 supported: 0
2017-01-27 11:21:35.516060 7fef07711940 4 rocksdb: ZSTD supported: 0
2017-01-27 11:21:35.516063 7fef07711940 4 rocksdb: Fast CRC32 supported: 0
2017-01-27 11:21:35.528293 7fef07711940 4 rocksdb: Recovering from manifest file: MANIFEST-000694
2017-01-27 11:21:35.528395 7fef07711940 4 rocksdb: --------------- Options for column family [default]:
2017-01-27 11:21:35.528404 7fef07711940 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
2017-01-27 11:21:35.528413 7fef07711940 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor
2017-01-27 11:21:35.528416 7fef07711940 4 rocksdb: Options.compaction_filter: None
2017-01-27 11:21:35.528417 7fef07711940 4 rocksdb: Options.compaction_filter_factory: None
2017-01-27 11:21:35.528420 7fef07711940 4 rocksdb: Options.memtable_factory: SkipListFactory
2017-01-27 11:21:35.528422 7fef07711940 4 rocksdb: Options.table_factory: BlockBasedTable
2017-01-27 11:21:35.528443 7fef07711940 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56477b2e0150)
cache_index_and_filter_blocks: 0
pin_l0_filter_and_index_blocks_in_cache: 0
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x56477b31a310
block_cache_size: 134217728
block_cache_compressed: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
filter_policy: nullptr
whole_key_filtering: 1
skip_table_builder_flush: 0
format_version: 2
2017-01-27 11:21:35.528452 7fef07711940 4 rocksdb: Options.write_buffer_size: 268435456
2017-01-27 11:21:35.528455 7fef07711940 4 rocksdb: Options.max_write_buffer_number: 4
2017-01-27 11:21:35.528457 7fef07711940 4 rocksdb: Options.compression: NoCompression
2017-01-27 11:21:35.528460 7fef07711940 4 rocksdb: Options.bottommost_compression: Disabled
2017-01-27 11:21:35.528464 7fef07711940 4 rocksdb: Options.prefix_extractor: nullptr
2017-01-27 11:21:35.528466 7fef07711940 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
2017-01-27 11:21:35.528468 7fef07711940 4 rocksdb: Options.num_levels: 7
2017-01-27 11:21:35.528471 7fef07711940 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
2017-01-27 11:21:35.528473 7fef07711940 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2017-01-27 11:21:35.528475 7fef07711940 4 rocksdb: Options.compression_opts.window_bits: -14
2017-01-27 11:21:35.528477 7fef07711940 4 rocksdb: Options.compression_opts.level: -1
2017-01-27 11:21:35.528479 7fef07711940 4 rocksdb: Options.compression_opts.strategy: 0
2017-01-27 11:21:35.528483 7fef07711940 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
2017-01-27 11:21:35.528485 7fef07711940 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2017-01-27 11:21:35.528487 7fef07711940 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2017-01-27 11:21:35.528489 7fef07711940 4 rocksdb: Options.level0_stop_writes_trigger: 24
2017-01-27 11:21:35.528492 7fef07711940 4 rocksdb: Options.target_file_size_base: 67108864
2017-01-27 11:21:35.528495 7fef07711940 4 rocksdb: Options.target_file_size_multiplier: 1
2017-01-27 11:21:35.528497 7fef07711940 4 rocksdb: Options.max_bytes_for_level_base: 268435456
2017-01-27 11:21:35.528499 7fef07711940 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2017-01-27 11:21:35.528501 7fef07711940 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
2017-01-27 11:21:35.528507 7fef07711940 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2017-01-27 11:21:35.528511 7fef07711940 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2017-01-27 11:21:35.528514 7fef07711940 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2017-01-27 11:21:35.528516 7fef07711940 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2017-01-27 11:21:35.528519 7fef07711940 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2017-01-27 11:21:35.528522 7fef07711940 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2017-01-27 11:21:35.528524 7fef07711940 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2017-01-27 11:21:35.528526 7fef07711940 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2017-01-27 11:21:35.528529 7fef07711940 4 rocksdb: Options.max_compaction_bytes: 1677721600
2017-01-27 11:21:35.528531 7fef07711940 4 rocksdb: Options.arena_block_size: 33554432
2017-01-27 11:21:35.528533 7fef07711940 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
2017-01-27 11:21:35.528536 7fef07711940 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
2017-01-27 11:21:35.528539 7fef07711940 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 1000
2017-01-27 11:21:35.528541 7fef07711940 4 rocksdb: Options.disable_auto_compactions: 0
2017-01-27 11:21:35.528543 7fef07711940 4 rocksdb: Options.verify_checksums_in_compaction: 1
2017-01-27 11:21:35.528545 7fef07711940 4 rocksdb: Options.compaction_style: 0
2017-01-27 11:21:35.528547 7fef07711940 4 rocksdb: Options.compaction_pri: 0
2017-01-27 11:21:35.528549 7fef07711940 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2017-01-27 11:21:35.528551 7fef07711940 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2017-01-27 11:21:35.528553 7fef07711940 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2017-01-27 11:21:35.528555 7fef07711940 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2017-01-27 11:21:35.528557 7fef07711940 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2017-01-27 11:21:35.528559 7fef07711940 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2017-01-27 11:21:35.528561 7fef07711940 4 rocksdb: Options.table_properties_collectors:
2017-01-27 11:21:35.528563 7fef07711940 4 rocksdb: Options.inplace_update_support: 0
2017-01-27 11:21:35.528566 7fef07711940 4 rocksdb: Options.inplace_update_num_locks: 10000
2017-01-27 11:21:35.528568 7fef07711940 4 rocksdb: Options.min_partial_merge_operands: 2
2017-01-27 11:21:35.528570 7fef07711940 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
2017-01-27 11:21:35.528574 7fef07711940 4 rocksdb: Options.memtable_huge_page_size: 0
2017-01-27 11:21:35.528575 7fef07711940 4 rocksdb: Options.bloom_locality: 0
2017-01-27 11:21:35.528577 7fef07711940 4 rocksdb: Options.max_successive_merges: 0
2017-01-27 11:21:35.528580 7fef07711940 4 rocksdb: Options.optimize_filters_for_hits: 0
2017-01-27 11:21:35.528582 7fef07711940 4 rocksdb: Options.paranoid_file_checks: 0
2017-01-27 11:21:35.528607 7fef07711940 4 rocksdb: Options.force_consistency_checks: 0
2017-01-27 11:21:35.528610 7fef07711940 4 rocksdb: Options.report_bg_io_stats: 0
2017-01-27 11:21:36.518000 7fef07711940 4 rocksdb: Recovered from manifest file:db/MANIFEST-000694 succeeded,manifest_file_number is 694, next_file_number is 696, last_sequence is 22819978, log_number is 0,prev_log_number is 0,max_column_family is 0
2017-01-27 11:21:36.518022 7fef07711940 4 rocksdb: Column family [default] (ID 0), log number is 693
2017-01-27 11:21:36.518480 7fef07711940 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1485476496518468, "job": 1, "event": "recovery_started", "log_files": [695]}
2017-01-27 11:21:36.518490 7fef07711940 4 rocksdb: Recovering log #695 mode 0
2017-01-27 11:21:36.518547 7fef07711940 4 rocksdb: Creating manifest 697
2017-01-27 11:21:36.564473 7fef07711940 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1485476496564463, "job": 1, "event": "recovery_finished"}
2017-01-27 11:21:36.598704 7fef07711940 4 rocksdb: DB pointer 0x56477b59a000
2017-01-27 11:21:36.598757 7fef07711940 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456
2017-01-27 11:21:36.613528 7fef07711940 1 freelist init
2017-01-27 11:21:51.652395 7fef07711940 0 <cls> /build/ceph-11.2.0/src/cls/cephfs/cls_cephfs.cc:198: loading cephfs
2017-01-27 11:21:51.656337 7fef07711940 0 _get_class not permitted to load kvs
2017-01-27 11:21:51.657169 7fef07711940 0 <cls> /build/ceph-11.2.0/src/cls/hello/cls_hello.cc:296: loading cls_hello
2017-01-27 11:21:51.657399 7fef07711940 0 _get_class not permitted to load lua
2017-01-27 11:21:51.678678 7fef07711940 0 osd.2 1214 crush map has features 2268850290688, adjusting msgr requires for clients
2017-01-27 11:21:51.678714 7fef07711940 0 osd.2 1214 crush map has features 2543728197632 was 8705, adjusting msgr requires for mons
2017-01-27 11:21:51.678725 7fef07711940 0 osd.2 1214 crush map has features 2543728197632, adjusting msgr requires for osds
2017-01-27 11:21:51.698985 7feeec44e700 1 bluefs reclaim_blocks bdev 1 want 0x80000000
2017-01-27 11:21:51.702959 7feeec44e700 -1 /build/ceph-11.2.0/src/os/bluestore/BitAllocator.cc: In function 'virtual void BitMapAreaIN::unreserve(int64_t, int64_t)' thread 7feeec44e700 time 2017-01-27 11:21:51.699301
/build/ceph-11.2.0/src/os/bluestore/BitAllocator.cc: 874: FAILED assert(m_reserved_blocks >= 0)
ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x82) [0x564771190152]
2: (BitMapAreaIN::unreserve(long, long)+0x98) [0x564770fa4348]
3: (BitMapAllocator::unreserve(unsigned long)+0x85) [0x564770fa3185]
4: (BlueFS::reclaim_blocks(unsigned int, unsigned long, std::vector<AllocExtent, mempool::pool_allocator<(mempool::pool_index_t)7, AllocExtent> >*)+0x652) [0x564770f6ca42]
5: (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, std::allocator<bluestore_pextent_t> >*)+0x360) [0x564770eac010]
6: (BlueStore::_kv_sync_thread()+0x1152) [0x564770eae602]
7: (BlueStore::KVSyncThread::entry()+0xd) [0x564770ed858d]
8: (()+0x8064) [0x7fef06333064]
9: (clone()+0x6d) [0x7fef041db62d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-255> 2017-01-27 11:21:35.324389 7fef07711940 5 asok(0x56477b404000) register_command perfcounters_dump hook 0x56477b2e4030
-254> 2017-01-27 11:21:35.324418 7fef07711940 5 asok(0x56477b404000) register_command 1 hook 0x56477b2e4030
-253> 2017-01-27 11:21:35.324428 7fef07711940 5 asok(0x56477b404000) register_command perf dump hook 0x56477b2e4030
-252> 2017-01-27 11:21:35.324433 7fef07711940 5 asok(0x56477b404000) register_command perfcounters_schema hook 0x56477b2e4030
-251> 2017-01-27 11:21:35.324440 7fef07711940 5 asok(0x56477b404000) register_command 2 hook 0x56477b2e4030
-250> 2017-01-27 11:21:35.324444 7fef07711940 5 asok(0x56477b404000) register_command perf schema hook 0x56477b2e4030
-249> 2017-01-27 11:21:35.324450 7fef07711940 5 asok(0x56477b404000) register_command perf reset hook 0x56477b2e4030
-248> 2017-01-27 11:21:35.324458 7fef07711940 5 asok(0x56477b404000) register_command config show hook 0x56477b2e4030
-247> 2017-01-27 11:21:35.324462 7fef07711940 5 asok(0x56477b404000) register_command config set hook 0x56477b2e4030
-246> 2017-01-27 11:21:35.324469 7fef07711940 5 asok(0x56477b404000) register_command config get hook 0x56477b2e4030
-245> 2017-01-27 11:21:35.324474 7fef07711940 5 asok(0x56477b404000) register_command config diff hook 0x56477b2e4030
-244> 2017-01-27 11:21:35.324482 7fef07711940 5 asok(0x56477b404000) register_command log flush hook 0x56477b2e4030
-243> 2017-01-27 11:21:35.324487 7fef07711940 5 asok(0x56477b404000) register_command log dump hook 0x56477b2e4030
-242> 2017-01-27 11:21:35.324495 7fef07711940 5 asok(0x56477b404000) register_command log reopen hook 0x56477b2e4030
-241> 2017-01-27 11:21:35.324510 7fef07711940 5 asok(0x56477b404000) register_command dump_mempools hook 0x56477b3f8c68
-240> 2017-01-27 11:21:35.333331 7fef07711940 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-239> 2017-01-27 11:21:35.333547 7fef07711940 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-238> 2017-01-27 11:21:35.333562 7fef07711940 0 set uid:gid to 64045:64045 (ceph:ceph)
-237> 2017-01-27 11:21:35.333581 7fef07711940 0 ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7), process ceph-osd, pid 3701964
-236> 2017-01-27 11:21:35.333822 7fef07711940 5 object store type is bluestore
-235> 2017-01-27 11:21:35.333858 7fef07711940 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
-234> 2017-01-27 11:21:35.335318 7fef0224a700 2 Event(0x56477b31ca80 nevent=5000 time_id=1).set_owner idx=0 owner=140664509867776
-233> 2017-01-27 11:21:35.335387 7fef01a49700 2 Event(0x56477b31ce80 nevent=5000 time_id=1).set_owner idx=1 owner=140664501475072
-232> 2017-01-27 11:21:35.335414 7fef01248700 2 Event(0x56477b31d280 nevent=5000 time_id=1).set_owner idx=2 owner=140664493082368
-231> 2017-01-27 11:21:35.335811 7fef07711940 2 Event(0x56477b31ca80 nevent=5000 time_id=1).wakeup
-230> 2017-01-27 11:21:35.335924 7fef07711940 1 -- 0.0.0.0:6800/3701964 _finish_bind bind my_inst.addr is 0.0.0.0:6800/3701964
-229> 2017-01-27 11:21:35.335941 7fef07711940 2 Event(0x56477b31ca80 nevent=5000 time_id=1).wakeup
-228> 2017-01-27 11:21:35.335990 7fef07711940 2 Event(0x56477b31ca80 nevent=5000 time_id=1).wakeup
-227> 2017-01-27 11:21:35.336027 7fef07711940 1 -- 0.0.0.0:6801/3701964 _finish_bind bind my_inst.addr is 0.0.0.0:6801/3701964
-226> 2017-01-27 11:21:35.336038 7fef07711940 2 Event(0x56477b31ca80 nevent=5000 time_id=1).wakeup
-225> 2017-01-27 11:21:35.336075 7fef07711940 2 Event(0x56477b31ca80 nevent=5000 time_id=1).wakeup
-224> 2017-01-27 11:21:35.336115 7fef07711940 2 Event(0x56477b31ca80 nevent=5000 time_id=1).wakeup
-223> 2017-01-27 11:21:35.336154 7fef07711940 1 -- 0.0.0.0:6802/3701964 _finish_bind bind my_inst.addr is 0.0.0.0:6802/3701964
-222> 2017-01-27 11:21:35.336165 7fef07711940 2 Event(0x56477b31ca80 nevent=5000 time_id=1).wakeup
-221> 2017-01-27 11:21:35.336201 7fef07711940 2 Event(0x56477b31ca80 nevent=5000 time_id=1).wakeup
-220> 2017-01-27 11:21:35.336240 7fef07711940 2 Event(0x56477b31ca80 nevent=5000 time_id=1).wakeup
-219> 2017-01-27 11:21:35.336280 7fef07711940 2 Event(0x56477b31ca80 nevent=5000 time_id=1).wakeup
-218> 2017-01-27 11:21:35.336321 7fef07711940 1 -- 0.0.0.0:6803/3701964 _finish_bind bind my_inst.addr is 0.0.0.0:6803/3701964
-217> 2017-01-27 11:21:35.336332 7fef07711940 0 pidfile_write: ignore empty --pid-file
-216> 2017-01-27 11:21:35.339124 7fef07711940 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
-215> 2017-01-27 11:21:35.339140 7fef07711940 1 get compressor snappy = 0
-214> 2017-01-27 11:21:35.339144 7fef07711940 1 load compressor snappy
-213> 2017-01-27 11:21:35.343741 7fef07711940 1 add compressor snappy 0x56477b53a930
-212> 2017-01-27 11:21:35.343761 7fef07711940 1 get compressor snappy = 0x56477b53a930
-211> 2017-01-27 11:21:35.343767 7fef07711940 1 load: compressor snappy loaded and registered
-210> 2017-01-27 11:21:35.343770 7fef07711940 1 get compressor snappy = 0x56477b53a930
-209> 2017-01-27 11:21:35.343786 7fef07711940 5 asok(0x56477b404000) init /var/run/ceph/ceph-osd.2.asok
-208> 2017-01-27 11:21:35.343806 7fef07711940 5 asok(0x56477b404000) bind_and_listen /var/run/ceph/ceph-osd.2.asok
-207> 2017-01-27 11:21:35.343987 7fef07711940 5 asok(0x56477b404000) register_command 0 hook 0x56477b2e0120
-206> 2017-01-27 11:21:35.344000 7fef07711940 5 asok(0x56477b404000) register_command version hook 0x56477b2e0120
-205> 2017-01-27 11:21:35.344009 7fef07711940 5 asok(0x56477b404000) register_command git_version hook 0x56477b2e0120
-204> 2017-01-27 11:21:35.344017 7fef07711940 5 asok(0x56477b404000) register_command help hook 0x56477b2e4490
-203> 2017-01-27 11:21:35.344025 7fef07711940 5 asok(0x56477b404000) register_command get_command_descriptions hook 0x56477b2e44a0
-202> 2017-01-27 11:21:35.344114 7feef742c700 5 asok(0x56477b404000) entry start
-201> 2017-01-27 11:21:35.344175 7fef07711940 10 monclient(hunting): build_initial_monmap
-200> 2017-01-27 11:21:35.352842 7fef07711940 0 load: jerasure load: lrc load: isa
-199> 2017-01-27 11:21:35.352896 7fef07711940 5 adding auth protocol: cephx
-198> 2017-01-27 11:21:35.352905 7fef07711940 5 adding auth protocol: cephx
-197> 2017-01-27 11:21:35.353354 7fef07711940 5 asok(0x56477b404000) register_command objecter_requests hook 0x56477b2e4530
-196> 2017-01-27 11:21:35.353431 7fef07711940 1 -- 0.0.0.0:6800/3701964 start start
-195> 2017-01-27 11:21:35.353446 7fef07711940 1 -- - start start
-194> 2017-01-27 11:21:35.353450 7fef07711940 1 -- 0.0.0.0:6803/3701964 start start
-193> 2017-01-27 11:21:35.353458 7fef07711940 1 -- 0.0.0.0:6802/3701964 start start
-192> 2017-01-27 11:21:35.353461 7fef07711940 1 -- 0.0.0.0:6801/3701964 start start
-191> 2017-01-27 11:21:35.353470 7fef07711940 1 -- - start start
-190> 2017-01-27 11:21:35.353627 7fef07711940 2 osd.2 0 mounting /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal
-189> 2017-01-27 11:21:35.353651 7fef07711940 1 bluestore(/var/lib/ceph/osd/ceph-2) mount path /var/lib/ceph/osd/ceph-2
-188> 2017-01-27 11:21:35.353726 7fef07711940 1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
-187> 2017-01-27 11:21:35.354569 7fef07711940 1 bdev(/var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
-186> 2017-01-27 11:21:35.354861 7fef07711940 1 bdev(/var/lib/ceph/osd/ceph-2/block) open size 2000293007360 (0x1d1bac11000, 1862 GB) block_size 4096 (4096 B) rotational
-185> 2017-01-27 11:21:35.355413 7fef07711940 1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
-184> 2017-01-27 11:21:35.356189 7fef07711940 1 bdev(/var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
-183> 2017-01-27 11:21:35.356343 7fef07711940 1 bdev(/var/lib/ceph/osd/ceph-2/block) open size 2000293007360 (0x1d1bac11000, 1862 GB) block_size 4096 (4096 B) rotational
-182> 2017-01-27 11:21:35.356353 7fef07711940 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block size 1862 GB
-181> 2017-01-27 11:21:35.356425 7fef07711940 1 bluefs mount
-180> 2017-01-27 11:21:35.515297 7fef07711940 0 set rocksdb option compression = kNoCompression
-179> 2017-01-27 11:21:35.515325 7fef07711940 0 set rocksdb option max_write_buffer_number = 4
-178> 2017-01-27 11:21:35.515337 7fef07711940 0 set rocksdb option min_write_buffer_number_to_merge = 1
-177> 2017-01-27 11:21:35.515351 7fef07711940 0 set rocksdb option recycle_log_file_num = 4
-176> 2017-01-27 11:21:35.515361 7fef07711940 0 set rocksdb option write_buffer_size = 268435456
-175> 2017-01-27 11:21:35.515432 7fef07711940 0 set rocksdb option compression = kNoCompression
-174> 2017-01-27 11:21:35.515446 7fef07711940 0 set rocksdb option max_write_buffer_number = 4
-173> 2017-01-27 11:21:35.515457 7fef07711940 0 set rocksdb option min_write_buffer_number_to_merge = 1
-172> 2017-01-27 11:21:35.515466 7fef07711940 0 set rocksdb option recycle_log_file_num = 4
-171> 2017-01-27 11:21:35.515476 7fef07711940 0 set rocksdb option write_buffer_size = 268435456
-170> 2017-01-27 11:21:35.515747 7fef07711940 4 rocksdb: RocksDB version: 5.0.0
-169> 2017-01-27 11:21:35.515764 7fef07711940 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
-168> 2017-01-27 11:21:35.515767 7fef07711940 4 rocksdb: Compile date Jan 19 2017
-167> 2017-01-27 11:21:35.515772 7fef07711940 4 rocksdb: DB SUMMARY
-166> 2017-01-27 11:21:35.515862 7fef07711940 4 rocksdb: CURRENT file: CURRENT
-165> 2017-01-27 11:21:35.515870 7fef07711940 4 rocksdb: IDENTITY file: IDENTITY
-164> 2017-01-27 11:21:35.515875 7fef07711940 4 rocksdb: MANIFEST file: MANIFEST-000694 size: 21330 Bytes
-163> 2017-01-27 11:21:35.515903 7fef07711940 4 rocksdb: SST files in db dir, Total Num: 72, files: 000219.sst 000341.sst 000394.sst 000435.sst 000499.sst 000500.sst 000515.sst 000516.sst 000517.sst
-162> 2017-01-27 11:21:35.515910 7fef07711940 4 rocksdb: Write Ahead Log file in db: 000695.log size: 0 ;
-161> 2017-01-27 11:21:35.515917 7fef07711940 4 rocksdb: Options.error_if_exists: 0
-160> 2017-01-27 11:21:35.515920 7fef07711940 4 rocksdb: Options.create_if_missing: 0
-159> 2017-01-27 11:21:35.515923 7fef07711940 4 rocksdb: Options.paranoid_checks: 1
-158> 2017-01-27 11:21:35.515925 7fef07711940 4 rocksdb: Options.env: 0x56477b543020
-157> 2017-01-27 11:21:35.515928 7fef07711940 4 rocksdb: Options.info_log: 0x56477b544160
-156> 2017-01-27 11:21:35.515930 7fef07711940 4 rocksdb: Options.max_open_files: -1
-155> 2017-01-27 11:21:35.515932 7fef07711940 4 rocksdb: Options.max_file_opening_threads: 16
-154> 2017-01-27 11:21:35.515933 7fef07711940 4 rocksdb: Options.disableDataSync: 0
-153> 2017-01-27 11:21:35.515936 7fef07711940 4 rocksdb: Options.use_fsync: 0
-152> 2017-01-27 11:21:35.515939 7fef07711940 4 rocksdb: Options.max_log_file_size: 0
-151> 2017-01-27 11:21:35.515941 7fef07711940 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
-150> 2017-01-27 11:21:35.515944 7fef07711940 4 rocksdb: Options.log_file_time_to_roll: 0
-149> 2017-01-27 11:21:35.515946 7fef07711940 4 rocksdb: Options.keep_log_file_num: 1000
-148> 2017-01-27 11:21:35.515948 7fef07711940 4 rocksdb: Options.recycle_log_file_num: 4
-147> 2017-01-27 11:21:35.515951 7fef07711940 4 rocksdb: Options.allow_os_buffer: 1
-146> 2017-01-27 11:21:35.515954 7fef07711940 4 rocksdb: Options.allow_mmap_reads: 0
-145> 2017-01-27 11:21:35.515956 7fef07711940 4 rocksdb: Options.allow_fallocate: 1
-144> 2017-01-27 11:21:35.515958 7fef07711940 4 rocksdb: Options.allow_mmap_writes: 0
-143> 2017-01-27 11:21:35.515960 7fef07711940 4 rocksdb: Options.use_direct_reads: 0
-142> 2017-01-27 11:21:35.515962 7fef07711940 4 rocksdb: Options.create_missing_column_families: 0
-141> 2017-01-27 11:21:35.515964 7fef07711940 4 rocksdb: Options.db_log_dir:
-140> 2017-01-27 11:21:35.515967 7fef07711940 4 rocksdb: Options.wal_dir: db
-139> 2017-01-27 11:21:35.515969 7fef07711940 4 rocksdb: Options.table_cache_numshardbits: 6
-138> 2017-01-27 11:21:35.515970 7fef07711940 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
-137> 2017-01-27 11:21:35.515973 7fef07711940 4 rocksdb: Options.max_subcompactions: 1
-136> 2017-01-27 11:21:35.515974 7fef07711940 4 rocksdb: Options.max_background_flushes: 1
-135> 2017-01-27 11:21:35.515977 7fef07711940 4 rocksdb: Options.WAL_ttl_seconds: 0
-134> 2017-01-27 11:21:35.515979 7fef07711940 4 rocksdb: Options.WAL_size_limit_MB: 0
-133> 2017-01-27 11:21:35.515981 7fef07711940 4 rocksdb: Options.manifest_preallocation_size: 4194304
-132> 2017-01-27 11:21:35.515983 7fef07711940 4 rocksdb: Options.allow_os_buffer: 1
-131> 2017-01-27 11:21:35.515986 7fef07711940 4 rocksdb: Options.allow_mmap_reads: 0
-130> 2017-01-27 11:21:35.515988 7fef07711940 4 rocksdb: Options.allow_mmap_writes: 0
-129> 2017-01-27 11:21:35.515990 7fef07711940 4 rocksdb: Options.is_fd_close_on_exec: 1
-128> 2017-01-27 11:21:35.515992 7fef07711940 4 rocksdb: Options.stats_dump_period_sec: 600
-127> 2017-01-27 11:21:35.515995 7fef07711940 4 rocksdb: Options.advise_random_on_open: 1
-126> 2017-01-27 11:21:35.515997 7fef07711940 4 rocksdb: Options.db_write_buffer_size: 0
-125> 2017-01-27 11:21:35.515999 7fef07711940 4 rocksdb: Options.access_hint_on_compaction_start: 1
-124> 2017-01-27 11:21:35.516001 7fef07711940 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
-123> 2017-01-27 11:21:35.516003 7fef07711940 4 rocksdb: Options.compaction_readahead_size: 0
-122> 2017-01-27 11:21:35.516005 7fef07711940 4 rocksdb: Options.random_access_max_buffer_size: 1048576
-121> 2017-01-27 11:21:35.516007 7fef07711940 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
-120> 2017-01-27 11:21:35.516009 7fef07711940 4 rocksdb: Options.use_adaptive_mutex: 0
-119> 2017-01-27 11:21:35.516011 7fef07711940 4 rocksdb: Options.rate_limiter: (nil)
-118> 2017-01-27 11:21:35.516013 7fef07711940 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
-117> 2017-01-27 11:21:35.516016 7fef07711940 4 rocksdb: Options.bytes_per_sync: 0
-116> 2017-01-27 11:21:35.516018 7fef07711940 4 rocksdb: Options.wal_bytes_per_sync: 0
-115> 2017-01-27 11:21:35.516020 7fef07711940 4 rocksdb: Options.wal_recovery_mode: 0
-114> 2017-01-27 11:21:35.516022 7fef07711940 4 rocksdb: Options.enable_thread_tracking: 0
-113> 2017-01-27 11:21:35.516023 7fef07711940 4 rocksdb: Options.allow_concurrent_memtable_write: 1
-112> 2017-01-27 11:21:35.516025 7fef07711940 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
-111> 2017-01-27 11:21:35.516027 7fef07711940 4 rocksdb: Options.write_thread_max_yield_usec: 100
-110> 2017-01-27 11:21:35.516029 7fef07711940 4 rocksdb: Options.write_thread_slow_yield_usec: 3
-109> 2017-01-27 11:21:35.516032 7fef07711940 4 rocksdb: Options.row_cache: None
-108> 2017-01-27 11:21:35.516034 7fef07711940 4 rocksdb: Options.wal_filter: None
-107> 2017-01-27 11:21:35.516037 7fef07711940 4 rocksdb: Options.avoid_flush_during_recovery: 0
-106> 2017-01-27 11:21:35.516039 7fef07711940 4 rocksdb: Options.base_background_compactions: 1
-105> 2017-01-27 11:21:35.516041 7fef07711940 4 rocksdb: Options.max_background_compactions: 1
-104> 2017-01-27 11:21:35.516043 7fef07711940 4 rocksdb: Options.avoid_flush_during_shutdown: 0
-103> 2017-01-27 11:21:35.516045 7fef07711940 4 rocksdb: Options.delayed_write_rate : 2097152
-102> 2017-01-27 11:21:35.516047 7fef07711940 4 rocksdb: Options.max_total_wal_size: 0
-101> 2017-01-27 11:21:35.516050 7fef07711940 4 rocksdb: Compression algorithms supported:
-100> 2017-01-27 11:21:35.516052 7fef07711940 4 rocksdb: Snappy supported: 0
-99> 2017-01-27 11:21:35.516054 7fef07711940 4 rocksdb: Zlib supported: 0
-98> 2017-01-27 11:21:35.516056 7fef07711940 4 rocksdb: Bzip supported: 0
-97> 2017-01-27 11:21:35.516058 7fef07711940 4 rocksdb: LZ4 supported: 0
-96> 2017-01-27 11:21:35.516060 7fef07711940 4 rocksdb: ZSTD supported: 0
-95> 2017-01-27 11:21:35.516063 7fef07711940 4 rocksdb: Fast CRC32 supported: 0
-94> 2017-01-27 11:21:35.528293 7fef07711940 4 rocksdb: Recovering from manifest file: MANIFEST-000694
-93> 2017-01-27 11:21:35.528395 7fef07711940 4 rocksdb: --------------- Options for column family [default]:
-92> 2017-01-27 11:21:35.528404 7fef07711940 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
-91> 2017-01-27 11:21:35.528413 7fef07711940 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor
-90> 2017-01-27 11:21:35.528416 7fef07711940 4 rocksdb: Options.compaction_filter: None
-89> 2017-01-27 11:21:35.528417 7fef07711940 4 rocksdb: Options.compaction_filter_factory: None
-88> 2017-01-27 11:21:35.528420 7fef07711940 4 rocksdb: Options.memtable_factory: SkipListFactory
-87> 2017-01-27 11:21:35.528422 7fef07711940 4 rocksdb: Options.table_factory: BlockBasedTable
-86> 2017-01-27 11:21:35.528443 7fef07711940 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56477b2e0150)
cache_index_and_filter_blocks: 0
pin_l0_filter_and_index_blocks_in_cache: 0
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x56477b31a310
block_cache_size: 134217728
block_cache_compressed: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
filter_policy: nullptr
whole_key_filtering: 1
skip_table_builder_flush: 0
format_version: 2
-85> 2017-01-27 11:21:35.528452 7fef07711940 4 rocksdb: Options.write_buffer_size: 268435456
-84> 2017-01-27 11:21:35.528455 7fef07711940 4 rocksdb: Options.max_write_buffer_number: 4
-83> 2017-01-27 11:21:35.528457 7fef07711940 4 rocksdb: Options.compression: NoCompression
-82> 2017-01-27 11:21:35.528460 7fef07711940 4 rocksdb: Options.bottommost_compression: Disabled
-81> 2017-01-27 11:21:35.528464 7fef07711940 4 rocksdb: Options.prefix_extractor: nullptr
-80> 2017-01-27 11:21:35.528466 7fef07711940 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
-79> 2017-01-27 11:21:35.528468 7fef07711940 4 rocksdb: Options.num_levels: 7
-78> 2017-01-27 11:21:35.528471 7fef07711940 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
-77> 2017-01-27 11:21:35.528473 7fef07711940 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
-76> 2017-01-27 11:21:35.528475 7fef07711940 4 rocksdb: Options.compression_opts.window_bits: -14
-75> 2017-01-27 11:21:35.528477 7fef07711940 4 rocksdb: Options.compression_opts.level: -1
-74> 2017-01-27 11:21:35.528479 7fef07711940 4 rocksdb: Options.compression_opts.strategy: 0
-73> 2017-01-27 11:21:35.528483 7fef07711940 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
-72> 2017-01-27 11:21:35.528485 7fef07711940 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
-71> 2017-01-27 11:21:35.528487 7fef07711940 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
-70> 2017-01-27 11:21:35.528489 7fef07711940 4 rocksdb: Options.level0_stop_writes_trigger: 24
-69> 2017-01-27 11:21:35.528492 7fef07711940 4 rocksdb: Options.target_file_size_base: 67108864
-68> 2017-01-27 11:21:35.528495 7fef07711940 4 rocksdb: Options.target_file_size_multiplier: 1
-67> 2017-01-27 11:21:35.528497 7fef07711940 4 rocksdb: Options.max_bytes_for_level_base: 268435456
-66> 2017-01-27 11:21:35.528499 7fef07711940 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
-65> 2017-01-27 11:21:35.528501 7fef07711940 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
-64> 2017-01-27 11:21:35.528507 7fef07711940 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
-63> 2017-01-27 11:21:35.528511 7fef07711940 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
-62> 2017-01-27 11:21:35.528514 7fef07711940 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
-61> 2017-01-27 11:21:35.528516 7fef07711940 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
-60> 2017-01-27 11:21:35.528519 7fef07711940 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
-59> 2017-01-27 11:21:35.528522 7fef07711940 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
-58> 2017-01-27 11:21:35.528524 7fef07711940 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
-57> 2017-01-27 11:21:35.528526 7fef07711940 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
-56> 2017-01-27 11:21:35.528529 7fef07711940 4 rocksdb: Options.max_compaction_bytes: 1677721600
-55> 2017-01-27 11:21:35.528531 7fef07711940 4 rocksdb: Options.arena_block_size: 33554432
-54> 2017-01-27 11:21:35.528533 7fef07711940 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
-53> 2017-01-27 11:21:35.528536 7fef07711940 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
-52> 2017-01-27 11:21:35.528539 7fef07711940 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 1000
-51> 2017-01-27 11:21:35.528541 7fef07711940 4 rocksdb: Options.disable_auto_compactions: 0
-50> 2017-01-27 11:21:35.528543 7fef07711940 4 rocksdb: Options.verify_checksums_in_compaction: 1
-49> 2017-01-27 11:21:35.528545 7fef07711940 4 rocksdb: Options.compaction_style: 0
-48> 2017-01-27 11:21:35.528547 7fef07711940 4 rocksdb: Options.compaction_pri: 0
-47> 2017-01-27 11:21:35.528549 7fef07711940 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
-46> 2017-01-27 11:21:35.528551 7fef07711940 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
-45> 2017-01-27 11:21:35.528553 7fef07711940 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
-44> 2017-01-27 11:21:35.528555 7fef07711940 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
-43> 2017-01-27 11:21:35.528557 7fef07711940 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
-42> 2017-01-27 11:21:35.528559 7fef07711940 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
-41> 2017-01-27 11:21:35.528561 7fef07711940 4 rocksdb: Options.table_properties_collectors:
-40> 2017-01-27 11:21:35.528563 7fef07711940 4 rocksdb: Options.inplace_update_support: 0
-39> 2017-01-27 11:21:35.528566 7fef07711940 4 rocksdb: Options.inplace_update_num_locks: 10000
-38> 2017-01-27 11:21:35.528568 7fef07711940 4 rocksdb: Options.min_partial_merge_operands: 2
-37> 2017-01-27 11:21:35.528570 7fef07711940 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
-36> 2017-01-27 11:21:35.528574 7fef07711940 4 rocksdb: Options.memtable_huge_page_size: 0
-35> 2017-01-27 11:21:35.528575 7fef07711940 4 rocksdb: Options.bloom_locality: 0
-34> 2017-01-27 11:21:35.528577 7fef07711940 4 rocksdb: Options.max_successive_merges: 0
-33> 2017-01-27 11:21:35.528580 7fef07711940 4 rocksdb: Options.optimize_filters_for_hits: 0
-32> 2017-01-27 11:21:35.528582 7fef07711940 4 rocksdb: Options.paranoid_file_checks: 0
-31> 2017-01-27 11:21:35.528607 7fef07711940 4 rocksdb: Options.force_consistency_checks: 0
-30> 2017-01-27 11:21:35.528610 7fef07711940 4 rocksdb: Options.report_bg_io_stats: 0
-29> 2017-01-27 11:21:36.518000 7fef07711940 4 rocksdb: Recovered from manifest file:db/MANIFEST-000694 succeeded,manifest_file_number is 694, next_file_number is 696, last_sequence is 22819978, log_number is 0,prev_log_number is 0,max_column_family is 0
-28> 2017-01-27 11:21:36.518022 7fef07711940 4 rocksdb: Column family [default] (ID 0), log number is 693
-27> 2017-01-27 11:21:36.518480 7fef07711940 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1485476496518468, "job": 1, "event": "recovery_started", "log_files": [695]}
-26> 2017-01-27 11:21:36.518490 7fef07711940 4 rocksdb: Recovering log #695 mode 0
-25> 2017-01-27 11:21:36.518547 7fef07711940 4 rocksdb: Creating manifest 697
-24> 2017-01-27 11:21:36.564473 7fef07711940 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1485476496564463, "job": 1, "event": "recovery_finished"}
-23> 2017-01-27 11:21:36.564723 7fef07711940 5 rocksdb: [JOB 2] Delete db//MANIFEST-000694 type=3 #694 -- OK
-22> 2017-01-27 11:21:36.564740 7fef07711940 5 rocksdb: [JOB 2] Delete db//000695.log type=0 #695 -- OK
-21> 2017-01-27 11:21:36.598704 7fef07711940 4 rocksdb: DB pointer 0x56477b59a000
-20> 2017-01-27 11:21:36.598757 7fef07711940 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456
-19> 2017-01-27 11:21:36.613528 7fef07711940 1 freelist init
-18> 2017-01-27 11:21:51.630939 7fef07711940 1 get compressor snappy = 0x56477b53a930
-17> 2017-01-27 11:21:51.630958 7fef07711940 2 osd.2 0 boot
-16> 2017-01-27 11:21:51.652190 7fef07711940 1 <cls> /build/ceph-11.2.0/src/cls/refcount/cls_refcount.cc:226: Loaded refcount class!
-15> 2017-01-27 11:21:51.652395 7fef07711940 0 <cls> /build/ceph-11.2.0/src/cls/cephfs/cls_cephfs.cc:198: loading cephfs
-14> 2017-01-27 11:21:51.655475 7fef07711940 1 <cls> /build/ceph-11.2.0/src/cls/rgw/cls_rgw.cc:3476: Loaded rgw class!
-13> 2017-01-27 11:21:51.655687 7fef07711940 1 <cls> /build/ceph-11.2.0/src/cls/statelog/cls_statelog.cc:300: Loaded log class!
-12> 2017-01-27 11:21:51.655878 7fef07711940 1 <cls> /build/ceph-11.2.0/src/cls/log/cls_log.cc:311: Loaded log class!
-11> 2017-01-27 11:21:51.656337 7fef07711940 0 _get_class not permitted to load kvs
-10> 2017-01-27 11:21:51.656495 7fef07711940 1 <cls> /build/ceph-11.2.0/src/cls/replica_log/cls_replica_log.cc:136: Loaded replica log class!
-9> 2017-01-27 11:21:51.656669 7fef07711940 1 <cls> /build/ceph-11.2.0/src/cls/timeindex/cls_timeindex.cc:254: Loaded timeindex class!
-8> 2017-01-27 11:21:51.656834 7fef07711940 1 <cls> /build/ceph-11.2.0/src/cls/user/cls_user.cc:368: Loaded user class!
-7> 2017-01-27 11:21:51.656990 7fef07711940 1 <cls> /build/ceph-11.2.0/src/cls/version/cls_version.cc:221: Loaded version class!
-6> 2017-01-27 11:21:51.657169 7fef07711940 0 <cls> /build/ceph-11.2.0/src/cls/hello/cls_hello.cc:296: loading cls_hello
-5> 2017-01-27 11:21:51.657399 7fef07711940 0 _get_class not permitted to load lua
-4> 2017-01-27 11:21:51.678678 7fef07711940 0 osd.2 1214 crush map has features 2268850290688, adjusting msgr requires for clients
-3> 2017-01-27 11:21:51.678714 7fef07711940 0 osd.2 1214 crush map has features 2543728197632 was 8705, adjusting msgr requires for mons
-2> 2017-01-27 11:21:51.678725 7fef07711940 0 osd.2 1214 crush map has features 2543728197632, adjusting msgr requires for osds
-1> 2017-01-27 11:21:51.698985 7feeec44e700 1 bluefs reclaim_blocks bdev 1 want 0x80000000
0> 2017-01-27 11:21:51.702959 7feeec44e700 -1 /build/ceph-11.2.0/src/os/bluestore/BitAllocator.cc: In function 'virtual void BitMapAreaIN::unreserve(int64_t, int64_t)' thread 7feeec44e700 time 2017-01-27 11:21:51.699301
/build/ceph-11.2.0/src/os/bluestore/BitAllocator.cc: 874: FAILED assert(m_reserved_blocks >= 0)
ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x82) [0x564771190152]
2: (BitMapAreaIN::unreserve(long, long)+0x98) [0x564770fa4348]
3: (BitMapAllocator::unreserve(unsigned long)+0x85) [0x564770fa3185]
4: (BlueFS::reclaim_blocks(unsigned int, unsigned long, std::vector<AllocExtent, mempool::pool_allocator<(mempool::pool_index_t)7, AllocExtent> >*)+0x652) [0x564770f6ca42]
5: (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, std::allocator<bluestore_pextent_t> >*)+0x360) [0x564770eac010]
6: (BlueStore::_kv_sync_thread()+0x1152) [0x564770eae602]
7: (BlueStore::KVSyncThread::entry()+0xd) [0x564770ed858d]
8: (()+0x8064) [0x7fef06333064]
9: (clone()+0x6d) [0x7fef041db62d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
2017-01-27 11:21:51.709311 7feeec44e700 -1 *** Caught signal (Aborted) **
in thread 7feeec44e700 thread_name:bstore_kv_sync
ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
1: (()+0x9419f7) [0x564770fae9f7]
2: (()+0xf890) [0x7fef0633a890]
3: (gsignal()+0x37) [0x7fef04128067]
4: (abort()+0x148) [0x7fef04129448]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x564771190326]
6: (BitMapAreaIN::unreserve(long, long)+0x98) [0x564770fa4348]
7: (BitMapAllocator::unreserve(unsigned long)+0x85) [0x564770fa3185]
8: (BlueFS::reclaim_blocks(unsigned int, unsigned long, std::vector<AllocExtent, mempool::pool_allocator<(mempool::pool_index_t)7, AllocExtent> >*)+0x652) [0x564770f6ca42]
9: (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, std::allocator<bluestore_pextent_t> >*)+0x360) [0x564770eac010]
10: (BlueStore::_kv_sync_thread()+0x1152) [0x564770eae602]
11: (BlueStore::KVSyncThread::entry()+0xd) [0x564770ed858d]
12: (()+0x8064) [0x7fef06333064]
13: (clone()+0x6d) [0x7fef041db62d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2017-01-27 11:21:51.709311 7feeec44e700 -1 *** Caught signal (Aborted) **
in thread 7feeec44e700 thread_name:bstore_kv_sync
ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
1: (()+0x9419f7) [0x564770fae9f7]
2: (()+0xf890) [0x7fef0633a890]
3: (gsignal()+0x37) [0x7fef04128067]
4: (abort()+0x148) [0x7fef04129448]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x564771190326]
6: (BitMapAreaIN::unreserve(long, long)+0x98) [0x564770fa4348]
7: (BitMapAllocator::unreserve(unsigned long)+0x85) [0x564770fa3185]
8: (BlueFS::reclaim_blocks(unsigned int, unsigned long, std::vector<AllocExtent, mempool::pool_allocator<(mempool::pool_index_t)7, AllocExtent> >*)+0x652) [0x564770f6ca42]
9: (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, std::allocator<bluestore_pextent_t> >*)+0x360) [0x564770eac010]
10: (BlueStore::_kv_sync_thread()+0x1152) [0x564770eae602]
11: (BlueStore::KVSyncThread::entry()+0xd) [0x564770ed858d]
12: (()+0x8064) [0x7fef06333064]
13: (clone()+0x6d) [0x7fef041db62d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
