On 8/17/17, 1:50 PM, "David Zafman" <dzafman@xxxxxxxxxx> wrote:
I wouldn't run the OSDs without identifying what is going on. I suggest
you stop the OSD and run fsck using ceph-objectstore-tool:
ceph-objectstore-tool --data-path .... --op fsck
If you don't get any informative messages add --debug to get detail logging.
David
Thanks, David for your input. It is interesting that all the 50 osds had the same fsck issue immediately after the upgrade.
Below is the log of manual fsck.
This is the error line in it:
fsck error: leaked some space;65536 bytes leaked
# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op fsck --debug
warning: line 36: 'bluestore_fsck_on_mount' in section 'global' redefined
2017-08-17 14:03:43.895377 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck (shallow) start
2017-08-17 14:03:43.895410 7fcb138f8280 1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
2017-08-17 14:03:43.895416 7fcb138f8280 1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
2017-08-17 14:03:43.895684 7fcb138f8280 1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
2017-08-17 14:03:43.895942 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes max 0.5 < ratio 0.99
2017-08-17 14:03:43.895957 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
2017-08-17 14:03:43.895990 7fcb138f8280 1 bdev create path /var/lib/ceph/osd/ceph-2/block.db type kernel
2017-08-17 14:03:43.895994 7fcb138f8280 1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open path /var/lib/ceph/osd/ceph-2/block.db
2017-08-17 14:03:43.896162 7fcb138f8280 1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) non-rotational
2017-08-17 14:03:43.896168 7fcb138f8280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block.db size 1024 MB
2017-08-17 14:03:43.896364 7fcb138f8280 1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
2017-08-17 14:03:43.896372 7fcb138f8280 1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
2017-08-17 14:03:43.896529 7fcb138f8280 1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
2017-08-17 14:03:43.896535 7fcb138f8280 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-2/block size 3725 GB
2017-08-17 14:03:43.896546 7fcb138f8280 1 bdev create path /var/lib/ceph/osd/ceph-2/block.wal type kernel
2017-08-17 14:03:43.896549 7fcb138f8280 1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open path /var/lib/ceph/osd/ceph-2/block.wal
2017-08-17 14:03:43.896733 7fcb138f8280 1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open size 10737418240 (0x280000000, 10240 MB) block_size 4096 (4096 B) non-rotational
2017-08-17 14:03:43.896739 7fcb138f8280 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-2/block.wal size 10240 MB
2017-08-17 14:03:43.896912 7fcb138f8280 1 bluefs mount
2017-08-17 14:03:43.902514 7fcb138f8280 0 set rocksdb option compaction_readahead_size = 2097152
2017-08-17 14:03:43.902525 7fcb138f8280 0 set rocksdb option compression = kNoCompression
2017-08-17 14:03:43.902533 7fcb138f8280 0 set rocksdb option max_write_buffer_number = 4
2017-08-17 14:03:43.902538 7fcb138f8280 0 set rocksdb option min_write_buffer_number_to_merge = 1
2017-08-17 14:03:43.902541 7fcb138f8280 0 set rocksdb option recycle_log_file_num = 4
2017-08-17 14:03:43.902544 7fcb138f8280 0 set rocksdb option writable_file_max_buffer_size = 0
2017-08-17 14:03:43.902548 7fcb138f8280 0 set rocksdb option write_buffer_size = 268435456
2017-08-17 14:03:43.902572 7fcb138f8280 0 set rocksdb option compaction_readahead_size = 2097152
2017-08-17 14:03:43.902577 7fcb138f8280 0 set rocksdb option compression = kNoCompression
2017-08-17 14:03:43.902580 7fcb138f8280 0 set rocksdb option max_write_buffer_number = 4
2017-08-17 14:03:43.902584 7fcb138f8280 0 set rocksdb option min_write_buffer_number_to_merge = 1
2017-08-17 14:03:43.902587 7fcb138f8280 0 set rocksdb option recycle_log_file_num = 4
2017-08-17 14:03:43.902590 7fcb138f8280 0 set rocksdb option writable_file_max_buffer_size = 0
2017-08-17 14:03:43.902593 7fcb138f8280 0 set rocksdb option write_buffer_size = 268435456
2017-08-17 14:03:43.902706 7fcb138f8280 4 rocksdb: RocksDB version: 5.4.0
2017-08-17 14:03:43.902713 7fcb138f8280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
2017-08-17 14:03:43.902715 7fcb138f8280 4 rocksdb: Compile date Aug 15 2017
2017-08-17 14:03:43.902719 7fcb138f8280 4 rocksdb: DB SUMMARY
2017-08-17 14:03:43.902727 7fcb138f8280 4 rocksdb: CURRENT file: CURRENT
2017-08-17 14:03:43.902729 7fcb138f8280 4 rocksdb: IDENTITY file: IDENTITY
2017-08-17 14:03:43.902735 7fcb138f8280 4 rocksdb: MANIFEST file: MANIFEST-000937 size: 1083 Bytes
2017-08-17 14:03:43.902739 7fcb138f8280 4 rocksdb: SST files in db dir, Total Num: 0, files:
2017-08-17 14:03:43.902745 7fcb138f8280 4 rocksdb: SST files in db.slow dir, Total Num: 6, files: 000187.sst 000199.sst 000219.sst 000220.sst 000934.sst 000935.sst
2017-08-17 14:03:43.902753 7fcb138f8280 4 rocksdb: Write Ahead Log file in db.wal: 000938.log size: 0 ;
2017-08-17 14:03:43.902755 7fcb138f8280 4 rocksdb: Options.error_if_exists: 0
2017-08-17 14:03:43.902756 7fcb138f8280 4 rocksdb: Options.create_if_missing: 0
2017-08-17 14:03:43.902757 7fcb138f8280 4 rocksdb: Options.paranoid_checks: 1
2017-08-17 14:03:43.902758 7fcb138f8280 4 rocksdb: Options.env: 0x5601f8a43e80
2017-08-17 14:03:43.902759 7fcb138f8280 4 rocksdb: Options.info_log: 0x5601f8a443e0
2017-08-17 14:03:43.902760 7fcb138f8280 4 rocksdb: Options.max_open_files: -1
2017-08-17 14:03:43.902762 7fcb138f8280 4 rocksdb: Options.max_file_opening_threads: 16
2017-08-17 14:03:43.902763 7fcb138f8280 4 rocksdb: Options.use_fsync: 0
2017-08-17 14:03:43.902764 7fcb138f8280 4 rocksdb: Options.max_log_file_size: 0
2017-08-17 14:03:43.902765 7fcb138f8280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
2017-08-17 14:03:43.902766 7fcb138f8280 4 rocksdb: Options.log_file_time_to_roll: 0
2017-08-17 14:03:43.902767 7fcb138f8280 4 rocksdb: Options.keep_log_file_num: 1000
2017-08-17 14:03:43.902768 7fcb138f8280 4 rocksdb: Options.recycle_log_file_num: 4
2017-08-17 14:03:43.902769 7fcb138f8280 4 rocksdb: Options.allow_fallocate: 1
2017-08-17 14:03:43.902770 7fcb138f8280 4 rocksdb: Options.allow_mmap_reads: 0
2017-08-17 14:03:43.902771 7fcb138f8280 4 rocksdb: Options.allow_mmap_writes: 0
2017-08-17 14:03:43.902771 7fcb138f8280 4 rocksdb: Options.use_direct_reads: 0
2017-08-17 14:03:43.902772 7fcb138f8280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
2017-08-17 14:03:43.902774 7fcb138f8280 4 rocksdb: Options.create_missing_column_families: 0
2017-08-17 14:03:43.902781 7fcb138f8280 4 rocksdb: Options.db_log_dir:
2017-08-17 14:03:43.902783 7fcb138f8280 4 rocksdb: Options.wal_dir: db.wal
2017-08-17 14:03:43.902784 7fcb138f8280 4 rocksdb: Options.table_cache_numshardbits: 6
2017-08-17 14:03:43.902784 7fcb138f8280 4 rocksdb: Options.max_subcompactions: 1
2017-08-17 14:03:43.902785 7fcb138f8280 4 rocksdb: Options.max_background_flushes: 1
2017-08-17 14:03:43.902786 7fcb138f8280 4 rocksdb: Options.WAL_ttl_seconds: 0
2017-08-17 14:03:43.902787 7fcb138f8280 4 rocksdb: Options.WAL_size_limit_MB: 0
2017-08-17 14:03:43.902788 7fcb138f8280 4 rocksdb: Options.manifest_preallocation_size: 4194304
2017-08-17 14:03:43.902789 7fcb138f8280 4 rocksdb: Options.is_fd_close_on_exec: 1
2017-08-17 14:03:43.902790 7fcb138f8280 4 rocksdb: Options.advise_random_on_open: 1
2017-08-17 14:03:43.902790 7fcb138f8280 4 rocksdb: Options.db_write_buffer_size: 0
2017-08-17 14:03:43.902791 7fcb138f8280 4 rocksdb: Options.access_hint_on_compaction_start: 1
2017-08-17 14:03:43.902792 7fcb138f8280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1
2017-08-17 14:03:43.902793 7fcb138f8280 4 rocksdb: Options.compaction_readahead_size: 2097152
2017-08-17 14:03:43.902794 7fcb138f8280 4 rocksdb: Options.random_access_max_buffer_size: 1048576
2017-08-17 14:03:43.902795 7fcb138f8280 4 rocksdb: Options.writable_file_max_buffer_size: 0
2017-08-17 14:03:43.902795 7fcb138f8280 4 rocksdb: Options.use_adaptive_mutex: 0
2017-08-17 14:03:43.902796 7fcb138f8280 4 rocksdb: Options.rate_limiter: (nil)
2017-08-17 14:03:43.902798 7fcb138f8280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
2017-08-17 14:03:43.902799 7fcb138f8280 4 rocksdb: Options.bytes_per_sync: 0
2017-08-17 14:03:43.902801 7fcb138f8280 4 rocksdb: Options.wal_bytes_per_sync: 0
2017-08-17 14:03:43.902802 7fcb138f8280 4 rocksdb: Options.wal_recovery_mode: 0
2017-08-17 14:03:43.902803 7fcb138f8280 4 rocksdb: Options.enable_thread_tracking: 0
2017-08-17 14:03:43.902804 7fcb138f8280 4 rocksdb: Options.allow_concurrent_memtable_write: 1
2017-08-17 14:03:43.902805 7fcb138f8280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
2017-08-17 14:03:43.902806 7fcb138f8280 4 rocksdb: Options.write_thread_max_yield_usec: 100
2017-08-17 14:03:43.902807 7fcb138f8280 4 rocksdb: Options.write_thread_slow_yield_usec: 3
2017-08-17 14:03:43.902808 7fcb138f8280 4 rocksdb: Options.row_cache: None
2017-08-17 14:03:43.902808 7fcb138f8280 4 rocksdb: Options.wal_filter: None
2017-08-17 14:03:43.902809 7fcb138f8280 4 rocksdb: Options.avoid_flush_during_recovery: 0
2017-08-17 14:03:43.902810 7fcb138f8280 4 rocksdb: Options.base_background_compactions: 1
2017-08-17 14:03:43.902811 7fcb138f8280 4 rocksdb: Options.max_background_compactions: 1
2017-08-17 14:03:43.902812 7fcb138f8280 4 rocksdb: Options.avoid_flush_during_shutdown: 0
2017-08-17 14:03:43.902813 7fcb138f8280 4 rocksdb: Options.delayed_write_rate : 16777216
2017-08-17 14:03:43.902814 7fcb138f8280 4 rocksdb: Options.max_total_wal_size: 0
2017-08-17 14:03:43.902816 7fcb138f8280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
2017-08-17 14:03:43.902817 7fcb138f8280 4 rocksdb: Options.stats_dump_period_sec: 600
2017-08-17 14:03:43.902818 7fcb138f8280 4 rocksdb: Compression algorithms supported:
2017-08-17 14:03:43.902819 7fcb138f8280 4 rocksdb: Snappy supported: 0
2017-08-17 14:03:43.902820 7fcb138f8280 4 rocksdb: Zlib supported: 0
2017-08-17 14:03:43.902821 7fcb138f8280 4 rocksdb: Bzip supported: 0
2017-08-17 14:03:43.902822 7fcb138f8280 4 rocksdb: LZ4 supported: 0
2017-08-17 14:03:43.902823 7fcb138f8280 4 rocksdb: ZSTD supported: 0
2017-08-17 14:03:43.902824 7fcb138f8280 4 rocksdb: Fast CRC32 supported: 1
2017-08-17 14:03:43.902920 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000937
2017-08-17 14:03:43.902958 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
2017-08-17 14:03:43.902960 7fcb138f8280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
2017-08-17 14:03:43.902965 7fcb138f8280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor
2017-08-17 14:03:43.902966 7fcb138f8280 4 rocksdb: Options.compaction_filter: None
2017-08-17 14:03:43.902967 7fcb138f8280 4 rocksdb: Options.compaction_filter_factory: None
2017-08-17 14:03:43.902969 7fcb138f8280 4 rocksdb: Options.memtable_factory: SkipListFactory
2017-08-17 14:03:43.902970 7fcb138f8280 4 rocksdb: Options.table_factory: BlockBasedTable
2017-08-17 14:03:43.902985 7fcb138f8280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601f8782208)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 1
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x5601f87cddd8
block_cache_name: LRUCache
block_cache_options:
capacity : 536870912
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
format_version: 2
2017-08-17 14:03:43.902995 7fcb138f8280 4 rocksdb: Options.write_buffer_size: 268435456
2017-08-17 14:03:43.902996 7fcb138f8280 4 rocksdb: Options.max_write_buffer_number: 4
2017-08-17 14:03:43.902997 7fcb138f8280 4 rocksdb: Options.compression: NoCompression
2017-08-17 14:03:43.902999 7fcb138f8280 4 rocksdb: Options.bottommost_compression: Disabled
2017-08-17 14:03:43.903000 7fcb138f8280 4 rocksdb: Options.prefix_extractor: nullptr
2017-08-17 14:03:43.903000 7fcb138f8280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
2017-08-17 14:03:43.903001 7fcb138f8280 4 rocksdb: Options.num_levels: 7
2017-08-17 14:03:43.903002 7fcb138f8280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
2017-08-17 14:03:43.903008 7fcb138f8280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
2017-08-17 14:03:43.903009 7fcb138f8280 4 rocksdb: Options.compression_opts.window_bits: -14
2017-08-17 14:03:43.903010 7fcb138f8280 4 rocksdb: Options.compression_opts.level: -1
2017-08-17 14:03:43.903010 7fcb138f8280 4 rocksdb: Options.compression_opts.strategy: 0
2017-08-17 14:03:43.903011 7fcb138f8280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
2017-08-17 14:03:43.903012 7fcb138f8280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
2017-08-17 14:03:43.903013 7fcb138f8280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
2017-08-17 14:03:43.903014 7fcb138f8280 4 rocksdb: Options.level0_stop_writes_trigger: 36
2017-08-17 14:03:43.903015 7fcb138f8280 4 rocksdb: Options.target_file_size_base: 67108864
2017-08-17 14:03:43.903015 7fcb138f8280 4 rocksdb: Options.target_file_size_multiplier: 1
2017-08-17 14:03:43.903016 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_base: 268435456
2017-08-17 14:03:43.903017 7fcb138f8280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2017-08-17 14:03:43.903018 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
2017-08-17 14:03:43.903021 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2017-08-17 14:03:43.903022 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2017-08-17 14:03:43.903023 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2017-08-17 14:03:43.903024 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2017-08-17 14:03:43.903025 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2017-08-17 14:03:43.903026 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2017-08-17 14:03:43.903026 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2017-08-17 14:03:43.903027 7fcb138f8280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
2017-08-17 14:03:43.903028 7fcb138f8280 4 rocksdb: Options.max_compaction_bytes: 1677721600
2017-08-17 14:03:43.903029 7fcb138f8280 4 rocksdb: Options.arena_block_size: 33554432
2017-08-17 14:03:43.903030 7fcb138f8280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
2017-08-17 14:03:43.903031 7fcb138f8280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
2017-08-17 14:03:43.903032 7fcb138f8280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
2017-08-17 14:03:43.903033 7fcb138f8280 4 rocksdb: Options.disable_auto_compactions: 0
2017-08-17 14:03:43.903034 7fcb138f8280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
2017-08-17 14:03:43.903035 7fcb138f8280 4 rocksdb: Options.compaction_pri: kByCompensatedSize
2017-08-17 14:03:43.903036 7fcb138f8280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
2017-08-17 14:03:43.903037 7fcb138f8280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2017-08-17 14:03:43.903038 7fcb138f8280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2017-08-17 14:03:43.903039 7fcb138f8280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2017-08-17 14:03:43.903040 7fcb138f8280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2017-08-17 14:03:43.903041 7fcb138f8280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2017-08-17 14:03:43.903042 7fcb138f8280 4 rocksdb: Options.table_properties_collectors:
2017-08-17 14:03:43.903043 7fcb138f8280 4 rocksdb: Options.inplace_update_support: 0
2017-08-17 14:03:43.903044 7fcb138f8280 4 rocksdb: Options.inplace_update_num_locks: 10000
2017-08-17 14:03:43.903045 7fcb138f8280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
2017-08-17 14:03:43.903046 7fcb138f8280 4 rocksdb: Options.memtable_huge_page_size: 0
2017-08-17 14:03:43.903047 7fcb138f8280 4 rocksdb: Options.bloom_locality: 0
2017-08-17 14:03:43.903048 7fcb138f8280 4 rocksdb: Options.max_successive_merges: 0
2017-08-17 14:03:43.903049 7fcb138f8280 4 rocksdb: Options.optimize_filters_for_hits: 0
2017-08-17 14:03:43.903050 7fcb138f8280 4 rocksdb: Options.paranoid_file_checks: 0
2017-08-17 14:03:43.903051 7fcb138f8280 4 rocksdb: Options.force_consistency_checks: 0
2017-08-17 14:03:43.903053 7fcb138f8280 4 rocksdb: Options.report_bg_io_stats: 0
2017-08-17 14:03:43.946483 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000937 succeeded,manifest_file_number is 937, next_file_number is 939, last_sequence is 64576158, log_number is 0,prev_log_number is 0,max_column_family is 0
2017-08-17 14:03:43.946492 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 932
2017-08-17 14:03:43.946530 7fcb138f8280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823946523, "job": 1, "event": "recovery_started", "log_files": [938]}
2017-08-17 14:03:43.946533 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:482] Recovering log #938 mode 0
2017-08-17 14:03:43.946541 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2395] Creating manifest 940
2017-08-17 14:03:43.947010 7fcb138f8280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823947008, "job": 1, "event": "recovery_finished"}
2017-08-17 14:03:43.947694 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5601f8a8ab00
2017-08-17 14:03:43.947722 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
2017-08-17 14:03:43.995973 7fcb138f8280 1 freelist init
2017-08-17 14:03:44.027637 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc opening allocation metadata
2017-08-17 14:03:45.927394 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc loaded 3701 G in 3919 extents
2017-08-17 14:03:46.426774 7fcb138f8280 1 bluefs fsck
2017-08-17 14:03:46.426792 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck walking object keyspace
2017-08-17 14:03:50.856077 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking shared_blobs
2017-08-17 14:03:50.856119 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking for stray omap data
2017-08-17 14:03:52.405816 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking deferred events
2017-08-17 14:03:52.405862 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking freelist vs allocated
2017-08-17 14:04:06.311748 7fcb138f8280 -1 bluestore(/var/lib/ceph/osd/ceph-2) fsck error: leaked some space;65536 bytes leaked
2017-08-17 14:04:06.410019 7fcb138f8280 1 freelist shutdown
2017-08-17 14:04:06.410080 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
2017-08-17 14:04:06.410464 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:343] Shutdown complete
2017-08-17 14:04:06.451387 7fcb138f8280 1 bluefs umount
2017-08-17 14:04:06.452720 7fcb138f8280 1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) close
2017-08-17 14:04:06.744745 7fcb138f8280 1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) close
2017-08-17 14:04:06.988729 7fcb138f8280 1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) close
2017-08-17 14:04:07.200717 7fcb138f8280 1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) close
2017-08-17 14:04:07.448859 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 23.553470 seconds
fsck found 1 errors
��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
