On Thu, 24 Aug 2017, Kamble, Nitin A wrote:
> On 8/23/17, 7:19 PM, "Sage Weil" <sweil@xxxxxxxxxx> wrote:
>
> Thanks! It's right at the end of the device, and I can reproduce it with
> vstart by making the block device not aligned to a 64k (min_alloc_size)
> boundary.
>
> sage
>
> Great!
> Does that mean ceph-disk needs to consider the min_alloc_size while
> creating the block device partitions?
BlueStore should be smart enough to handle it. And it appears to be, now.
Did you see my other email? Can you share what version you originally
used when the OSD was created? I think that's where the bug was.
Thanks!
sage
>
>
> Nitin
>
> On Thu, 24 Aug 2017, Kamble, Nitin A wrote:
>
> > Hi Sage,
> > The log file is uploaded. It gave id like this:
> >
> > ceph-post-file: 48a58181-5272-40ef-938c-2f9712443d10
> >
> > Thanks,
> > Nitin
> >
> >
> > On 8/23/17, 2:33 PM, "Sage Weil" <sweil@xxxxxxxxxx> wrote:
> >
> > On Wed, 23 Aug 2017, Kamble, Nitin A wrote:
> > > Hi Sage,
> > > My comments inline below.
> > >
> > > On 8/23/17, 12:15 PM, "Sage Weil" <sweil@xxxxxxxxxx> wrote:
> > >
> > > Hi Nitin,
> > >
> > > As David mentioned, it's a non-fatal error, so disabling fsck on mount is
> > > a fine workaround. (Out of curiousity, how did fsck get enabled in the
> > > first place? It should be off by default.)
> > >
> > > I find it is enabled by default.
> >
> > Which version are you running? Master branch has
> >
> > Option("bluestore_fsck_on_mount", Option::TYPE_BOOL, Option::LEVEL_DEV)
> > .set_default(false)
> >
> > > If you don't mind, I have a branch that will additionally spit out exactly
> > > which extent(s) were leaked:
> > >
> > > https://github.com/liewegas/ceph/commit/wip-bluestore-fsck-leaks-luminous
> > >
> > > Would you mind running fsck with that branch and send me the log?
> > >
> > > Sure, I can try it out. Looks like I will only need
> > > ceph-objectstore-tool replaced in my ceph setup, correct?
> >
> > Right. Thanks!
> >
> > > CEPH_ARGS="--log-file c" ceph-objectstore-tool --op fsck --data-path ...
> > >
> > > One with --debug-bluestore 20 in the CEPH_ARGS would be ideal if it's not
> > > too big. You can ceph-post-file the log.
> > >
> > > What is ceph-post-file thingy? I have not used it before.
> >
> > It will upload a file or files to a server the developers have access to
> > (any developers with access to the sepia lab) and spit out a uuid they can
> > use to identify which upload is yours.
> >
> > sage
> >
> > >
> > > Thanks,
> > > Nitin
> > >
> > > Thanks!
> > > sage
> > >
> > > On Thu, 17 Aug 2017, Kamble, Nitin A wrote:
> > >
> > > >
> > > >
> > > > On 8/17/17, 1:50 PM, "David Zafman" <dzafman@xxxxxxxxxx> wrote:
> > > >
> > > >
> > > > I wouldn't run the OSDs without identifying what is going on. I suggest
> > > > you stop the OSD and run fsck using ceph-objectstore-tool:
> > > >
> > > >
> > > > ceph-objectstore-tool --data-path .... --op fsck
> > > >
> > > > If you don't get any informative messages add --debug to get detail logging.
> > > >
> > > >
> > > > David
> > > >
> > > >
> > > > Thanks, David for your input. It is interesting that all the 50 osds had the same fsck issue immediately after the upgrade.
> > > >
> > > > Below is the log of manual fsck.
> > > >
> > > > This is the error line in it:
> > > > fsck error: leaked some space;65536 bytes leaked
> > > >
> > > >
> > > > # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op fsck --debug
> > > > warning: line 36: 'bluestore_fsck_on_mount' in section 'global' redefined
> > > > 2017-08-17 14:03:43.895377 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck (shallow) start
> > > > 2017-08-17 14:03:43.895410 7fcb138f8280 1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
> > > > 2017-08-17 14:03:43.895416 7fcb138f8280 1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
> > > > 2017-08-17 14:03:43.895684 7fcb138f8280 1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
> > > > 2017-08-17 14:03:43.895942 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes max 0.5 < ratio 0.99
> > > > 2017-08-17 14:03:43.895957 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
> > > > 2017-08-17 14:03:43.895990 7fcb138f8280 1 bdev create path /var/lib/ceph/osd/ceph-2/block.db type kernel
> > > > 2017-08-17 14:03:43.895994 7fcb138f8280 1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open path /var/lib/ceph/osd/ceph-2/block.db
> > > > 2017-08-17 14:03:43.896162 7fcb138f8280 1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) non-rotational
> > > > 2017-08-17 14:03:43.896168 7fcb138f8280 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block.db size 1024 MB
> > > > 2017-08-17 14:03:43.896364 7fcb138f8280 1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
> > > > 2017-08-17 14:03:43.896372 7fcb138f8280 1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
> > > > 2017-08-17 14:03:43.896529 7fcb138f8280 1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
> > > > 2017-08-17 14:03:43.896535 7fcb138f8280 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-2/block size 3725 GB
> > > > 2017-08-17 14:03:43.896546 7fcb138f8280 1 bdev create path /var/lib/ceph/osd/ceph-2/block.wal type kernel
> > > > 2017-08-17 14:03:43.896549 7fcb138f8280 1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open path /var/lib/ceph/osd/ceph-2/block.wal
> > > > 2017-08-17 14:03:43.896733 7fcb138f8280 1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open size 10737418240 (0x280000000, 10240 MB) block_size 4096 (4096 B) non-rotational
> > > > 2017-08-17 14:03:43.896739 7fcb138f8280 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-2/block.wal size 10240 MB
> > > > 2017-08-17 14:03:43.896912 7fcb138f8280 1 bluefs mount
> > > > 2017-08-17 14:03:43.902514 7fcb138f8280 0 set rocksdb option compaction_readahead_size = 2097152
> > > > 2017-08-17 14:03:43.902525 7fcb138f8280 0 set rocksdb option compression = kNoCompression
> > > > 2017-08-17 14:03:43.902533 7fcb138f8280 0 set rocksdb option max_write_buffer_number = 4
> > > > 2017-08-17 14:03:43.902538 7fcb138f8280 0 set rocksdb option min_write_buffer_number_to_merge = 1
> > > > 2017-08-17 14:03:43.902541 7fcb138f8280 0 set rocksdb option recycle_log_file_num = 4
> > > > 2017-08-17 14:03:43.902544 7fcb138f8280 0 set rocksdb option writable_file_max_buffer_size = 0
> > > > 2017-08-17 14:03:43.902548 7fcb138f8280 0 set rocksdb option write_buffer_size = 268435456
> > > > 2017-08-17 14:03:43.902572 7fcb138f8280 0 set rocksdb option compaction_readahead_size = 2097152
> > > > 2017-08-17 14:03:43.902577 7fcb138f8280 0 set rocksdb option compression = kNoCompression
> > > > 2017-08-17 14:03:43.902580 7fcb138f8280 0 set rocksdb option max_write_buffer_number = 4
> > > > 2017-08-17 14:03:43.902584 7fcb138f8280 0 set rocksdb option min_write_buffer_number_to_merge = 1
> > > > 2017-08-17 14:03:43.902587 7fcb138f8280 0 set rocksdb option recycle_log_file_num = 4
> > > > 2017-08-17 14:03:43.902590 7fcb138f8280 0 set rocksdb option writable_file_max_buffer_size = 0
> > > > 2017-08-17 14:03:43.902593 7fcb138f8280 0 set rocksdb option write_buffer_size = 268435456
> > > > 2017-08-17 14:03:43.902706 7fcb138f8280 4 rocksdb: RocksDB version: 5.4.0
> > > >
> > > > 2017-08-17 14:03:43.902713 7fcb138f8280 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
> > > > 2017-08-17 14:03:43.902715 7fcb138f8280 4 rocksdb: Compile date Aug 15 2017
> > > > 2017-08-17 14:03:43.902719 7fcb138f8280 4 rocksdb: DB SUMMARY
> > > >
> > > > 2017-08-17 14:03:43.902727 7fcb138f8280 4 rocksdb: CURRENT file: CURRENT
> > > >
> > > > 2017-08-17 14:03:43.902729 7fcb138f8280 4 rocksdb: IDENTITY file: IDENTITY
> > > >
> > > > 2017-08-17 14:03:43.902735 7fcb138f8280 4 rocksdb: MANIFEST file: MANIFEST-000937 size: 1083 Bytes
> > > >
> > > > 2017-08-17 14:03:43.902739 7fcb138f8280 4 rocksdb: SST files in db dir, Total Num: 0, files:
> > > >
> > > > 2017-08-17 14:03:43.902745 7fcb138f8280 4 rocksdb: SST files in db.slow dir, Total Num: 6, files: 000187.sst 000199.sst 000219.sst 000220.sst 000934.sst 000935.sst
> > > >
> > > > 2017-08-17 14:03:43.902753 7fcb138f8280 4 rocksdb: Write Ahead Log file in db.wal: 000938.log size: 0 ;
> > > >
> > > > 2017-08-17 14:03:43.902755 7fcb138f8280 4 rocksdb: Options.error_if_exists: 0
> > > > 2017-08-17 14:03:43.902756 7fcb138f8280 4 rocksdb: Options.create_if_missing: 0
> > > > 2017-08-17 14:03:43.902757 7fcb138f8280 4 rocksdb: Options.paranoid_checks: 1
> > > > 2017-08-17 14:03:43.902758 7fcb138f8280 4 rocksdb: Options.env: 0x5601f8a43e80
> > > > 2017-08-17 14:03:43.902759 7fcb138f8280 4 rocksdb: Options.info_log: 0x5601f8a443e0
> > > > 2017-08-17 14:03:43.902760 7fcb138f8280 4 rocksdb: Options.max_open_files: -1
> > > > 2017-08-17 14:03:43.902762 7fcb138f8280 4 rocksdb: Options.max_file_opening_threads: 16
> > > > 2017-08-17 14:03:43.902763 7fcb138f8280 4 rocksdb: Options.use_fsync: 0
> > > > 2017-08-17 14:03:43.902764 7fcb138f8280 4 rocksdb: Options.max_log_file_size: 0
> > > > 2017-08-17 14:03:43.902765 7fcb138f8280 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
> > > > 2017-08-17 14:03:43.902766 7fcb138f8280 4 rocksdb: Options.log_file_time_to_roll: 0
> > > > 2017-08-17 14:03:43.902767 7fcb138f8280 4 rocksdb: Options.keep_log_file_num: 1000
> > > > 2017-08-17 14:03:43.902768 7fcb138f8280 4 rocksdb: Options.recycle_log_file_num: 4
> > > > 2017-08-17 14:03:43.902769 7fcb138f8280 4 rocksdb: Options.allow_fallocate: 1
> > > > 2017-08-17 14:03:43.902770 7fcb138f8280 4 rocksdb: Options.allow_mmap_reads: 0
> > > > 2017-08-17 14:03:43.902771 7fcb138f8280 4 rocksdb: Options.allow_mmap_writes: 0
> > > > 2017-08-17 14:03:43.902771 7fcb138f8280 4 rocksdb: Options.use_direct_reads: 0
> > > > 2017-08-17 14:03:43.902772 7fcb138f8280 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
> > > > 2017-08-17 14:03:43.902774 7fcb138f8280 4 rocksdb: Options.create_missing_column_families: 0
> > > > 2017-08-17 14:03:43.902781 7fcb138f8280 4 rocksdb: Options.db_log_dir:
> > > > 2017-08-17 14:03:43.902783 7fcb138f8280 4 rocksdb: Options.wal_dir: db.wal
> > > > 2017-08-17 14:03:43.902784 7fcb138f8280 4 rocksdb: Options.table_cache_numshardbits: 6
> > > > 2017-08-17 14:03:43.902784 7fcb138f8280 4 rocksdb: Options.max_subcompactions: 1
> > > > 2017-08-17 14:03:43.902785 7fcb138f8280 4 rocksdb: Options.max_background_flushes: 1
> > > > 2017-08-17 14:03:43.902786 7fcb138f8280 4 rocksdb: Options.WAL_ttl_seconds: 0
> > > > 2017-08-17 14:03:43.902787 7fcb138f8280 4 rocksdb: Options.WAL_size_limit_MB: 0
> > > > 2017-08-17 14:03:43.902788 7fcb138f8280 4 rocksdb: Options.manifest_preallocation_size: 4194304
> > > > 2017-08-17 14:03:43.902789 7fcb138f8280 4 rocksdb: Options.is_fd_close_on_exec: 1
> > > > 2017-08-17 14:03:43.902790 7fcb138f8280 4 rocksdb: Options.advise_random_on_open: 1
> > > > 2017-08-17 14:03:43.902790 7fcb138f8280 4 rocksdb: Options.db_write_buffer_size: 0
> > > > 2017-08-17 14:03:43.902791 7fcb138f8280 4 rocksdb: Options.access_hint_on_compaction_start: 1
> > > > 2017-08-17 14:03:43.902792 7fcb138f8280 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1
> > > > 2017-08-17 14:03:43.902793 7fcb138f8280 4 rocksdb: Options.compaction_readahead_size: 2097152
> > > > 2017-08-17 14:03:43.902794 7fcb138f8280 4 rocksdb: Options.random_access_max_buffer_size: 1048576
> > > > 2017-08-17 14:03:43.902795 7fcb138f8280 4 rocksdb: Options.writable_file_max_buffer_size: 0
> > > > 2017-08-17 14:03:43.902795 7fcb138f8280 4 rocksdb: Options.use_adaptive_mutex: 0
> > > > 2017-08-17 14:03:43.902796 7fcb138f8280 4 rocksdb: Options.rate_limiter: (nil)
> > > > 2017-08-17 14:03:43.902798 7fcb138f8280 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
> > > > 2017-08-17 14:03:43.902799 7fcb138f8280 4 rocksdb: Options.bytes_per_sync: 0
> > > > 2017-08-17 14:03:43.902801 7fcb138f8280 4 rocksdb: Options.wal_bytes_per_sync: 0
> > > > 2017-08-17 14:03:43.902802 7fcb138f8280 4 rocksdb: Options.wal_recovery_mode: 0
> > > > 2017-08-17 14:03:43.902803 7fcb138f8280 4 rocksdb: Options.enable_thread_tracking: 0
> > > > 2017-08-17 14:03:43.902804 7fcb138f8280 4 rocksdb: Options.allow_concurrent_memtable_write: 1
> > > > 2017-08-17 14:03:43.902805 7fcb138f8280 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
> > > > 2017-08-17 14:03:43.902806 7fcb138f8280 4 rocksdb: Options.write_thread_max_yield_usec: 100
> > > > 2017-08-17 14:03:43.902807 7fcb138f8280 4 rocksdb: Options.write_thread_slow_yield_usec: 3
> > > > 2017-08-17 14:03:43.902808 7fcb138f8280 4 rocksdb: Options.row_cache: None
> > > > 2017-08-17 14:03:43.902808 7fcb138f8280 4 rocksdb: Options.wal_filter: None
> > > > 2017-08-17 14:03:43.902809 7fcb138f8280 4 rocksdb: Options.avoid_flush_during_recovery: 0
> > > > 2017-08-17 14:03:43.902810 7fcb138f8280 4 rocksdb: Options.base_background_compactions: 1
> > > > 2017-08-17 14:03:43.902811 7fcb138f8280 4 rocksdb: Options.max_background_compactions: 1
> > > > 2017-08-17 14:03:43.902812 7fcb138f8280 4 rocksdb: Options.avoid_flush_during_shutdown: 0
> > > > 2017-08-17 14:03:43.902813 7fcb138f8280 4 rocksdb: Options.delayed_write_rate : 16777216
> > > > 2017-08-17 14:03:43.902814 7fcb138f8280 4 rocksdb: Options.max_total_wal_size: 0
> > > > 2017-08-17 14:03:43.902816 7fcb138f8280 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
> > > > 2017-08-17 14:03:43.902817 7fcb138f8280 4 rocksdb: Options.stats_dump_period_sec: 600
> > > > 2017-08-17 14:03:43.902818 7fcb138f8280 4 rocksdb: Compression algorithms supported:
> > > > 2017-08-17 14:03:43.902819 7fcb138f8280 4 rocksdb: Snappy supported: 0
> > > > 2017-08-17 14:03:43.902820 7fcb138f8280 4 rocksdb: Zlib supported: 0
> > > > 2017-08-17 14:03:43.902821 7fcb138f8280 4 rocksdb: Bzip supported: 0
> > > > 2017-08-17 14:03:43.902822 7fcb138f8280 4 rocksdb: LZ4 supported: 0
> > > > 2017-08-17 14:03:43.902823 7fcb138f8280 4 rocksdb: ZSTD supported: 0
> > > > 2017-08-17 14:03:43.902824 7fcb138f8280 4 rocksdb: Fast CRC32 supported: 1
> > > > 2017-08-17 14:03:43.902920 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000937
> > > >
> > > > 2017-08-17 14:03:43.902958 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
> > > >
> > > > 2017-08-17 14:03:43.902960 7fcb138f8280 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
> > > > 2017-08-17 14:03:43.902965 7fcb138f8280 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor
> > > > 2017-08-17 14:03:43.902966 7fcb138f8280 4 rocksdb: Options.compaction_filter: None
> > > > 2017-08-17 14:03:43.902967 7fcb138f8280 4 rocksdb: Options.compaction_filter_factory: None
> > > > 2017-08-17 14:03:43.902969 7fcb138f8280 4 rocksdb: Options.memtable_factory: SkipListFactory
> > > > 2017-08-17 14:03:43.902970 7fcb138f8280 4 rocksdb: Options.table_factory: BlockBasedTable
> > > > 2017-08-17 14:03:43.902985 7fcb138f8280 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601f8782208)
> > > > cache_index_and_filter_blocks: 1
> > > > cache_index_and_filter_blocks_with_high_priority: 1
> > > > pin_l0_filter_and_index_blocks_in_cache: 1
> > > > index_type: 0
> > > > hash_index_allow_collision: 1
> > > > checksum: 1
> > > > no_block_cache: 0
> > > > block_cache: 0x5601f87cddd8
> > > > block_cache_name: LRUCache
> > > > block_cache_options:
> > > > capacity : 536870912
> > > > num_shard_bits : 4
> > > > strict_capacity_limit : 0
> > > > high_pri_pool_ratio: 0.000
> > > > block_cache_compressed: (nil)
> > > > persistent_cache: (nil)
> > > > block_size: 4096
> > > > block_size_deviation: 10
> > > > block_restart_interval: 16
> > > > index_block_restart_interval: 1
> > > > filter_policy: rocksdb.BuiltinBloomFilter
> > > > whole_key_filtering: 1
> > > > format_version: 2
> > > >
> > > > 2017-08-17 14:03:43.902995 7fcb138f8280 4 rocksdb: Options.write_buffer_size: 268435456
> > > > 2017-08-17 14:03:43.902996 7fcb138f8280 4 rocksdb: Options.max_write_buffer_number: 4
> > > > 2017-08-17 14:03:43.902997 7fcb138f8280 4 rocksdb: Options.compression: NoCompression
> > > > 2017-08-17 14:03:43.902999 7fcb138f8280 4 rocksdb: Options.bottommost_compression: Disabled
> > > > 2017-08-17 14:03:43.903000 7fcb138f8280 4 rocksdb: Options.prefix_extractor: nullptr
> > > > 2017-08-17 14:03:43.903000 7fcb138f8280 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
> > > > 2017-08-17 14:03:43.903001 7fcb138f8280 4 rocksdb: Options.num_levels: 7
> > > > 2017-08-17 14:03:43.903002 7fcb138f8280 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
> > > > 2017-08-17 14:03:43.903008 7fcb138f8280 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
> > > > 2017-08-17 14:03:43.903009 7fcb138f8280 4 rocksdb: Options.compression_opts.window_bits: -14
> > > > 2017-08-17 14:03:43.903010 7fcb138f8280 4 rocksdb: Options.compression_opts.level: -1
> > > > 2017-08-17 14:03:43.903010 7fcb138f8280 4 rocksdb: Options.compression_opts.strategy: 0
> > > > 2017-08-17 14:03:43.903011 7fcb138f8280 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
> > > > 2017-08-17 14:03:43.903012 7fcb138f8280 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
> > > > 2017-08-17 14:03:43.903013 7fcb138f8280 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
> > > > 2017-08-17 14:03:43.903014 7fcb138f8280 4 rocksdb: Options.level0_stop_writes_trigger: 36
> > > > 2017-08-17 14:03:43.903015 7fcb138f8280 4 rocksdb: Options.target_file_size_base: 67108864
> > > > 2017-08-17 14:03:43.903015 7fcb138f8280 4 rocksdb: Options.target_file_size_multiplier: 1
> > > > 2017-08-17 14:03:43.903016 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_base: 268435456
> > > > 2017-08-17 14:03:43.903017 7fcb138f8280 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
> > > > 2017-08-17 14:03:43.903018 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
> > > > 2017-08-17 14:03:43.903021 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
> > > > 2017-08-17 14:03:43.903022 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
> > > > 2017-08-17 14:03:43.903023 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
> > > > 2017-08-17 14:03:43.903024 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
> > > > 2017-08-17 14:03:43.903025 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
> > > > 2017-08-17 14:03:43.903026 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
> > > > 2017-08-17 14:03:43.903026 7fcb138f8280 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
> > > > 2017-08-17 14:03:43.903027 7fcb138f8280 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
> > > > 2017-08-17 14:03:43.903028 7fcb138f8280 4 rocksdb: Options.max_compaction_bytes: 1677721600
> > > > 2017-08-17 14:03:43.903029 7fcb138f8280 4 rocksdb: Options.arena_block_size: 33554432
> > > > 2017-08-17 14:03:43.903030 7fcb138f8280 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
> > > > 2017-08-17 14:03:43.903031 7fcb138f8280 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
> > > > 2017-08-17 14:03:43.903032 7fcb138f8280 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
> > > > 2017-08-17 14:03:43.903033 7fcb138f8280 4 rocksdb: Options.disable_auto_compactions: 0
> > > > 2017-08-17 14:03:43.903034 7fcb138f8280 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
> > > > 2017-08-17 14:03:43.903035 7fcb138f8280 4 rocksdb: Options.compaction_pri: kByCompensatedSize
> > > > 2017-08-17 14:03:43.903036 7fcb138f8280 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
> > > > 2017-08-17 14:03:43.903037 7fcb138f8280 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
> > > > 2017-08-17 14:03:43.903038 7fcb138f8280 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
> > > > 2017-08-17 14:03:43.903039 7fcb138f8280 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
> > > > 2017-08-17 14:03:43.903040 7fcb138f8280 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
> > > > 2017-08-17 14:03:43.903041 7fcb138f8280 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
> > > > 2017-08-17 14:03:43.903042 7fcb138f8280 4 rocksdb: Options.table_properties_collectors:
> > > > 2017-08-17 14:03:43.903043 7fcb138f8280 4 rocksdb: Options.inplace_update_support: 0
> > > > 2017-08-17 14:03:43.903044 7fcb138f8280 4 rocksdb: Options.inplace_update_num_locks: 10000
> > > > 2017-08-17 14:03:43.903045 7fcb138f8280 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
> > > > 2017-08-17 14:03:43.903046 7fcb138f8280 4 rocksdb: Options.memtable_huge_page_size: 0
> > > > 2017-08-17 14:03:43.903047 7fcb138f8280 4 rocksdb: Options.bloom_locality: 0
> > > > 2017-08-17 14:03:43.903048 7fcb138f8280 4 rocksdb: Options.max_successive_merges: 0
> > > > 2017-08-17 14:03:43.903049 7fcb138f8280 4 rocksdb: Options.optimize_filters_for_hits: 0
> > > > 2017-08-17 14:03:43.903050 7fcb138f8280 4 rocksdb: Options.paranoid_file_checks: 0
> > > > 2017-08-17 14:03:43.903051 7fcb138f8280 4 rocksdb: Options.force_consistency_checks: 0
> > > > 2017-08-17 14:03:43.903053 7fcb138f8280 4 rocksdb: Options.report_bg_io_stats: 0
> > > > 2017-08-17 14:03:43.946483 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000937 succeeded,manifest_file_number is 937, next_file_number is 939, last_sequence is 64576158, log_number is 0,prev_log_number is 0,max_column_family is 0
> > > >
> > > > 2017-08-17 14:03:43.946492 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 932
> > > >
> > > > 2017-08-17 14:03:43.946530 7fcb138f8280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823946523, "job": 1, "event": "recovery_started", "log_files": [938]}
> > > > 2017-08-17 14:03:43.946533 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:482] Recovering log #938 mode 0
> > > > 2017-08-17 14:03:43.946541 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2395] Creating manifest 940
> > > >
> > > > 2017-08-17 14:03:43.947010 7fcb138f8280 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823947008, "job": 1, "event": "recovery_finished"}
> > > > 2017-08-17 14:03:43.947694 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5601f8a8ab00
> > > > 2017-08-17 14:03:43.947722 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
> > > > 2017-08-17 14:03:43.995973 7fcb138f8280 1 freelist init
> > > > 2017-08-17 14:03:44.027637 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc opening allocation metadata
> > > > 2017-08-17 14:03:45.927394 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc loaded 3701 G in 3919 extents
> > > > 2017-08-17 14:03:46.426774 7fcb138f8280 1 bluefs fsck
> > > > 2017-08-17 14:03:46.426792 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck walking object keyspace
> > > > 2017-08-17 14:03:50.856077 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking shared_blobs
> > > > 2017-08-17 14:03:50.856119 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking for stray omap data
> > > > 2017-08-17 14:03:52.405816 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking deferred events
> > > > 2017-08-17 14:03:52.405862 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking freelist vs allocated
> > > > 2017-08-17 14:04:06.311748 7fcb138f8280 -1 bluestore(/var/lib/ceph/osd/ceph-2) fsck error: leaked some space;65536 bytes leaked
> > > > 2017-08-17 14:04:06.410019 7fcb138f8280 1 freelist shutdown
> > > > 2017-08-17 14:04:06.410080 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
> > > > 2017-08-17 14:04:06.410464 7fcb138f8280 4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:343] Shutdown complete
> > > > 2017-08-17 14:04:06.451387 7fcb138f8280 1 bluefs umount
> > > > 2017-08-17 14:04:06.452720 7fcb138f8280 1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) close
> > > > 2017-08-17 14:04:06.744745 7fcb138f8280 1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) close
> > > > 2017-08-17 14:04:06.988729 7fcb138f8280 1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) close
> > > > 2017-08-17 14:04:07.200717 7fcb138f8280 1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) close
> > > > 2017-08-17 14:04:07.448859 7fcb138f8280 1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 23.553470 seconds
> > > > fsck found 1 errors
> > > >
> > > >
> > > >
> > > > N?????r??y??????X??ǧv???){.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i
> > >
> > > N?????r??y??????X??ǧv???){.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i
> >
> >
>
>
