Hi Sage,
This happened on production ceph cluster. Ceph mons were suddenly went
offline. Ceph osds are running.
When we check the logs while restarting the ceph-mon services we got
above error. Complete log entry is attached as
'ceph_mon_start_error.txt'.
Issue started when we change size of one of the test pools to 1. Pool
is having size as 1. But the associated crush rule is having min size
set to 2. Can this cause the issue ?
As a workaround we we trying to change the crush rule and inject the
crush map. But still crush map is not getting changed.
/usr/bin/ceph-monstore-tool /var/lib/ceph/mon/ceph-cephmon
rewrite-crush -- --crush ./osdmap_crush.updated_compiled
Do we have an option to change pool size in the osdmap directly inside
ceph-mon ? Or remove the pool reference.
Thanks & Regards,
Brayan
On Mon, Aug 6, 2018 at 6:50 PM, Sage Weil <sage@xxxxxxxxxxxx> wrote:
> Moving to ceph-devel.
>
> On Mon, 6 Aug 2018, Brayan Perera wrote:
>> Dear All,
>>
>> We have encounter an issue on ceph mons, where all the ceph-mon instances
>> went offline with following error.
>>
>>
>> =======================
>> 0> 2018-08-06 18:20:56.373266 7f9e4b98d700 -1 *** Caught signal
>> (Segmentation fault) **
>> in thread 7f9e4b98d700 thread_name:cpu_tp
>>
>> ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous
>> (stable)
>> 1: (()+0x8f51b1) [0x560c971ab1b1]
>> 2: (()+0xf6d0) [0x7f9e541106d0]
>> 3: (void std::vector<pg_t,
>> mempool::pool_allocator<(mempool::pool_index_t)16, pg_t>
>> >::_M_emplace_back_aux<pg_t const&>(pg_t const&)+0x69) [0x560c96fb5fd9]
>> 4: (OSDMapMapping::_build_rmap(OSDMap const&)+0x209) [0x560c96fb5389]
>> 5: (OSDMapMapping::_finish(OSDMap const&)+0x11) [0x560c96fb53c1]
>> 6: (ParallelPGMapper::Job::finish_one()+0x82) [0x560c96fb4332]
>> 7: (ParallelPGMapper::WQ::_process(ParallelPGMapper::Item*,
>> ThreadPool::TPHandle&)+0x7f) [0x560c96fb43ff]
>> 8: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa8e) [0x560c96eaf59e]
>> 9: (ThreadPool::WorkThread::entry()+0x10) [0x560c96eb0480]
>> 10: (()+0x7e25) [0x7f9e54108e25]
>> 11: (clone()+0x6d) [0x7f9e512febad]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
>> to interpret this.
>> ============================
>
> I haven't seen this one before. I take it it happens reliably when you
> try to start the OSD? Can you reproduce the bug and generate a mon log
> file by setting 'debug mon = 20'?
>
> Thanks!
> sage
>
>>
>>
>> Before this we have change one of the test pools size to 1 from 3 for
>> testing purpose.
>>
>> Is there a way to recover this without loosing the data ?
>>
>>
>> Thanks in advance.
>>
>> Regards,
>> Brayan
>>
>>
>>
>>
>>
>>
>> --
>> Best Regards,
>> Brayan Perera
>>
--
Best Regards,
Brayan Perera
--- begin dump of recent events ---
-383> 2018-08-06 21:43:40.433705 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command perfcounters_dump hook 0x55d71e76a190
-382> 2018-08-06 21:43:40.433723 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command 1 hook 0x55d71e76a190
-381> 2018-08-06 21:43:40.433728 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command perf dump hook 0x55d71e76a190
-380> 2018-08-06 21:43:40.433731 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command perfcounters_schema hook 0x55d71e76a190
-379> 2018-08-06 21:43:40.433734 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command perf histogram dump hook 0x55d71e76a190
-378> 2018-08-06 21:43:40.433737 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command 2 hook 0x55d71e76a190
-377> 2018-08-06 21:43:40.433740 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command perf schema hook 0x55d71e76a190
-376> 2018-08-06 21:43:40.433745 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command perf histogram schema hook 0x55d71e76a190
-375> 2018-08-06 21:43:40.433750 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command perf reset hook 0x55d71e76a190
-374> 2018-08-06 21:43:40.433753 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command config show hook 0x55d71e76a190
-373> 2018-08-06 21:43:40.433756 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command config help hook 0x55d71e76a190
-372> 2018-08-06 21:43:40.433760 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command config set hook 0x55d71e76a190
-371> 2018-08-06 21:43:40.433763 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command config get hook 0x55d71e76a190
-370> 2018-08-06 21:43:40.433767 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command config diff hook 0x55d71e76a190
-369> 2018-08-06 21:43:40.433770 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command config diff get hook 0x55d71e76a190
-368> 2018-08-06 21:43:40.433773 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command log flush hook 0x55d71e76a190
-367> 2018-08-06 21:43:40.433776 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command log dump hook 0x55d71e76a190
-366> 2018-08-06 21:43:40.433783 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command log reopen hook 0x55d71e76a190
-365> 2018-08-06 21:43:40.433798 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command dump_mempools hook 0x55d71e982e48
-364> 2018-08-06 21:43:40.447749 7f4bc94f7ec0 0 set uid:gid to 1001:1001 (ceph:ceph)
-363> 2018-08-06 21:43:40.447781 7f4bc94f7ec0 0 ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable), process (unknown), pid 7181
-362> 2018-08-06 21:43:40.455447 7f4bc94f7ec0 0 pidfile_write: ignore empty --pid-file
-361> 2018-08-06 21:43:40.498197 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) init /var/run/ceph/ceph-mon.cephmon.asok
-360> 2018-08-06 21:43:40.498228 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) bind_and_listen /var/run/ceph/ceph-mon.cephmon.asok
-359> 2018-08-06 21:43:40.498402 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command 0 hook 0x55d71e768180
-358> 2018-08-06 21:43:40.498422 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command version hook 0x55d71e768180
-357> 2018-08-06 21:43:40.498426 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command git_version hook 0x55d71e768180
-356> 2018-08-06 21:43:40.498431 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command help hook 0x55d71e76a2a0
-355> 2018-08-06 21:43:40.498438 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command get_command_descriptions hook 0x55d71e76a2b0
-354> 2018-08-06 21:43:40.498499 7f4bc3971700 5 asok(0x55d71e7b81c0) entry start
-353> 2018-08-06 21:43:40.507896 7f4bc94f7ec0 0 load: jerasure load: lrc load: isa
-352> 2018-08-06 21:43:40.509639 7f4bc94f7ec0 0 set rocksdb option compression = kNoCompression
-351> 2018-08-06 21:43:40.509660 7f4bc94f7ec0 0 set rocksdb option write_buffer_size = 33554432
-350> 2018-08-06 21:43:40.509687 7f4bc94f7ec0 0 set rocksdb option compression = kNoCompression
-349> 2018-08-06 21:43:40.509690 7f4bc94f7ec0 0 set rocksdb option write_buffer_size = 33554432
-348> 2018-08-06 21:43:40.509893 7f4bc94f7ec0 4 rocksdb: RocksDB version: 5.4.0
-347> 2018-08-06 21:43:40.509918 7f4bc94f7ec0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@
-346> 2018-08-06 21:43:40.509920 7f4bc94f7ec0 4 rocksdb: Compile date Apr 23 2018
-345> 2018-08-06 21:43:40.509923 7f4bc94f7ec0 4 rocksdb: DB SUMMARY
-344> 2018-08-06 21:43:40.524586 7f4bc94f7ec0 4 rocksdb: CURRENT file: CURRENT
-343> 2018-08-06 21:43:40.524604 7f4bc94f7ec0 4 rocksdb: IDENTITY file: IDENTITY
-342> 2018-08-06 21:43:40.524620 7f4bc94f7ec0 4 rocksdb: MANIFEST file: MANIFEST-093649 size: 359 Bytes
-341> 2018-08-06 21:43:40.524624 7f4bc94f7ec0 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-cephmon/store.db dir, Total Num: 5, files: 093617.sst 093618.sst 093620.sst 093621.sst 093639.sst
-340> 2018-08-06 21:43:40.524627 7f4bc94f7ec0 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-cephmon/store.db: 093650.log size: 0 ;
-339> 2018-08-06 21:43:40.524630 7f4bc94f7ec0 4 rocksdb: Options.error_if_exists: 0
-338> 2018-08-06 21:43:40.524642 7f4bc94f7ec0 4 rocksdb: Options.create_if_missing: 0
-337> 2018-08-06 21:43:40.524643 7f4bc94f7ec0 4 rocksdb: Options.paranoid_checks: 1
-336> 2018-08-06 21:43:40.524644 7f4bc94f7ec0 4 rocksdb: Options.env: 0x55d71cc4c140
-335> 2018-08-06 21:43:40.524645 7f4bc94f7ec0 4 rocksdb: Options.info_log: 0x55d71ea089e0
-334> 2018-08-06 21:43:40.524646 7f4bc94f7ec0 4 rocksdb: Options.max_open_files: -1
-333> 2018-08-06 21:43:40.524647 7f4bc94f7ec0 4 rocksdb: Options.max_file_opening_threads: 16
-332> 2018-08-06 21:43:40.524648 7f4bc94f7ec0 4 rocksdb: Options.use_fsync: 0
-331> 2018-08-06 21:43:40.524649 7f4bc94f7ec0 4 rocksdb: Options.max_log_file_size: 0
-330> 2018-08-06 21:43:40.524651 7f4bc94f7ec0 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615
-329> 2018-08-06 21:43:40.524652 7f4bc94f7ec0 4 rocksdb: Options.log_file_time_to_roll: 0
-328> 2018-08-06 21:43:40.524653 7f4bc94f7ec0 4 rocksdb: Options.keep_log_file_num: 1000
-327> 2018-08-06 21:43:40.524654 7f4bc94f7ec0 4 rocksdb: Options.recycle_log_file_num: 0
-326> 2018-08-06 21:43:40.524655 7f4bc94f7ec0 4 rocksdb: Options.allow_fallocate: 1
-325> 2018-08-06 21:43:40.524656 7f4bc94f7ec0 4 rocksdb: Options.allow_mmap_reads: 0
-324> 2018-08-06 21:43:40.524657 7f4bc94f7ec0 4 rocksdb: Options.allow_mmap_writes: 0
-323> 2018-08-06 21:43:40.524658 7f4bc94f7ec0 4 rocksdb: Options.use_direct_reads: 0
-322> 2018-08-06 21:43:40.524658 7f4bc94f7ec0 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0
-321> 2018-08-06 21:43:40.524659 7f4bc94f7ec0 4 rocksdb: Options.create_missing_column_families: 0
-320> 2018-08-06 21:43:40.524660 7f4bc94f7ec0 4 rocksdb: Options.db_log_dir:
-319> 2018-08-06 21:43:40.524661 7f4bc94f7ec0 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-cephmon/store.db
-318> 2018-08-06 21:43:40.524662 7f4bc94f7ec0 4 rocksdb: Options.table_cache_numshardbits: 6
-317> 2018-08-06 21:43:40.524663 7f4bc94f7ec0 4 rocksdb: Options.max_subcompactions: 1
-316> 2018-08-06 21:43:40.524664 7f4bc94f7ec0 4 rocksdb: Options.max_background_flushes: 1
-315> 2018-08-06 21:43:40.524665 7f4bc94f7ec0 4 rocksdb: Options.WAL_ttl_seconds: 0
-314> 2018-08-06 21:43:40.524666 7f4bc94f7ec0 4 rocksdb: Options.WAL_size_limit_MB: 0
-313> 2018-08-06 21:43:40.524667 7f4bc94f7ec0 4 rocksdb: Options.manifest_preallocation_size: 4194304
-312> 2018-08-06 21:43:40.524668 7f4bc94f7ec0 4 rocksdb: Options.is_fd_close_on_exec: 1
-311> 2018-08-06 21:43:40.524669 7f4bc94f7ec0 4 rocksdb: Options.advise_random_on_open: 1
-310> 2018-08-06 21:43:40.524669 7f4bc94f7ec0 4 rocksdb: Options.db_write_buffer_size: 0
-309> 2018-08-06 21:43:40.524670 7f4bc94f7ec0 4 rocksdb: Options.access_hint_on_compaction_start: 1
-308> 2018-08-06 21:43:40.524671 7f4bc94f7ec0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0
-307> 2018-08-06 21:43:40.524672 7f4bc94f7ec0 4 rocksdb: Options.compaction_readahead_size: 0
-306> 2018-08-06 21:43:40.524672 7f4bc94f7ec0 4 rocksdb: Options.random_access_max_buffer_size: 1048576
-305> 2018-08-06 21:43:40.524673 7f4bc94f7ec0 4 rocksdb: Options.writable_file_max_buffer_size: 1048576
-304> 2018-08-06 21:43:40.524700 7f4bc94f7ec0 4 rocksdb: Options.use_adaptive_mutex: 0
-303> 2018-08-06 21:43:40.524702 7f4bc94f7ec0 4 rocksdb: Options.rate_limiter: (nil)
-302> 2018-08-06 21:43:40.524703 7f4bc94f7ec0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0
-301> 2018-08-06 21:43:40.524705 7f4bc94f7ec0 4 rocksdb: Options.bytes_per_sync: 0
-300> 2018-08-06 21:43:40.524706 7f4bc94f7ec0 4 rocksdb: Options.wal_bytes_per_sync: 0
-299> 2018-08-06 21:43:40.524706 7f4bc94f7ec0 4 rocksdb: Options.wal_recovery_mode: 2
-298> 2018-08-06 21:43:40.524707 7f4bc94f7ec0 4 rocksdb: Options.enable_thread_tracking: 0
-297> 2018-08-06 21:43:40.524708 7f4bc94f7ec0 4 rocksdb: Options.allow_concurrent_memtable_write: 1
-296> 2018-08-06 21:43:40.524709 7f4bc94f7ec0 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1
-295> 2018-08-06 21:43:40.524710 7f4bc94f7ec0 4 rocksdb: Options.write_thread_max_yield_usec: 100
-294> 2018-08-06 21:43:40.524718 7f4bc94f7ec0 4 rocksdb: Options.write_thread_slow_yield_usec: 3
-293> 2018-08-06 21:43:40.524719 7f4bc94f7ec0 4 rocksdb: Options.row_cache: None
-292> 2018-08-06 21:43:40.524720 7f4bc94f7ec0 4 rocksdb: Options.wal_filter: None
-291> 2018-08-06 21:43:40.524721 7f4bc94f7ec0 4 rocksdb: Options.avoid_flush_during_recovery: 0
-290> 2018-08-06 21:43:40.524722 7f4bc94f7ec0 4 rocksdb: Options.base_background_compactions: 1
-289> 2018-08-06 21:43:40.524723 7f4bc94f7ec0 4 rocksdb: Options.max_background_compactions: 1
-288> 2018-08-06 21:43:40.524724 7f4bc94f7ec0 4 rocksdb: Options.avoid_flush_during_shutdown: 0
-287> 2018-08-06 21:43:40.524725 7f4bc94f7ec0 4 rocksdb: Options.delayed_write_rate : 16777216
-286> 2018-08-06 21:43:40.524726 7f4bc94f7ec0 4 rocksdb: Options.max_total_wal_size: 0
-285> 2018-08-06 21:43:40.524727 7f4bc94f7ec0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000
-284> 2018-08-06 21:43:40.524728 7f4bc94f7ec0 4 rocksdb: Options.stats_dump_period_sec: 600
-283> 2018-08-06 21:43:40.524729 7f4bc94f7ec0 4 rocksdb: Compression algorithms supported:
-282> 2018-08-06 21:43:40.524732 7f4bc94f7ec0 4 rocksdb: Snappy supported: 0
-281> 2018-08-06 21:43:40.524734 7f4bc94f7ec0 4 rocksdb: Zlib supported: 0
-280> 2018-08-06 21:43:40.524734 7f4bc94f7ec0 4 rocksdb: Bzip supported: 0
-279> 2018-08-06 21:43:40.524735 7f4bc94f7ec0 4 rocksdb: LZ4 supported: 0
-278> 2018-08-06 21:43:40.524736 7f4bc94f7ec0 4 rocksdb: ZSTD supported: 0
-277> 2018-08-06 21:43:40.524740 7f4bc94f7ec0 4 rocksdb: Fast CRC32 supported: 1
-276> 2018-08-06 21:43:40.531242 7f4bc94f7ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5/rpm/el7/BUILD/ceph-12.2.5/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-093649
-275> 2018-08-06 21:43:40.532721 7f4bc94f7ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5/rpm/el7/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
-274> 2018-08-06 21:43:40.532744 7f4bc94f7ec0 4 rocksdb: Options.comparator: leveldb.BytewiseComparator
-273> 2018-08-06 21:43:40.532748 7f4bc94f7ec0 4 rocksdb: Options.merge_operator:
-272> 2018-08-06 21:43:40.532750 7f4bc94f7ec0 4 rocksdb: Options.compaction_filter: None
-271> 2018-08-06 21:43:40.532751 7f4bc94f7ec0 4 rocksdb: Options.compaction_filter_factory: None
-270> 2018-08-06 21:43:40.532753 7f4bc94f7ec0 4 rocksdb: Options.memtable_factory: SkipListFactory
-269> 2018-08-06 21:43:40.532754 7f4bc94f7ec0 4 rocksdb: Options.table_factory: BlockBasedTable
-268> 2018-08-06 21:43:40.532793 7f4bc94f7ec0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d71e768168)
cache_index_and_filter_blocks: 1
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 1
index_type: 0
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x55d71e825278
block_cache_name: LRUCache
block_cache_options:
capacity : 134217728
num_shard_bits : 4
strict_capacity_limit : 0
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
filter_policy: rocksdb.BuiltinBloomFilter
whole_key_filtering: 1
format_version: 2
-267> 2018-08-06 21:43:40.532807 7f4bc94f7ec0 4 rocksdb: Options.write_buffer_size: 33554432
-266> 2018-08-06 21:43:40.532809 7f4bc94f7ec0 4 rocksdb: Options.max_write_buffer_number: 2
-265> 2018-08-06 21:43:40.532811 7f4bc94f7ec0 4 rocksdb: Options.compression: NoCompression
-264> 2018-08-06 21:43:40.532813 7f4bc94f7ec0 4 rocksdb: Options.bottommost_compression: Disabled
-263> 2018-08-06 21:43:40.532815 7f4bc94f7ec0 4 rocksdb: Options.prefix_extractor: nullptr
-262> 2018-08-06 21:43:40.532826 7f4bc94f7ec0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr
-261> 2018-08-06 21:43:40.532828 7f4bc94f7ec0 4 rocksdb: Options.num_levels: 7
-260> 2018-08-06 21:43:40.532829 7f4bc94f7ec0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1
-259> 2018-08-06 21:43:40.532830 7f4bc94f7ec0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0
-258> 2018-08-06 21:43:40.532831 7f4bc94f7ec0 4 rocksdb: Options.compression_opts.window_bits: -14
-257> 2018-08-06 21:43:40.532833 7f4bc94f7ec0 4 rocksdb: Options.compression_opts.level: -1
-256> 2018-08-06 21:43:40.532834 7f4bc94f7ec0 4 rocksdb: Options.compression_opts.strategy: 0
-255> 2018-08-06 21:43:40.532836 7f4bc94f7ec0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0
-254> 2018-08-06 21:43:40.532837 7f4bc94f7ec0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4
-253> 2018-08-06 21:43:40.532838 7f4bc94f7ec0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20
-252> 2018-08-06 21:43:40.532839 7f4bc94f7ec0 4 rocksdb: Options.level0_stop_writes_trigger: 36
-251> 2018-08-06 21:43:40.532840 7f4bc94f7ec0 4 rocksdb: Options.target_file_size_base: 67108864
-250> 2018-08-06 21:43:40.532842 7f4bc94f7ec0 4 rocksdb: Options.target_file_size_multiplier: 1
-249> 2018-08-06 21:43:40.532843 7f4bc94f7ec0 4 rocksdb: Options.max_bytes_for_level_base: 268435456
-248> 2018-08-06 21:43:40.532845 7f4bc94f7ec0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
-247> 2018-08-06 21:43:40.532846 7f4bc94f7ec0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000
-246> 2018-08-06 21:43:40.532862 7f4bc94f7ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
-245> 2018-08-06 21:43:40.532864 7f4bc94f7ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
-244> 2018-08-06 21:43:40.532865 7f4bc94f7ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
-243> 2018-08-06 21:43:40.532869 7f4bc94f7ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
-242> 2018-08-06 21:43:40.532871 7f4bc94f7ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
-241> 2018-08-06 21:43:40.532872 7f4bc94f7ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
-240> 2018-08-06 21:43:40.532873 7f4bc94f7ec0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
-239> 2018-08-06 21:43:40.532874 7f4bc94f7ec0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8
-238> 2018-08-06 21:43:40.532876 7f4bc94f7ec0 4 rocksdb: Options.max_compaction_bytes: 1677721600
-237> 2018-08-06 21:43:40.532877 7f4bc94f7ec0 4 rocksdb: Options.arena_block_size: 4194304
-236> 2018-08-06 21:43:40.532878 7f4bc94f7ec0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736
-235> 2018-08-06 21:43:40.532880 7f4bc94f7ec0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944
-234> 2018-08-06 21:43:40.532881 7f4bc94f7ec0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100
-233> 2018-08-06 21:43:40.532882 7f4bc94f7ec0 4 rocksdb: Options.disable_auto_compactions: 0
-232> 2018-08-06 21:43:40.532884 7f4bc94f7ec0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel
-231> 2018-08-06 21:43:40.532886 7f4bc94f7ec0 4 rocksdb: Options.compaction_pri: kByCompensatedSize
-230> 2018-08-06 21:43:40.532888 7f4bc94f7ec0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1
-229> 2018-08-06 21:43:40.532889 7f4bc94f7ec0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
-228> 2018-08-06 21:43:40.532891 7f4bc94f7ec0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
-227> 2018-08-06 21:43:40.532892 7f4bc94f7ec0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
-226> 2018-08-06 21:43:40.532900 7f4bc94f7ec0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
-225> 2018-08-06 21:43:40.532901 7f4bc94f7ec0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
-224> 2018-08-06 21:43:40.532903 7f4bc94f7ec0 4 rocksdb: Options.table_properties_collectors:
-223> 2018-08-06 21:43:40.532904 7f4bc94f7ec0 4 rocksdb: Options.inplace_update_support: 0
-222> 2018-08-06 21:43:40.532905 7f4bc94f7ec0 4 rocksdb: Options.inplace_update_num_locks: 10000
-221> 2018-08-06 21:43:40.532907 7f4bc94f7ec0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000
-220> 2018-08-06 21:43:40.532909 7f4bc94f7ec0 4 rocksdb: Options.memtable_huge_page_size: 0
-219> 2018-08-06 21:43:40.532911 7f4bc94f7ec0 4 rocksdb: Options.bloom_locality: 0
-218> 2018-08-06 21:43:40.532912 7f4bc94f7ec0 4 rocksdb: Options.max_successive_merges: 0
-217> 2018-08-06 21:43:40.532913 7f4bc94f7ec0 4 rocksdb: Options.optimize_filters_for_hits: 0
-216> 2018-08-06 21:43:40.532914 7f4bc94f7ec0 4 rocksdb: Options.paranoid_file_checks: 0
-215> 2018-08-06 21:43:40.532924 7f4bc94f7ec0 4 rocksdb: Options.force_consistency_checks: 0
-214> 2018-08-06 21:43:40.532925 7f4bc94f7ec0 4 rocksdb: Options.report_bg_io_stats: 0
-213> 2018-08-06 21:43:40.546977 7f4bc94f7ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5/rpm/el7/BUILD/ceph-12.2.5/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/ceph/mon/ceph-cephmon/store.db/MANIFEST-093649 succeeded,manifest_file_number is 93649, next_file_number is 93651, last_sequence is 32923615, log_number is 0,prev_log_number is 0,max_column_family is 0
-212> 2018-08-06 21:43:40.547000 7f4bc94f7ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5/rpm/el7/BUILD/ceph-12.2.5/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 93648
-211> 2018-08-06 21:43:40.551641 7f4bc94f7ec0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1533563020551629, "job": 1, "event": "recovery_started", "log_files": [93650]}
-210> 2018-08-06 21:43:40.551652 7f4bc94f7ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5/rpm/el7/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_open.cc:482] Recovering log #93650 mode 2
-209> 2018-08-06 21:43:40.554398 7f4bc94f7ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5/rpm/el7/BUILD/ceph-12.2.5/src/rocksdb/db/version_set.cc:2395] Creating manifest 93652
-208> 2018-08-06 21:43:40.581403 7f4bc94f7ec0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1533563020581400, "job": 1, "event": "recovery_finished"}
-207> 2018-08-06 21:43:40.591342 7f4bc94f7ec0 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5/rpm/el7/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete /var/lib/ceph/mon/ceph-cephmon/store.db//MANIFEST-093649 type=3 #93649 -- OK
-206> 2018-08-06 21:43:40.594297 7f4bc94f7ec0 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5/rpm/el7/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 2] Delete /var/lib/ceph/mon/ceph-cephmon/store.db//093650.log type=0 #93650 -- OK
-205> 2018-08-06 21:43:40.622058 7f4bc94f7ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.5/rpm/el7/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x55d71eb04000
-204> 2018-08-06 21:43:40.622526 7f4bc94f7ec0 10 obtain_monmap
-203> 2018-08-06 21:43:41.903421 7f4bc94f7ec0 10 obtain_monmap read last committed monmap ver 1
-202> 2018-08-06 21:43:41.903816 7f4bbb91a700 2 Event(0x55d71ea12880 nevent=5000 time_id=1).set_owner idx=0 owner=139963246159616
-201> 2018-08-06 21:43:41.903900 7f4bbc11b700 2 Event(0x55d71ea0dc80 nevent=5000 time_id=1).set_owner idx=1 owner=139963254552320
-200> 2018-08-06 21:43:41.903929 7f4bbc91c700 2 Event(0x55d71ea13680 nevent=5000 time_id=1).set_owner idx=2 owner=139963262945024
-199> 2018-08-06 21:43:41.904038 7f4bc94f7ec0 0 starting mon.cephmon rank 6 at public addr 192.168.253.242:6789/0 at bind addr 192.168.253.242:6789/0 mon_data /var/lib/ceph/mon/ceph-cephmon fsid 484eca8f-c17b-41fd-83eb-314052ccd8ea
-198> 2018-08-06 21:43:41.904249 7f4bc94f7ec0 1 -- 192.168.253.242:6789/0 learned_addr learned my addr 192.168.253.242:6789/0
-197> 2018-08-06 21:43:41.904263 7f4bc94f7ec0 1 -- 192.168.253.242:6789/0 _finish_bind bind my_inst.addr is 192.168.253.242:6789/0
-196> 2018-08-06 21:43:41.904286 7f4bc94f7ec0 0 starting mon.cephmon rank 6 at 192.168.253.242:6789/0 mon_data /var/lib/ceph/mon/ceph-cephmon fsid 484eca8f-c17b-41fd-83eb-314052ccd8ea
-195> 2018-08-06 21:43:41.904313 7f4bc94f7ec0 5 adding auth protocol: cephx
-194> 2018-08-06 21:43:41.904316 7f4bc94f7ec0 5 adding auth protocol: cephx
-193> 2018-08-06 21:43:41.904353 7f4bc94f7ec0 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
-192> 2018-08-06 21:43:41.904357 7f4bc94f7ec0 10 log_channel(audit) update_config to_monitors: true to_syslog: false syslog_facility: local0 prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
-191> 2018-08-06 21:43:41.904738 7f4bc94f7ec0 1 mon.cephmon@-1(probing) e1 preinit fsid 484eca8f-c17b-41fd-83eb-314052ccd8ea
-190> 2018-08-06 21:43:41.904799 7f4bc94f7ec0 10 mon.cephmon@-1(probing) e1 check_fsid cluster_uuid contains '484eca8f-c17b-41fd-83eb-314052ccd8ea'
-189> 2018-08-06 21:43:41.904846 7f4bc94f7ec0 10 mon.cephmon@-1(probing) e1 features compat={},rocompat={},incompat={1=initial feature set (~v.18),3=single paxos with k/v store (v0.?),4=support erasure code pools,5=new-style osdmap encoding,6=support isa/lrc erasure code,7=support shec erasure code,8=support monmap features,9=luminous ondisk layout}
-188> 2018-08-06 21:43:41.904869 7f4bc94f7ec0 10 mon.cephmon@-1(probing) e1 calc_quorum_requirements required_features 153140804152475648
-187> 2018-08-06 21:43:41.904872 7f4bc94f7ec0 10 mon.cephmon@-1(probing) e1 required_features 153140804152475648
-186> 2018-08-06 21:43:41.904879 7f4bc94f7ec0 10 mon.cephmon@-1(probing) e1 has_ever_joined = 1
-185> 2018-08-06 21:43:41.931861 7f4bc94f7ec0 10 mon.cephmon@-1(probing) e1 sync_last_committed_floor 0
-184> 2018-08-06 21:43:41.931900 7f4bc94f7ec0 10 mon.cephmon@-1(probing) e1 init_paxos
-183> 2018-08-06 21:43:41.933581 7f4bc94f7ec0 1 mon.cephmon@-1(probing).mds e0 Unable to load 'last_metadata'
-182> 2018-08-06 21:43:41.933632 7f4bc94f7ec0 10 mon.cephmon@-1(probing).health init
-181> 2018-08-06 21:43:41.933637 7f4bc94f7ec0 10 mon.cephmon@-1(probing) e1 refresh_from_paxos
-180> 2018-08-06 21:43:41.933662 7f4bc94f7ec0 1 mon.cephmon@-1(probing).paxosservice(pgmap 1..2) refresh upgraded, format 0 -> 1
-179> 2018-08-06 21:43:41.933668 7f4bc94f7ec0 1 mon.cephmon@-1(probing).pg v0 on_upgrade discarding in-core PGMap
-178> 2018-08-06 21:43:41.933694 7f4bc94f7ec0 10 mon.cephmon@-1(probing).pg v0 update_from_paxos deleted, clearing in-memory PGMap
-177> 2018-08-06 21:43:41.933719 7f4bc94f7ec0 10 mon.cephmon@-1(probing).mds e0 update_from_paxos version 1, my e 0
-176> 2018-08-06 21:43:41.933748 7f4bc94f7ec0 10 mon.cephmon@-1(probing).mds e0 update_from_paxos got 1
-175> 2018-08-06 21:43:41.933764 7f4bc94f7ec0 4 mon.cephmon@-1(probing).mds e1 new map
-174> 2018-08-06 21:43:41.933765 7f4bc94f7ec0 0 mon.cephmon@-1(probing).mds e1 print_map
e1
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2}
legacy client fscid: -1
No filesystems configured
-173> 2018-08-06 21:43:41.933787 7f4bc94f7ec0 10 mon.cephmon@-1(probing).mds e1 update_logger
-172> 2018-08-06 21:43:41.933876 7f4bc94f7ec0 15 mon.cephmon@-1(probing).osd e0 update_from_paxos paxos e 6337, my e 0
-171> 2018-08-06 21:43:41.933943 7f4bc94f7ec0 7 mon.cephmon@-1(probing).osd e0 update_from_paxos loading latest full map e6337
-170> 2018-08-06 21:43:41.934383 7f4bc94f7ec0 7 mon.cephmon@-1(probing).osd e6337 update_from_paxos loading creating_pgs last_scan_epoch 6336 with 0 pgs
-169> 2018-08-06 21:43:41.934388 7f4bc94f7ec0 10 mon.cephmon@-1(probing).osd e6337 update_from_paxos pgservice is mgrstat
-168> 2018-08-06 21:43:41.934392 7f4bc94f7ec0 10 mon.cephmon@-1(probing).osd e6337 check_osdmap_subs
-167> 2018-08-06 21:43:41.934399 7f4bc94f7ec0 10 mon.cephmon@-1(probing).osd e6337 share_map_with_random_osd no up osd on our session map
-166> 2018-08-06 21:43:41.934403 7f4bc94f7ec0 10 mon.cephmon@-1(probing).osd e6337 update_logger
-165> 2018-08-06 21:43:41.934411 7f4bc94f7ec0 0 mon.cephmon@-1(probing).osd e6337 crush map has features 288514051259236352, adjusting msgr requires
-164> 2018-08-06 21:43:41.934415 7f4bc94f7ec0 0 mon.cephmon@-1(probing).osd e6337 crush map has features 288514051259236352, adjusting msgr requires
-163> 2018-08-06 21:43:41.934418 7f4bc94f7ec0 0 mon.cephmon@-1(probing).osd e6337 crush map has features 1009089991638532096, adjusting msgr requires
-162> 2018-08-06 21:43:41.934420 7f4bc94f7ec0 0 mon.cephmon@-1(probing).osd e6337 crush map has features 288514051259236352, adjusting msgr requires
-161> 2018-08-06 21:43:41.934448 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 85 [0,256)
-160> 2018-08-06 21:43:41.934466 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 89 [0,32)
-159> 2018-08-06 21:43:41.934467 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 90 [0,32)
-158> 2018-08-06 21:43:41.934468 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 91 [0,32)
-157> 2018-08-06 21:43:41.934469 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 92 [0,128)
-156> 2018-08-06 21:43:41.934470 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 93 [0,32)
-155> 2018-08-06 21:43:41.934470 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 94 [0,64)
-154> 2018-08-06 21:43:41.934471 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 95 [0,32)
-153> 2018-08-06 21:43:41.934472 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 96 [0,32)
-152> 2018-08-06 21:43:41.934474 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 97 [0,32)
-151> 2018-08-06 21:43:41.934475 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 98 [0,32)
-150> 2018-08-06 21:43:41.934476 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 99 [0,32)
-149> 2018-08-06 21:43:41.934476 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 100 [0,32)
-148> 2018-08-06 21:43:41.934477 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 101 [0,32)
-147> 2018-08-06 21:43:41.934478 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 102 [0,32)
-146> 2018-08-06 21:43:41.934479 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 103 [0,32)
-145> 2018-08-06 21:43:41.934480 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 104 [0,128)
-144> 2018-08-06 21:43:41.934480 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 105 [0,1024)
-143> 2018-08-06 21:43:41.934481 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 106 [0,256)
-142> 2018-08-06 21:43:41.934482 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 107 [0,128)
-141> 2018-08-06 21:43:41.934483 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 108 [0,128)
-140> 2018-08-06 21:43:41.934484 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 109 [0,128)
-139> 2018-08-06 21:43:41.934485 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 110 [0,128)
-138> 2018-08-06 21:43:41.934485 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 111 [0,16)
-137> 2018-08-06 21:43:41.934486 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 112 [0,16)
-136> 2018-08-06 21:43:41.934487 7f4bc94f7ec0 20 queue 0x55d71ec8d6c0 113 [0,100)
-135> 2018-08-06 21:43:41.934487 7f4bc94f7ec0 10 mon.cephmon@-1(probing).osd e6337 start_mapping started mapping job 0x55d71ec8d6c0 at 2018-08-06 21:43:41.934422
-134> 2018-08-06 21:43:41.934571 7f4bc94f7ec0 10 mon.cephmon@-1(probing).log v370257 update_from_paxos
-133> 2018-08-06 21:43:41.934591 7f4bc94f7ec0 10 mon.cephmon@-1(probing).log v370257 update_from_paxos version 370257 summary v 0
-132> 2018-08-06 21:43:41.934596 7f4bc94f7ec0 10 mon.cephmon@-1(probing).log v370257 update_from_paxos latest full 370256
-131> 2018-08-06 21:43:41.934615 7f4bc94f7ec0 7 mon.cephmon@-1(probing).log v370257 update_from_paxos loading summary e370256
-130> 2018-08-06 21:43:41.934700 7f4bc94f7ec0 7 mon.cephmon@-1(probing).log v370257 update_from_paxos loaded summary e370256
-129> 2018-08-06 21:43:41.934716 7f4bc94f7ec0 7 mon.cephmon@-1(probing).log v370257 update_from_paxos applying incremental log 370257 2018-08-06 13:19:33.722976 mon.cephmon mon.6 192.168.253.242:6789/0 138 : audit [INF] from='client.? 192.168.253.242:0/4111570017' entity='client.admin' cmd=[{"var": "size", "prefix": "osd pool set", "pool": "moc_test", "val": "1"}]: dispatch
-128> 2018-08-06 21:43:41.934737 7f4bc94f7ec0 20 mon.cephmon@-1(probing).log v370257 update_from_paxos logging for channel 'audit' to file '/var/log/ceph/ceph.audit.log'
-127> 2018-08-06 21:43:41.934752 7f4bc94f7ec0 7 mon.cephmon@-1(probing).log v370257 update_from_paxos applying incremental log 370257 2018-08-06 13:19:33.725952 mon.storage01moc mon.0 192.168.253.163:6789/0 3071 : audit [INF] from='client.1785433 -' entity='client.admin' cmd=[{"var": "size", "prefix": "osd pool set", "pool": "moc_test", "val": "1"}]: dispatch
-126> 2018-08-06 21:43:41.934758 7f4bc94f7ec0 20 mon.cephmon@-1(probing).log v370257 update_from_paxos logging for channel 'audit' to file '/var/log/ceph/ceph.audit.log'
-125> 2018-08-06 21:43:41.934766 7f4bc94f7ec0 7 mon.cephmon@-1(probing).log v370257 update_from_paxos applying incremental log 370257 2018-08-06 13:19:33.931419 mon.storage03roc mon.5 192.168.253.205:6789/0 106 : audit [DBG] from='client.? 192.168.253.203:0/2060363278' entity='client.bootstrap-osd' cmd=[{"prefix": "status", "format": "json"}]: dispatch
-124> 2018-08-06 21:43:41.934772 7f4bc94f7ec0 20 mon.cephmon@-1(probing).log v370257 update_from_paxos logging for channel 'audit' to file '/var/log/ceph/ceph.audit.log'
-123> 2018-08-06 21:43:41.934777 7f4bc94f7ec0 7 mon.cephmon@-1(probing).log v370257 update_from_paxos applying incremental log 370257 2018-08-06 13:19:34.543738 mon.storage03moc mon.2 192.168.253.165:6789/0 204 : audit [DBG] from='client.? 192.168.253.39:0/2016799470' entity='client.openstack' cmd=[{,",p,r,e,f,i,x,",:,",d,f,",,, ,",f,o,r,m,a,t,",:,",j,s,o,n,",}]: dispatch
-122> 2018-08-06 21:43:41.934783 7f4bc94f7ec0 20 mon.cephmon@-1(probing).log v370257 update_from_paxos logging for channel 'audit' to file '/var/log/ceph/ceph.audit.log'
-121> 2018-08-06 21:43:41.934787 7f4bc94f7ec0 15 mon.cephmon@-1(probing).log v370257 update_from_paxos logging for 1 channels
-120> 2018-08-06 21:43:41.934788 7f4bc94f7ec0 15 mon.cephmon@-1(probing).log v370257 update_from_paxos channel 'audit' logging 471 bytes
-119> 2018-08-06 21:43:41.934841 7f4bc94f7ec0 10 mon.cephmon@-1(probing).log v370257 check_subs
-118> 2018-08-06 21:43:41.934945 7f4bc94f7ec0 1 mon.cephmon@-1(probing).paxosservice(auth 1751..1973) refresh upgraded, format 0 -> 2
-117> 2018-08-06 21:43:41.934950 7f4bc94f7ec0 10 mon.cephmon@-1(probing).auth v1973 update_from_paxos
-116> 2018-08-06 21:43:41.934957 7f4bc94f7ec0 10 mon.cephmon@-1(probing).auth v1973 update_from_paxos version 1973 keys ver 0 latest 1961
-115> 2018-08-06 21:43:41.934972 7f4bc94f7ec0 7 mon.cephmon@-1(probing).auth v1973 update_from_paxos loading summary e 1961
-114> 2018-08-06 21:43:41.934973 7f4bc94f7ec0 7 mon.cephmon@-1(probing).auth v1973 update_from_paxos latest length 10146
-113> 2018-08-06 21:43:41.935779 7f4bc94f7ec0 10 mon.cephmon@-1(probing).auth v1973 update_from_paxos key server version 1961
-112> 2018-08-06 21:43:41.935809 7f4bc94f7ec0 20 mon.cephmon@-1(probing).auth v1973 update_from_paxos walking through version 1962 len 58
-111> 2018-08-06 21:43:41.935843 7f4bc94f7ec0 20 mon.cephmon@-1(probing).auth v1973 update_from_paxos walking through version 1963 len 801
-110> 2018-08-06 21:43:41.935955 7f4bc94f7ec0 20 mon.cephmon@-1(probing).auth v1973 update_from_paxos walking through version 1964 len 801
-109> 2018-08-06 21:43:41.936123 7f4bc94f7ec0 20 mon.cephmon@-1(probing).auth v1973 update_from_paxos walking through version 1965 len 19
-108> 2018-08-06 21:43:41.936143 7f4bc94f7ec0 20 mon.cephmon@-1(probing).auth v1973 update_from_paxos walking through version 1966 len 801
-107> 2018-08-06 21:43:41.936251 7f4bc94f7ec0 20 mon.cephmon@-1(probing).auth v1973 update_from_paxos walking through version 1967 len 19
-106> 2018-08-06 21:43:41.936260 7f4bc94f7ec0 20 mon.cephmon@-1(probing).auth v1973 update_from_paxos walking through version 1968 len 801
-105> 2018-08-06 21:43:41.936360 7f4bc94f7ec0 20 mon.cephmon@-1(probing).auth v1973 update_from_paxos walking through version 1969 len 19
-104> 2018-08-06 21:43:41.936371 7f4bc94f7ec0 20 mon.cephmon@-1(probing).auth v1973 update_from_paxos walking through version 1970 len 19
-103> 2018-08-06 21:43:41.936375 7f4bc94f7ec0 20 mon.cephmon@-1(probing).auth v1973 update_from_paxos walking through version 1971 len 19
-102> 2018-08-06 21:43:41.936380 7f4bc94f7ec0 20 mon.cephmon@-1(probing).auth v1973 update_from_paxos walking through version 1972 len 19
-101> 2018-08-06 21:43:41.936385 7f4bc94f7ec0 20 mon.cephmon@-1(probing).auth v1973 update_from_paxos walking through version 1973 len 801
-100> 2018-08-06 21:43:41.936469 7f4bc94f7ec0 10 mon.cephmon@-1(probing).auth v1973 update_from_paxos() last_allocated_id=1834096 max_global_id=1834096 format_version 2
-99> 2018-08-06 21:43:41.936505 7f4bc94f7ec0 4 mon.cephmon@-1(probing).mgr e0 loading version 1
-98> 2018-08-06 21:43:41.936520 7f4bc94f7ec0 4 mon.cephmon@-1(probing).mgr e1 active server: -(0)
-97> 2018-08-06 21:43:41.936530 7f4bc94f7ec0 4 mon.cephmon@-1(probing).mgr e1 mkfs or daemon transitioned to available, loading commands
-96> 2018-08-06 21:43:41.936611 7f4bc94f7ec0 4 mgrc handle_mgr_map Got map version 1
-95> 2018-08-06 21:43:41.936622 7f4bc94f7ec0 4 mgrc handle_mgr_map Active mgr is now -
-94> 2018-08-06 21:43:41.936625 7f4bc94f7ec0 4 mgrc reconnect No active mgr available yet
-93> 2018-08-06 21:43:41.936639 7f4bc94f7ec0 10 mon.cephmon@-1(probing).mgrstat 2821726
-92> 2018-08-06 21:43:41.936749 7f4bc94f7ec0 10 mon.cephmon@-1(probing).mgrstat update_from_paxos v2821726 service_map e1
-91> 2018-08-06 21:43:41.936753 7f4bc94f7ec0 10 mon.cephmon@-1(probing).mgrstat check_subs
-90> 2018-08-06 21:43:41.936755 7f4bc94f7ec0 20 mon.cephmon@-1(probing).mgrstat update_logger
-89> 2018-08-06 21:43:41.936785 7f4bc94f7ec0 10 mon.cephmon@-1(probing).health update_from_paxos
-88> 2018-08-06 21:43:41.936801 7f4bc94f7ec0 20 mon.cephmon@-1(probing).health dump:{
"quorum_health": {},
"leader_health": {}
}
-87> 2018-08-06 21:43:41.936887 7f4bc94f7ec0 10 mon.cephmon@-1(probing).health(0) init
-86> 2018-08-06 21:43:41.942165 7f4bc94f7ec0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/ceph-cephmon/keyring
-85> 2018-08-06 21:43:41.942177 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command mon_status hook 0x55d71e76ab40
-84> 2018-08-06 21:43:41.942182 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command quorum_status hook 0x55d71e76ab40
-83> 2018-08-06 21:43:41.942185 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command sync_force hook 0x55d71e76ab40
-82> 2018-08-06 21:43:41.942187 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command add_bootstrap_peer_hint hook 0x55d71e76ab40
-81> 2018-08-06 21:43:41.942189 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command quorum enter hook 0x55d71e76ab40
-80> 2018-08-06 21:43:41.942191 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command quorum exit hook 0x55d71e76ab40
-79> 2018-08-06 21:43:41.942194 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command ops hook 0x55d71e76ab40
-78> 2018-08-06 21:43:41.942198 7f4bc94f7ec0 5 asok(0x55d71e7b81c0) register_command sessions hook 0x55d71e76ab40
-77> 2018-08-06 21:43:41.942225 7f4bc94f7ec0 1 -- 192.168.253.242:6789/0 start start
-76> 2018-08-06 21:43:41.942255 7f4bc94f7ec0 1 -- - start start
-75> 2018-08-06 21:43:41.942258 7f4bc94f7ec0 2 mon.cephmon@-1(probing) e1 init
-74> 2018-08-06 21:43:41.942389 7f4bc94f7ec0 1 Processor -- start
-73> 2018-08-06 21:43:41.942513 7f4bc1926700 20 _process 0x55d71ec8d6c0 85 [0,256)
-72> 2018-08-06 21:43:41.942534 7f4bc0924700 20 _process 0x55d71ec8d6c0 89 [0,32)
-71> 2018-08-06 21:43:41.942910 7f4bc0924700 20 _process 0x55d71ec8d6c0 90 [0,32)
-70> 2018-08-06 21:43:41.943241 7f4bc0924700 20 _process 0x55d71ec8d6c0 91 [0,32)
-69> 2018-08-06 21:43:41.943570 7f4bc0924700 20 _process 0x55d71ec8d6c0 92 [0,128)
-68> 2018-08-06 21:43:41.943980 7f4bc1926700 20 _process 0x55d71ec8d6c0 93 [0,32)
-67> 2018-08-06 21:43:41.944171 7f4bc1926700 20 _process 0x55d71ec8d6c0 94 [0,64)
-66> 2018-08-06 21:43:41.944508 7f4bc1926700 20 _process 0x55d71ec8d6c0 95 [0,32)
-65> 2018-08-06 21:43:41.944674 7f4bc1926700 20 _process 0x55d71ec8d6c0 96 [0,32)
-64> 2018-08-06 21:43:41.944850 7f4bc0924700 20 _process 0x55d71ec8d6c0 97 [0,32)
-63> 2018-08-06 21:43:41.944868 7f4bc1926700 20 _process 0x55d71ec8d6c0 98 [0,32)
-62> 2018-08-06 21:43:41.945018 7f4bc1926700 20 _process 0x55d71ec8d6c0 99 [0,32)
-61> 2018-08-06 21:43:41.945191 7f4bc0924700 20 _process 0x55d71ec8d6c0 100 [0,32)
-60> 2018-08-06 21:43:41.945212 7f4bc1926700 20 _process 0x55d71ec8d6c0 101 [0,32)
-59> 2018-08-06 21:43:41.945388 7f4bc1926700 20 _process 0x55d71ec8d6c0 102 [0,32)
-58> 2018-08-06 21:43:41.945515 7f4bc0924700 20 _process 0x55d71ec8d6c0 103 [0,32)
-57> 2018-08-06 21:43:41.945560 7f4bc1926700 20 _process 0x55d71ec8d6c0 104 [0,128)
-56> 2018-08-06 21:43:41.945848 7f4bc0924700 20 _process 0x55d71ec8d6c0 105 [0,1024)
-55> 2018-08-06 21:43:41.946199 7f4bc1926700 20 _process 0x55d71ec8d6c0 106 [0,256)
-54> 2018-08-06 21:43:41.947564 7f4bc1926700 20 _process 0x55d71ec8d6c0 107 [0,128)
-53> 2018-08-06 21:43:41.948245 7f4bc1926700 20 _process 0x55d71ec8d6c0 108 [0,128)
-52> 2018-08-06 21:43:41.948898 7f4bc1926700 20 _process 0x55d71ec8d6c0 109 [0,128)
-51> 2018-08-06 21:43:41.949567 7f4bc1926700 20 _process 0x55d71ec8d6c0 110 [0,128)
-50> 2018-08-06 21:43:41.950224 7f4bc1926700 20 _process 0x55d71ec8d6c0 111 [0,16)
-49> 2018-08-06 21:43:41.950246 7f4bc1926700 20 _process 0x55d71ec8d6c0 112 [0,16)
-48> 2018-08-06 21:43:41.950323 7f4bc1926700 20 _process 0x55d71ec8d6c0 113 [0,100)
-47> 2018-08-06 21:43:41.950909 7f4bc94f7ec0 1 Processor -- start
-46> 2018-08-06 21:43:41.950939 7f4bc94f7ec0 10 mon.cephmon@-1(probing) e1 bootstrap
-45> 2018-08-06 21:43:41.950942 7f4bc94f7ec0 10 mon.cephmon@-1(probing) e1 sync_reset_requester
-44> 2018-08-06 21:43:41.950943 7f4bc94f7ec0 10 mon.cephmon@-1(probing) e1 unregister_cluster_logger - not registered
-43> 2018-08-06 21:43:41.950944 7f4bc94f7ec0 10 mon.cephmon@-1(probing) e1 cancel_probe_timeout (none scheduled)
-42> 2018-08-06 21:43:41.950946 7f4bc94f7ec0 0 mon.cephmon@-1(probing) e1 my rank is now 6 (was -1)
-41> 2018-08-06 21:43:41.950948 7f4bc94f7ec0 1 -- 192.168.253.242:6789/0 shutdown_connections
-40> 2018-08-06 21:43:41.950955 7f4bc94f7ec0 10 mon.cephmon@6(probing) e1 _reset
-39> 2018-08-06 21:43:41.950955 7f4bc94f7ec0 10 mon.cephmon@6(probing) e1 cancel_probe_timeout (none scheduled)
-38> 2018-08-06 21:43:41.950956 7f4bc94f7ec0 10 mon.cephmon@6(probing) e1 timecheck_finish
-37> 2018-08-06 21:43:41.950959 7f4bc94f7ec0 15 mon.cephmon@6(probing) e1 health_tick_stop
-36> 2018-08-06 21:43:41.950960 7f4bc94f7ec0 15 mon.cephmon@6(probing) e1 health_interval_stop
-35> 2018-08-06 21:43:41.950961 7f4bc94f7ec0 10 mon.cephmon@6(probing) e1 scrub_event_cancel
-34> 2018-08-06 21:43:41.950964 7f4bc94f7ec0 10 mon.cephmon@6(probing) e1 scrub_reset
-33> 2018-08-06 21:43:41.950973 7f4bc94f7ec0 10 mon.cephmon@6(probing) e1 cancel_probe_timeout (none scheduled)
-32> 2018-08-06 21:43:41.950978 7f4bc94f7ec0 10 mon.cephmon@6(probing) e1 reset_probe_timeout 0x55d71ed1f7a0 after 2 seconds
-31> 2018-08-06 21:43:41.950995 7f4bc94f7ec0 10 mon.cephmon@6(probing) e1 probing other monitors
-30> 2018-08-06 21:43:41.950999 7f4bc94f7ec0 1 -- 192.168.253.242:6789/0 _send_message--> mon.0 192.168.253.163:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name cephmon) v6 -- ?+0 0x55d71ed18f00
-29> 2018-08-06 21:43:41.951023 7f4bc94f7ec0 1 -- 192.168.253.242:6789/0 --> 192.168.253.163:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name cephmon) v6 -- 0x55d71ed18f00 con 0
-28> 2018-08-06 21:43:41.951032 7f4bc94f7ec0 1 -- 192.168.253.242:6789/0 _send_message--> mon.1 192.168.253.164:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name cephmon) v6 -- ?+0 0x55d71ed19180
-27> 2018-08-06 21:43:41.951048 7f4bc94f7ec0 1 -- 192.168.253.242:6789/0 --> 192.168.253.164:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name cephmon) v6 -- 0x55d71ed19180 con 0
-26> 2018-08-06 21:43:41.951054 7f4bc94f7ec0 1 -- 192.168.253.242:6789/0 _send_message--> mon.2 192.168.253.165:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name cephmon) v6 -- ?+0 0x55d71ed19400
-25> 2018-08-06 21:43:41.951065 7f4bc94f7ec0 1 -- 192.168.253.242:6789/0 --> 192.168.253.165:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name cephmon) v6 -- 0x55d71ed19400 con 0
-24> 2018-08-06 21:43:41.951069 7f4bc94f7ec0 1 -- 192.168.253.242:6789/0 _send_message--> mon.3 192.168.253.203:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name cephmon) v6 -- ?+0 0x55d71ed19680
-23> 2018-08-06 21:43:41.951074 7f4bc94f7ec0 1 -- 192.168.253.242:6789/0 --> 192.168.253.203:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name cephmon) v6 -- 0x55d71ed19680 con 0
-22> 2018-08-06 21:43:41.951086 7f4bc94f7ec0 1 -- 192.168.253.242:6789/0 _send_message--> mon.4 192.168.253.204:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name cephmon) v6 -- ?+0 0x55d71ed18c80
-21> 2018-08-06 21:43:41.951092 7f4bc94f7ec0 1 -- 192.168.253.242:6789/0 --> 192.168.253.204:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name cephmon) v6 -- 0x55d71ed18c80 con 0
-20> 2018-08-06 21:43:41.951095 7f4bc94f7ec0 1 -- 192.168.253.242:6789/0 _send_message--> mon.5 192.168.253.205:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name cephmon) v6 -- ?+0 0x55d71ed18a00
-19> 2018-08-06 21:43:41.951101 7f4bc94f7ec0 1 -- 192.168.253.242:6789/0 --> 192.168.253.205:6789/0 -- mon_probe(probe 484eca8f-c17b-41fd-83eb-314052ccd8ea name cephmon) v6 -- 0x55d71ed18a00 con 0
-18> 2018-08-06 21:43:41.951528 7f4bbb91a700 1 -- 192.168.253.242:6789/0 >> 192.168.253.164:6789/0 conn(0x55d71eb2c000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed
-17> 2018-08-06 21:43:41.951552 7f4bbb91a700 2 -- 192.168.253.242:6789/0 >> 192.168.253.164:6789/0 conn(0x55d71eb2c000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused!
-16> 2018-08-06 21:43:41.951594 7f4bbc11b700 1 -- 192.168.253.242:6789/0 >> 192.168.253.165:6789/0 conn(0x55d71eb2d800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed
-15> 2018-08-06 21:43:41.951601 7f4bbc11b700 2 -- 192.168.253.242:6789/0 >> 192.168.253.165:6789/0 conn(0x55d71eb2d800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused!
-14> 2018-08-06 21:43:41.951618 7f4bbc91c700 1 -- 192.168.253.242:6789/0 >> 192.168.253.163:6789/0 conn(0x55d71ee04800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed
-13> 2018-08-06 21:43:41.951624 7f4bbc91c700 2 -- 192.168.253.242:6789/0 >> 192.168.253.163:6789/0 conn(0x55d71ee04800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused!
-12> 2018-08-06 21:43:41.951645 7f4bbf922700 10 mon.cephmon@6(probing) e1 ms_handle_refused 0x55d71eb2c000 192.168.253.164:6789/0
-11> 2018-08-06 21:43:41.951651 7f4bbf922700 10 mon.cephmon@6(probing) e1 ms_handle_refused 0x55d71eb2d800 192.168.253.165:6789/0
-10> 2018-08-06 21:43:41.951654 7f4bbf922700 10 mon.cephmon@6(probing) e1 ms_handle_refused 0x55d71ee04800 192.168.253.163:6789/0
-9> 2018-08-06 21:43:41.952006 7f4bbb91a700 1 -- 192.168.253.242:6789/0 >> 192.168.253.204:6789/0 conn(0x55d71eb30800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed
-8> 2018-08-06 21:43:41.952053 7f4bbb91a700 2 -- 192.168.253.242:6789/0 >> 192.168.253.204:6789/0 conn(0x55d71eb30800 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused!
-7> 2018-08-06 21:43:41.952088 7f4bbc91c700 1 -- 192.168.253.242:6789/0 >> 192.168.253.203:6789/0 conn(0x55d71eb2f000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed
-6> 2018-08-06 21:43:41.952092 7f4bbc91c700 2 -- 192.168.253.242:6789/0 >> 192.168.253.203:6789/0 conn(0x55d71eb2f000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused!
-5> 2018-08-06 21:43:41.952107 7f4bbf922700 10 mon.cephmon@6(probing) e1 ms_handle_refused 0x55d71eb30800 192.168.253.204:6789/0
-4> 2018-08-06 21:43:41.952112 7f4bbf922700 10 mon.cephmon@6(probing) e1 ms_handle_refused 0x55d71eb2f000 192.168.253.203:6789/0
-3> 2018-08-06 21:43:41.952120 7f4bbc11b700 1 -- 192.168.253.242:6789/0 >> 192.168.253.205:6789/0 conn(0x55d71ee24000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection reconnect failed
-2> 2018-08-06 21:43:41.952123 7f4bbc11b700 2 -- 192.168.253.242:6789/0 >> 192.168.253.205:6789/0 conn(0x55d71ee24000 :-1 s=STATE_CONNECTING_RE pgs=0 cs=0 l=0)._process_connection connection refused!
-1> 2018-08-06 21:43:41.952150 7f4bbf922700 10 mon.cephmon@6(probing) e1 ms_handle_refused 0x55d71ee24000 192.168.253.205:6789/0
0> 2018-08-06 21:43:41.961973 7f4bc0924700 -1 *** Caught signal (Segmentation fault) **
in thread 7f4bc0924700 thread_name:cpu_tp
ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
1: (()+0x8f51b1) [0x55d71441c1b1]
2: (()+0xf6d0) [0x7f4bc88a66d0]
3: (void std::vector<pg_t, mempool::pool_allocator<(mempool::pool_index_t)16, pg_t> >::_M_emplace_back_aux<pg_t const&>(pg_t const&)+0x69) [0x55d714226fd9]
4: (OSDMapMapping::_build_rmap(OSDMap const&)+0x209) [0x55d714226389]
5: (OSDMapMapping::_finish(OSDMap const&)+0x11) [0x55d7142263c1]
6: (ParallelPGMapper::Job::finish_one()+0x82) [0x55d714225332]
7: (ParallelPGMapper::WQ::_process(ParallelPGMapper::Item*, ThreadPool::TPHandle&)+0x7f) [0x55d7142253ff]
8: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa8e) [0x55d71412059e]
9: (ThreadPool::WorkThread::entry()+0x10) [0x55d714121480]
10: (()+0x7e25) [0x7f4bc889ee25]
11: (clone()+0x6d) [0x7f4bc5a94bad]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
20/20 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mon.cephmon.log
--- end dump of recent events ---
