Hi, Folks,
I just encountered an OSD being down and can not be up again. Below atttached is the log messages. Anyone can tell what is wrong with the OSD? and what should i do?
thanks in advance,
Samuel
***********************************************************************
# tail -500 /var/log/ceph/ceph-osd.0.log.522
2019-05-22 09:23:35.139017 7f9d18f71d80 0 set rocksdb option min_write_buffer_number_to_merge = 32
2019-05-22 09:23:35.139019 7f9d18f71d80 0 set rocksdb option recycle_log_file_num = 64
2019-05-22 09:23:35.139023 7f9d18f71d80 0 set rocksdb option target_file_size_base = 4MB
2019-05-22 09:23:35.139025 7f9d18f71d80 0 set rocksdb option write_buffer_size = 4MB
2019-05-22 09:23:35.139058 7f9d18f71d80 0 set rocksdb option compaction_readahead_size = 2MB
2019-05-22 09:23:35.139063 7f9d18f71d80 0 set rocksdb option compaction_style = kCompactionStyleLevel
2019-05-22 09:23:35.139067 7f9d18f71d80 0 set rocksdb option compaction_threads = 32
2019-05-22 09:23:35.139069 7f9d18f71d80 0 set rocksdb option compression = kNoCompression
2019-05-22 09:23:35.139072 7f9d18f71d80 0 set rocksdb option flusher_threads = 8
2019-05-22 09:23:35.139074 7f9d18f71d80 0 set rocksdb option level0_file_num_compaction_trigger = 64
2019-05-22 09:23:35.139077 7f9d18f71d80 0 set rocksdb option level0_slowdown_writes_trigger = 128
2019-05-22 09:23:35.139079 7f9d18f71d80 0 set rocksdb option level0_stop_writes_trigger = 256
2019-05-22 09:23:35.139081 7f9d18f71d80 0 set rocksdb option max_background_compactions = 64
2019-05-22 09:23:35.139084 7f9d18f71d80 0 set rocksdb option max_bytes_for_level_base = 6GB
2019-05-22 09:23:35.139086 7f9d18f71d80 0 set rocksdb option max_write_buffer_number = 64
2019-05-22 09:23:35.139088 7f9d18f71d80 0 set rocksdb option min_write_buffer_number_to_merge = 32
2019-05-22 09:23:35.139090 7f9d18f71d80 0 set rocksdb option recycle_log_file_num = 64
2019-05-22 09:23:35.139093 7f9d18f71d80 0 set rocksdb option target_file_size_base = 4MB
2019-05-22 09:23:35.139095 7f9d18f71d80 0 set rocksdb option write_buffer_size = 4MB
2019-05-22 09:23:35.171770 7f9d18f71d80 0 <cls> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs
2019-05-22 09:23:35.172018 7f9d18f71d80 0 <cls> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/cls/hello/cls_hello.cc:296: loading cls_hello
2019-05-22 09:23:35.172865 7f9d18f71d80 0 _get_class not permitted to load kvs
2019-05-22 09:23:35.173387 7f9d18f71d80 0 _get_class not permitted to load lua
2019-05-22 09:23:35.178435 7f9d18f71d80 0 _get_class not permitted to load sdk
2019-05-22 09:23:35.179798 7f9d18f71d80 0 osd.0 723 crush map has features 288514051259236352, adjusting msgr requires for clients
2019-05-22 09:23:35.179806 7f9d18f71d80 0 osd.0 723 crush map has features 288514051259236352 was 8705, adjusting msgr requires for mons
2019-05-22 09:23:35.179809 7f9d18f71d80 0 osd.0 723 crush map has features 1009089991638532096, adjusting msgr requires for osds
2019-05-22 09:23:35.245033 7f9d18f71d80 0 osd.0 723 load_pgs
2019-05-22 09:25:01.427911 7f9d18f71d80 -1 *** Caught signal (Segmentation fault) **
in thread 7f9d18f71d80 thread_name:ceph-osd
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
1: (()+0xa64ee1) [0x55fc16d31ee1]
2: (()+0xf6d0) [0x7f9d1621e6d0]
3: (std::__detail::_List_node_base::_M_hook(std::__detail::_List_node_base*)+0x4) [0x7f9d15b5ce54]
4: (pg_log_t::pg_log_t(pg_log_t const&)+0x109) [0x55fc168d1e79]
5: (PGLog::IndexedLog::operator=(PGLog::IndexedLog const&)+0x29) [0x55fc168dcb29]
6: (void PGLog::read_log_and_missing<pg_missing_set<true> >(ObjectStore*, coll_t, coll_t, ghobject_t, pg_info_t const&, PGLog::IndexedLog&, pg_missing_set<true>&, bool, std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >&, bool, bool*, DoutPrefixProvider const*, std::set<std::string, std::less<std::string>, std::allocator<std::string> >*, bool)+0x793) [0x55fc168ddde3]
7: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x52b) [0x55fc1687fc2b]
8: (OSD::load_pgs()+0x9b4) [0x55fc167cdd64]
9: (OSD::init()+0x2169) [0x55fc167ec8a9]
10: (main()+0x2d07) [0x55fc166edef7]
11: (__libc_start_main()+0xf5) [0x7f9d1522b445]
12: (()+0x4c0dc3) [0x55fc1678ddc3]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2019-05-22 09:23:35.139017 7f9d18f71d80 0 set rocksdb option min_write_buffer_number_to_merge = 32
2019-05-22 09:23:35.139019 7f9d18f71d80 0 set rocksdb option recycle_log_file_num = 64
2019-05-22 09:23:35.139023 7f9d18f71d80 0 set rocksdb option target_file_size_base = 4MB
2019-05-22 09:23:35.139025 7f9d18f71d80 0 set rocksdb option write_buffer_size = 4MB
2019-05-22 09:23:35.139058 7f9d18f71d80 0 set rocksdb option compaction_readahead_size = 2MB
2019-05-22 09:23:35.139063 7f9d18f71d80 0 set rocksdb option compaction_style = kCompactionStyleLevel
2019-05-22 09:23:35.139067 7f9d18f71d80 0 set rocksdb option compaction_threads = 32
2019-05-22 09:23:35.139069 7f9d18f71d80 0 set rocksdb option compression = kNoCompression
2019-05-22 09:23:35.139072 7f9d18f71d80 0 set rocksdb option flusher_threads = 8
2019-05-22 09:23:35.139074 7f9d18f71d80 0 set rocksdb option level0_file_num_compaction_trigger = 64
2019-05-22 09:23:35.139077 7f9d18f71d80 0 set rocksdb option level0_slowdown_writes_trigger = 128
2019-05-22 09:23:35.139079 7f9d18f71d80 0 set rocksdb option level0_stop_writes_trigger = 256
2019-05-22 09:23:35.139081 7f9d18f71d80 0 set rocksdb option max_background_compactions = 64
2019-05-22 09:23:35.139084 7f9d18f71d80 0 set rocksdb option max_bytes_for_level_base = 6GB
2019-05-22 09:23:35.139086 7f9d18f71d80 0 set rocksdb option max_write_buffer_number = 64
2019-05-22 09:23:35.139088 7f9d18f71d80 0 set rocksdb option min_write_buffer_number_to_merge = 32
2019-05-22 09:23:35.139090 7f9d18f71d80 0 set rocksdb option recycle_log_file_num = 64
2019-05-22 09:23:35.139093 7f9d18f71d80 0 set rocksdb option target_file_size_base = 4MB
2019-05-22 09:23:35.139095 7f9d18f71d80 0 set rocksdb option write_buffer_size = 4MB
2019-05-22 09:23:35.171770 7f9d18f71d80 0 <cls> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs
2019-05-22 09:23:35.172018 7f9d18f71d80 0 <cls> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/cls/hello/cls_hello.cc:296: loading cls_hello
2019-05-22 09:23:35.172865 7f9d18f71d80 0 _get_class not permitted to load kvs
2019-05-22 09:23:35.173387 7f9d18f71d80 0 _get_class not permitted to load lua
2019-05-22 09:23:35.178435 7f9d18f71d80 0 _get_class not permitted to load sdk
2019-05-22 09:23:35.179798 7f9d18f71d80 0 osd.0 723 crush map has features 288514051259236352, adjusting msgr requires for clients
2019-05-22 09:23:35.179806 7f9d18f71d80 0 osd.0 723 crush map has features 288514051259236352 was 8705, adjusting msgr requires for mons
2019-05-22 09:23:35.179809 7f9d18f71d80 0 osd.0 723 crush map has features 1009089991638532096, adjusting msgr requires for osds
2019-05-22 09:23:35.245033 7f9d18f71d80 0 osd.0 723 load_pgs
2019-05-22 09:25:01.427911 7f9d18f71d80 -1 *** Caught signal (Segmentation fault) **
in thread 7f9d18f71d80 thread_name:ceph-osd
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
1: (()+0xa64ee1) [0x55fc16d31ee1]
2: (()+0xf6d0) [0x7f9d1621e6d0]
3: (std::__detail::_List_node_base::_M_hook(std::__detail::_List_node_base*)+0x4) [0x7f9d15b5ce54]
4: (pg_log_t::pg_log_t(pg_log_t const&)+0x109) [0x55fc168d1e79]
5: (PGLog::IndexedLog::operator=(PGLog::IndexedLog const&)+0x29) [0x55fc168dcb29]
6: (void PGLog::read_log_and_missing<pg_missing_set<true> >(ObjectStore*, coll_t, coll_t, ghobject_t, pg_info_t const&, PGLog::IndexedLog&, pg_missing_set<true>&, bool, std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >&, bool, bool*, DoutPrefixProvider const*, std::set<std::string, std::less<std::string>, std::allocator<std::string> >*, bool)+0x793) [0x55fc168ddde3]
7: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x52b) [0x55fc1687fc2b]
8: (OSD::load_pgs()+0x9b4) [0x55fc167cdd64]
9: (OSD::init()+0x2169) [0x55fc167ec8a9]
10: (main()+0x2d07) [0x55fc166edef7]
11: (__libc_start_main()+0xf5) [0x7f9d1522b445]
12: (()+0x4c0dc3) [0x55fc1678ddc3]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
