Hi, Igor, thanks for the tip. Dmesg does not say any suspicious information. I will investigate whether hardware has any problem or not. best regards, samuel huxiaoyu@xxxxxxxxxxxx 发件人： Igor Fedotov 发送时间： 2020-03-19 12:07 收件人： huxiaoyu@xxxxxxxxxxxx; ceph-users; ceph-users 主题： Re: OSDs continuously restarting under load Hi, Samuel, I've never seen that sort of signal in the real life: 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) ** I suppose this has some hardware roots. Have you checked dmesg output? Just in case, here is some info on "Bus Error" signal, may be it will provide some insight: https://en.wikipedia.org/wiki/Bus_error Thanks, Igor On 3/18/2020 5:06 PM, huxiaoyu@xxxxxxxxxxxx wrote: > Hello， folks, > > I am trying to add a ceph node into an existing ceph cluster. Once the reweight of newly-added OSD on the new node exceed 0.4 somewhere, the osd becomes unresponsive and restarting, eventually go down. > > What could be the problem? Any suggestion would be highly appreciated. > > best regards, > > samuel > > **************************************************** > root@node81:/var/log/ceph# > root@node81:/var/log/ceph# > root@node81:/var/log/ceph# > root@node81:/var/log/ceph# ceph osd df > ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS > 12 hybrid 1.00000 1.00000 3.81TiB 38.3GiB 3.77TiB 0.98 1.32 316 > 13 hybrid 1.00000 1.00000 3.81TiB 37.6GiB 3.77TiB 0.96 1.29 308 > 14 hybrid 1.00000 1.00000 3.81TiB 36.9GiB 3.77TiB 0.95 1.27 301 > 15 hybrid 1.00000 1.00000 3.81TiB 37.1GiB 3.77TiB 0.95 1.28 297 > 0 hybrid 1.00000 1.00000 3.81TiB 37.6GiB 3.77TiB 0.96 1.29 305 > 1 hybrid 1.00000 1.00000 3.81TiB 38.2GiB 3.77TiB 0.98 1.31 309 > 2 hybrid 1.00000 1.00000 3.81TiB 37.4GiB 3.77TiB 0.96 1.29 296 > 3 hybrid 1.00000 1.00000 3.81TiB 37.9GiB 3.77TiB 0.97 1.30 303 > 4 hdd 0.20000 1.00000 3.42TiB 10.5GiB 3.41TiB 0.30 0.40 0 > 5 hdd 0.20000 1.00000 3.42TiB 9.63GiB 3.41TiB 0.28 0.37 87 > 6 hdd 0.20000 1.00000 3.42TiB 1.91GiB 3.42TiB 0.05 0.07 0 > 7 hdd 0.20000 1.00000 3.42TiB 11.3GiB 3.41TiB 0.32 0.43 83 > 16 hdd 0.39999 1.00000 1.79TiB 16.3GiB 1.78TiB 0.89 1.19 142 > TOTAL 45.9TiB 351GiB 45.6TiB 0.75 > > ------------------------------------------------------------------------------------ 日志 > > root@node81:/var/log/ceph# cat ceph-osd.6.log | grep load_pgs > 2020-03-18 18:33:57.808747 2000b556000 0 osd.6 0 load_pgs > 2020-03-18 18:33:57.808763 2000b556000 0 osd.6 0 load_pgs opened 0 pgs > -1324> 2020-03-18 18:33:57.808747 2000b556000 0 osd.6 0 load_pgs > -1323> 2020-03-18 18:33:57.808763 2000b556000 0 osd.6 0 load_pgs opened 0 pgs > 2020-03-18 18:35:04.363341 20003270000 0 osd.6 5222 load_pgs > 2020-03-18 18:36:15.318489 20003270000 0 osd.6 5222 load_pgs opened 202 pgs > -466> 2020-03-18 18:35:04.363341 20003270000 0 osd.6 5222 load_pgs > -465> 2020-03-18 18:36:15.318489 20003270000 0 osd.6 5222 load_pgs opened 202 pgs > 2020-03-18 18:36:32.367450 2000326e000 0 osd.6 5236 load_pgs > 2020-03-18 18:37:40.747347 2000326e000 0 osd.6 5236 load_pgs opened 177 pgs > -422> 2020-03-18 18:36:32.367450 2000326e000 0 osd.6 5236 load_pgs > -421> 2020-03-18 18:37:40.747347 2000326e000 0 osd.6 5236 load_pgs opened 177 pgs > 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs > 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 pgs > -67> 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs > -66> 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 pgs > > > 2020-03-18 18:39:09.483868 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2043:::rbd_data.8a738625558ec.00000000000056a3:head have 3291'557 flags = none tried to add 3291'557 flags = none > 2020-03-18 18:39:09.483882 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2a18:::rbd_data.9177446e87ccd.00000000000010f8:head have 4738'731 flags = none tried to add 4738'731 flags = none > 2020-03-18 18:39:09.483896 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47fc7a4:::rbd_data.58f426b8b4567.0000000000000221:head have 1789'169 flags = delete tried to add 1789'169 flags = delete > 2020-03-18 18:39:20.985370 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.11:0/3129700933 conn(0x200140cb3f0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > 2020-03-18 18:39:21.495101 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.12:0/4111063261 conn(0x200140c55a0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > 2020-03-18 18:39:21.495101 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.13:0/464497787 conn(0x200140fd4b0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > 2020-03-18 18:39:21.629021 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4088469422 conn(0x20014100b10 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) ** > in thread 201e35fdb40 thread_name:tp_osd_tp > > ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable) > 1: (()+0x145882c) [0x2000245882c] > 2: (()+0x19890) [0x2000c54b890] > 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] > 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] > 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) [0x200022c7a6c] > 6: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) [0x20001c19ee0] > 7: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] > 8: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x730) [0x2000202e970] > 9: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x59c) [0x200020442bc] > 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x94) [0x20001ecea74] > 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x814) [0x20001de1384] > 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) [0x20001b817d4] > 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0xb8) [0x20001f98968] > 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] > 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0xab4) [0x200024d60a4] > 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x28) [0x200024da278] > 17: (Thread::entry_wrapper()+0xec) [0x20002769b4c] > 18: (Thread::_entry_func(void*)+0x20) [0x20002769ba0] > 19: (()+0x80fc) [0x2000c53a0fc] > 20: (()+0x119854) [0x2000f2ad854] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > --- begin dump of recent events --- > -147> 2020-03-18 18:37:51.039443 2000f374000 5 asok(0x2000cd7f230) register_command perfcounters_dump hook 0x2000ce09e40 > -146> 2020-03-18 18:37:51.039716 2000f374000 5 asok(0x2000cd7f230) register_command 1 hook 0x2000ce09e40 > -145> 2020-03-18 18:37:51.039736 2000f374000 5 asok(0x2000cd7f230) register_command perf dump hook 0x2000ce09e40 > -144> 2020-03-18 18:37:51.039769 2000f374000 5 asok(0x2000cd7f230) register_command perfcounters_schema hook 0x2000ce09e40 > -143> 2020-03-18 18:37:51.039789 2000f374000 5 asok(0x2000cd7f230) register_command perf histogram dump hook 0x2000ce09e40 > -142> 2020-03-18 18:37:51.039807 2000f374000 5 asok(0x2000cd7f230) register_command 2 hook 0x2000ce09e40 > -141> 2020-03-18 18:37:51.039823 2000f374000 5 asok(0x2000cd7f230) register_command perf schema hook 0x2000ce09e40 > -140> 2020-03-18 18:37:51.039843 2000f374000 5 asok(0x2000cd7f230) register_command perf histogram schema hook 0x2000ce09e40 > -139> 2020-03-18 18:37:51.039863 2000f374000 5 asok(0x2000cd7f230) register_command perf reset hook 0x2000ce09e40 > -138> 2020-03-18 18:37:51.039881 2000f374000 5 asok(0x2000cd7f230) register_command config show hook 0x2000ce09e40 > -137> 2020-03-18 18:37:51.039899 2000f374000 5 asok(0x2000cd7f230) register_command config help hook 0x2000ce09e40 > -136> 2020-03-18 18:37:51.039928 2000f374000 5 asok(0x2000cd7f230) register_command config set hook 0x2000ce09e40 > -135> 2020-03-18 18:37:51.039949 2000f374000 5 asok(0x2000cd7f230) register_command config get hook 0x2000ce09e40 > -134> 2020-03-18 18:37:51.039967 2000f374000 5 asok(0x2000cd7f230) register_command config diff hook 0x2000ce09e40 > -133> 2020-03-18 18:37:51.039985 2000f374000 5 asok(0x2000cd7f230) register_command config diff get hook 0x2000ce09e40 > -132> 2020-03-18 18:37:51.040005 2000f374000 5 asok(0x2000cd7f230) register_command log flush hook 0x2000ce09e40 > -131> 2020-03-18 18:37:51.040021 2000f374000 5 asok(0x2000cd7f230) register_command log dump hook 0x2000ce09e40 > -130> 2020-03-18 18:37:51.040038 2000f374000 5 asok(0x2000cd7f230) register_command log reopen hook 0x2000ce09e40 > -129> 2020-03-18 18:37:51.040189 2000f374000 5 asok(0x2000cd7f230) register_command dump_mempools hook 0x2000ce0d038 > -128> 2020-03-18 18:37:51.099580 2000f374000 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb > -127> 2020-03-18 18:37:51.102046 2000f374000 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb > -126> 2020-03-18 18:37:51.102148 2000f374000 0 ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable), process ceph-osd, pid 1159687 > -125> 2020-03-18 18:37:51.109860 2000f374000 0 pidfile_write: ignore empty --pid-file > -124> 2020-03-18 18:37:51.130256 2000f374000 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb > -123> 2020-03-18 18:37:51.183798 2000f374000 0 load: jerasure load: lrc > -122> 2020-03-18 18:37:51.184330 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel > -121> 2020-03-18 18:37:51.184492 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block > -120> 2020-03-18 18:37:51.184741 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway > -119> 2020-03-18 18:37:51.185841 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational > -118> 2020-03-18 18:37:51.186232 2000f374000 1 bdev(0x2000cf49800 /var/lib/ceph/osd/ceph-6/block) close > -117> 2020-03-18 18:37:51.498718 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel > -116> 2020-03-18 18:37:51.498748 2000f374000 1 bdev(0x2000cf5d560 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block > -115> 2020-03-18 18:37:51.498811 2000f374000 1 bdev(0x2000cf5d560 /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway > -114> 2020-03-18 18:37:51.499234 2000f374000 1 bdev(0x2000cf5d560 /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational > -113> 2020-03-18 18:37:51.500074 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block.db type kernel > -112> 2020-03-18 18:37:51.500096 2000f374000 1 bdev(0x2000cf5e6a0 /var/lib/ceph/osd/ceph-6/block.db) open path /var/lib/ceph/osd/ceph-6/block.db > -111> 2020-03-18 18:37:51.500170 2000f374000 1 bdev(0x2000cf5e6a0 /var/lib/ceph/osd/ceph-6/block.db) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway > -110> 2020-03-18 18:37:51.500815 2000f374000 1 bdev(0x2000cf5e6a0 /var/lib/ceph/osd/ceph-6/block.db) open size 39998980096 (0x950200000, 38146 MB) block_size 4096 (4096 B) rotational > -109> 2020-03-18 18:37:51.502625 2000f374000 1 bdev create path /var/lib/ceph/osd/ceph-6/block type kernel > -108> 2020-03-18 18:37:51.502651 2000f374000 1 bdev(0x2000cf5ed80 /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block > -107> 2020-03-18 18:37:51.502718 2000f374000 1 bdev(0x2000cf5ed80 /var/lib/ceph/osd/ceph-6/block) open backing device/file reports st_blksize 8192, using bdev_block_size 4096 anyway > -106> 2020-03-18 18:37:51.503137 2000f374000 1 bdev(0x2000cf5ed80 /var/lib/ceph/osd/ceph-6/block) open size 3758096384000 (0x36b00000000, 3500 GB) block_size 4096 (4096 B) rotational > -105> 2020-03-18 18:37:51.549269 2000f374000 0 set rocksdb option compaction_readahead_size = 2MB > -104> 2020-03-18 18:37:51.549349 2000f374000 0 set rocksdb option compaction_style = kCompactionStyleLevel > -103> 2020-03-18 18:37:51.552610 2000f374000 0 set rocksdb option compaction_threads = 32 > -102> 2020-03-18 18:37:51.552652 2000f374000 0 set rocksdb option compression = kNoCompression > -101> 2020-03-18 18:37:51.553442 2000f374000 0 set rocksdb option flusher_threads = 8 > -100> 2020-03-18 18:37:51.553508 2000f374000 0 set rocksdb option level0_file_num_compaction_trigger = 64 > -99> 2020-03-18 18:37:51.553536 2000f374000 0 set rocksdb option level0_slowdown_writes_trigger = 128 > -98> 2020-03-18 18:37:51.553559 2000f374000 0 set rocksdb option level0_stop_writes_trigger = 256 > -97> 2020-03-18 18:37:51.553579 2000f374000 0 set rocksdb option max_background_compactions = 64 > -96> 2020-03-18 18:37:51.553601 2000f374000 0 set rocksdb option max_bytes_for_level_base = 2GB > -95> 2020-03-18 18:37:51.553624 2000f374000 0 set rocksdb option max_write_buffer_number = 64 > -94> 2020-03-18 18:37:51.553646 2000f374000 0 set rocksdb option min_write_buffer_number_to_merge = 32 > -93> 2020-03-18 18:37:51.553665 2000f374000 0 set rocksdb option recycle_log_file_num = 64 > -92> 2020-03-18 18:37:51.553687 2000f374000 0 set rocksdb option target_file_size_base = 4MB > -91> 2020-03-18 18:37:51.553708 2000f374000 0 set rocksdb option write_buffer_size = 4MB > -90> 2020-03-18 18:37:51.553892 2000f374000 0 set rocksdb option compaction_readahead_size = 2MB > -89> 2020-03-18 18:37:51.553923 2000f374000 0 set rocksdb option compaction_style = kCompactionStyleLevel > -88> 2020-03-18 18:37:51.553948 2000f374000 0 set rocksdb option compaction_threads = 32 > -87> 2020-03-18 18:37:51.553973 2000f374000 0 set rocksdb option compression = kNoCompression > -86> 2020-03-18 18:37:51.553994 2000f374000 0 set rocksdb option flusher_threads = 8 > -85> 2020-03-18 18:37:51.554016 2000f374000 0 set rocksdb option level0_file_num_compaction_trigger = 64 > -84> 2020-03-18 18:37:51.554043 2000f374000 0 set rocksdb option level0_slowdown_writes_trigger = 128 > -83> 2020-03-18 18:37:51.554065 2000f374000 0 set rocksdb option level0_stop_writes_trigger = 256 > -82> 2020-03-18 18:37:51.554084 2000f374000 0 set rocksdb option max_background_compactions = 64 > -81> 2020-03-18 18:37:51.554106 2000f374000 0 set rocksdb option max_bytes_for_level_base = 2GB > -80> 2020-03-18 18:37:51.554133 2000f374000 0 set rocksdb option max_write_buffer_number = 64 > -79> 2020-03-18 18:37:51.554154 2000f374000 0 set rocksdb option min_write_buffer_number_to_merge = 32 > -78> 2020-03-18 18:37:51.554174 2000f374000 0 set rocksdb option recycle_log_file_num = 64 > -77> 2020-03-18 18:37:51.554196 2000f374000 0 set rocksdb option target_file_size_base = 4MB > -76> 2020-03-18 18:37:51.554232 2000f374000 0 set rocksdb option write_buffer_size = 4MB > -75> 2020-03-18 18:37:56.382110 2000f374000 0 <cls> /home/deepin/hhao/srccode/ceph-12.2.7/src/cls/hello/cls_hello.cc:296: loading cls_hello > -74> 2020-03-18 18:37:56.383845 2000f374000 0 _get_class not permitted to load lua > -73> 2020-03-18 18:37:56.386594 2000f374000 0 _get_class not permitted to load sdk > -72> 2020-03-18 18:37:56.395800 2000f374000 0 _get_class not permitted to load kvs > -71> 2020-03-18 18:37:56.398226 2000f374000 0 <cls> /home/deepin/hhao/srccode/ceph-12.2.7/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs > -70> 2020-03-18 18:37:56.433293 2000f374000 0 osd.6 5247 crush map has features 432629239337189376, adjusting msgr requires for clients > -69> 2020-03-18 18:37:56.433330 2000f374000 0 osd.6 5247 crush map has features 432629239337189376 was 8705, adjusting msgr requires for mons > -68> 2020-03-18 18:37:56.433357 2000f374000 0 osd.6 5247 crush map has features 1009089991640629248, adjusting msgr requires for osds > -67> 2020-03-18 18:37:56.579371 2000f374000 0 osd.6 5247 load_pgs > -66> 2020-03-18 18:39:03.376838 2000f374000 0 osd.6 5247 load_pgs opened 170 pgs > -65> 2020-03-18 18:39:03.377040 2000f374000 0 osd.6 5247 using weightedpriority op queue with priority op cut off at 196. > -64> 2020-03-18 18:39:03.413901 2000f374000 -1 osd.6 5247 log_to_monitors {default=true} > -63> 2020-03-18 18:39:03.663128 2000f374000 0 osd.6 5247 done with init, starting boot process > -62> 2020-03-18 18:39:03.663856 201d65fdb40 4 mgrc handle_mgr_map Got map version 34 > -61> 2020-03-18 18:39:03.663947 201d65fdb40 4 mgrc handle_mgr_map Active mgr is now 192.168.230.120:6808/44007 > -60> 2020-03-18 18:39:03.663972 201d65fdb40 4 mgrc reconnect Starting new session with 192.168.230.120:6808/44007 > -59> 2020-03-18 18:39:03.667814 201d65fdb40 4 mgrc handle_mgr_configure stats_period=5 > -58> 2020-03-18 18:39:03.667831 201d65fdb40 4 mgrc handle_mgr_configure updated stats threshold: 5 > -57> 2020-03-18 18:39:03.752253 2000dae5b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6802/33592 conn(0x20014018040 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -56> 2020-03-18 18:39:03.754040 2000fc61b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6800/33864 conn(0x2001403ee60 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -55> 2020-03-18 18:39:03.762441 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6804/13410 conn(0x20014079940 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -54> 2020-03-18 18:39:03.762860 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6806/13743 conn(0x20014083980 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -53> 2020-03-18 18:39:03.765775 2000fc61b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6806/12423 conn(0x2001407e030 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -52> 2020-03-18 18:39:03.767284 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.121:6804/11599 conn(0x2001408c660 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -51> 2020-03-18 18:39:03.769434 2000dae5b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6800/12205 conn(0x20014087ff0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -50> 2020-03-18 18:39:03.775190 2000ec1fb40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.120:6802/12991 conn(0x2001409d300 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -49> 2020-03-18 18:39:04.009755 2000dae5b40 0 -- 192.168.240.122:6806/1159687 >> 192.168.240.122:6804/1159466 conn(0x200140b3420 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer > -48> 2020-03-18 18:39:05.177544 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.202:0/3091162658 conn(0x200140cd900 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > -47> 2020-03-18 18:39:05.402465 2000dae5b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4289863819 conn(0x200140d8500 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > -46> 2020-03-18 18:39:09.483237 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b442b93f:::rbd_data.911772ae8944a.0000000000002aa7:head have 3097'452 flags = none tried to add 3097'452 flags = none > -45> 2020-03-18 18:39:09.483318 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b445877b:::rbd_data.2fa7e6b8b4567.000000000000002d:head have 1915'212 flags = none tried to add 1915'212 flags = none > -44> 2020-03-18 18:39:09.483336 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44674a4:::rbd_data.110706b8b4567.0000000000000659:head have 1915'213 flags = none tried to add 1915'213 flags = none > -43> 2020-03-18 18:39:09.483351 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44757d1:::rbd_data.cd282238e1f29.0000000000009ea2:head have 5165'734 flags = none tried to add 5165'734 flags = none > -42> 2020-03-18 18:39:09.483366 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4490029:::rbd_data.8d8146b8b4567.00000000000080a0:head have 2855'272 flags = none tried to add 2855'272 flags = none > -41> 2020-03-18 18:39:09.483381 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44950ae:::rbd_data.5f15f625558ec.0000000000009fd2:head have 1915'214 flags = none tried to add 1915'214 flags = none > -40> 2020-03-18 18:39:09.483395 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b449ce0d:::rbd_data.9117a327b23c6.00000000000074a6:head have 3798'560 flags = none tried to add 3798'560 flags = none > -39> 2020-03-18 18:39:09.483409 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44a69ad:::rbd_data.8b4a76b8b4567.000000000000017a:head have 2197'242 flags = none tried to add 2197'242 flags = none > -38> 2020-03-18 18:39:09.483423 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44bbb34:::rbd_data.8922074b0dc51.00000000000098a5:head have 3099'543 flags = delete tried to add 3099'543 flags = delete > -37> 2020-03-18 18:39:09.483438 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44be196:::rbd_data.cd8e219495cff.00000000000192a0:head have 5169'1101 flags = delete tried to add 5169'1101 flags = delete > -36> 2020-03-18 18:39:09.483454 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44cd1c7:::rbd_data.5f15f625558ec.000000000000820b:head have 1915'215 flags = none tried to add 1915'215 flags = none > -35> 2020-03-18 18:39:09.483469 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44e04c0:::rbd_data.cd66c46e87ccd.0000000000015aa4:head have 5175'1280 flags = none tried to add 5175'1280 flags = none > -34> 2020-03-18 18:39:09.483483 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b44fa767:::rbd_data.ccf5e625558ec.00000000000150a0:head have 5166'844 flags = delete tried to add 5166'844 flags = delete > -33> 2020-03-18 18:39:09.483497 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4518cb0:::rbd_data.589572ae8944a.0000000000000433:head have 1805'174 flags = delete tried to add 1805'174 flags = delete > -32> 2020-03-18 18:39:09.483511 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4539e4a:::rbd_data.5f15f625558ec.000000000000031a:head have 1915'216 flags = none tried to add 1915'216 flags = none > -31> 2020-03-18 18:39:09.483525 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4540dfd:::rbd_data.cd66c46e87ccd.000000000001d6a6:head have 5175'1283 flags = none tried to add 5175'1283 flags = none > -30> 2020-03-18 18:39:09.483539 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45474ee:::rbd_data.2fa7e6b8b4567.000000000000003a:head have 1915'217 flags = none tried to add 1915'217 flags = none > -29> 2020-03-18 18:39:09.483553 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45506f1:::rbd_data.9177446e87ccd.00000000000036c3:head have 5175'1284 flags = none tried to add 5175'1284 flags = none > -28> 2020-03-18 18:39:09.483567 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4576e29:::rbd_data.589572ae8944a.0000000000001207:head have 1805'179 flags = delete tried to add 1805'179 flags = delete > -27> 2020-03-18 18:39:09.483582 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4578057:::rbd_data.ccf5e625558ec.00000000000096a5:head have 5166'785 flags = delete tried to add 5166'785 flags = delete > -26> 2020-03-18 18:39:09.483595 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b457fe94:::rbd_data.589572ae8944a.0000000000004e27:head have 1805'200 flags = delete tried to add 1805'200 flags = delete > -25> 2020-03-18 18:39:09.483610 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4592563:::rbd_data.9117a327b23c6.000000000000b2a0:head have 3894'562 flags = none tried to add 3894'562 flags = none > -24> 2020-03-18 18:39:09.483625 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45bb3ff:::rbd_data.c6c99507ed7ab.000000000000aea0:head have 5172'1141 flags = none tried to add 5172'1141 flags = none > -23> 2020-03-18 18:39:09.483639 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45d7c66:::rbd_data.110706b8b4567.0000000000000e07:head have 1915'218 flags = none tried to add 1915'218 flags = none > -22> 2020-03-18 18:39:09.483654 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45e1a62:::rbd_data.589572ae8944a.0000000000004733:head have 1805'198 flags = delete tried to add 1805'198 flags = delete > -21> 2020-03-18 18:39:09.483667 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b45f03a3:::rbd_data.914c22ae8944a.00000000000058a8:head have 3908'593 flags = delete tried to add 3908'593 flags = delete > -20> 2020-03-18 18:39:09.483681 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4604fdd:::rbd_data.c6c99507ed7ab.00000000000004ca:head have 5176'1523 flags = none tried to add 5176'1523 flags = none > -19> 2020-03-18 18:39:09.483695 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4636d5b:::rbd_data.cd282238e1f29.000000000000b6a1:head have 5165'735 flags = none tried to add 5165'735 flags = none > -18> 2020-03-18 18:39:09.483710 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b463cd0a:::rbd_data.9117a327b23c6.000000000000c0a8:head have 3908'563 flags = none tried to add 3908'563 flags = none > -17> 2020-03-18 18:39:09.483724 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b468ca37:::rbd_data.9177446e87ccd.000000000000040a:head have 4427'730 flags = none tried to add 4427'730 flags = none > -16> 2020-03-18 18:39:09.483738 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4696438:::rbd_data.9177446e87ccd.0000000000000c53:head have 4743'732 flags = none tried to add 4743'732 flags = none > -15> 2020-03-18 18:39:09.483752 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46be061:::rbd_data.c70aa2eb141f2.0000000000001ea6:head have 5175'1287 flags = none tried to add 5175'1287 flags = none > -14> 2020-03-18 18:39:09.483766 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46bfc4e:::rbd_data.cda772ae8944a.00000000000020a4:head have 5172'1144 flags = none tried to add 5172'1144 flags = none > -13> 2020-03-18 18:39:09.483782 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46e7efd:::rbd_data.110706b8b4567.0000000000001c01:head have 1915'219 flags = none tried to add 1915'219 flags = none > -12> 2020-03-18 18:39:09.483796 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b46ece7b:::rbd_data.8ff1766334873.00000000000000ae:head have 3092'344 flags = delete tried to add 3092'344 flags = delete > -11> 2020-03-18 18:39:09.483811 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b4786571:::rbd_data.8f6b3643c9869.00000000000040a0:head have 3092'381 flags = delete tried to add 3092'381 flags = delete > -10> 2020-03-18 18:39:09.483825 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47888b8:::rbd_data.589572ae8944a.0000000000004534:head have 1805'197 flags = delete tried to add 1805'197 flags = delete > -9> 2020-03-18 18:39:09.483840 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b478b833:::rbd_data.589572ae8944a.00000000000048ff:head have 1805'199 flags = delete tried to add 1805'199 flags = delete > -8> 2020-03-18 18:39:09.483854 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47ec645:::rbd_data.8b4a76b8b4567.00000000000001d2:head have 2197'271 flags = none tried to add 2197'271 flags = none > -7> 2020-03-18 18:39:09.483868 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2043:::rbd_data.8a738625558ec.00000000000056a3:head have 3291'557 flags = none tried to add 3291'557 flags = none > -6> 2020-03-18 18:39:09.483882 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47f2a18:::rbd_data.9177446e87ccd.00000000000010f8:head have 4738'731 flags = none tried to add 4738'731 flags = none > -5> 2020-03-18 18:39:09.483896 201df5fdb40 0 0x201c4c90c90 4.22d unexpected need for 4:b47fc7a4:::rbd_data.58f426b8b4567.0000000000000221:head have 1789'169 flags = delete tried to add 1789'169 flags = delete > -4> 2020-03-18 18:39:20.985370 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.11:0/3129700933 conn(0x200140cb3f0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > -3> 2020-03-18 18:39:21.495101 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.12:0/4111063261 conn(0x200140c55a0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > -2> 2020-03-18 18:39:21.495101 2000fc61b40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.13:0/464497787 conn(0x200140fd4b0 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > -1> 2020-03-18 18:39:21.629021 2000ec1fb40 0 -- 192.168.230.122:6806/1159687 >> 192.168.230.201:0/4088469422 conn(0x20014100b10 :6806 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer > 0> 2020-03-18 18:39:26.426584 201e35fdb40 -1 *** Caught signal (Bus error) ** > in thread 201e35fdb40 thread_name:tp_osd_tp > > ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable) > 1: (()+0x145882c) [0x2000245882c] > 2: (()+0x19890) [0x2000c54b890] > 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] > 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] > 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) [0x200022c7a6c] > 6: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) [0x20001c19ee0] > 7: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] > 8: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x730) [0x2000202e970] > 9: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x59c) [0x200020442bc] > 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x94) [0x20001ecea74] > 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x814) [0x20001de1384] > 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) [0x20001b817d4] > 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0xb8) [0x20001f98968] > 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] > 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0xab4) [0x200024d60a4] > 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x28) [0x200024da278] > 17: (Thread::entry_wrapper()+0xec) [0x20002769b4c] > 18: (Thread::_entry_func(void*)+0x20) [0x20002769ba0] > 19: (()+0x80fc) [0x2000c53a0fc] > 20: (()+0x119854) [0x2000f2ad854] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > > > huxiaoyu@xxxxxxxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |