Hi all, We're midway through an update from 13.2.6 to 13.2.7 and started getting OSDs crashing regularly like this [1]. Does anyone obviously know what the issue is? (Maybe https://github.com/ceph/ceph/pull/26448/files ?) Or is it some temporary problem while we still have v13.2.6 and v13.2.7 osds running concurrently? Thanks! Dan [1] 2019-12-03 15:53:51.817 7ff3a3d39700 -1 osd.1384 2758889 build_incremental_map_msg missing incremental map 2758889 2019-12-03 15:53:51.817 7ff3a453a700 -1 osd.1384 2758889 build_incremental_map_msg missing incremental map 2758889 2019-12-03 15:53:51.817 7ff3a453a700 -1 osd.1384 2758889 build_incremental_map_msg unable to load latest map 2758889 2019-12-03 15:53:51.822 7ff3a453a700 -1 *** Caught signal (Aborted) ** in thread 7ff3a453a700 thread_name:tp_osd_tp ceph version 13.2.7 (71bd687b6e8b9424dd5e5974ed542595d8977416) mimic (stable) 1: (()+0xf5f0) [0x7ff3c620b5f0] 2: (gsignal()+0x37) [0x7ff3c522b337] 3: (abort()+0x148) [0x7ff3c522ca28] 4: (OSDService::build_incremental_map_msg(unsigned int, unsigned int, OSDSuperblock&)+0x767) [0x555d60e8d797] 5: (OSDService::send_incremental_map(unsigned int, Connection*, std::shared_ptr<OSDMap const>&)+0x39e) [0x555d60e8dbee] 6: (OSDService::share_map_peer(int, Connection*, std::shared_ptr<OSDMap const>)+0x159) [0x555d60e8eda9] 7: (OSDService::send_message_osd_cluster(int, Message*, unsigned int)+0x1a5) [0x555d60e8f085] 8: (ReplicatedBackend::issue_op(hobject_t const&, eversion_t const&, unsigned long, osd_reqid_t, eversion_t, eversion_t, hobject_t, hobject_t, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, boost::optional<pg_hit_set_history_t>&, ReplicatedBackend::InProgressOp*, ObjectStore::Transaction&)+0x452) [0x555d6116e522] 9: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, boost::optional<pg_hit_set_history_t>&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr<OpRequest>)+0x6f5) [0x555d6117ed85] 10: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xd62) [0x555d60ff5142] 11: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xf12) [0x555d61035902] 12: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3679) [0x555d610397a9] 13: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xc99) [0x555d6103d869] 14: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1b7) [0x555d60e8e8a7] 15: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x555d611144c2] 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x592) [0x555d60eb25f2] 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3d3) [0x7ff3c929f5b3] 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7ff3c92a01a0] 19: (()+0x7e65) [0x7ff3c6203e65] 20: (clone()+0x6d) [0x7ff3c52f388d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |