I created https://tracker.ceph.com/issues/43106 and we're downgrading our osds back to 13.2.6. -- dan On Tue, Dec 3, 2019 at 4:09 PM Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote: > > Hi all, > > We're midway through an update from 13.2.6 to 13.2.7 and started > getting OSDs crashing regularly like this [1]. > Does anyone obviously know what the issue is? (Maybe > https://github.com/ceph/ceph/pull/26448/files ?) > Or is it some temporary problem while we still have v13.2.6 and > v13.2.7 osds running concurrently? > > Thanks! > > Dan > > [1] > > 2019-12-03 15:53:51.817 7ff3a3d39700 -1 osd.1384 2758889 > build_incremental_map_msg missing incremental map 2758889 > 2019-12-03 15:53:51.817 7ff3a453a700 -1 osd.1384 2758889 > build_incremental_map_msg missing incremental map 2758889 > 2019-12-03 15:53:51.817 7ff3a453a700 -1 osd.1384 2758889 > build_incremental_map_msg unable to load latest map 2758889 > 2019-12-03 15:53:51.822 7ff3a453a700 -1 *** Caught signal (Aborted) ** > in thread 7ff3a453a700 thread_name:tp_osd_tp > > ceph version 13.2.7 (71bd687b6e8b9424dd5e5974ed542595d8977416) mimic (stable) > 1: (()+0xf5f0) [0x7ff3c620b5f0] > 2: (gsignal()+0x37) [0x7ff3c522b337] > 3: (abort()+0x148) [0x7ff3c522ca28] > 4: (OSDService::build_incremental_map_msg(unsigned int, unsigned int, > OSDSuperblock&)+0x767) [0x555d60e8d797] > 5: (OSDService::send_incremental_map(unsigned int, Connection*, > std::shared_ptr<OSDMap const>&)+0x39e) [0x555d60e8dbee] > 6: (OSDService::share_map_peer(int, Connection*, > std::shared_ptr<OSDMap const>)+0x159) [0x555d60e8eda9] > 7: (OSDService::send_message_osd_cluster(int, Message*, unsigned > int)+0x1a5) [0x555d60e8f085] > 8: (ReplicatedBackend::issue_op(hobject_t const&, eversion_t const&, > unsigned long, osd_reqid_t, eversion_t, eversion_t, hobject_t, > hobject_t, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > > const&, boost::optional<pg_hit_set_history_t>&, > ReplicatedBackend::InProgressOp*, ObjectStore::Transaction&)+0x452) > [0x555d6116e522] > 9: (ReplicatedBackend::submit_transaction(hobject_t const&, > object_stat_sum_t const&, eversion_t const&, > std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, > eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t, > std::allocator<pg_log_entry_t> > const&, > boost::optional<pg_hit_set_history_t>&, Context*, unsigned long, > osd_reqid_t, boost::intrusive_ptr<OpRequest>)+0x6f5) [0x555d6117ed85] > 10: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, > PrimaryLogPG::OpContext*)+0xd62) [0x555d60ff5142] > 11: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xf12) > [0x555d61035902] > 12: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3679) > [0x555d610397a9] > 13: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, > ThreadPool::TPHandle&)+0xc99) [0x555d6103d869] > 14: (OSD::dequeue_op(boost::intrusive_ptr<PG>, > boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1b7) > [0x555d60e8e8a7] > 15: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, > ThreadPool::TPHandle&)+0x62) [0x555d611144c2] > 16: (OSD::ShardedOpWQ::_process(unsigned int, > ceph::heartbeat_handle_d*)+0x592) [0x555d60eb25f2] > 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3d3) > [0x7ff3c929f5b3] > 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7ff3c92a01a0] > 19: (()+0x7e65) [0x7ff3c6203e65] > 20: (clone()+0x6d) [0x7ff3c52f388d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |