Thanks David and Marius for your input. Can you share which versions of Ceph you've experienced this with? And what is the use-case: RGW, CephFS, RBD, mixed ... ? Is there anything special about your ceph usage ? small/large files? num osds ? Num PGs ? When was the cluster created? Thanks! dan On Wed, Jan 12, 2022 at 6:47 AM Marius Leustean <marius.leus@xxxxxxxxx> wrote: > > Had the same issue after a pg_num increase. > > Indeed the convenient solution was to add the needed memory (either a Swap partition or physical RAM). > Things will get back to normal after the initial start, you won’t have to keep that extra ram into your storage nodes. > > This is a really annoying issue, and I never found a proper fix to it. You may run into this when ceph decides to change the pg_num. I hope it will be fixed soon. > > On Wed, 12 Jan 2022 at 04:44, David Yang <gmydw1118@xxxxxxxxx> wrote: >> >> Hi, I have also encountered this problem before, I did not do other >> operations, just added a ssd as large as possible to create a swap >> partition. >> >> At the most when osd is restored, a storage node uses up 2T of swap. Then >> after the osd boots back to normal, the memory will be released and return >> to normal usage. >> >> Dan van der Ster <dan@xxxxxxxxxxxxxx> 于2022年1月12日周三 02:14写道： >> >> > Hi, >> > >> > It sounds like https://tracker.ceph.com/issues/53729 >> > >> > -- Dan >> > >> > >> > On Tue., Jan. 11, 2022, 18:32 Konstantin Larin, <klarin@xxxxxxxxxxxxxxxxxx >> > > >> > wrote: >> > >> > > Hi all, >> > > >> > > We have a problem with our 3 node all-in-one cluster (15.2.15). >> > > >> > > There are 16 OSDs on each node, 16 HDDs for data and 4 SSDs for DB. >> > > >> > > At some point 2 nodes suffered simultaneous power outage with another >> > > subsequent power outage on one of these nodes. Power outage lasted about >> > > an hour. This seemingly also triggered autoscaler that reduced pg_num on >> > > busy pools. >> > > >> > > After that there were numerous OOMs that killed OSDs, even on remaining >> > > node. >> > > >> > > We tried to start OSDs one by one, but starting any single OSD results >> > > in it going into a loop using almost 200 GB of RAM and then aborting >> > > itself, then restarting by systemd. >> > > >> > > The largest numbers in dump_mempools shortly before crash are >> > > buffer_anon which is over 100 GB and osd_pglog which is about 10 GB. >> > > >> > > We have tried manual compaction of OSD DB (ceph-kvstore-tool) and >> > > trimming PG log (ceph-objectstore-tool). This has not changed anything. >> > > >> > > OSDs have common traceback after aborting: >> > > >> > > 0> 2022-01-11T11:11:13.644+0200 7f52f01b3700 -1 *** Caught signal >> > > (Aborted) ** >> > > in thread 7f52f01b3700 thread_name:tp_osd_tp >> > > >> > > ceph version 15.2.15 (2dfb18841cfecc2f7eb7eb2afd65986ca4d95985) >> > > octopus (stable) >> > > 1: (()+0x12c20) [0x7f5313954c20] >> > > 2: (gsignal()+0x10f) [0x7f53125b337f] >> > > 3: (abort()+0x127) [0x7f531259ddb5] >> > > 4: (()+0x9009b) [0x7f5312f6b09b] >> > > 5: (()+0x9653c) [0x7f5312f7153c] >> > > 6: (()+0x96597) [0x7f5312f71597] >> > > 7: (()+0x967f8) [0x7f5312f717f8] >> > > 8: (ceph::buffer::v15_2_0::list::refill_append_space(unsigned >> > > int)+0x200) [0x561cfe88f3a0] >> > > 9: (ceph::buffer::v15_2_0::list::append_hole(unsigned int)+0x8b) >> > > [0x561cfe88f69b] >> > > 10: (pg_log_dup_t::encode(ceph::buffer::v15_2_0::list&) const+0x38) >> > > [0x561cfe2a1a28] >> > > 11: (PGLog::_write_log_and_missing(ceph::os::Transaction&, >> > > std::map<std::__cxx11::basic_string<char, std::char_traits<char>, >> > > std::allocator<char> >, ceph::buffer::v15_2_0::list, >> > > std::less<std::__cxx11::basic_string<char, std::char_trait >> > > s<char>, std::allocator<char> > >, >> > > std::allocator<std::pair<std::__cxx11::basic_string<char, >> > > std::char_traits<char>, std::allocator<char> > const, >> > > ceph::buffer::v15_2_0::list> > >*, pg_log_t&, coll_t const&, ghobject_t >> > > const&, eversion_t >> > > , eversion_t, eversion_t, std::set<eversion_t, std::less<eversion_t>, >> > > std::allocator<eversion_t> >&&, >> > > std::set<std::__cxx11::basic_string<char, std::char_traits<char>, >> > > std::allocator<char> >, std::less<std::__cxx11::basic_string<char, st >> > > d::char_traits<char>, std::allocator<char> > >, >> > > std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, >> > > std::allocator<char> > > >&&, pg_missing_set<true> const&, bool, bool, >> > > bool, eversion_t, eversion_t, eversion_t, bool >> > > *, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, >> > > std::allocator<char> >, std::less<std::__cxx11::basic_string<char, >> > > std::char_traits<char>, std::allocator<char> > >, >> > > std::allocator<std::__cxx11::basic_string<char, std >> > > ::char_traits<char>, std::allocator<char> > > >*)+0xd7c) [0x561cfe1719ec] >> > > 12: (PGLog::write_log_and_missing(ceph::os::Transaction&, >> > > std::map<std::__cxx11::basic_string<char, std::char_traits<char>, >> > > std::allocator<char> >, ceph::buffer::v15_2_0::list, >> > > std::less<std::__cxx11::basic_string<char, std::char_traits >> > > <char>, std::allocator<char> > >, >> > > std::allocator<std::pair<std::__cxx11::basic_string<char, >> > > std::char_traits<char>, std::allocator<char> > const, >> > > ceph::buffer::v15_2_0::list> > >*, coll_t const&, ghobject_t const&, >> > > bool)+0x132) [0x561cfe >> > > 17c2c2] >> > > 13: (PG::prepare_write(pg_info_t&, pg_info_t&, PastIntervals&, PGLog&, >> > > bool, bool, bool, ceph::os::Transaction&)+0x1a6) [0x561cfe130226] >> > > 14: (PeeringState::write_if_dirty(ceph::os::Transaction&)+0x70) >> > > [0x561cfe305ce0] >> > > 15: (OSD::split_pgs(PG*, std::set<spg_t, std::less<spg_t>, >> > > std::allocator<spg_t> > const&, std::set<boost::intrusive_ptr<PG>, >> > > std::less<boost::intrusive_ptr<PG> >, >> > > std::allocator<boost::intrusive_ptr<PG> > >*, std::shared_ptr<OSDMap con >> > > st>, std::shared_ptr<OSDMap const>, PeeringCtx&)+0x57b) [0x561cfe095abb] >> > > 16: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, >> > > PeeringCtx&)+0x70c) [0x561cfe0c2e5c] >> > > 17: (OSD::dequeue_peering_evt(OSDShard*, PG*, >> > > std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xa4) >> > > [0x561cfe0c4bb4] >> > > 18: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, >> > > boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x56) [0x561cfe2f6c76] >> > > 19: (OSD::ShardedOpWQ::_process(unsigned int, >> > > ceph::heartbeat_handle_d*)+0x12ef) [0x561cfe0b7a5f] >> > > 20: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) >> > > [0x561cfe6f6204] >> > > 21: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) >> > [0x561cfe6f8e64] >> > > 22: (()+0x817a) [0x7f531394a17a] >> > > 23: (clone()+0x43) [0x7f5312678dc3] >> > > >> > > There is also a strange thing that ceph osd tree reports some OSDs are >> > > up, when in fact they are not running. >> > > >> > > Could you please suggest on this issue? >> > > >> > > Regards, >> > > Konstantin >> > > _______________________________________________ >> > > ceph-users mailing list -- ceph-users@xxxxxxx >> > > To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > > >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users@xxxxxxx >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |