Re: Lots of OSDs crashlooping (DRAFT - feedback?)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



oh jeez, sorry about the subject line - I forgot to change it after asking
a coworker to review the message.  This is not a draft.

On Mon, Jan 24, 2022 at 6:44 PM Benjamin Staffin <bstaffin@xxxxxxxxxxxxxxx>
wrote:

> I have a cluster where 46 out of 120 OSDs have begun crash looping with
> the same stack trace (see pasted output below).  The cluster is in a very
> bad state with this many OSDs down, unsurprisingly.
>
> The day before this problem showed up, the k8s cluster was under extreme
> memory pressure and a lot of pods were OOM killed, including some of the
> Ceph OSDs, but after the memory pressure abated everything seemed to
> stabilize for about a day.
>
> Then we attempted to set a 4gb memory limit on the OSD pods, because they
> had been using upwards of 100gb of ram(!) per OSD after about a month of
> uptime, and this was a contributing factor in the cluster-wide OOM
> situation.  Everything seemed fine for a few minutes after Rook rolled out
> the memory limit, but then OSDs gradually started to crash, a few at a
> time, up to about 30 of them.  At this point I reverted the memory limit,
> but I don't think the OSDs were hitting their memory limits at all.  In an
> attempt to stabilize the cluster, we eventually the Rook operator and set
> the osd norebalance, nobackfill, noout, and norecover flags, but at this
> point there were 46 OSDs down and pools were hitting BackFillFull.
>
> This is a Rook-ceph deployment on bare-metal kubernetes cluster of 12
> nodes.  Each node has two 7TiB nvme disks dedicated to Ceph, and we have 5
> BlueStore OSDs per nvme disk (so around 1.4TiB per OSD, which ough to be
> fine with a 4gb memory target, right?).  The crash we're seeing looks very
> much like the one in this bug report:
> https://tracker.ceph.com/issues/52220
>
> I don't know how to proceed from here, so any advice would be very much
> appreciated.
>
> Ceph version: 16.2.6
> Rook version: 1.7.6
> Kubernetes version: 1.21.5
> Kernel version: 5.4.156-1.el7.elrepo.x86_64
> Distro: CentOS 7.9
>
> I've also attached the full log output from one of the crashing OSDs, in
> case that is of any use.
>
> ----begin stack trace paste----
> debug     -1> 2022-01-24T22:09:09.405+0000 7ff8b4315700 -1
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/osd/ECUtil.cc:
> In function 'void ECUtil::HashInfo::append(uint64_t, std::map<int,
> ceph::buffer::v15_2_0::list>&)' thread 7ff8b4315700 time
> 2022-01-24T22:09:09.398961+0000
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/osd/ECUtil.cc:
> 169: FAILED ceph_assert(to_append.size() == cumulative_shard_hashes.size())
>
>  ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific
> (stable)
>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x158) [0x564f88db554c]
>  2: ceph-osd(+0x56a766) [0x564f88db5766]
>  3: (ECUtil::HashInfo::append(unsigned long, std::map<int,
> ceph::buffer::v15_2_0::list, std::less<int>, std::allocator<std::pair<int
> const, ceph::buffer::v15_2_0::list> > >&)+0x14b) [0x564f8910ca0b]
>  4: (encode_and_write(pg_t, hobject_t const&, ECUtil::stripe_info_t
> const&, std::shared_ptr<ceph::ErasureCodeInterface>&, std::set<int,
> std::less<int>, std::allocator<int> > const&, unsigned long,
> ceph::buffer::v15_2_0::list, unsigned int,
> std::shared_ptr<ECUtil::HashInfo>, interval_map<unsigned long,
> ceph::buffer::v15_2_0::list, bl_split_merge>&, std::map<shard_id_t,
> ceph::os::Transaction, std::less<shard_id_t>,
> std::allocator<std::pair<shard_id_t const, ceph::os::Transaction> > >*,
> DoutPrefixProvider*)+0x6ec) [0x564f8929fa7c]
>  5: ceph-osd(+0xa5a611) [0x564f892a5611]
>  6: (ECTransaction::generate_transactions(ECTransaction::WritePlan&,
> std::shared_ptr<ceph::ErasureCodeInterface>&, pg_t, ECUtil::stripe_info_t
> const&, std::map<hobject_t, interval_map<unsigned long,
> ceph::buffer::v15_2_0::list, bl_split_merge>, std::less<hobject_t>,
> std::allocator<std::pair<hobject_t const, interval_map<unsigned long,
> ceph::buffer::v15_2_0::list, bl_split_merge> > > > const&,
> std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> >&,
> std::map<hobject_t, interval_map<unsigned long,
> ceph::buffer::v15_2_0::list, bl_split_merge>, std::less<hobject_t>,
> std::allocator<std::pair<hobject_t const, interval_map<unsigned long,
> ceph::buffer::v15_2_0::list, bl_split_merge> > > >*, std::map<shard_id_t,
> ceph::os::Transaction, std::less<shard_id_t>,
> std::allocator<std::pair<shard_id_t const, ceph::os::Transaction> > >*,
> std::set<hobject_t, std::less<hobject_t>, std::allocator<hobject_t> >*,
> std::set<hobject_t, std::less<hobject_t>, std::allocator<hobject_t> >*,
> DoutPrefixProvider*, ceph_release_t)+0x7db) [0x564f892a6dcb]
>  7: (ECBackend::try_reads_to_commit()+0x468) [0x564f8927ec28]
>  8: (ECBackend::check_ops()+0x24) [0x564f89281cd4]
>  9: (CallClientContexts::finish(std::pair<RecoveryMessages*,
> ECBackend::read_result_t&>&)+0x1278) [0x564f8929d338]
>  10: (ECBackend::complete_read_op(ECBackend::ReadOp&,
> RecoveryMessages*)+0x8f) [0x564f8926dfaf]
>  11: (ECBackend::handle_sub_read_reply(pg_shard_t, ECSubReadReply&,
> RecoveryMessages*, ZTracer::Trace const&)+0x1196) [0x564f89287106]
>  12: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x18f)
> [0x564f89287bdf]
>  13: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x52)
> [0x564f8908dd12]
>  14: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&,
> ThreadPool::TPHandle&)+0x5de) [0x564f89030d6e]
>  15: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
> boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x309)
> [0x564f88eba1b9]
>  16: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*,
> boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x68) [0x564f89117868]
>  17: (OSD::ShardedOpWQ::_process(unsigned int,
> ceph::heartbeat_handle_d*)+0xa58) [0x564f88eda1e8]
>  18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4)
> [0x564f895456c4]
>  19: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x564f89548364]
>  20: /lib64/libpthread.so.0(+0x814a) [0x7ff8db40e14a]
>  21: clone()
>
> debug      0> 2022-01-24T22:09:09.411+0000 7ff8b4315700 -1 *** Caught
> signal (Aborted) **
>  in thread 7ff8b4315700 thread_name:tp_osd_tp
> ----end paste----
>
> # ceph status
>   cluster:
>     id:     a262fadd-b995-4861-9cb0-06c1f1eddaf7
>     health: HEALTH_ERR
>             1 MDSs report slow metadata IOs
>             1277/19235302 objects unfound (0.007%)
>             noout,nobackfill,norebalance,norecover flag(s) set
>             1 backfillfull osd(s)
>             46 osds down
>             15 nearfull osd(s)
>             Reduced data availability: 1470 pgs inactive, 577 pgs down,
> 1615 pgs stale
>             Possible data damage: 22 pgs recovery_unfound
>             Degraded data redundancy: 18956079/115409079 objects degraded
> (16.425%), 1942 pgs degraded, 1941 pgs undersized
>             13 pool(s) backfillfull
>             1309 daemons have recently crashed
>
>   services:
>     mon: 3 daemons, quorum b,c,d (age 2d)
>     mgr: a(active, since 2d)
>     mds: 1/1 daemons up, 1 hot standby
>     osd: 120 osds: 74 up (since 27s), 120 in (since 38m); 1817 remapped pgs
>          flags noout,nobackfill,norebalance,norecover
>
>   data:
>     volumes: 1/1 healthy
>     pools:   13 pools, 3234 pgs
>     objects: 19.24M objects, 72 TiB
>     usage:   80 TiB used, 41 TiB / 122 TiB avail
>     pgs:     45.455% pgs not active
>              18956079/115409079 objects degraded (16.425%)
>              2329606/115409079 objects misplaced (2.019%)
>              1277/19235302 objects unfound (0.007%)
>              326 undersized+degraded+remapped+backfill_wait+peered
>              325 active+clean
>              325 stale+active+undersized+degraded+remapped+backfill_wait
>              319 active+undersized+degraded+remapped+backfill_wait
>              311 stale+undersized+degraded+remapped+backfill_wait+peered
>              302 stale+active+clean
>              278 stale+down+remapped
>              217 down+remapped
>              149 active+recovery_wait+undersized+degraded+remapped
>              127 stale+active+recovery_wait+undersized+degraded+remapped
>              119 stale+recovery_wait+undersized+degraded+remapped+peered
>              107 recovery_wait+undersized+degraded+remapped+peered
>              57  active+undersized+degraded
>              50  stale+active+undersized+degraded
>              46  down
>              36  stale+down
>              31  active+remapped+backfill_wait
>              30  stale+active+remapped+backfill_wait
>              10  active+recovery_unfound+degraded
>              9   stale+active+undersized+remapped+backfill_wait
>              7   stale+undersized+degraded+remapped+backfilling+peered
>              6   active+undersized
>              6   undersized+degraded+remapped+backfilling+peered
>              5   active+undersized+remapped+backfill_wait
>              5   stale+active+recovery_unfound+degraded
>              4   stale+active+recovery_wait+degraded
>              4   active+recovery_wait+degraded+remapped
>              3   undersized+remapped+backfill_wait+peered
>              3   recovery_unfound+undersized+degraded+remapped+peered
>              3   stale+recovery_unfound+undersized+degraded+remapped+peered
>              3   stale+undersized+degraded+peered
>              3   stale+undersized+remapped+backfill_wait+peered
>              2   active+recovery_wait+degraded
>              2   stale+active+recovery_wait+degraded+remapped
>              1   recovery_wait+undersized+degraded+peered
>              1   active+recovery_unfound+undersized+degraded
>              1   active+clean+remapped
>              1   stale+recovery_wait+undersized+degraded+peered
>
>
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux