Hmm, this ceph_assert hasn't shown up in my email before. It looks like there may be a soft-state bug in Octopus. Can you file a ticket at tracker.ceph.com with the bcaktrace and osd log file? We can direct that to the RADOS team to check out. -Greg On Sat, Aug 28, 2021 at 7:13 AM Amudhan P <amudhan83@xxxxxxxxx> wrote: > > Hi, > > I am having a peculiar problem with my ceph octopus cluster. 2 weeks ago I > had an issue that started like too many scrub error and later random OSD > stopped which lead to pg corrupt, replica missing. since it's a testing > cluster I wanted to understand the issue. > I tried to recover PG but it didn't help. when I use `set norecover, > norebalance, nodown` OSD service running without stopping. > > I have gone thru the steps in ceph osd troubleshooting but nothing helps or > leads to finding the issue. > > I have mailed earlier but couldn't get any solution. > > Any help would be appreciated to find out the issue. > > *error msg in one of the OSD which failed.* > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.7/rpm/el8/BUILD/ > ceph-15.2.7/src/osd/OSD.cc: 9521: FAILED ceph_assert(started <= > reserved_pushes) > > ceph version 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) octopus > (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x158) [0x55fcb6621dbe] > 2: (()+0x504fd8) [0x55fcb6621fd8] > 3: (OSD::do_recovery(PG*, unsigned int, unsigned long, > ThreadPool::TPHandle&)+0x5f5) [0x55fcb6704c25] > 4: (ceph::osd::scheduler::PGRecovery::run(OSD*, OSDShard*, > boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x1d) [0x55fcb6960a3d] > 5: (OSD::ShardedOpWQ::_process(unsigned int, > ceph::heartbeat_handle_d*)+0x12ef) [0x55fcb67224df] > 6: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) > [0x55fcb6d5b224] > 7: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55fcb6d5de84] > 8: (()+0x82de) [0x7f04c1b1c2de] > 9: (clone()+0x43) [0x7f04c0853e83] > > 0> 2021-08-28T13:53:37.444+0000 7f04a128d700 -1 *** Caught signal > (Aborted) ** > in thread 7f04a128d700 thread_name:tp_osd_tp > > ceph version 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) octopus > (stable) > 1: (()+0x12dd0) [0x7f04c1b26dd0] > 2: (gsignal()+0x10f) [0x7f04c078f70f] > 3: (abort()+0x127) [0x7f04c0779b25] > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x1a9) [0x55fcb6621e0f] > 5: (()+0x504fd8) [0x55fcb6621fd8] > 6: (OSD::do_recovery(PG*, unsigned int, unsigned long, > ThreadPool::TPHandle&)+0x5f5) [0x55fcb6704c25] > 7: (ceph::osd::scheduler::PGRecovery::run(OSD*, OSDShard*, > boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x1d) [0x55fcb6960a3d] > 8: (OSD::ShardedOpWQ::_process(unsigned int, > ceph::heartbeat_handle_d*)+0x12ef) [0x55fcb67224df] > 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) > [0x55fcb6d5b224] > 10: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55fcb6d5de84] > 11: (()+0x82de) [0x7f04c1b1c2de] > 12: (clone()+0x43) [0x7f04c0853e83] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed > to interpret this. > > > Thanks > Amudhan > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |