Re: Multiple OSD crashing a lot

[Blade Doyle <blade.doyle@xxxxxxxxx>]

Sadly, the update to 0.94.6 did not solve the issue.  I still can't get one of my OSD to run at all.  I have included the crash report below.

It looks like the following assert fails:
https://github.com/ceph/ceph/blob/v0.94.6/src/osd/ReplicatedPG.cc  line 10495

ObjectContextRef obc = get_object_context(oid, false);
assert(obc);

I already tried rebuilding the OS and updated ceph on that node, so there must be some problem with the OSD data itself.

At this point I will choose between hacking that assert out of the source code and see if I can just ignore the problem, or I might mark that OSD as "lost" and hope for minimal data loss.  I imagine skipping the assert will probably just lead to a SEGV a few lines later, so im leaning toward the "lost" osd solution.  Suggestions welcome.

Blade.

(0.94.6 failure stack trace follows)

root@node8:~# /usr/bin/ceph-osd -i 2 --pid-file /var/run/ceph/osd.2.pid -c /etc/ceph/ceph.conf --cluster ceph -f
starting osd.2 at :/0 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal
2016-04-23 11:47:14.294809 b68c7000 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2016-04-23 11:47:22.388962 b68c7000 -1 osd.2 338888 log_to_monitors {default=true}
osd/ReplicatedPG.cc: In function 'void ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)' thread 9d0bb350 time 2016-04-23 11:47:26.570268
osd/ReplicatedPG.cc: 10495: FAILED assert(obc)
 ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x70) [0x7774c8]
 2: (ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)+0xa48) [0x44f8f0]
 3: (ReplicatedPG::hit_set_persist()+0xf18) [0x450a78]
 4: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0xd9c) [0x45c788]
 5: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x554) [0x3fa028]
 6: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a8) [0x2707a0]
 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x428) [0x270e70]
 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x740) [0x7696a0]
 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x76c1f4]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2016-04-23 11:47:26.585694 9d0bb350 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)' thread 9d0bb350 time 2016-04-23 11:47:26.570268
osd/ReplicatedPG.cc: 10495: FAILED assert(obc)

 ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x70) [0x7774c8]
 2: (ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)+0xa48) [0x44f8f0]
 3: (ReplicatedPG::hit_set_persist()+0xf18) [0x450a78]
 4: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0xd9c) [0x45c788]
 5: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x554) [0x3fa028]
 6: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a8) [0x2707a0]
 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x428) [0x270e70]
 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x740) [0x7696a0]
 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x76c1f4]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

 -3175> 2016-04-23 11:47:14.294809 b68c7000 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
 -2425> 2016-04-23 11:47:22.388962 b68c7000 -1 osd.2 338888 log_to_monitors {default=true}
     0> 2016-04-23 11:47:26.585694 9d0bb350 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)' thread 9d0bb350 time 2016-04-23 11:47:26.570268
osd/ReplicatedPG.cc: 10495: FAILED assert(obc)

 ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x70) [0x7774c8]
 2: (ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)+0xa48) [0x44f8f0]
 3: (ReplicatedPG::hit_set_persist()+0xf18) [0x450a78]
 4: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0xd9c) [0x45c788]
 5: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x554) [0x3fa028]
 6: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a8) [0x2707a0]
 7: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x428) [0x270e70]
 8: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x740) [0x7696a0]
 9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x76c1f4]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

terminate called after throwing an instance of 'ceph::FailedAssertion'
*** Caught signal (Aborted) **
 in thread 9d0bb350
 ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
 1: /usr/bin/ceph-osd() [0x69764c]
 2: (__default_sa_restorer()+0) [0xb694ed10]
 3: (gsignal()+0x38) [0xb694daa8]
2016-04-23 11:47:26.856125 9d0bb350 -1 *** Caught signal (Aborted) **
 in thread 9d0bb350

 ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
 1: /usr/bin/ceph-osd() [0x69764c]
 2: (__default_sa_restorer()+0) [0xb694ed10]
 3: (gsignal()+0x38) [0xb694daa8]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

     0> 2016-04-23 11:47:26.856125 9d0bb350 -1 *** Caught signal (Aborted) **
 in thread 9d0bb350

 ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
 1: /usr/bin/ceph-osd() [0x69764c]
 2: (__default_sa_restorer()+0) [0xb694ed10]
 3: (gsignal()+0x38) [0xb694daa8]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Aborted

On Thu, Apr 21, 2016 at 12:40 AM, Blade Doyle <blade.doyle@xxxxxxxxx> wrote:

That was a poor example, because it was an older version of ceph and the clock was not set correctly.   But I don't think either of those things causes the problem because I see it on multiple nodes:

root@node8:/var/log/ceph# grep hit_set_trim ceph-osd.2.log | wc -l
2524
root@node8:/var/log/ceph# ceph --version
ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)
root@node8:/var/log/ceph# date
Wed 20 Apr 23:58:59 PDT 2016

I saw this:  http://tracker.ceph.com/issues/9732 so I set all the timezones.  Didn't solve the problem.  I am building 0.94.6 anyway.

Thanks,

Blade.

On Wed, Apr 20, 2016 at 12:37 AM, Blade Doyle <blade.doyle@xxxxxxxxx> wrote:

I get a lot of osd crash with the following stack - suggestion please:

     0> 1969-12-31 16:04:55.455688 83ccf410 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)' thread 83ccf410 time 295.324905
osd/ReplicatedPG.cc: 11011: FAILED assert(obc)

 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)
 1: (ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)+0x3f9) [0xb6c625e6]
 2: (ReplicatedPG::hit_set_persist()+0x8bf) [0xb6c62fb4]
 3: (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>)+0xc97) [0xb6c6eb2c]
 4: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x439) [0xb6c2f01a]
 5: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x22b) [0xb6b0b984]
 6: (OSD::OpWQ::_process(boost::intrusive_ptr<PG>, ThreadPool::TPHandle&)+0x13d) [0xb6b1ccf6]
 7: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest> >, boost::intrusive_ptr<PG> >::_void_process(void*, ThreadPool::TPHandle&)+0x6b) [0xb6b4692c]
 8: (ThreadPool::worker(ThreadPool::WorkThread*)+0xb93) [0xb6e152bc]
 9: (ThreadPool::WorkThread::entry()+0x9) [0xb6e15aea]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Blade.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com