Hi Kefu, The following runs on OpenStack and the next branch http://integration.ceph.dachary.org:8081/ubuntu-2015-07-21_00:04:04-rados-next---basic-openstack/ and 15 out of the 16 dead jobs (timed out after 3 hours) are from rados/thrash. A rados suite run on next dated a few days ago in the sepia lab ( http://pulpito.ceph.com/teuthology-2015-07-15_21:00:10-rados-next-distro-basic-multi/ ) also has a few dead jobs but only two of them are from rados/thrash. Cheers On 20/07/2015 16:23, Loic Dachary wrote: > More information about this run. I'll run a rados suite on master on OpenStack to get a baseline of what we should expect. > > http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/12/ > http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/14/ > http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/15/ > http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/17/ > http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/20/ > http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/21/ > http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/22/ > http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/23/ > http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/26/ > http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/28/ > http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/2/ > http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/5/ > http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/6/ > http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/7/ > http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/9/ > > I see > > 2015-07-20T10:02:10.567 INFO:tasks.ceph.osd.5.ovh165019.stderr:osd/ReplicatedPG.cc: In function 'bool ReplicatedPG::is_degraded_or_backfilling_object(const hobject_t&)' thread 7f2af94df700 time 2015-07-20 10:02:10.481916 > 2015-07-20T10:02:10.567 INFO:tasks.ceph.osd.5.ovh165019.stderr:osd/ReplicatedPG.cc: 412: FAILED assert(!actingbackfill.empty()) > 2015-07-20T10:02:10.567 INFO:tasks.ceph.osd.5.ovh165019.stderr: ceph version 9.0.2-799-gba9c2ae (ba9c2ae4bffd3fd7b26a2e0ce843913b77940b8a) > 2015-07-20T10:02:10.568 INFO:tasks.ceph.osd.5.ovh165019.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xc45d1b] > 2015-07-20T10:02:10.568 INFO:tasks.ceph.osd.5.ovh165019.stderr: 2: ceph-osd() [0x88535d] > 2015-07-20T10:02:10.568 INFO:tasks.ceph.osd.5.ovh165019.stderr: 3: (ReplicatedPG::hit_set_remove_all()+0x7c) [0x8b039c] > 2015-07-20T10:02:10.568 INFO:tasks.ceph.osd.5.ovh165019.stderr: 4: (ReplicatedPG::on_pool_change()+0x161) [0x8b1a21] > 2015-07-20T10:02:10.569 INFO:tasks.ceph.osd.5.ovh165019.stderr: 5: (PG::handle_advance_map(std::tr1::shared_ptr<OSDMap const>, std::tr1::shared_ptr<OSDMap const>, std::vector<int, std::allocator<int> >&, int, std::vector<int, std::allocator<int> >&, int, PG::RecoveryCtx*)+0x60c) [0x8348fc] > 2015-07-20T10:02:10.569 INFO:tasks.ceph.osd.5.ovh165019.stderr: 6: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x2c3) [0x6dcc73] > 2015-07-20T10:02:10.569 INFO:tasks.ceph.osd.5.ovh165019.stderr: 7: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x1f1) [0x6dd721] > 2015-07-20T10:02:10.572 INFO:tasks.ceph.osd.5.ovh165019.stderr: 8: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x18) [0x7328d8] > 2015-07-20T10:02:10.573 INFO:tasks.ceph.osd.5.ovh165019.stderr: 9: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa5e) [0xc3677e] > 2015-07-20T10:02:10.573 INFO:tasks.ceph.osd.5.ovh165019.stderr: 10: (ThreadPool::WorkThread::entry()+0x10) [0xc37820] > 2015-07-20T10:02:10.573 INFO:tasks.ceph.osd.5.ovh165019.stderr: 11: (()+0x8182) [0x7f2b149e3182] > 2015-07-20T10:02:10.573 INFO:tasks.ceph.osd.5.ovh165019.stderr: 12: (clone()+0x6d) [0x7f2b12d2847d] > > > In > > http://149.202.164.239/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/24/ > > I see the same error as below. > > In > > http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/8/ > > it looks like the run was about to finish, just took a long time, and should be ignored as a false negative. > > On 20/07/2015 14:52, Loic Dachary wrote: >> Hi, >> >> I checked one of the timeout (dead) at http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/ >> >> http://149.202.164.239/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/10/config.yaml >> timeed out because of >> >> >> Paste2 >> >> Create Paste >> Followup Paste >> QR >> >> sd.5 since back 2015-07-20 10:45:28.566308 front 2015-07-20 10:45:28.566308 (cutoff 2015-07-20 10:45:33.823074) >> 2015-07-20T10:47:13.921 INFO:tasks.ceph.osd.4.ovh164254.stderr:2015-07-20 10:47:13.899770 7fb4be171700 -1 osd.4 655 heartbeat_check: no reply from osd.5 since back 2015-07-20 10:45:30.719801 front 2015-07-20 10:45:30.719801 (cutoff 2015-07-20 10:45:33.899763) >> 2015-07-20T10:47:15.023 INFO:tasks.ceph.osd.1.ovh164253.stderr:osd/ReplicatedPG.cc: In function 'virtual void ReplicatedPG::op_applied(const eversion_t&)' thread 7f92f0244700 time 2015-07-20 10:47:14.998470 >> 2015-07-20T10:47:15.024 INFO:tasks.ceph.osd.1.ovh164253.stderr:osd/ReplicatedPG.cc: 7311: FAILED assert(applied_version <= info.last_update) >> 2015-07-20T10:47:15.025 INFO:tasks.ceph.osd.1.ovh164253.stderr: ceph version 9.0.2-799-gba9c2ae (ba9c2ae4bffd3fd7b26a2e0ce843913b77940b8a) >> 2015-07-20T10:47:15.025 INFO:tasks.ceph.osd.1.ovh164253.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xc45d1b] >> 2015-07-20T10:47:15.025 INFO:tasks.ceph.osd.1.ovh164253.stderr: 2: (ReplicatedPG::op_applied(eversion_t const&)+0x6dc) [0x8741ac] >> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 3: (ReplicatedBackend::op_applied(ReplicatedBackend::InProgressOp*)+0xd0) [0xa5cfe0] >> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 4: (Context::complete(int)+0x9) [0x6f4649] >> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 5: (ReplicatedPG::BlessedContext::finish(int)+0x94) [0x8dec54] >> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 6: (Context::complete(int)+0x9) [0x6f4649] >> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 7: (void finish_contexts<Context>(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0x94) [0x7351d4] >> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 8: (C_ContextsBase<Context, Context>::complete(int)+0x9) [0x6f4e89] >> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 9: (Finisher::finisher_thread_entry()+0x158) [0xb6f2b8] >> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 10: (()+0x8182) [0x7f92ff4e7182] >> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 11: (clone()+0x6d) [0x7f92fd82c47d] >> 2015-07-20T10:47:15.027 INFO:tasks.ceph.osd.1.ovh164253.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. >> 2015-07-20T10:47:15.038 INFO:tasks.ceph.osd.1.ovh164253.stderr:2015-07-20 10:47:15.005862 7f92f0244700 -1 osd/ReplicatedPG.cc: In function 'virtual void ReplicatedPG::op_applied(const eversion_t&)' thread 7f92f0244700 time 2015-07-20 10:47:14.998470 >> 2015-07-20T10:47:15.039 INFO:tasks.ceph.osd.1.ovh164253.stderr:osd/ReplicatedPG.cc: 7311: FAILED assert(applied_version <= info.last_update) >> 2015-07-20T10:47:15.039 INFO:tasks.ceph.osd.1.ovh164253.stderr: >> 2015-07-20T10:47:15.039 INFO:tasks.ceph.osd.1.ovh164253.stderr: ceph version 9.0.2-799-gba9c2ae (ba9c2ae4bffd3fd7b26a2e0ce843913b77940b8a) >> 2015-07-20T10:47:15.039 INFO:tasks.ceph.osd.1.ovh164253.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xc45d1b] >> 2015-07-20T10:47:15.039 INFO:tasks.ceph.osd.1.ovh164253.stderr: 2: (ReplicatedPG::op_applied(eversion_t const&)+0x6dc) [0x8741ac] >> 2015-07-20T10:47:15.039 INFO:tasks.ceph.osd.1.ovh164253.stderr: 3: (ReplicatedBackend::op_applied(ReplicatedBackend::InProgressOp*)+0xd0) [0xa5cfe0] >> 2015-07-20T10:47:15.039 INFO:tasks.ceph.osd.1.ovh164253.stderr: 4: (Context::complete(int)+0x9) [0x6f4649] >> 2015-07-20T10:47:15.039 INFO:tasks.ceph.osd.1.ovh164253.stderr: 5: (ReplicatedPG::BlessedContext::finish(int)+0x94) [0x8dec54] >> 2015-07-20T10:47:15.040 INFO:tasks.ceph.osd.1.ovh164253.stderr: 6: (Context::complete(int)+0x9) [0x6f4649] >> 2015-07-20T10:47:15.040 INFO:tasks.ceph.osd.1.ovh164253.stderr: 7: (void finish_contexts<Context>(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0x94) [0x7351d4] >> 2015-07-20T10:47:15.040 INFO:tasks.ceph.osd.1.ovh164253.stderr: 8: (C_ContextsBase<Context, Context>::complete(int)+0x9) [0x6f4e89] >> 2015-07-20T10:47:15.040 INFO:tasks.ceph.osd.1.ovh164253.stderr: 9: (Finisher::finisher_thread_entry()+0x158) [0xb6f2b8] >> 2015-07-20T10:47:15.040 INFO:tasks.ceph.osd.1.ovh164253.stderr: 10: (()+0x8182) [0x7f92ff4e7182] >> 2015-07-20T10:47:15.040 INFO:tasks.ceph.osd.1.ovh164253.stderr: 11: (clone()+0x6d) [0x7f92fd82c47d] >> 2015-07-20T10:47:15.040 INFO:tasks.ceph.osd.1.ovh164253.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. >> 2015-07-20T10:47:15.041 INFO:tasks.ceph.osd.1.ovh164253.stderr: >> 2015-07-20T10:47:15.212 INFO:tasks.ceph.osd.1.ovh164253.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion' >> 2015-07-20T10:47:15.212 INFO:tasks.ceph.osd.1.ovh164253.stderr:*** Caught signal (Aborted) ** >> 2015-07-20T10:47:15.212 INFO:tasks.ceph.osd.1.ovh164253.stderr: in thread 7f92f0244700 >> 2015-07-20T10:47:15.217 INFO:tasks.ceph.osd.1.ovh164253.stderr: ceph version 9.0.2-799-gba9c2ae (ba9c2ae4bffd3fd7b26a2e0ce843913b77940b8a) >> 2015-07-20T10:47:15.217 INFO:tasks.ceph.osd.1.ovh164253.stderr: 1: ceph-osd() [0xb49fba] >> 2015-07-20T10:47:15.217 INFO:tasks.ceph.osd.1.ovh164253.stderr: 2: (()+0x10340) [0x7f92ff4ef340] >> 2015-07-20T10:47:15.218 INFO:tasks.ceph.osd.1.ovh164253.stderr: 3: (gsignal()+0x39) [0x7f92fd768cc9] >> 2015-07-20T10:47:15.218 INFO:tasks.ceph.osd.1.ovh164253.stderr: 4: (abort()+0x148) [0x7f92fd76c0d8] >> 2015-07-20T10:47:15.218 INFO:tasks.ceph.osd.1.ovh164253.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f92fe073535] >> 2015-07-20T10:47:15.218 INFO:tasks.ceph.osd.1.ovh164253.stderr: 6: (()+0x5e6d6) [0x7f92fe0716d6] >> 2015-07-20T10:47:15.218 INFO:tasks.ceph.osd.1.ovh164253.stderr: 7: (()+0x5e703) [0x7f92fe071703] >> 2015-07-20T10:47:15.219 INFO:tasks.ceph.osd.1.ovh164253.stderr: 8: (()+0x5e922) [0x7f92fe071922] >> 2015-07-20T10:47:15.219 INFO:tasks.ceph.osd.1.ovh164253.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xc45f08] >> 2015-07-20T10:47:15.219 INFO:tasks.ceph.osd.1.ovh164253.stderr: 10: (ReplicatedPG::op_applied(eversion_t const&)+0x6dc) [0x8741ac] >> 2015-07-20T10:47:15.219 INFO:tasks.ceph.osd.1.ovh164253.stderr: 11: (ReplicatedBackend::op_applied(ReplicatedBackend::InProgressOp*)+0xd0) [0xa5cfe0] >> 2015-07-20T10:47:15.219 INFO:tasks.ceph.osd.1.ovh164253.stderr: 12: (Context::complete(int)+0x9) [0x6f4649] >> 2015-07-20T10:47:15.219 INFO:tasks.ceph.osd.1.ovh164253.stderr: 13: (ReplicatedPG::BlessedContext::finish(int)+0x94) [0x8dec54] >> 2015-07-20T10:47:15.219 INFO:tasks.ceph.osd.1.ovh164253.stderr: 14: (Context::complete(int)+0x9) [0x6f4649] >> 2015-07-20T10:47:15.219 INFO:tasks.ceph.osd.1.ovh164253.stderr: 15: (void finish_contexts<Context>(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0x94) [0x7351d4] >> 2015-07-20T10:47:15.220 INFO:tasks.ceph.osd.1.ovh164253.stderr: 16: (C_ContextsBase<Context, Context>::complete(int)+0x9) [0x6f4e89] >> 2015-07-20T10:47:15.220 INFO:tasks.ceph.osd.1.ovh164253.stderr: 17: (Finisher::finisher_thread_entry()+0x158) [0xb6f2b8] >> 2015-07-20T10:47:15.220 INFO:tasks.ceph.osd.1.ovh164253.stderr: 18: (()+0x8182) [0x7f92ff4e7182] >> 2015-07-20T10:47:15.220 INFO:tasks.ceph.osd.1.ovh164253.stderr: 19: (clone()+0x6d) [0x7f92fd82c47d] >> 2015-07-20T10:47:15.221 INFO:tasks.ceph.osd.1.ovh164253.stderr:2015-07-20 10:47:15.197571 7f92f0244700 -1 *** Caught signal (Aborted) ** >> 2015-07-20T10:47:15.221 INFO:tasks.ceph.osd.1.ovh164253.stderr: in thread 7f92f0244700 >> 2015-07-20T10:47:15.221 INFO:tasks.ceph.osd.1.ovh164253.stderr: >> 2015-07-20T10:47:15.221 INFO:tasks.ceph.osd.1.ovh164253.stderr: ceph version 9.0.2-799-gba9c2ae (ba9c2ae4bffd3fd7b26a2e0ce843913b77940b8a) >> 2015-07-20T10:47:15.221 INFO:tasks.ceph.osd.1.ovh164253.stderr: 1: ceph-osd() [0xb49fba] >> 2015-07-20T10:47:15.222 INFO:tasks.ceph.osd.1.ovh164253.stderr: 2: (()+0x10340) [0x7f92ff4ef340] >> 2015-07-20T10:47:15.222 INFO:tasks.ceph.osd.1.ovh164253.stderr: 3: (gsignal()+0x39) [0x7f92fd768cc9] >> 2015-07-20T10:47:15.222 INFO:tasks.ceph.osd.1.ovh164253.stderr: 4: (abort()+0x148) [0x7f92fd76c0d8] >> 2015-07-20T10:47:15.222 INFO:tasks.ceph.osd.1.ovh164253.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f92fe073535] >> 2015-07-20T10:47:15.222 INFO:tasks.ceph.osd.1.ovh164253.stderr: 6: (()+0x5e6d6) [0x7f92fe0716d6] >> 2015-07-20T10:47:15.222 INFO:tasks.ceph.osd.1.ovh164253.stderr: 7: (()+0x5e703) [0x7f92fe071703] >> 2015-07-20T10:47:15.222 INFO:tasks.ceph.osd.1.ovh164253.stderr: 8: (()+0x5e922) [0x7f92fe071922] >> 2015-07-20T10:47:15.223 INFO:tasks.ceph.osd.1.ovh164253.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xc45f08] >> 2015-07-20T10:47:15.223 INFO:tasks.ceph.osd.1.ovh164253.stderr: 10: (ReplicatedPG::op_applied(eversion_t const&)+0x6dc) [0x8741ac] >> 2015-07-20T10:47:15.223 INFO:tasks.ceph.osd.1.ovh164253.stderr: 11: (ReplicatedBackend::op_applied(ReplicatedBackend::InProgressOp*)+0xd0) [0xa5cfe0] >> 2015-07-20T10:47:15.223 INFO:tasks.ceph.osd.1.ovh164253.stderr: 12: (Context::complete(int)+0x9) [0x6f4649] >> 2015-07-20T10:47:15.223 INFO:tasks.ceph.osd.1.ovh164253.stderr: 13: (ReplicatedPG::BlessedContext::finish(int)+0x94) [0x8dec54] >> 2015-07-20T10:47:15.223 INFO:tasks.ceph.osd.1.ovh164253.stderr: 14: (Context::complete(int)+0x9) [0x6f4649] >> 2015-07-20T10:47:15.223 INFO:tasks.ceph.osd.1.ovh164253.stderr: 15: (void finish_contexts<Context>(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0x94) [0x7351d4] >> 2015-07-20T10:47:15.224 INFO:tasks.ceph.osd.1.ovh164253.stderr: 16: (C_ContextsBase<Context, Context>::complete(int)+0x9) [0x6f4e89] >> 2015-07-20T10:47:15.224 INFO:tasks.ceph.osd.1.ovh164253.stderr: 17: (Finisher::finisher_thread_entry()+0x158) [0xb6f2b8] >> 2015-07-20T10:47:15.224 INFO:tasks.ceph.osd.1.ovh164253.stderr: 18: (()+0x8182) [0x7f92ff4e7182] >> 2015-07-20T10:47:15.224 INFO:tasks.ceph.osd.1.ovh164253.stderr: 19: (clone()+0x6d) [0x7f92fd82c47d] >> 2015-07-20T10:47:15.224 INFO:tasks.ceph.osd.1.ovh164253.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. >> 2015-07-20T10:47:15.224 INFO:tasks.ceph.osd.1.ovh164253.stderr: >> 2015-07-20T10:47:15.238 INFO:tasks.ceph.osd.1.ovh164253.stderr: -172> 2015-07-20 10:47:15.197571 7f92f0244700 -1 *** Caught signal (Aborted) ** >> 2015-07-20T10:47:15.239 INFO:tasks.ceph.osd.1.ovh164253.stderr: in thread 7f92f0244700 >> 2015-07-20T10:47:15.239 INFO:tasks.ceph.osd.1.ovh164253.stderr: >> 2015-07-20T10:47:15.239 INFO:tasks.ceph.osd.1.ovh164253.stderr: ceph version 9.0.2-799-gba9c2ae (ba9c2ae4bffd3fd7b26a2e0ce843913b77940b8a) >> 2015-07-20T10:47:15.239 INFO:tasks.ceph.osd.1.ovh164253.stderr: 1: ceph-osd() [0xb49fba] >> 2015-07-20T10:47:15.239 INFO:tasks.ceph.osd.1.ovh164253.stderr: 2: (()+0x10340) [0x7f92ff4ef340] >> 2015-07-20T10:47:15.240 INFO:tasks.ceph.osd.1.ovh164253.stderr: 3: (gsignal()+0x39) [0x7f92fd768cc9] >> 2015-07-20T10:47:15.240 INFO:tasks.ceph.osd.1.ovh164253.stderr: 4: (abort()+0x148) [0x7f92fd76c0d8] >> 2015-07-20T10:47:15.240 INFO:tasks.ceph.osd.1.ovh164253.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f92fe073535] >> 2015-07-20T10:47:15.240 INFO:tasks.ceph.osd.1.ovh164253.stderr: 6: (()+0x5e6d6) [0x7f92fe0716d6] >> 2015-07-20T10:47:15.240 INFO:tasks.ceph.osd.1.ovh164253.stderr: 7: (()+0x5e703) [0x7f92fe071703] >> 2015-07-20T10:47:15.240 INFO:tasks.ceph.osd.1.ovh164253.stderr: 8: (()+0x5e922) [0x7f92fe071922] >> 2015-07-20T10:47:15.241 INFO:tasks.ceph.osd.1.ovh164253.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xc45f08] >> 2015-07-20T10:47:15.241 INFO:tasks.ceph.osd.1.ovh164253.stderr: 10: (ReplicatedPG::op_applied(eversion_t const&)+0x6dc) [0x8741ac] >> 2015-07-20T10:47:15.241 INFO:tasks.ceph.osd.1.ovh164253.stderr: 11: (ReplicatedBackend::op_applied(ReplicatedBackend::InProgressOp*)+0xd0) [0xa5cfe0] >> 2015-07-20T10:47:15.241 INFO:tasks.ceph.osd.1.ovh164253.stderr: 12: (Context::complete(int)+0x9) [0x6f4649] >> 2015-07-20T10:47:15.242 INFO:tasks.ceph.osd.1.ovh164253.stderr: 13: (ReplicatedPG::BlessedContext::finish(int)+0x94) [0x8dec54] >> 2015-07-20T10:47:15.242 INFO:tasks.ceph.osd.1.ovh164253.stderr: 14: (Context::complete(int)+0x9) [0x6f4649] >> 2015-07-20T10:47:15.242 INFO:tasks.ceph.osd.1.ovh164253.stderr: 15: (void finish_contexts<Context>(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0x94) [0x7351d4] >> 2015-07-20T10:47:15.242 INFO:tasks.ceph.osd.1.ovh164253.stderr: 16: (C_ContextsBase<Context, Context>::complete(int)+0x9) [0x6f4e89] >> 2015-07-20T10:47:15.242 INFO:tasks.ceph.osd.1.ovh164253.stderr: 17: (Finisher::finisher_thread_entry()+0x158) [0xb6f2b8] >> 2015-07-20T10:47:15.243 INFO:tasks.ceph.osd.1.ovh164253.stderr: 18: (()+0x8182) [0x7f92ff4e7182] >> 2015-07-20T10:47:15.243 INFO:tasks.ceph.osd.1.ovh164253.stderr: 19: (clone()+0x6d) [0x7f92fd82c47d] >> 2015-07-20T10:47:15.243 INFO:tasks.ceph.osd.1.ovh164253.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. >> 2015-07-20T10:47:15.243 INFO:tasks.ceph.osd.1.ovh164253.stderr: >> 2015-07-20T10:47:15.494 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 2] out_osds: [0, 4, 3] dead_osds: [5] live_osds: [4, 1, 3, 2, 0] >> 2015-07-20T10:47:15.494 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 >> 2015-07-20T10:47:15.494 INFO:tasks.thrashosds.thrasher:Reviving osd 5 >> 2015-07-20T10:47:15.494 INFO:tasks.ceph.osd.5:Restarting daemon >> >> >> © 2006 - 2015 Paste2.org. >> Follow paste2.org on Twitter >> >> >> as found in >> 149.202.164.239/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/10/teuthology.log >> >> description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml >> clusters/fixed-2.yaml fs/ext4.yaml msgr-failures/few.yaml thrashers/default.yaml >> workloads/cache.yaml} >> >> Not sure if this is virtual machine related just yet (I did an almost clean run of rados but that was hammer). >> >> http://integration.ceph.dachary.org:8081/ubuntu-2015-07-19_17:29:05-rados-hammer---basic-openstack/ >> + re-run of failed/dead at >> http://integration.ceph.dachary.org:8081/ubuntu-2015-07-19_23:34:04-rados-hammer---basic-openstack/ >> >> Cheers >> > -- Loïc Dachary, Artisan Logiciel Libre
Attachment:
signature.asc
Description: OpenPGP digital signature
