Note however that only one of the dead (timed out) job has an assert (looks like it's because the file system is not as it should, which is expected since there are no attached disks to the instances, therefore no way for the job to mkfs the file system of choice). All others timed out just because they either need more disk or just more time. On 21/07/2015 09:30, Loic Dachary wrote: > Hi Kefu, > > The following runs on OpenStack and the next branch http://integration.ceph.dachary.org:8081/ubuntu-2015-07-21_00:04:04-rados-next---basic-openstack/ and 15 out of the 16 dead jobs (timed out after 3 hours) are from rados/thrash. A rados suite run on next dated a few days ago in the sepia lab ( http://pulpito.ceph.com/teuthology-2015-07-15_21:00:10-rados-next-distro-basic-multi/ ) also has a few dead jobs but only two of them are from rados/thrash. > > Cheers > > > On 20/07/2015 16:23, Loic Dachary wrote: >> More information about this run. I'll run a rados suite on master on OpenStack to get a baseline of what we should expect. >> >> http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/12/ >> http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/14/ >> http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/15/ >> http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/17/ >> http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/20/ >> http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/21/ >> http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/22/ >> http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/23/ >> http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/26/ >> http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/28/ >> http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/2/ >> http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/5/ >> http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/6/ >> http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/7/ >> http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/9/ >> >> I see >> >> 2015-07-20T10:02:10.567 INFO:tasks.ceph.osd.5.ovh165019.stderr:osd/ReplicatedPG.cc: In function 'bool ReplicatedPG::is_degraded_or_backfilling_object(const hobject_t&)' thread 7f2af94df700 time 2015-07-20 10:02:10.481916 >> 2015-07-20T10:02:10.567 INFO:tasks.ceph.osd.5.ovh165019.stderr:osd/ReplicatedPG.cc: 412: FAILED assert(!actingbackfill.empty()) >> 2015-07-20T10:02:10.567 INFO:tasks.ceph.osd.5.ovh165019.stderr: ceph version 9.0.2-799-gba9c2ae (ba9c2ae4bffd3fd7b26a2e0ce843913b77940b8a) >> 2015-07-20T10:02:10.568 INFO:tasks.ceph.osd.5.ovh165019.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xc45d1b] >> 2015-07-20T10:02:10.568 INFO:tasks.ceph.osd.5.ovh165019.stderr: 2: ceph-osd() [0x88535d] >> 2015-07-20T10:02:10.568 INFO:tasks.ceph.osd.5.ovh165019.stderr: 3: (ReplicatedPG::hit_set_remove_all()+0x7c) [0x8b039c] >> 2015-07-20T10:02:10.568 INFO:tasks.ceph.osd.5.ovh165019.stderr: 4: (ReplicatedPG::on_pool_change()+0x161) [0x8b1a21] >> 2015-07-20T10:02:10.569 INFO:tasks.ceph.osd.5.ovh165019.stderr: 5: (PG::handle_advance_map(std::tr1::shared_ptr<OSDMap const>, std::tr1::shared_ptr<OSDMap const>, std::vector<int, std::allocator<int> >&, int, std::vector<int, std::allocator<int> >&, int, PG::RecoveryCtx*)+0x60c) [0x8348fc] >> 2015-07-20T10:02:10.569 INFO:tasks.ceph.osd.5.ovh165019.stderr: 6: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x2c3) [0x6dcc73] >> 2015-07-20T10:02:10.569 INFO:tasks.ceph.osd.5.ovh165019.stderr: 7: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x1f1) [0x6dd721] >> 2015-07-20T10:02:10.572 INFO:tasks.ceph.osd.5.ovh165019.stderr: 8: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x18) [0x7328d8] >> 2015-07-20T10:02:10.573 INFO:tasks.ceph.osd.5.ovh165019.stderr: 9: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa5e) [0xc3677e] >> 2015-07-20T10:02:10.573 INFO:tasks.ceph.osd.5.ovh165019.stderr: 10: (ThreadPool::WorkThread::entry()+0x10) [0xc37820] >> 2015-07-20T10:02:10.573 INFO:tasks.ceph.osd.5.ovh165019.stderr: 11: (()+0x8182) [0x7f2b149e3182] >> 2015-07-20T10:02:10.573 INFO:tasks.ceph.osd.5.ovh165019.stderr: 12: (clone()+0x6d) [0x7f2b12d2847d] >> >> >> In >> >> http://149.202.164.239/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/24/ >> >> I see the same error as below. >> >> In >> >> http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/8/ >> >> it looks like the run was about to finish, just took a long time, and should be ignored as a false negative. >> >> On 20/07/2015 14:52, Loic Dachary wrote: >>> Hi, >>> >>> I checked one of the timeout (dead) at http://149.202.164.239:8081/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/ >>> >>> http://149.202.164.239/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/10/config.yaml >>> timeed out because of >>> >>> >>> Paste2 >>> >>> Create Paste >>> Followup Paste >>> QR >>> >>> sd.5 since back 2015-07-20 10:45:28.566308 front 2015-07-20 10:45:28.566308 (cutoff 2015-07-20 10:45:33.823074) >>> 2015-07-20T10:47:13.921 INFO:tasks.ceph.osd.4.ovh164254.stderr:2015-07-20 10:47:13.899770 7fb4be171700 -1 osd.4 655 heartbeat_check: no reply from osd.5 since back 2015-07-20 10:45:30.719801 front 2015-07-20 10:45:30.719801 (cutoff 2015-07-20 10:45:33.899763) >>> 2015-07-20T10:47:15.023 INFO:tasks.ceph.osd.1.ovh164253.stderr:osd/ReplicatedPG.cc: In function 'virtual void ReplicatedPG::op_applied(const eversion_t&)' thread 7f92f0244700 time 2015-07-20 10:47:14.998470 >>> 2015-07-20T10:47:15.024 INFO:tasks.ceph.osd.1.ovh164253.stderr:osd/ReplicatedPG.cc: 7311: FAILED assert(applied_version <= info.last_update) >>> 2015-07-20T10:47:15.025 INFO:tasks.ceph.osd.1.ovh164253.stderr: ceph version 9.0.2-799-gba9c2ae (ba9c2ae4bffd3fd7b26a2e0ce843913b77940b8a) >>> 2015-07-20T10:47:15.025 INFO:tasks.ceph.osd.1.ovh164253.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xc45d1b] >>> 2015-07-20T10:47:15.025 INFO:tasks.ceph.osd.1.ovh164253.stderr: 2: (ReplicatedPG::op_applied(eversion_t const&)+0x6dc) [0x8741ac] >>> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 3: (ReplicatedBackend::op_applied(ReplicatedBackend::InProgressOp*)+0xd0) [0xa5cfe0] >>> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 4: (Context::complete(int)+0x9) [0x6f4649] >>> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 5: (ReplicatedPG::BlessedContext::finish(int)+0x94) [0x8dec54] >>> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 6: (Context::complete(int)+0x9) [0x6f4649] >>> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 7: (void finish_contexts<Context>(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0x94) [0x7351d4] >>> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 8: (C_ContextsBase<Context, Context>::complete(int)+0x9) [0x6f4e89] >>> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 9: (Finisher::finisher_thread_entry()+0x158) [0xb6f2b8] >>> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 10: (()+0x8182) [0x7f92ff4e7182] >>> 2015-07-20T10:47:15.026 INFO:tasks.ceph.osd.1.ovh164253.stderr: 11: (clone()+0x6d) [0x7f92fd82c47d] >>> 2015-07-20T10:47:15.027 INFO:tasks.ceph.osd.1.ovh164253.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. >>> 2015-07-20T10:47:15.038 INFO:tasks.ceph.osd.1.ovh164253.stderr:2015-07-20 10:47:15.005862 7f92f0244700 -1 osd/ReplicatedPG.cc: In function 'virtual void ReplicatedPG::op_applied(const eversion_t&)' thread 7f92f0244700 time 2015-07-20 10:47:14.998470 >>> 2015-07-20T10:47:15.039 INFO:tasks.ceph.osd.1.ovh164253.stderr:osd/ReplicatedPG.cc: 7311: FAILED assert(applied_version <= info.last_update) >>> 2015-07-20T10:47:15.039 INFO:tasks.ceph.osd.1.ovh164253.stderr: >>> 2015-07-20T10:47:15.039 INFO:tasks.ceph.osd.1.ovh164253.stderr: ceph version 9.0.2-799-gba9c2ae (ba9c2ae4bffd3fd7b26a2e0ce843913b77940b8a) >>> 2015-07-20T10:47:15.039 INFO:tasks.ceph.osd.1.ovh164253.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xc45d1b] >>> 2015-07-20T10:47:15.039 INFO:tasks.ceph.osd.1.ovh164253.stderr: 2: (ReplicatedPG::op_applied(eversion_t const&)+0x6dc) [0x8741ac] >>> 2015-07-20T10:47:15.039 INFO:tasks.ceph.osd.1.ovh164253.stderr: 3: (ReplicatedBackend::op_applied(ReplicatedBackend::InProgressOp*)+0xd0) [0xa5cfe0] >>> 2015-07-20T10:47:15.039 INFO:tasks.ceph.osd.1.ovh164253.stderr: 4: (Context::complete(int)+0x9) [0x6f4649] >>> 2015-07-20T10:47:15.039 INFO:tasks.ceph.osd.1.ovh164253.stderr: 5: (ReplicatedPG::BlessedContext::finish(int)+0x94) [0x8dec54] >>> 2015-07-20T10:47:15.040 INFO:tasks.ceph.osd.1.ovh164253.stderr: 6: (Context::complete(int)+0x9) [0x6f4649] >>> 2015-07-20T10:47:15.040 INFO:tasks.ceph.osd.1.ovh164253.stderr: 7: (void finish_contexts<Context>(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0x94) [0x7351d4] >>> 2015-07-20T10:47:15.040 INFO:tasks.ceph.osd.1.ovh164253.stderr: 8: (C_ContextsBase<Context, Context>::complete(int)+0x9) [0x6f4e89] >>> 2015-07-20T10:47:15.040 INFO:tasks.ceph.osd.1.ovh164253.stderr: 9: (Finisher::finisher_thread_entry()+0x158) [0xb6f2b8] >>> 2015-07-20T10:47:15.040 INFO:tasks.ceph.osd.1.ovh164253.stderr: 10: (()+0x8182) [0x7f92ff4e7182] >>> 2015-07-20T10:47:15.040 INFO:tasks.ceph.osd.1.ovh164253.stderr: 11: (clone()+0x6d) [0x7f92fd82c47d] >>> 2015-07-20T10:47:15.040 INFO:tasks.ceph.osd.1.ovh164253.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. >>> 2015-07-20T10:47:15.041 INFO:tasks.ceph.osd.1.ovh164253.stderr: >>> 2015-07-20T10:47:15.212 INFO:tasks.ceph.osd.1.ovh164253.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion' >>> 2015-07-20T10:47:15.212 INFO:tasks.ceph.osd.1.ovh164253.stderr:*** Caught signal (Aborted) ** >>> 2015-07-20T10:47:15.212 INFO:tasks.ceph.osd.1.ovh164253.stderr: in thread 7f92f0244700 >>> 2015-07-20T10:47:15.217 INFO:tasks.ceph.osd.1.ovh164253.stderr: ceph version 9.0.2-799-gba9c2ae (ba9c2ae4bffd3fd7b26a2e0ce843913b77940b8a) >>> 2015-07-20T10:47:15.217 INFO:tasks.ceph.osd.1.ovh164253.stderr: 1: ceph-osd() [0xb49fba] >>> 2015-07-20T10:47:15.217 INFO:tasks.ceph.osd.1.ovh164253.stderr: 2: (()+0x10340) [0x7f92ff4ef340] >>> 2015-07-20T10:47:15.218 INFO:tasks.ceph.osd.1.ovh164253.stderr: 3: (gsignal()+0x39) [0x7f92fd768cc9] >>> 2015-07-20T10:47:15.218 INFO:tasks.ceph.osd.1.ovh164253.stderr: 4: (abort()+0x148) [0x7f92fd76c0d8] >>> 2015-07-20T10:47:15.218 INFO:tasks.ceph.osd.1.ovh164253.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f92fe073535] >>> 2015-07-20T10:47:15.218 INFO:tasks.ceph.osd.1.ovh164253.stderr: 6: (()+0x5e6d6) [0x7f92fe0716d6] >>> 2015-07-20T10:47:15.218 INFO:tasks.ceph.osd.1.ovh164253.stderr: 7: (()+0x5e703) [0x7f92fe071703] >>> 2015-07-20T10:47:15.219 INFO:tasks.ceph.osd.1.ovh164253.stderr: 8: (()+0x5e922) [0x7f92fe071922] >>> 2015-07-20T10:47:15.219 INFO:tasks.ceph.osd.1.ovh164253.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xc45f08] >>> 2015-07-20T10:47:15.219 INFO:tasks.ceph.osd.1.ovh164253.stderr: 10: (ReplicatedPG::op_applied(eversion_t const&)+0x6dc) [0x8741ac] >>> 2015-07-20T10:47:15.219 INFO:tasks.ceph.osd.1.ovh164253.stderr: 11: (ReplicatedBackend::op_applied(ReplicatedBackend::InProgressOp*)+0xd0) [0xa5cfe0] >>> 2015-07-20T10:47:15.219 INFO:tasks.ceph.osd.1.ovh164253.stderr: 12: (Context::complete(int)+0x9) [0x6f4649] >>> 2015-07-20T10:47:15.219 INFO:tasks.ceph.osd.1.ovh164253.stderr: 13: (ReplicatedPG::BlessedContext::finish(int)+0x94) [0x8dec54] >>> 2015-07-20T10:47:15.219 INFO:tasks.ceph.osd.1.ovh164253.stderr: 14: (Context::complete(int)+0x9) [0x6f4649] >>> 2015-07-20T10:47:15.219 INFO:tasks.ceph.osd.1.ovh164253.stderr: 15: (void finish_contexts<Context>(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0x94) [0x7351d4] >>> 2015-07-20T10:47:15.220 INFO:tasks.ceph.osd.1.ovh164253.stderr: 16: (C_ContextsBase<Context, Context>::complete(int)+0x9) [0x6f4e89] >>> 2015-07-20T10:47:15.220 INFO:tasks.ceph.osd.1.ovh164253.stderr: 17: (Finisher::finisher_thread_entry()+0x158) [0xb6f2b8] >>> 2015-07-20T10:47:15.220 INFO:tasks.ceph.osd.1.ovh164253.stderr: 18: (()+0x8182) [0x7f92ff4e7182] >>> 2015-07-20T10:47:15.220 INFO:tasks.ceph.osd.1.ovh164253.stderr: 19: (clone()+0x6d) [0x7f92fd82c47d] >>> 2015-07-20T10:47:15.221 INFO:tasks.ceph.osd.1.ovh164253.stderr:2015-07-20 10:47:15.197571 7f92f0244700 -1 *** Caught signal (Aborted) ** >>> 2015-07-20T10:47:15.221 INFO:tasks.ceph.osd.1.ovh164253.stderr: in thread 7f92f0244700 >>> 2015-07-20T10:47:15.221 INFO:tasks.ceph.osd.1.ovh164253.stderr: >>> 2015-07-20T10:47:15.221 INFO:tasks.ceph.osd.1.ovh164253.stderr: ceph version 9.0.2-799-gba9c2ae (ba9c2ae4bffd3fd7b26a2e0ce843913b77940b8a) >>> 2015-07-20T10:47:15.221 INFO:tasks.ceph.osd.1.ovh164253.stderr: 1: ceph-osd() [0xb49fba] >>> 2015-07-20T10:47:15.222 INFO:tasks.ceph.osd.1.ovh164253.stderr: 2: (()+0x10340) [0x7f92ff4ef340] >>> 2015-07-20T10:47:15.222 INFO:tasks.ceph.osd.1.ovh164253.stderr: 3: (gsignal()+0x39) [0x7f92fd768cc9] >>> 2015-07-20T10:47:15.222 INFO:tasks.ceph.osd.1.ovh164253.stderr: 4: (abort()+0x148) [0x7f92fd76c0d8] >>> 2015-07-20T10:47:15.222 INFO:tasks.ceph.osd.1.ovh164253.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f92fe073535] >>> 2015-07-20T10:47:15.222 INFO:tasks.ceph.osd.1.ovh164253.stderr: 6: (()+0x5e6d6) [0x7f92fe0716d6] >>> 2015-07-20T10:47:15.222 INFO:tasks.ceph.osd.1.ovh164253.stderr: 7: (()+0x5e703) [0x7f92fe071703] >>> 2015-07-20T10:47:15.222 INFO:tasks.ceph.osd.1.ovh164253.stderr: 8: (()+0x5e922) [0x7f92fe071922] >>> 2015-07-20T10:47:15.223 INFO:tasks.ceph.osd.1.ovh164253.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xc45f08] >>> 2015-07-20T10:47:15.223 INFO:tasks.ceph.osd.1.ovh164253.stderr: 10: (ReplicatedPG::op_applied(eversion_t const&)+0x6dc) [0x8741ac] >>> 2015-07-20T10:47:15.223 INFO:tasks.ceph.osd.1.ovh164253.stderr: 11: (ReplicatedBackend::op_applied(ReplicatedBackend::InProgressOp*)+0xd0) [0xa5cfe0] >>> 2015-07-20T10:47:15.223 INFO:tasks.ceph.osd.1.ovh164253.stderr: 12: (Context::complete(int)+0x9) [0x6f4649] >>> 2015-07-20T10:47:15.223 INFO:tasks.ceph.osd.1.ovh164253.stderr: 13: (ReplicatedPG::BlessedContext::finish(int)+0x94) [0x8dec54] >>> 2015-07-20T10:47:15.223 INFO:tasks.ceph.osd.1.ovh164253.stderr: 14: (Context::complete(int)+0x9) [0x6f4649] >>> 2015-07-20T10:47:15.223 INFO:tasks.ceph.osd.1.ovh164253.stderr: 15: (void finish_contexts<Context>(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0x94) [0x7351d4] >>> 2015-07-20T10:47:15.224 INFO:tasks.ceph.osd.1.ovh164253.stderr: 16: (C_ContextsBase<Context, Context>::complete(int)+0x9) [0x6f4e89] >>> 2015-07-20T10:47:15.224 INFO:tasks.ceph.osd.1.ovh164253.stderr: 17: (Finisher::finisher_thread_entry()+0x158) [0xb6f2b8] >>> 2015-07-20T10:47:15.224 INFO:tasks.ceph.osd.1.ovh164253.stderr: 18: (()+0x8182) [0x7f92ff4e7182] >>> 2015-07-20T10:47:15.224 INFO:tasks.ceph.osd.1.ovh164253.stderr: 19: (clone()+0x6d) [0x7f92fd82c47d] >>> 2015-07-20T10:47:15.224 INFO:tasks.ceph.osd.1.ovh164253.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. >>> 2015-07-20T10:47:15.224 INFO:tasks.ceph.osd.1.ovh164253.stderr: >>> 2015-07-20T10:47:15.238 INFO:tasks.ceph.osd.1.ovh164253.stderr: -172> 2015-07-20 10:47:15.197571 7f92f0244700 -1 *** Caught signal (Aborted) ** >>> 2015-07-20T10:47:15.239 INFO:tasks.ceph.osd.1.ovh164253.stderr: in thread 7f92f0244700 >>> 2015-07-20T10:47:15.239 INFO:tasks.ceph.osd.1.ovh164253.stderr: >>> 2015-07-20T10:47:15.239 INFO:tasks.ceph.osd.1.ovh164253.stderr: ceph version 9.0.2-799-gba9c2ae (ba9c2ae4bffd3fd7b26a2e0ce843913b77940b8a) >>> 2015-07-20T10:47:15.239 INFO:tasks.ceph.osd.1.ovh164253.stderr: 1: ceph-osd() [0xb49fba] >>> 2015-07-20T10:47:15.239 INFO:tasks.ceph.osd.1.ovh164253.stderr: 2: (()+0x10340) [0x7f92ff4ef340] >>> 2015-07-20T10:47:15.240 INFO:tasks.ceph.osd.1.ovh164253.stderr: 3: (gsignal()+0x39) [0x7f92fd768cc9] >>> 2015-07-20T10:47:15.240 INFO:tasks.ceph.osd.1.ovh164253.stderr: 4: (abort()+0x148) [0x7f92fd76c0d8] >>> 2015-07-20T10:47:15.240 INFO:tasks.ceph.osd.1.ovh164253.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f92fe073535] >>> 2015-07-20T10:47:15.240 INFO:tasks.ceph.osd.1.ovh164253.stderr: 6: (()+0x5e6d6) [0x7f92fe0716d6] >>> 2015-07-20T10:47:15.240 INFO:tasks.ceph.osd.1.ovh164253.stderr: 7: (()+0x5e703) [0x7f92fe071703] >>> 2015-07-20T10:47:15.240 INFO:tasks.ceph.osd.1.ovh164253.stderr: 8: (()+0x5e922) [0x7f92fe071922] >>> 2015-07-20T10:47:15.241 INFO:tasks.ceph.osd.1.ovh164253.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xc45f08] >>> 2015-07-20T10:47:15.241 INFO:tasks.ceph.osd.1.ovh164253.stderr: 10: (ReplicatedPG::op_applied(eversion_t const&)+0x6dc) [0x8741ac] >>> 2015-07-20T10:47:15.241 INFO:tasks.ceph.osd.1.ovh164253.stderr: 11: (ReplicatedBackend::op_applied(ReplicatedBackend::InProgressOp*)+0xd0) [0xa5cfe0] >>> 2015-07-20T10:47:15.241 INFO:tasks.ceph.osd.1.ovh164253.stderr: 12: (Context::complete(int)+0x9) [0x6f4649] >>> 2015-07-20T10:47:15.242 INFO:tasks.ceph.osd.1.ovh164253.stderr: 13: (ReplicatedPG::BlessedContext::finish(int)+0x94) [0x8dec54] >>> 2015-07-20T10:47:15.242 INFO:tasks.ceph.osd.1.ovh164253.stderr: 14: (Context::complete(int)+0x9) [0x6f4649] >>> 2015-07-20T10:47:15.242 INFO:tasks.ceph.osd.1.ovh164253.stderr: 15: (void finish_contexts<Context>(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0x94) [0x7351d4] >>> 2015-07-20T10:47:15.242 INFO:tasks.ceph.osd.1.ovh164253.stderr: 16: (C_ContextsBase<Context, Context>::complete(int)+0x9) [0x6f4e89] >>> 2015-07-20T10:47:15.242 INFO:tasks.ceph.osd.1.ovh164253.stderr: 17: (Finisher::finisher_thread_entry()+0x158) [0xb6f2b8] >>> 2015-07-20T10:47:15.243 INFO:tasks.ceph.osd.1.ovh164253.stderr: 18: (()+0x8182) [0x7f92ff4e7182] >>> 2015-07-20T10:47:15.243 INFO:tasks.ceph.osd.1.ovh164253.stderr: 19: (clone()+0x6d) [0x7f92fd82c47d] >>> 2015-07-20T10:47:15.243 INFO:tasks.ceph.osd.1.ovh164253.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. >>> 2015-07-20T10:47:15.243 INFO:tasks.ceph.osd.1.ovh164253.stderr: >>> 2015-07-20T10:47:15.494 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 2] out_osds: [0, 4, 3] dead_osds: [5] live_osds: [4, 1, 3, 2, 0] >>> 2015-07-20T10:47:15.494 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 >>> 2015-07-20T10:47:15.494 INFO:tasks.thrashosds.thrasher:Reviving osd 5 >>> 2015-07-20T10:47:15.494 INFO:tasks.ceph.osd.5:Restarting daemon >>> >>> >>> © 2006 - 2015 Paste2.org. >>> Follow paste2.org on Twitter >>> >>> >>> as found in >>> 149.202.164.239/ubuntu-2015-07-20_09:21:01-rados-wip-kefu-testing---basic-openstack/10/teuthology.log >>> >>> description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml >>> clusters/fixed-2.yaml fs/ext4.yaml msgr-failures/few.yaml thrashers/default.yaml >>> workloads/cache.yaml} >>> >>> Not sure if this is virtual machine related just yet (I did an almost clean run of rados but that was hammer). >>> >>> http://integration.ceph.dachary.org:8081/ubuntu-2015-07-19_17:29:05-rados-hammer---basic-openstack/ >>> + re-run of failed/dead at >>> http://integration.ceph.dachary.org:8081/ubuntu-2015-07-19_23:34:04-rados-hammer---basic-openstack/ >>> >>> Cheers >>> >> > -- Loïc Dachary, Artisan Logiciel Libre
Attachment:
signature.asc
Description: OpenPGP digital signature
