The OSD daemon is getting back EIO when it tries to do a read. Sounds like your disk is going bad. -Greg PS: This question is a good fit for the new ceph-users list. :) On Sunday, February 10, 2013 at 9:45 AM, Olivier Bonvalet wrote: > Hi, > > I have an OSD which often stopped (ceph 0.56.2), with that in logs : > > 446 stamp 2013-02-10 18:37:27.559777) v2 ==== 47+0+0 (4068038983 0 0) > 0x11e028c0 con 0x573d6e0 > -3> 2013-02-10 18:37:27.561618 7f1c765d5700 1 -- > 192.168.42.1:0/5824 <== osd.31 192.168.42.3:6811/23050 129 ==== > osd_ping(ping_reply e13446 stamp 2013-02-10 18:37:27.559777) v2 ==== 47 > +0+0 (4068038983 0 0) 0x73be380 con 0x573d420 > -2> 2013-02-10 18:37:27.562674 7f1c765d5700 1 -- > 192.168.42.1:0/5824 <== osd.1 192.168.42.2:6803/7458 129 ==== > osd_ping(ping_reply e13446 stamp 2013-02-10 18:37:27.559777) v2 ==== 47 > +0+0 (4068038983 0 0) 0x6bd8a80 con 0x573dc60 > -1> 2013-02-10 18:37:28.217626 7f1c805e9700 5 osd.12 13444 tick > 0> 2013-02-10 18:37:28.552692 7f1c725cd700 -1 os/FileStore.cc (http://FileStore.cc): In > function 'virtual int FileStore::read(coll_t, const hobject_t&, > uint64_t, size_t, ceph::bufferlist&)' thread 7f1c725cd700 time > 2013-02-10 18:37:28.537715 > os/FileStore.cc (http://FileStore.cc): 2732: FAILED assert(!m_filestore_fail_eio || got != -5) > > ceph version () > 1: (FileStore::read(coll_t, hobject_t const&, unsigned long, unsigned > long, ceph::buffer::list&)+0x462) [0x725f92] > 2: (PG::_scan_list(ScrubMap&, std::vector<hobject_t, > std::allocator<hobject_t> >&, bool)+0x371) [0x685da1] > 3: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, > bool)+0x29b) [0x6866bb] > 4: (PG::replica_scrub(MOSDRepScrub*)+0x8e9) [0x6952b9] > 5: (OSD::RepScrubWQ::_process(MOSDRepScrub*)+0xc2) [0x6410a2] > 6: (ThreadPool::worker(ThreadPool::WorkThread*)+0x879) [0x80f9e9] > 7: (ThreadPool::WorkThread::entry()+0x10) [0x8121f0] > 8: (()+0x68ca) [0x7f1c852f48ca] > 9: (clone()+0x6d) [0x7f1c83e23b6d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 0/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 0/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/ 5 hadoop > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 100000 > max_new 1000 > log_file /var/log/ceph/osd.12.log > --- end dump of recent events --- > 2013-02-10 18:37:29.236649 7f1c725cd700 -1 *** Caught signal (Aborted) > ** > in thread 7f1c725cd700 > > ceph version () > 1: /usr/bin/ceph-osd() [0x7a0db9] > 2: (()+0xeff0) [0x7f1c852fcff0] > 3: (gsignal()+0x35) [0x7f1c83d861b5] > 4: (abort()+0x180) [0x7f1c83d88fc0] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7f1c8461adc5] > 6: (()+0xcb166) [0x7f1c84619166] > 7: (()+0xcb193) [0x7f1c84619193] > 8: (()+0xcb28e) [0x7f1c8461928e] > 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x7c9) [0x8f3fc9] > 10: (FileStore::read(coll_t, hobject_t const&, unsigned long, unsigned > long, ceph::buffer::list&)+0x462) [0x725f92] > 11: (PG::_scan_list(ScrubMap&, std::vector<hobject_t, > std::allocator<hobject_t> >&, bool)+0x371) [0x685da1] > 12: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, > bool)+0x29b) [0x6866bb] > 13: (PG::replica_scrub(MOSDRepScrub*)+0x8e9) [0x6952b9] > 14: (OSD::RepScrubWQ::_process(MOSDRepScrub*)+0xc2) [0x6410a2] > 15: (ThreadPool::worker(ThreadPool::WorkThread*)+0x879) [0x80f9e9] > 16: (ThreadPool::WorkThread::entry()+0x10) [0x8121f0] > 17: (()+0x68ca) [0x7f1c852f48ca] > 18: (clone()+0x6d) [0x7f1c83e23b6d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > --- begin dump of recent events --- > -1> 2013-02-10 18:37:29.217778 7f1c805e9700 5 osd.12 13444 tick > 0> 2013-02-10 18:37:29.236649 7f1c725cd700 -1 *** Caught signal > (Aborted) ** > in thread 7f1c725cd700 > > ceph version () > 1: /usr/bin/ceph-osd() [0x7a0db9] > 2: (()+0xeff0) [0x7f1c852fcff0] > 3: (gsignal()+0x35) [0x7f1c83d861b5] > 4: (abort()+0x180) [0x7f1c83d88fc0] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7f1c8461adc5] > 6: (()+0xcb166) [0x7f1c84619166] > 7: (()+0xcb193) [0x7f1c84619193] > 8: (()+0xcb28e) [0x7f1c8461928e] > 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x7c9) [0x8f3fc9] > 10: (FileStore::read(coll_t, hobject_t const&, unsigned long, unsigned > long, ceph::buffer::list&)+0x462) [0x725f92] > 11: (PG::_scan_list(ScrubMap&, std::vector<hobject_t, > std::allocator<hobject_t> >&, bool)+0x371) [0x685da1] > 12: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, > bool)+0x29b) [0x6866bb] > 13: (PG::replica_scrub(MOSDRepScrub*)+0x8e9) [0x6952b9] > 14: (OSD::RepScrubWQ::_process(MOSDRepScrub*)+0xc2) [0x6410a2] > 15: (ThreadPool::worker(ThreadPool::WorkThread*)+0x879) [0x80f9e9] > 16: (ThreadPool::WorkThread::entry()+0x10) [0x8121f0] > 17: (()+0x68ca) [0x7f1c852f48ca] > 18: (clone()+0x6d) [0x7f1c83e23b6d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 0/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 0/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/ 5 hadoop > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 100000 > max_new 1000 > log_file /var/log/ceph/osd.12.log > --- end dump of recent events --- > > > > What should I do ? > > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx (mailto:majordomo@xxxxxxxxxxxxxxx) > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
