Wow... all that verbose stuff is «just» a read error ? But ok, I will continue on the ceph-users list then ;) Le dimanche 10 février 2013 à 10:10 -0800, Gregory Farnum a écrit : > The OSD daemon is getting back EIO when it tries to do a read. Sounds like your disk is going bad. > -Greg > > PS: This question is a good fit for the new ceph-users list. :) > > > On Sunday, February 10, 2013 at 9:45 AM, Olivier Bonvalet wrote: > > > Hi, > > > > I have an OSD which often stopped (ceph 0.56.2), with that in logs : > > > > 446 stamp 2013-02-10 18:37:27.559777) v2 ==== 47+0+0 (4068038983 0 0) > > 0x11e028c0 con 0x573d6e0 > > -3> 2013-02-10 18:37:27.561618 7f1c765d5700 1 -- > > 192.168.42.1:0/5824 <== osd.31 192.168.42.3:6811/23050 129 ==== > > osd_ping(ping_reply e13446 stamp 2013-02-10 18:37:27.559777) v2 ==== 47 > > +0+0 (4068038983 0 0) 0x73be380 con 0x573d420 > > -2> 2013-02-10 18:37:27.562674 7f1c765d5700 1 -- > > 192.168.42.1:0/5824 <== osd.1 192.168.42.2:6803/7458 129 ==== > > osd_ping(ping_reply e13446 stamp 2013-02-10 18:37:27.559777) v2 ==== 47 > > +0+0 (4068038983 0 0) 0x6bd8a80 con 0x573dc60 > > -1> 2013-02-10 18:37:28.217626 7f1c805e9700 5 osd.12 13444 tick > > 0> 2013-02-10 18:37:28.552692 7f1c725cd700 -1 os/FileStore.cc (http://FileStore.cc): In > > function 'virtual int FileStore::read(coll_t, const hobject_t&, > > uint64_t, size_t, ceph::bufferlist&)' thread 7f1c725cd700 time > > 2013-02-10 18:37:28.537715 > > os/FileStore.cc (http://FileStore.cc): 2732: FAILED assert(!m_filestore_fail_eio || got != -5) > > > > ceph version () > > 1: (FileStore::read(coll_t, hobject_t const&, unsigned long, unsigned > > long, ceph::buffer::list&)+0x462) [0x725f92] > > 2: (PG::_scan_list(ScrubMap&, std::vector<hobject_t, > > std::allocator<hobject_t> >&, bool)+0x371) [0x685da1] > > 3: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, > > bool)+0x29b) [0x6866bb] > > 4: (PG::replica_scrub(MOSDRepScrub*)+0x8e9) [0x6952b9] > > 5: (OSD::RepScrubWQ::_process(MOSDRepScrub*)+0xc2) [0x6410a2] > > 6: (ThreadPool::worker(ThreadPool::WorkThread*)+0x879) [0x80f9e9] > > 7: (ThreadPool::WorkThread::entry()+0x10) [0x8121f0] > > 8: (()+0x68ca) [0x7f1c852f48ca] > > 9: (clone()+0x6d) [0x7f1c83e23b6d] > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > > needed to interpret this. > > > > --- logging levels --- > > 0/ 5 none > > 0/ 1 lockdep > > 0/ 1 context > > 1/ 1 crush > > 1/ 5 mds > > 1/ 5 mds_balancer > > 1/ 5 mds_locker > > 1/ 5 mds_log > > 1/ 5 mds_log_expire > > 1/ 5 mds_migrator > > 0/ 1 buffer > > 0/ 1 timer > > 0/ 1 filer > > 0/ 1 striper > > 0/ 1 objecter > > 0/ 5 rados > > 0/ 5 rbd > > 0/ 5 journaler > > 0/ 5 objectcacher > > 0/ 5 client > > 0/ 5 osd > > 0/ 5 optracker > > 0/ 5 objclass > > 1/ 3 filestore > > 1/ 3 journal > > 0/ 5 ms > > 1/ 5 mon > > 0/10 monc > > 0/ 5 paxos > > 0/ 5 tp > > 1/ 5 auth > > 1/ 5 crypto > > 1/ 1 finisher > > 1/ 5 heartbeatmap > > 1/ 5 perfcounter > > 1/ 5 rgw > > 1/ 5 hadoop > > 1/ 5 javaclient > > 1/ 5 asok > > 1/ 1 throttle > > -2/-2 (syslog threshold) > > -1/-1 (stderr threshold) > > max_recent 100000 > > max_new 1000 > > log_file /var/log/ceph/osd.12.log > > --- end dump of recent events --- > > 2013-02-10 18:37:29.236649 7f1c725cd700 -1 *** Caught signal (Aborted) > > ** > > in thread 7f1c725cd700 > > > > ceph version () > > 1: /usr/bin/ceph-osd() [0x7a0db9] > > 2: (()+0xeff0) [0x7f1c852fcff0] > > 3: (gsignal()+0x35) [0x7f1c83d861b5] > > 4: (abort()+0x180) [0x7f1c83d88fc0] > > 5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7f1c8461adc5] > > 6: (()+0xcb166) [0x7f1c84619166] > > 7: (()+0xcb193) [0x7f1c84619193] > > 8: (()+0xcb28e) [0x7f1c8461928e] > > 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char > > const*)+0x7c9) [0x8f3fc9] > > 10: (FileStore::read(coll_t, hobject_t const&, unsigned long, unsigned > > long, ceph::buffer::list&)+0x462) [0x725f92] > > 11: (PG::_scan_list(ScrubMap&, std::vector<hobject_t, > > std::allocator<hobject_t> >&, bool)+0x371) [0x685da1] > > 12: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, > > bool)+0x29b) [0x6866bb] > > 13: (PG::replica_scrub(MOSDRepScrub*)+0x8e9) [0x6952b9] > > 14: (OSD::RepScrubWQ::_process(MOSDRepScrub*)+0xc2) [0x6410a2] > > 15: (ThreadPool::worker(ThreadPool::WorkThread*)+0x879) [0x80f9e9] > > 16: (ThreadPool::WorkThread::entry()+0x10) [0x8121f0] > > 17: (()+0x68ca) [0x7f1c852f48ca] > > 18: (clone()+0x6d) [0x7f1c83e23b6d] > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > > needed to interpret this. > > > > --- begin dump of recent events --- > > -1> 2013-02-10 18:37:29.217778 7f1c805e9700 5 osd.12 13444 tick > > 0> 2013-02-10 18:37:29.236649 7f1c725cd700 -1 *** Caught signal > > (Aborted) ** > > in thread 7f1c725cd700 > > > > ceph version () > > 1: /usr/bin/ceph-osd() [0x7a0db9] > > 2: (()+0xeff0) [0x7f1c852fcff0] > > 3: (gsignal()+0x35) [0x7f1c83d861b5] > > 4: (abort()+0x180) [0x7f1c83d88fc0] > > 5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7f1c8461adc5] > > 6: (()+0xcb166) [0x7f1c84619166] > > 7: (()+0xcb193) [0x7f1c84619193] > > 8: (()+0xcb28e) [0x7f1c8461928e] > > 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char > > const*)+0x7c9) [0x8f3fc9] > > 10: (FileStore::read(coll_t, hobject_t const&, unsigned long, unsigned > > long, ceph::buffer::list&)+0x462) [0x725f92] > > 11: (PG::_scan_list(ScrubMap&, std::vector<hobject_t, > > std::allocator<hobject_t> >&, bool)+0x371) [0x685da1] > > 12: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, > > bool)+0x29b) [0x6866bb] > > 13: (PG::replica_scrub(MOSDRepScrub*)+0x8e9) [0x6952b9] > > 14: (OSD::RepScrubWQ::_process(MOSDRepScrub*)+0xc2) [0x6410a2] > > 15: (ThreadPool::worker(ThreadPool::WorkThread*)+0x879) [0x80f9e9] > > 16: (ThreadPool::WorkThread::entry()+0x10) [0x8121f0] > > 17: (()+0x68ca) [0x7f1c852f48ca] > > 18: (clone()+0x6d) [0x7f1c83e23b6d] > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > > needed to interpret this. > > > > --- logging levels --- > > 0/ 5 none > > 0/ 1 lockdep > > 0/ 1 context > > 1/ 1 crush > > 1/ 5 mds > > 1/ 5 mds_balancer > > 1/ 5 mds_locker > > 1/ 5 mds_log > > 1/ 5 mds_log_expire > > 1/ 5 mds_migrator > > 0/ 1 buffer > > 0/ 1 timer > > 0/ 1 filer > > 0/ 1 striper > > 0/ 1 objecter > > 0/ 5 rados > > 0/ 5 rbd > > 0/ 5 journaler > > 0/ 5 objectcacher > > 0/ 5 client > > 0/ 5 osd > > 0/ 5 optracker > > 0/ 5 objclass > > 1/ 3 filestore > > 1/ 3 journal > > 0/ 5 ms > > 1/ 5 mon > > 0/10 monc > > 0/ 5 paxos > > 0/ 5 tp > > 1/ 5 auth > > 1/ 5 crypto > > 1/ 1 finisher > > 1/ 5 heartbeatmap > > 1/ 5 perfcounter > > 1/ 5 rgw > > 1/ 5 hadoop > > 1/ 5 javaclient > > 1/ 5 asok > > 1/ 1 throttle > > -2/-2 (syslog threshold) > > -1/-1 (stderr threshold) > > max_recent 100000 > > max_new 1000 > > log_file /var/log/ceph/osd.12.log > > --- end dump of recent events --- > > > > > > > > What should I do ? > > > > > > > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majordomo@xxxxxxxxxxxxxxx (mailto:majordomo@xxxxxxxxxxxxxxx) > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
