On Sun, 10 Feb 2013, Olivier Bonvalet wrote: > And so, how can I temporarily disable that OSD without blocking all the > cluster ? > A balance will take hours, and everything will be unaivable during that > time. Everything should be available during the rebalance. But, if you want to avoid the rebalance, you can leave the osd daemon down and marked in. Eventually the data needs to be re-replicated somewhere, though, so it's a question of whether you want to do that in place (wipe the drive and re-add) or replace it. > Since there is replication, can I instruct clients to use only > replicas ? > > And that IO error, came from the journal (SSD) or storage (HDD) ? HDD in this case. I would check kern.log or dmesg to see if it is a media error or a corrupt file system. If it's the latter, you can reformat and re-use the disk. sage > > Thanks, > > Olivier B. > > > Le dimanche 10 f?vrier 2013 ? 20:52 +0100, Olivier Bonvalet a ?crit : > > Wow... all that verbose stuff is ?just? a read error ? > > > > But ok, I will continue on the ceph-users list then ;) > > > > > > Le dimanche 10 f?vrier 2013 ? 10:10 -0800, Gregory Farnum a ?crit : > > > The OSD daemon is getting back EIO when it tries to do a read. Sounds like your disk is going bad. > > > -Greg > > > > > > PS: This question is a good fit for the new ceph-users list. :) > > > > > > > > > On Sunday, February 10, 2013 at 9:45 AM, Olivier Bonvalet wrote: > > > > > > > Hi, > > > > > > > > I have an OSD which often stopped (ceph 0.56.2), with that in logs : > > > > > > > > 446 stamp 2013-02-10 18:37:27.559777) v2 ==== 47+0+0 (4068038983 0 0) > > > > 0x11e028c0 con 0x573d6e0 > > > > -3> 2013-02-10 18:37:27.561618 7f1c765d5700 1 -- > > > > 192.168.42.1:0/5824 <== osd.31 192.168.42.3:6811/23050 129 ==== > > > > osd_ping(ping_reply e13446 stamp 2013-02-10 18:37:27.559777) v2 ==== 47 > > > > +0+0 (4068038983 0 0) 0x73be380 con 0x573d420 > > > > -2> 2013-02-10 18:37:27.562674 7f1c765d5700 1 -- > > > > 192.168.42.1:0/5824 <== osd.1 192.168.42.2:6803/7458 129 ==== > > > > osd_ping(ping_reply e13446 stamp 2013-02-10 18:37:27.559777) v2 ==== 47 > > > > +0+0 (4068038983 0 0) 0x6bd8a80 con 0x573dc60 > > > > -1> 2013-02-10 18:37:28.217626 7f1c805e9700 5 osd.12 13444 tick > > > > 0> 2013-02-10 18:37:28.552692 7f1c725cd700 -1 os/FileStore.cc (http://FileStore.cc): In > > > > function 'virtual int FileStore::read(coll_t, const hobject_t&, > > > > uint64_t, size_t, ceph::bufferlist&)' thread 7f1c725cd700 time > > > > 2013-02-10 18:37:28.537715 > > > > os/FileStore.cc (http://FileStore.cc): 2732: FAILED assert(!m_filestore_fail_eio || got != -5) > > > > > > > > ceph version () > > > > 1: (FileStore::read(coll_t, hobject_t const&, unsigned long, unsigned > > > > long, ceph::buffer::list&)+0x462) [0x725f92] > > > > 2: (PG::_scan_list(ScrubMap&, std::vector<hobject_t, > > > > std::allocator<hobject_t> >&, bool)+0x371) [0x685da1] > > > > 3: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, > > > > bool)+0x29b) [0x6866bb] > > > > 4: (PG::replica_scrub(MOSDRepScrub*)+0x8e9) [0x6952b9] > > > > 5: (OSD::RepScrubWQ::_process(MOSDRepScrub*)+0xc2) [0x6410a2] > > > > 6: (ThreadPool::worker(ThreadPool::WorkThread*)+0x879) [0x80f9e9] > > > > 7: (ThreadPool::WorkThread::entry()+0x10) [0x8121f0] > > > > 8: (()+0x68ca) [0x7f1c852f48ca] > > > > 9: (clone()+0x6d) [0x7f1c83e23b6d] > > > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > > > > needed to interpret this. > > > > > > > > --- logging levels --- > > > > 0/ 5 none > > > > 0/ 1 lockdep > > > > 0/ 1 context > > > > 1/ 1 crush > > > > 1/ 5 mds > > > > 1/ 5 mds_balancer > > > > 1/ 5 mds_locker > > > > 1/ 5 mds_log > > > > 1/ 5 mds_log_expire > > > > 1/ 5 mds_migrator > > > > 0/ 1 buffer > > > > 0/ 1 timer > > > > 0/ 1 filer > > > > 0/ 1 striper > > > > 0/ 1 objecter > > > > 0/ 5 rados > > > > 0/ 5 rbd > > > > 0/ 5 journaler > > > > 0/ 5 objectcacher > > > > 0/ 5 client > > > > 0/ 5 osd > > > > 0/ 5 optracker > > > > 0/ 5 objclass > > > > 1/ 3 filestore > > > > 1/ 3 journal > > > > 0/ 5 ms > > > > 1/ 5 mon > > > > 0/10 monc > > > > 0/ 5 paxos > > > > 0/ 5 tp > > > > 1/ 5 auth > > > > 1/ 5 crypto > > > > 1/ 1 finisher > > > > 1/ 5 heartbeatmap > > > > 1/ 5 perfcounter > > > > 1/ 5 rgw > > > > 1/ 5 hadoop > > > > 1/ 5 javaclient > > > > 1/ 5 asok > > > > 1/ 1 throttle > > > > -2/-2 (syslog threshold) > > > > -1/-1 (stderr threshold) > > > > max_recent 100000 > > > > max_new 1000 > > > > log_file /var/log/ceph/osd.12.log > > > > --- end dump of recent events --- > > > > 2013-02-10 18:37:29.236649 7f1c725cd700 -1 *** Caught signal (Aborted) > > > > ** > > > > in thread 7f1c725cd700 > > > > > > > > ceph version () > > > > 1: /usr/bin/ceph-osd() [0x7a0db9] > > > > 2: (()+0xeff0) [0x7f1c852fcff0] > > > > 3: (gsignal()+0x35) [0x7f1c83d861b5] > > > > 4: (abort()+0x180) [0x7f1c83d88fc0] > > > > 5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7f1c8461adc5] > > > > 6: (()+0xcb166) [0x7f1c84619166] > > > > 7: (()+0xcb193) [0x7f1c84619193] > > > > 8: (()+0xcb28e) [0x7f1c8461928e] > > > > 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char > > > > const*)+0x7c9) [0x8f3fc9] > > > > 10: (FileStore::read(coll_t, hobject_t const&, unsigned long, unsigned > > > > long, ceph::buffer::list&)+0x462) [0x725f92] > > > > 11: (PG::_scan_list(ScrubMap&, std::vector<hobject_t, > > > > std::allocator<hobject_t> >&, bool)+0x371) [0x685da1] > > > > 12: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, > > > > bool)+0x29b) [0x6866bb] > > > > 13: (PG::replica_scrub(MOSDRepScrub*)+0x8e9) [0x6952b9] > > > > 14: (OSD::RepScrubWQ::_process(MOSDRepScrub*)+0xc2) [0x6410a2] > > > > 15: (ThreadPool::worker(ThreadPool::WorkThread*)+0x879) [0x80f9e9] > > > > 16: (ThreadPool::WorkThread::entry()+0x10) [0x8121f0] > > > > 17: (()+0x68ca) [0x7f1c852f48ca] > > > > 18: (clone()+0x6d) [0x7f1c83e23b6d] > > > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > > > > needed to interpret this. > > > > > > > > --- begin dump of recent events --- > > > > -1> 2013-02-10 18:37:29.217778 7f1c805e9700 5 osd.12 13444 tick > > > > 0> 2013-02-10 18:37:29.236649 7f1c725cd700 -1 *** Caught signal > > > > (Aborted) ** > > > > in thread 7f1c725cd700 > > > > > > > > ceph version () > > > > 1: /usr/bin/ceph-osd() [0x7a0db9] > > > > 2: (()+0xeff0) [0x7f1c852fcff0] > > > > 3: (gsignal()+0x35) [0x7f1c83d861b5] > > > > 4: (abort()+0x180) [0x7f1c83d88fc0] > > > > 5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7f1c8461adc5] > > > > 6: (()+0xcb166) [0x7f1c84619166] > > > > 7: (()+0xcb193) [0x7f1c84619193] > > > > 8: (()+0xcb28e) [0x7f1c8461928e] > > > > 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char > > > > const*)+0x7c9) [0x8f3fc9] > > > > 10: (FileStore::read(coll_t, hobject_t const&, unsigned long, unsigned > > > > long, ceph::buffer::list&)+0x462) [0x725f92] > > > > 11: (PG::_scan_list(ScrubMap&, std::vector<hobject_t, > > > > std::allocator<hobject_t> >&, bool)+0x371) [0x685da1] > > > > 12: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, > > > > bool)+0x29b) [0x6866bb] > > > > 13: (PG::replica_scrub(MOSDRepScrub*)+0x8e9) [0x6952b9] > > > > 14: (OSD::RepScrubWQ::_process(MOSDRepScrub*)+0xc2) [0x6410a2] > > > > 15: (ThreadPool::worker(ThreadPool::WorkThread*)+0x879) [0x80f9e9] > > > > 16: (ThreadPool::WorkThread::entry()+0x10) [0x8121f0] > > > > 17: (()+0x68ca) [0x7f1c852f48ca] > > > > 18: (clone()+0x6d) [0x7f1c83e23b6d] > > > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > > > > needed to interpret this. > > > > > > > > --- logging levels --- > > > > 0/ 5 none > > > > 0/ 1 lockdep > > > > 0/ 1 context > > > > 1/ 1 crush > > > > 1/ 5 mds > > > > 1/ 5 mds_balancer > > > > 1/ 5 mds_locker > > > > 1/ 5 mds_log > > > > 1/ 5 mds_log_expire > > > > 1/ 5 mds_migrator > > > > 0/ 1 buffer > > > > 0/ 1 timer > > > > 0/ 1 filer > > > > 0/ 1 striper > > > > 0/ 1 objecter > > > > 0/ 5 rados > > > > 0/ 5 rbd > > > > 0/ 5 journaler > > > > 0/ 5 objectcacher > > > > 0/ 5 client > > > > 0/ 5 osd > > > > 0/ 5 optracker > > > > 0/ 5 objclass > > > > 1/ 3 filestore > > > > 1/ 3 journal > > > > 0/ 5 ms > > > > 1/ 5 mon > > > > 0/10 monc > > > > 0/ 5 paxos > > > > 0/ 5 tp > > > > 1/ 5 auth > > > > 1/ 5 crypto > > > > 1/ 1 finisher > > > > 1/ 5 heartbeatmap > > > > 1/ 5 perfcounter > > > > 1/ 5 rgw > > > > 1/ 5 hadoop > > > > 1/ 5 javaclient > > > > 1/ 5 asok > > > > 1/ 1 throttle > > > > -2/-2 (syslog threshold) > > > > -1/-1 (stderr threshold) > > > > max_recent 100000 > > > > max_new 1000 > > > > log_file /var/log/ceph/osd.12.log > > > > --- end dump of recent events --- > > > > > > > > > > > > > > > > What should I do ? > > > > > > > > > > > > > > > > -- > > > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > > > the body of a message to majordomo@xxxxxxxxxxxxxxx (mailto:majordomo@xxxxxxxxxxxxxxx) > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > > > > > > > > > > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |