Hi Sam, perhaps you have overlooked my comments further down, beginning with "been there" ? ;) If so, please have a look, cause I'm clueless 8-) On 12/10/2012 11:48 AM, Oliver Francke wrote:
Hi Sam, helpful input.. and... not so... On 12/07/2012 10:18 PM, Samuel Just wrote:Ah... unfortunately doing a repair in these 6 cases would probably result in the wrong object surviving. It should work, but it might corrupt the rbd image contents. If the images are expendable, you could repair and then delete the images. The red flag here is that the "known size" is smaller than the other size. This indicates that it most likely chose the wrong file as the "correct" one since rbd image blocks usually get bigger over time. To fix this, you will need to manually copy the file for the larger of the two object replicas to replace the smaller of the two object replicas. For the first, soid 87c96f10/rb.0.47d9b.1014b7b4.0000000002df/head//65 in pg 65.10: 1) Find the object on the primary and the replica (from above, primary is 12 and replica is 40). You can use find in the primary and replica current/65.10_head directories to look for a file matching *rb.0.47d9b.1014b7b4.0000000002df*). The file name should be 'rb.0.47d9b.1014b7b4.0000000002df__head_87C96F10__65' I think. 2) Stop the primary and replica osds 3) Compare the file sizes for the two files -- you should find that the file sizes do not match. 4) Replace the smaller file with the larger one (you'll probably want to keep a copy of the smaller one around just in case). 5) Restart the osds and scrub pg 65.10 -- the pg should come up clean (possibly with a relatively harmless stat mismatch)been there. on OSD.12 it's-rw-r--r-- 1 root root 699904 Dec 9 06:25 rb.0.47d9b.1014b7b4.0000000002df__head_87C96F10__41on OSD.40:-rw-r--r-- 1 root root 4194304 Dec 9 06:25 rb.0.47d9b.1014b7b4.0000000002df__head_87C96F10__41going by a short glance into the file, there are some readable syslog-entries, in both files. For the bad luck in this example, the shorter file contains the more current entries?!What exactly happens, if I try to copy or export the file? Which block will be chosen?VM is running as I'm writing, so flexibility reduced. Regards, Oliver.If this worked our correctly, you can repeat for the other 5 cases. Let me know if you have any questions. -SamOn Fri, Dec 7, 2012 at 11:09 AM, Oliver Francke <Oliver.Francke@xxxxxxxx> wrote:Hi Sam, Am 07.12.2012 um 19:37 schrieb Samuel Just <sam.just@xxxxxxxxxxx>:That is very likely to be one of the merge_log bugs fixed between 0.48 and 0.55. I could confirm with a stacktrace from gdb with line numbers or the remainder of the logging dumped when the daemon crashed. My understanding of your situation is that currently all pgs are active+clean but you are missing some rbd image headers and some rbd images appear to be corrupted. Is that accurate? -Samthnx for droppig in. Uhm almost correct, there are now 6 pg in state inconsistent: HEALTH_WARN 6 pgs inconsistent pg 65.da is active+clean+inconsistent, acting [1,33] pg 65.d7 is active+clean+inconsistent, acting [13,42] pg 65.10 is active+clean+inconsistent, acting [12,40] pg 65.f is active+clean+inconsistent, acting [13,31] pg 65.75 is active+clean+inconsistent, acting [1,33] pg 65.6a is active+clean+inconsistent, acting [13,31] I know which images are affected, but does a repair help?0 log [ERR] : 65.10 osd.40: soid 87c96f10/rb.0.47d9b.1014b7b4.0000000002df/head//65 size 4194304 != known size 699904 0 log [ERR] : 65.6a osd.31: soid 19a2526a/rb.0.2dcf2.1da2a31e.000000000737/head//65 size 4191744 != known size 2757632 0 log [ERR] : 65.75 osd.33: soid 20550575/rb.0.2d520.5c17a6e3.000000000339/head//65 size 4194304 != known size 1238016 0 log [ERR] : 65.d7 osd.42: soid fa3a5d7/rb.0.2c2a8.12ec359d.00000000205c/head//65 size 4194304 != known size 1382912 0 log [ERR] : 65.da osd.33: soid c2a344da/rb.0.2be17.cb4bd69.000000000081/head//65 size 4191744 != known size 1815552 0 log [ERR] : 65.f osd.31: soid e8d2430f/rb.0.2d1e9.1339c5dd.000000000c41/head//65 size 2424832 != known size 2331648of make things worse?I could only check 14 out of 20 OSD's so far, cause from two older nodes a scrub leads to slow-requests… > couple of minutes, so VM's got stalled… customers pressing the "reset-button", so losing caches…Comments welcome, Oliver.On Fri, Dec 7, 2012 at 6:39 AM, Oliver Francke <Oliver.Francke@xxxxxxxx> wrote:Hi,is the following a "known one", too? Would be good to get it out of my head:/var/log/ceph/ceph-osd.40.log.1.gz: 1: /usr/bin/ceph-osd() [0x706c59]/var/log/ceph/ceph-osd.40.log.1.gz: 2: (()+0xeff0) [0x7f7f306c0ff0]/var/log/ceph/ceph-osd.40.log.1.gz: 3: (gsignal()+0x35) [0x7f7f2f35f1b5] /var/log/ceph/ceph-osd.40.log.1.gz: 4: (abort()+0x180) [0x7f7f2f361fc0]/var/log/ceph/ceph-osd.40.log.1.gz: 5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7f7f2fbf3dc5] /var/log/ceph/ceph-osd.40.log.1.gz: 6: (()+0xcb166) [0x7f7f2fbf2166] /var/log/ceph/ceph-osd.40.log.1.gz: 7: (()+0xcb193) [0x7f7f2fbf2193] /var/log/ceph/ceph-osd.40.log.1.gz: 8: (()+0xcb28e) [0x7f7f2fbf228e]/var/log/ceph/ceph-osd.40.log.1.gz: 9: (ceph::__ceph_assert_fail(charconst*, char const*, int, char const*)+0x793) [0x77e903] /var/log/ceph/ceph-osd.40.log.1.gz: 10: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, int)+0x1de3) [0x63db93] /var/log/ceph/ceph-osd.40.log.1.gz: 11:(PG::RecoveryState::Stray::react(PG::RecoveryState::MLogRec const&)+0x2cc)[0x63e00c] /var/log/ceph/ceph-osd.40.log.1.gz: 12: (boost::statechart::simple_state<PG::RecoveryState::Stray,PG::RecoveryState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,mpl_::na, mpl_::na, mpl_::na>,(boost::statechart::history_mode)0>::react_impl(boost::statechart::event_baseconst&, void const*)+0x203) [0x658a63] /var/log/ceph/ceph-osd.40.log.1.gz: 13:(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine,PG::RecoveryState::Initial, std::allocator<void>,boost::statechart::null_exception_translator>::process_event(boost::statechart::event_baseconst&)+0x6b) [0x650b4b] /var/log/ceph/ceph-osd.40.log.1.gz: 14:(PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x190)[0x60a520] /var/log/ceph/ceph-osd.40.log.1.gz: 15:(OSD::handle_pg_log(std::tr1::shared_ptr<OpRequest>)+0x666) [0x5c62e6]/var/log/ceph/ceph-osd.40.log.1.gz: 16: (OSD::dispatch_op(std::tr1::shared_ptr<OpRequest>)+0x11b) [0x5c6f3b]/var/log/ceph/ceph-osd.40.log.1.gz: 17: (OSD::_dispatch(Message*)+0x173)[0x5d1983]/var/log/ceph/ceph-osd.40.log.1.gz: 18: (OSD::ms_dispatch(Message*)+0x184)[0x5d2254] /var/log/ceph/ceph-osd.40.log.1.gz: 19: (SimpleMessenger::DispatchQueue::entry()+0x5e9) [0x7d3c09] /var/log/ceph/ceph-osd.40.log.1.gz: 20: (SimpleMessenger::dispatch_entry()+0x15) [0x7d5195] /var/log/ceph/ceph-osd.40.log.1.gz: 21: (SimpleMessenger::DispatchThread::entry()+0xd) [0x726bad] /var/log/ceph/ceph-osd.40.log.1.gz: 22: (()+0x68ca) [0x7f7f306b88ca]/var/log/ceph/ceph-osd.40.log.1.gz: 23: (clone()+0x6d) [0x7f7f2f3fc92d]Thnx for looking, Oliver. -- Oliver Francke filoo GmbH Moltkestraße 25a 33330 Gütersloh HRB4355 AG Gütersloh Geschäftsführer: S.Grewing | J.Rehpöhler | C.Kunz Folgen Sie uns auf Twitter: http://twitter.com/filoogmbh --To unsubscribe from this list: send the line "unsubscribe ceph-devel" inthe body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html--To unsubscribe from this list: send the line "unsubscribe ceph-devel" inthe body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
-- Oliver Francke filoo GmbH Moltkestraße 25a 33330 Gütersloh HRB4355 AG Gütersloh Geschäftsführer: S.Grewing | J.Rehpöhler | C.Kunz Folgen Sie uns auf Twitter: http://twitter.com/filoogmbh -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
