On Fri, Sep 06, 2013 at 01:12:21PM +1000, Chris Dunlop wrote: > On Thu, Sep 05, 2013 at 07:55:52PM -0700, Sage Weil wrote: >> On Fri, 6 Sep 2013, Chris Dunlop wrote: >>> Hi Sage, >>> >>> Does this answer your question? >>> >>> 2013-09-06 09:30:19.813811 7f0ae8cbc700 0 log [INF] : applying configuration change: internal_safe_to_start_threads = 'true' >>> 2013-09-06 09:33:28.303658 7f0ae94bd700 0 log [ERR] : 2.12 osd.7: soid 56987a12/rb.0.17d9b.2ae8944a.000000001e11/head//2 extra attr _, extra attr snapset >>> 2013-09-06 09:33:28.303685 7f0ae94bd700 0 log [ERR] : repair 2.12 56987a12/rb.0.17d9b.2ae8944a.000000001e11/head//2 no 'snapset' attr >>> 2013-09-06 09:34:45.138468 7f0ae94bd700 0 log [ERR] : 2.12 repair stat mismatch, got 2722/2723 objects, 339/339 clones, 11307104768/11311299072 bytes. >>> 2013-09-06 09:34:45.142215 7f0ae94bd700 0 log [ERR] : 2.12 repair 0 missing, 1 inconsistent objects >>> 2013-09-06 09:34:45.206621 7f0ae94bd700 -1 *** Caught signal (Aborted) ** >>> >>> I've just attached the full 'debug_osd 0/10' log to the bug report. >> >> This suggests to me that the object on osd.6 is missing those xattrs; can >> you confirm with getfattr -d on the in osd.6's data directory? > > I haven't yet wrapped my head around how to translate an oid > like those above into a underlying file system object. What > directory should I be looking at? Found it: b5# cd /var/lib/ceph/osd/ceph-6/current b5# find 2.12* | grep -i 17d9b.2ae8944a.000000001e11 2.12_head/DIR_2/DIR_1/DIR_A/rb.0.17d9b.2ae8944a.000000001e11__head_56987A12__2 b5# getfattr -d 2.12_head/DIR_2/DIR_1/DIR_A/rb.0.17d9b.2ae8944a.000000001e11__head_56987A12__2 <<< ...crickets... >>> vs. b4# cd /var/lib/ceph/osd/ceph-7/current b4# getfattr -d 2.12_head/DIR_2/DIR_1/DIR_A/rb.0.17d9b.2ae8944a.000000001e11__head_56987A12__2 # file: 2.12_head/DIR_2/DIR_1/DIR_A/rb.0.17d9b.2ae8944a.000000001e11__head_56987A12__2 user.ceph._=0sCgjhAAAABANBAAAAAAAAACAAAAByYi4wLjE3ZDliLjJhZTg5NDRhLjAwMDAwMDAwMWUxMf7/////////EnqYVgAAAAAAAgAAAAAAAAAEAxAAAAACAAAAAAAAAP////8AAAAAAAAAAEInCgAAAAAAuEsAAEEnCgAAAAAAuEsAAAICFQAAAAgTmwEAAAAAAHD1AgAAAAAAAAAAAAAAQAAAAAAAyY4dUpjCTSACAhUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABCJwoAAAAAALhLAAAA user.ceph.snapset=0sAgIZAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAA== >> If that is indeed the case, you should be able to move the object out of >> the way (don't delete it, just in case) and then do the repair. The osd.6 >> should recover by copying the object from osd.7 (which has the needed >> xattrs). Bobtail is smart enough to recover missing objects but not to >> recover just missing xattrs. > > Do you want me to hold off on any repairs to allow tracking down > the crash, or is the current code sufficiently different that > there's little point? Repaired! ...but why does it take multiple rounds? b5# mv 2.12_head/DIR_2/DIR_1/DIR_A/rb.0.17d9b.2ae8944a.000000001e11__head_56987A12__2 .. b5# ceph pg repair 2.12 b5# while ceph -s | grep -q scrubbing; do sleep 60; done b5# tail /var/log/ceph/ceph-osd.6.log 2013-09-06 15:02:13.751160 7f6ccc5ae700 0 log [ERR] : 2.12 osd.6 missing 56987a12/rb.0.17d9b.2ae8944a.000000001e11/head//2 2013-09-06 15:04:15.286711 7f6ccc5ae700 0 log [ERR] : 2.12 repair stat mismatch, got 2723/2724 objects, 339/339 clones, 11311299072/11315493376 bytes. 2013-09-06 15:04:15.286766 7f6ccc5ae700 0 log [ERR] : 2.12 repair 1 missing, 0 inconsistent objects 2013-09-06 15:04:15.286823 7f6ccc5ae700 0 log [ERR] : 2.12 repair 2 errors, 2 fixed 2013-09-06 15:04:20.778377 7f6ccc5ae700 0 log [ERR] : 2.12 scrub stat mismatch, got 2724/2723 objects, 339/339 clones, 11315493376/11311299072 bytes. 2013-09-06 15:04:20.778383 7f6ccc5ae700 0 log [ERR] : 2.12 scrub 1 errors b5# ceph pg dump | grep inconsistent 2.12 2723 0 0 0 11311299072 159103 159103 active+clean+inconsistent 2013-09-06 15:04:20.778413 20121'690883 20128'7941893 [6,7] [6,7] 20121'690883 2013-09-06 15:04:20.778387 20121'690883 2013-09-06 15:04:15.286835 b5# ceph pg repair 2.12 b5# while ceph -s | grep -q scrubbing; do sleep 60; done b5# tail /var/log/ceph/ceph-osd.6.log 2013-09-06 15:07:30.461959 7f6ccc5ae700 0 log [ERR] : 2.12 repair stat mismatch, got 2724/2723 objects, 339/339 clones, 11315493376/11311299072 bytes. 2013-09-06 15:07:30.461991 7f6ccc5ae700 0 log [ERR] : 2.12 repair 1 errors, 1 fixed b5# ceph pg dump | grep inconsistent 2.12 2724 0 0 0 11315493376 159580 159580 active+clean+inconsistent 2013-09-06 15:07:30.462039 20129'690886 20128'7942171 [6,7] [6,7] 20129'690886 2013-09-06 15:07:30.461995 20129'690886 2013-09-06 15:07:30.461995 b5# ceph pg repair 2.12 b5# while ceph -s | grep -q scrubbing; do sleep 60; done b5# tail /var/log/ceph/ceph-osd.6.log 2013-09-06 15:09:36.993049 7f6ccc5ae700 0 log [INF] : 2.12 repair ok, 0 fixed # ceph pg dump | grep inconsistent <<< ...crickets... >>> Chris -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
