Hi Henry,
On Thu, 10 Mar 2011, Henry Chang wrote:
> Hi all,
>
> I was tracing the osd recovery code recently. I found that the
> PG::merge_log function seems not working properly (see the logs in the
> bottom).
> The object 10000000fbf.0000046b should not be added to osd6's missing
> list after osd6 merged osd3's log+backlog. I try to fix it as follows:
>
> diff --git a/src/osd/PG.cc b/src/osd/PG.cc
> index 2778591..671a37d 100644
> --- a/src/osd/PG.cc
> +++ b/src/osd/PG.cc
> @@ -351,7 +351,8 @@ void PG::merge_log(ObjectStore::Transaction& t,
> if (p->version <= log.head) {
> dout(10) << "merge_log split point is " << *p << dendl;
>
> - if (p->version == log.head)
> + if (old_objects.find(p->soid) != old_objects.end() &&
> + old_objects[p->soid]->version == p->version)
> p++; // move past the split point, if it also exists
> in our old log...
> break;
> }
>
> Is it a right fix for this issue? (I am running with the stable branch.)
That looks like the correct fix. I changed the patch a bit to use an
iterator and avoid a double lookup, and pushed to stable.
Thanks!
sage
> --
> Henry Chang
>
>
> 2011-03-08 16:18:44.765612 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 lcod 0'0
> stray] my log = log(27'14,27'17]+backlog
> 2011-03-08 16:18:44.765629 25'1 (0'0) b 100000003f1.00000137/head by
> client4111.1:1320 2011-03-08 12:41:45.787200 indexed
> 2011-03-08 16:18:44.765643 27'2 (0'0) b 100000003f1.00000471/head by
> client4111.1:3885 2011-03-08 12:55:46.341185 indexed
> 2011-03-08 16:18:44.765657 27'3 (0'0) b 10000000bd4.00000017/head by
> client4120.1:217 2011-03-08 15:21:10.475753 indexed
> 2011-03-08 16:18:44.765671 27'5 (0'0) b 10000000fbf.00000160/head by
> client4113.1:3178 2011-03-08 15:22:59.209544 indexed
> 2011-03-08 16:18:44.765685 27'7 (0'0) b 10000000bd4.00000253/head by
> client4120.1:5429 2011-03-08 15:24:03.796691 indexed
> 2011-03-08 16:18:44.765699 27'8 (0'0) b 10000000fbf.000002e5/head by
> client4113.1:6680 2011-03-08 15:24:47.665824 indexed
> 2011-03-08 16:18:44.765712 27'9 (0'0) b 100000007dc.0000031a/head by
> client4107.1:7232 2011-03-08 15:24:50.757166 indexed
> 2011-03-08 16:18:44.765726 27'10 (0'0) b 10000000bd4.00000304/head by
> client4120.1:7054 2011-03-08 15:24:56.641569 indexed
> 2011-03-08 16:18:44.765740 27'11 (0'0) b 10000000fbf.0000046b/head by
> client4113.1:10190 2011-03-08 15:26:43.835520 indexed
> 2011-03-08 16:18:44.765753 27'12 (27'4) b 10000000bd4.00000122/head by
> client4120.1:16840 2011-03-08 16:05:30.061403 indexed
> 2011-03-08 16:18:44.765767 27'15 (27'14) m 10000000bd4.000001f9/head
> by client4120.1:16882 2011-03-08 16:05:30.061403 indexed
> 2011-03-08 16:18:44.765781 27'16 (27'15) m 10000000bd4.000001f9/head
> by client4120.1:17145 2011-03-08 16:06:08.641234 indexed
> 2011-03-08 16:18:44.765795 27'17 (27'16) m 10000000bd4.000001f9/head
> by client4120.1:17146 2011-03-08 16:06:08.641234 indexed
> 2011-03-08 16:18:44.765805 2011-03-08 16:18:44.765822 g[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 lcod 0'0
> stray] osd3 log = log(60'24,60'27]+backlog
> 2011-03-08 16:18:44.765837 25'1 (0'0) b 100000003f1.00000137/head by
> client4111.1:1320 2011-03-08 12:41:45.787200
> 2011-03-08 16:18:44.765850 27'2 (0'0) b 100000003f1.00000471/head by
> client4111.1:3885 2011-03-08 12:55:46.341185
> 2011-03-08 16:18:44.765863 27'3 (0'0) b 10000000bd4.00000017/head by
> client4120.1:217 2011-03-08 15:21:10.475753
> 2011-03-08 16:18:44.765876 27'5 (0'0) b 10000000fbf.00000160/head by
> client4113.1:3178 2011-03-08 15:22:59.209544
> 2011-03-08 16:18:44.765889 27'7 (0'0) b 10000000bd4.00000253/head by
> client4120.1:5429 2011-03-08 15:24:03.796691
> 2011-03-08 16:18:44.765902 27'8 (0'0) b 10000000fbf.000002e5/head by
> client4113.1:6680 2011-03-08 15:24:47.665824
> 2011-03-08 16:18:44.765915 27'9 (0'0) b 100000007dc.0000031a/head by
> client4107.1:7232 2011-03-08 15:24:50.757166
> 2011-03-08 16:18:44.765928 27'10 (0'0) b 10000000bd4.00000304/head by
> client4120.1:7054 2011-03-08 15:24:56.641569
> 2011-03-08 16:18:44.765941 27'11 (0'0) b 10000000fbf.0000046b/head by
> client4113.1:10190 2011-03-08 15:26:43.835520
> 2011-03-08 16:18:44.765954 60'22 (27'12) b 10000000bd4.00000122/head
> by client4107.1:17748 2011-03-08 16:14:46.795745
> 2011-03-08 16:18:44.765966 60'25 (60'24) m 10000000bd4.000001f9/head
> by client4107.1:17789 2011-03-08 16:14:46.795745
> 2011-03-08 16:18:44.765980 60'26 (60'25) m 10000000bd4.000001f9/head
> by client4107.1:18108 2011-03-08 16:15:19.394651
> 2011-03-08 16:18:44.765992 60'27 (60'26) m 10000000bd4.000001f9/head
> by client4107.1:18109 2011-03-08 16:15:19.394651
> 2011-03-08 16:18:44.766002 2011-03-08 16:18:44.766029 7f74783d6710
> osd6 75 pg[0.1b( v 27'17 (27'14,27'17]+backlog n=11 ec=2 les=73
> 61/67/6) [3,6] r=1 lcod 0'0 stray] merge_log log(60'24,60'27]+backlog
> from osd3 into log(27'14,27'17]+backlog
> 2011-03-08 16:18:44.766094 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray] merge_log split point
> is 27'11 (0'0) b 10000000fbf.0000046b/head by client4113.1:10190
> 2011-03-08 15:26:43.835520
> 2011-03-08 16:18:44.766127 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray] merge_log merging 27'11
> (0'0) b 10000000fbf.0000046b/head by client4113.1:10190 2011-03-08
> 15:26:43.835520
> 2011-03-08 16:18:44.766160 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray m=1] merge_log merging
> 60'22 (27'12) b 10000000bd4.00000122/head by client4107.1:17748
> 2011-03-08 16:14:46.795745
> 2011-03-08 16:18:44.766191 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray m=2] merge_log merging
> 60'25 (60'24) m 10000000bd4.000001f9/head by client4107.1:17789
> 2011-03-08 16:14:46.795745
> 2011-03-08 16:18:44.766224 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray m=3] merge_log merging
> 60'26 (60'25) m 10000000bd4.000001f9/head by client4107.1:18108
> 2011-03-08 16:15:19.394651
> 2011-03-08 16:18:44.766256 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray m=3] merge_log merging
> 60'27 (60'26) m 10000000bd4.000001f9/head by client4107.1:18109
> 2011-03-08 16:15:19.394651
> 2011-03-08 16:18:44.766294 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray m=3] merge_old_entry
> had 25'1 (0'0) b 100000003f1.00000137/head by client4111.1:1320
> 2011-03-08 12:41:45.787200 new 25'1 (0'0) b 100000003f1.00000137/head
> by client4111.1:1320 2011-03-08 12:41:45.787200 : same
> 2011-03-08 16:18:44.766328 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray m=3] merge_old_entry
> had 27'2 (0'0) b 100000003f1.00000471/head by client4111.1:3885
> 2011-03-08 12:55:46.341185 new 27'2 (0'0) b 100000003f1.00000471/head
> by client4111.1:3885 2011-03-08 12:55:46.341185 : same
> 2011-03-08 16:18:44.766363 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray m=3] merge_old_entry
> had 27'3 (0'0) b 10000000bd4.00000017/head by client4120.1:217
> 2011-03-08 15:21:10.475753 new 27'3 (0'0) b 10000000bd4.00000017/head
> by client4120.1:217 2011-03-08 15:21:10.475753 : same
> 2011-03-08 16:18:44.766397 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray m=3] merge_old_entry
> had 27'5 (0'0) b 10000000fbf.00000160/head by client4113.1:3178
> 2011-03-08 15:22:59.209544 new 27'5 (0'0) b 10000000fbf.00000160/head
> by client4113.1:3178 2011-03-08 15:22:59.209544 : same
> 2011-03-08 16:18:44.766432 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray m=3] merge_old_entry
> had 27'7 (0'0) b 10000000bd4.00000253/head by client4120.1:5429
> 2011-03-08 15:24:03.796691 new 27'7 (0'0) b 10000000bd4.00000253/head
> by client4120.1:5429 2011-03-08 15:24:03.796691 : same
> 2011-03-08 16:18:44.766465 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray m=3] merge_old_entry
> had 27'8 (0'0) b 10000000fbf.000002e5/head by client4113.1:6680
> 2011-03-08 15:24:47.665824 new 27'8 (0'0) b 10000000fbf.000002e5/head
> by client4113.1:6680 2011-03-08 15:24:47.665824 : same
> 2011-03-08 16:18:44.766510 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray m=3] merge_old_entry
> had 27'9 (0'0) b 100000007dc.0000031a/head by client4107.1:7232
> 2011-03-08 15:24:50.757166 new 27'9 (0'0) b 100000007dc.0000031a/head
> by client4107.1:7232 2011-03-08 15:24:50.757166 : same
> 2011-03-08 16:18:44.766545 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray m=3] merge_old_entry
> had 27'10 (0'0) b 10000000bd4.00000304/head by client4120.1:7054
> 2011-03-08 15:24:56.641569 new 27'10 (0'0) b 10000000bd4.00000304/head
> by client4120.1:7054 2011-03-08 15:24:56.641569 : same
> 2011-03-08 16:18:44.766580 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray m=3] merge_old_entry
> had 27'11 (0'0) b 10000000fbf.0000046b/head by client4113.1:10190
> 2011-03-08 15:26:43.835520 new 27'11 (0'0) b 10000000fbf.0000046b/head
> by client4113.1:10190 2011-03-08 15:26:43.835520 : same
> 2011-03-08 16:18:44.766614 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray m=3] merge_old_entry
> had 27'12 (27'4) b 10000000bd4.00000122/head by client4120.1:16840
> 2011-03-08 16:05:30.061403 new 60'22 (27'12) b
> 10000000bd4.00000122/head by client4107.1:17748 2011-03-08
> 16:14:46.795745 : older, missing
> 2011-03-08 16:18:44.766650 7f74783d6710 osd6 75 pg[0.1b( v 27'17
> (27'14,27'17]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 (log bound
> mismatch, actual=[25'1,60'27]) lcod 0'0 stray m=3] merge_old_entry
> had 27'17 (27'16) m 10000000bd4.000001f9/head by client4120.1:17146
> 2011-03-08 16:06:08.641234 new 60'27 (60'26) m
> 10000000bd4.000001f9/head by client4107.1:18109 2011-03-08
> 16:15:19.394651 : older, missing
> 2011-03-08 16:18:44.766681 7f74783d6710 osd6 75 pg[0.1b( v 60'27 lc
> 27'17 (60'24,60'27]+backlog n=11 ec=2 les=73 61/67/6) [3,6] r=1 lcod
> 0'0 stray m=3] merge_log result log(60'24,60'27]+backlog missing(3)
> changed=1
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
