On Sun, 16 Mar 2014, Kevinsky Dy wrote: > Hi, > > I currently have 2 OSDs that won't start and it's preventing my > cluster from running my VMs. > My cluster is running on: > # ceph -v > ceph version 0.67.7 (d7ab4244396b57aac8b7e80812115bbd079e6b73) > > My OSD's are found on different hosts as per the default CRUSH Map > rules and I got the logs from the problem OSDs as follows > > osd.1: > -10> 2014-03-16 11:51:25.183093 7f50923e4780 20 read_log > 12794'1377363 (12794'1377357) modify > 5ee8f77d/rb.0.1f7b.238e1f29.0000000118fb/head//2 by > client.1077470.0:376704 2014-03-15 12:37:43.175319 > -9> 2014-03-16 11:51:25.183122 7f50923e4780 20 read_log > 12794'1377364 (12794'1377362) modify > af8f477d/rb.0.24f4.238e1f29.00000000e378/head//2 by > client.1074837.0:1295631 2014-03-15 12:38:03.613604 > -8> 2014-03-16 11:51:25.183146 7f50923e4780 20 read_log > 12794'1377365 (12794'1377348) modify > cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by > client.1077379.0:1396709 2014-03-15 12:38:33.720354 > -7> 2014-03-16 11:51:25.183179 7f50923e4780 20 read_log > 12794'1377366 (12794'1377365) modify > cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by > client.1077379.0:1396712 2014-03-15 12:38:33.726419 > -6> 2014-03-16 11:51:25.183207 7f50923e4780 20 read_log > 12794'1377367 (12794'1377366) modify > cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by > client.1077379.0:1397305 2014-03-15 12:39:09.863260 > -5> 2014-03-16 11:51:25.183231 7f50923e4780 20 read_log > 12794'1377368 (12794'1377367) modify > cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by > client.1077379.0:1398903 2014-03-15 12:40:13.096258 > -4> 2014-03-16 11:51:25.183258 7f50923e4780 20 read_log > 12794'1377369 (12794'1377363) modify > 5ee8f77d/rb.0.1f7b.238e1f29.0000000118fb/head//2 by > client.1077470.0:377159 2014-03-15 12:40:13.105469 > -3> 2014-03-16 11:51:25.183282 7f50923e4780 20 read_log > 12794'1377370 (12794'1377360) modify > 19463f7d/rb.0.ecb29.238e1f29.000000000101/head//2 by > client.1058212.1:358750 2014-03-15 12:40:24.998076 > -2> 2014-03-16 11:51:25.183309 7f50923e4780 20 read_log > 12794'1377371 (12794'1377368) modify > cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by > client.1077379.0:1399253 2014-03-15 12:40:28.134624 > -1> 2014-03-16 11:51:25.183333 7f50923e4780 20 read_log > 13740'1377371 (12605'1364064) modify > 94c6137d/rb.0.d07d.2ae8944a.000000002524/head//2 by > client.1088028.0:10498 2014-03-16 00:06:12.643968 > 0> 2014-03-16 11:51:25.185685 7f50923e4780 -1 osd/PGLog.cc: In > function 'static bool PGLog::read_log(ObjectStore*, coll_t, hobject_t, > const pg_info_t&, std::map<eversion_t, hobject_t>&, > PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&, > std::set<std::basic_string<char> >*)' thread 7f50923e4780 time > 2014-03-16 11:51:25.183350 > osd/PGLog.cc: 677: FAILED assert(last_e.version.version < e.version.version) Any idea what might have happened to get the cluster in this state? Was logging turned up leading up to the crash? The version is an (epoch, v) pair and v is normally monotonically increasing. I don't think much relies on this, though, so I pushed a wip-dumpling-sloppy-log branch that just drops this assert; I suspect it will come up after that. sage > > osd.2: > -10> 2014-03-16 11:28:45.015366 7fe5a7539780 20 read_log 12794'1377363 > (12794'1377357) modify > 5ee8f77d/rb.0.1f7b.238e1f29.0000000118fb/head//2 by > client.1077470.0:376704 2014-03-15 12:37:43.175319 > -9> 2014-03-16 11:28:45.015381 7fe5a7539780 20 read_log > 12794'1377364 (12794'1377362) modify > af8f477d/rb.0.24f4.238e1f29.00000000e378/head//2 by > client.1074837.0:1295631 2014-03-15 12:38:03.613604 > -8> 2014-03-16 11:28:45.015394 7fe5a7539780 20 read_log > 12794'1377365 (12794'1377348) modify > cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by > client.1077379.0:1396709 2014-03-15 12:38:33.720354 > -7> 2014-03-16 11:28:45.015405 7fe5a7539780 20 read_log > 12794'1377366 (12794'1377365) modify > cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by > client.1077379.0:1396712 2014-03-15 12:38:33.726419 > -6> 2014-03-16 11:28:45.015418 7fe5a7539780 20 read_log > 12794'1377367 (12794'1377366) modify > cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by > client.1077379.0:1397305 2014-03-15 12:39:09.863260 > -5> 2014-03-16 11:28:45.015428 7fe5a7539780 20 read_log > 12794'1377368 (12794'1377367) modify > cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by > client.1077379.0:1398903 2014-03-15 12:40:13.096258 > -4> 2014-03-16 11:28:45.015441 7fe5a7539780 20 read_log > 12794'1377369 (12794'1377363) modify > 5ee8f77d/rb.0.1f7b.238e1f29.0000000118fb/head//2 by > client.1077470.0:377159 2014-03-15 12:40:13.105469 > -3> 2014-03-16 11:28:45.015452 7fe5a7539780 20 read_log > 12794'1377370 (12794'1377360) modify > 19463f7d/rb.0.ecb29.238e1f29.000000000101/head//2 by > client.1058212.1:358750 2014-03-15 12:40:24.998076 > -2> 2014-03-16 11:28:45.015464 7fe5a7539780 20 read_log > 12794'1377371 (12794'1377368) modify > cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by > client.1077379.0:1399253 2014-03-15 12:40:28.134624 > -1> 2014-03-16 11:28:45.015475 7fe5a7539780 20 read_log > 13740'1377371 (12605'1364064) modify > 94c6137d/rb.0.d07d.2ae8944a.000000002524/head//2 by > client.1088028.0:10498 2014-03-16 00:06:12.643968 > 0> 2014-03-16 11:28:45.016656 7fe5a7539780 -1 osd/PGLog.cc: In > function 'static bool PGLog::read_log(ObjectStore*, coll_t, hobject_t, > const pg_info_t&, std::map<eversion_t, hobject_t>&, > PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&, > std::set<std::basic_string<char> >*)' thread 7fe5a7539780 time > 2014-03-16 11:28:45.015497 > osd/PGLog.cc: 677: FAILED assert(last_e.version.version < e.version.version) > > It seems that the latest version of the object taken from omap is not > newer than the ones represented in the logs and this failed assert is > preventing those 2 OSDs to start. > > Here's a link to the code. > https://github.com/ceph/ceph/blob/dumpling/src/osd/PGLog.cc#L667 > > Any thoughts on how to recover from this? > > Thanks, > Skye Dy > > P.S. I apologize to those who will receive this multiple times. > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
