Hi Mart, Wido...
A disclaimer: Not really an expert, just a regular site admin
sharing my experience.
At the beginning of the thread you give the idea that only osd.68
has problems dealing with the problematic PG 3.117. If that is
indeed the case, you could simply mark that osd.68 down and remove
it from the cluster. This will trigger Ceph to replicate all PGs in
osd.68 to other osds based on other PG replicas.
However, In the last email, you seem to give the idea that it is the
PG 3.117 which has problems, which makes all osds which share that
PG also problematic. Because of that you marked all osds sharing
that PG as down.
Before actually trying something more drastic, I would go for a more
classic approach. For example, what happens if you turn only one osd
up? I would start with osd.74 since you suspect of problems in
osd.68 and osd.55 was the reason for the dump message bellow. If it
still aborts than it means that the PG might have been replicated
everywhere with 'bad' data.
The drastic approach (If you do not care about data on that PG), is
to mark those osds has down, and force that PG to be recreated using
'ceph pg force_create_pg 3.117'. Based on my previous experience,
once I've recreated a PG, 'ceph pg dump_stuck stale' showed that PG
is creating state forever. To make it right, I had to restart the
proper osds. But, as you stated, you then have to deal with data
corruption at the VMs level... Maybe that is a problem, maybe it
isn't...
Hope that helps
Cheers
Goncalo
On 03/31/2016 12:26 PM, Mart van Santen
wrote:
Hello,
Well unfortunately the problem is not really solved. Yes, we
managed to get to a good health state at some point, when a client
hits some specific data, the osd process crashes with below
errors. The 3 OSD which handle 3.117, the PG with problems, are
currently down and reweighted them to 0, so non-affected PGs are
currently rebuild on other OSDs
If I put them crashed osd up, the do crash again within a few
minutes.
As I'm a bit afraid for the data in this PG, I think we want to
recreate the PG with empty data and discard the old disks. I
understand I will get datacorruption on serveral RBDs in this
case, but we will try to solve that and rebuild the affected VMs.
Does this makes sense and what are the best next steps?
Regards,
Mart
works
-34> 2016-03-31 03:07:56.932800 7f8e43829700 3 osd.55
122203 handle_osd_map epochs [122203,122203], i have 122203, src
has [120245,122203]
-33> 2016-03-31 03:07:56.932837 7f8e43829700 1 --
[2a00:c6c0:0:122::105]:6822/11703 <== osd.45
[2a00:c6c0:0:122::103]:6800/1852 7 ==== pg_info(1 pgs
e122202:3.117) v4 ==== 919+0+0 (3389909573 0 0) 0x528bc00 con
0x1200a840
-32> 2016-03-31 03:07:56.932855 7f8e43829700 5 -- op
tracker -- seq: 22, time: 2016-03-31 03:07:56.932770, event:
header_read, op: pg_info(1 pgs e122202:3.117)
-31> 2016-03-31 03:07:56.932869 7f8e43829700 5 -- op
tracker -- seq: 22, time: 2016-03-31 03:07:56.932771, event:
throttled, op: pg_info(1 pgs e122202:3.117)
-30> 2016-03-31 03:07:56.932878 7f8e43829700 5 -- op
tracker -- seq: 22, time: 2016-03-31 03:07:56.932822, event:
all_read, op: pg_info(1 pgs e122202:3.117)
-29> 2016-03-31 03:07:56.932886 7f8e43829700 5 -- op
tracker -- seq: 22, time: 2016-03-31 03:07:56.932851, event:
dispatched, op: pg_info(1 pgs e122202:3.117)
-28> 2016-03-31 03:07:56.932895 7f8e43829700 5 -- op
tracker -- seq: 22, time: 2016-03-31 03:07:56.932895, event:
waiting_for_osdmap, op: pg_info(1 pgs e122202:3.117)
-27> 2016-03-31 03:07:56.932912 7f8e43829700 5 -- op
tracker -- seq: 22, time: 2016-03-31 03:07:56.932912, event:
started, op: pg_info(1 pgs e122202:3.117)
-26> 2016-03-31 03:07:56.932947 7f8e43829700 5 -- op
tracker -- seq: 22, time: 2016-03-31 03:07:56.932947, event: done,
op: pg_info(1 pgs e122202:3.117)
-25> 2016-03-31 03:07:56.933022 7f8e3c01a700 1 --
[2a00:c6c0:0:122::105]:6822/11703 -->
[2a00:c6c0:0:122::103]:6800/1852 -- osd_map(122203..122203 src has
121489..122203) v3 -- ?+0 0x11c7fd40 con 0x1200a840
-24> 2016-03-31 03:07:56.933041 7f8e3c01a700 1 --
[2a00:c6c0:0:122::105]:6822/11703 -->
[2a00:c6c0:0:122::103]:6800/1852 -- pg_info(1 pgs e122203:3.117)
v4 -- ?+0 0x528bde0 con 0x1200a840
-23> 2016-03-31 03:07:56.933111 7f8e3c01a700 1 --
[2a00:c6c0:0:122::105]:6822/11703 -->
[2a00:c6c0:0:122::105]:6810/3568 -- osd_map(122203..122203 src has
121489..122203) v3 -- ?+0 0x12200d00 con 0x1209d4a0
-22> 2016-03-31 03:07:56.933125 7f8e3c01a700 1 --
[2a00:c6c0:0:122::105]:6822/11703 -->
[2a00:c6c0:0:122::105]:6810/3568 -- pg_info(1 pgs e122203:3.117)
v4 -- ?+0 0x5288960 con 0x1209d4a0
-21> 2016-03-31 03:07:56.933154 7f8e3c01a700 1 --
[2a00:c6c0:0:122::105]:6822/11703 -->
[2a00:c6c0:0:122::108]:6816/1002847 -- pg_info(1 pgs
e122203:3.117) v4 -- ?+0 0x5288d20 con 0x101a19c0
-20> 2016-03-31 03:07:56.933212 7f8e3c01a700 5 osd.55
pg_epoch: 122203 pg[3.117( v 122193'1898519
(108032'1895437,122193'1898519] local-les=122202 n=2789 ec=23736
les/c 122202/122047 122062/122201/122201) [72,54,45]/[55] r=0
lpr=122201 pi=122046-122200/51 bft=45,54,72 crt=122133'1898514
lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] on
activate: bft=45,54,72 from 0//0//-1
-19> 2016-03-31 03:07:56.933232 7f8e3c01a700 5 osd.55
pg_epoch: 122203 pg[3.117( v 122193'1898519
(108032'1895437,122193'1898519] local-les=122202 n=2789 ec=23736
les/c 122202/122047 122062/122201/122201) [72,54,45]/[55] r=0
lpr=122201 pi=122046-122200/51 bft=45,54,72 crt=122133'1898514
lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target
shard 45 from 0//0//-1
-18> 2016-03-31 03:07:56.933244 7f8e3c01a700 5 osd.55
pg_epoch: 122203 pg[3.117( v 122193'1898519
(108032'1895437,122193'1898519] local-les=122202 n=2789 ec=23736
les/c 122202/122047 122062/122201/122201) [72,54,45]/[55] r=0
lpr=122201 pi=122046-122200/51 bft=45,54,72 crt=122133'1898514
lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target
shard 54 from 0//0//-1
-17> 2016-03-31 03:07:56.933255 7f8e3c01a700 5 osd.55
pg_epoch: 122203 pg[3.117( v 122193'1898519
(108032'1895437,122193'1898519] local-les=122202 n=2789 ec=23736
les/c 122202/122047 122062/122201/122201) [72,54,45]/[55] r=0
lpr=122201 pi=122046-122200/51 bft=45,54,72 crt=122133'1898514
lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped] target
shard 72 from 0//0//-1
-16> 2016-03-31 03:07:56.933283 7f8e3680f700 5 -- op
tracker -- seq: 20, time: 2016-03-31 03:07:56.933283, event:
reached_pg, op: osd_op(client.776466.1:190178605
rbd_data.900a62ae8944a.0000000000000829 [set-alloc-hint
object_size 4194304 write_size 4194304,write 8192~8192] 3.b1492517
RETRY=1 snapc 8b3=[8b3] ondisk+retry+write e122203)
-15> 2016-03-31 03:07:56.933507 7f8e3680f700 5 -- op
tracker -- seq: 20, time: 2016-03-31 03:07:56.933507, event:
started, op: osd_op(client.776466.1:190178605
rbd_data.900a62ae8944a.0000000000000829 [set-alloc-hint
object_size 4194304 write_size 4194304,write 8192~8192] 3.b1492517
RETRY=1 snapc 8b3=[8b3] ondisk+retry+write e122203)
-14> 2016-03-31 03:07:56.933648 7f8e3680f700 5 -- op
tracker -- seq: 20, time: 2016-03-31 03:07:56.933648, event:
waiting for subops from 45,54,72, op:
osd_op(client.776466.1:190178605
rbd_data.900a62ae8944a.0000000000000829 [set-alloc-hint
object_size 4194304 write_size 4194304,write 8192~8192] 3.b1492517
RETRY=1 snapc 8b3=[8b3] ondisk+retry+write e122203)
-13> 2016-03-31 03:07:56.933682 7f8e3680f700 1 --
[2a00:c6c0:0:122::105]:6822/11703 -->
[2a00:c6c0:0:122::103]:6800/1852 --
osd_repop(client.776466.1:190178605 3.117
b1492517/rbd_data.900a62ae8944a.0000000000000829/head//3 v
122203'1898521) v1 -- ?+46 0x11e96400 con 0x1200a840
-12> 2016-03-31 03:07:56.933712 7f8e3680f700 1 --
[2a00:c6c0:0:122::105]:6822/11703 -->
[2a00:c6c0:0:122::105]:6810/3568 --
osd_repop(client.776466.1:190178605 3.117
b1492517/rbd_data.900a62ae8944a.0000000000000829/head//3 v
122203'1898521) v1 -- ?+46 0x11e96a00 con 0x1209d4a0
-11> 2016-03-31 03:07:56.933735 7f8e3680f700 1 --
[2a00:c6c0:0:122::105]:6822/11703 -->
[2a00:c6c0:0:122::108]:6816/1002847 --
osd_repop(client.776466.1:190178605 3.117
b1492517/rbd_data.900a62ae8944a.0000000000000829/head//3 v
122203'1898521) v1 -- ?+46 0x11e97600 con 0x101a19c0
-10> 2016-03-31 03:07:56.935173 7f8e30ef5700 1 --
[2a00:c6c0:0:122::105]:6822/11703 <== osd.72
[2a00:c6c0:0:122::108]:6816/1002847 9 ====
osd_repop_reply(client.776466.1:190178605 3.117 ondisk, result =
0) v1 ==== 83+0+0 (405786713 0 0) 0x11e66d00 con 0x101a19c0
-9> 2016-03-31 03:07:56.935212 7f8e30ef5700 5 -- op
tracker -- seq: 23, time: 2016-03-31 03:07:56.935087, event:
header_read, op: osd_repop_reply(client.776466.1:190178605 3.117
ondisk, result = 0)
-8> 2016-03-31 03:07:56.935224 7f8e30ef5700 5 -- op
tracker -- seq: 23, time: 2016-03-31 03:07:56.935090, event:
throttled, op: osd_repop_reply(client.776466.1:190178605 3.117
ondisk, result = 0)
-7> 2016-03-31 03:07:56.935234 7f8e30ef5700 5 -- op
tracker -- seq: 23, time: 2016-03-31 03:07:56.935162, event:
all_read, op: osd_repop_reply(client.776466.1:190178605 3.117
ondisk, result = 0)
-6> 2016-03-31 03:07:56.935245 7f8e30ef5700 5 -- op
tracker -- seq: 23, time: 0.000000, event: dispatched, op:
osd_repop_reply(client.776466.1:190178605 3.117 ondisk, result =
0)
-5> 2016-03-31 03:07:56.936129 7f8e2dfc6700 1 --
[2a00:c6c0:0:122::105]:6822/11703 <== osd.45
[2a00:c6c0:0:122::103]:6800/1852 8 ====
osd_repop_reply(client.776466.1:190178605 3.117 ondisk, result =
0) v1 ==== 83+0+0 (3967999676 0 0) 0x11c7fd40 con 0x1200a840
-4> 2016-03-31 03:07:56.936150 7f8e2dfc6700 5 -- op
tracker -- seq: 24, time: 2016-03-31 03:07:56.936086, event:
header_read, op: osd_repop_reply(client.776466.1:190178605 3.117
ondisk, result = 0)
-3> 2016-03-31 03:07:56.936159 7f8e2dfc6700 5 -- op
tracker -- seq: 24, time: 2016-03-31 03:07:56.936087, event:
throttled, op: osd_repop_reply(client.776466.1:190178605 3.117
ondisk, result = 0)
-2> 2016-03-31 03:07:56.936166 7f8e2dfc6700 5 -- op
tracker -- seq: 24, time: 2016-03-31 03:07:56.936124, event:
all_read, op: osd_repop_reply(client.776466.1:190178605 3.117
ondisk, result = 0)
-1> 2016-03-31 03:07:56.936172 7f8e2dfc6700 5 -- op
tracker -- seq: 24, time: 0.000000, event: dispatched, op:
osd_repop_reply(client.776466.1:190178605 3.117 ondisk, result =
0)
0> 2016-03-31 03:07:56.940165 7f8e3680f700 -1
osd/SnapMapper.cc: In function 'void SnapMapper::add_oid(const
hobject_t&, const std::set<snapid_t>&,
MapCacher::Transaction<std::basic_string<char>,
ceph::buffer::list>*)' thread 7f8e3680f700 time 2016-03-31
03:07:56.933983
osd/SnapMapper.cc: 228: FAILED assert(r == -2)
ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x8b) [0xba8b8b]
2: (SnapMapper::add_oid(hobject_t const&,
std::set<snapid_t, std::less<snapid_t>,
std::allocator<snapid_t> > const&,
MapCacher::Transaction<std::string,
ceph::buffer::list>*)+0x61e) [0x72137e]
3: (PG::update_snap_map(std::vector<pg_log_entry_t,
std::allocator<pg_log_entry_t> > const&,
ObjectStore::Transaction&)+0x402) [0x7d25c2]
4: (PG::append_log(std::vector<pg_log_entry_t,
std::allocator<pg_log_entry_t> > const&, eversion_t,
eversion_t, ObjectStore::Transaction&, bool)+0x4e8) [0x7d2c68]
5: (ReplicatedPG::log_operation(std::vector<pg_log_entry_t,
std::allocator<pg_log_entry_t> > const&,
boost::optional<pg_hit_set_history_t>&, eversion_t
const&, eversion_t const&, bool,
ObjectStore::Transaction*)+0xba) [0x899eca]
6: (ReplicatedBackend::submit_transaction(hobject_t const&,
eversion_t const&, PGBackend::PGTransaction*, eversion_t
const&, eversion_t const&, std::vector<pg_log_entry_t,
std::allocator<pg_log_entry_t> > const&,
boost::optional<pg_hit_set_history_t>&, Context*,
Context*, Context*, unsigned long, osd_reqid_t,
std::tr1::shared_ptr<OpRequest>)+0x77c) [0x9f06cc]
7: (ReplicatedPG::issue_repop(ReplicatedPG::RepGather*)+0x7aa)
[0x8391aa]
8: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0xbdd)
[0x88792d]
9:
(ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x4559)
[0x88cee9]
10:
(ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&,
ThreadPool::TPHandle&)+0x66a) [0x82702a]
11: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
std::tr1::shared_ptr<OpRequest>,
ThreadPool::TPHandle&)+0x3bd) [0x6961dd]
12: (OSD::ShardedOpWQ::_process(unsigned int,
ceph::heartbeat_handle_d*)+0x338) [0x696708]
13: (ShardedThreadPool::shardedthreadpool_worker(unsigned
int)+0x875) [0xb98555]
14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10)
[0xb9a670]
15: (()+0x8182) [0x7f8e57f6c182]
16: (clone()+0x6d) [0x7f8e564d747d]
NOTE: a copy of the executable, or `objdump -rdS
<executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.55.log
--- end dump of recent events ---
2016-03-31 03:07:56.960104 7f8e3680f700 -1 *** Caught signal
(Aborted) **
in thread 7f8e3680f700
ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
1: /usr/bin/ceph-osd() [0xaaff6a]
2: (()+0x10340) [0x7f8e57f74340]
3: (gsignal()+0x39) [0x7f8e56413cc9]
4: (abort()+0x148) [0x7f8e564170d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155)
[0x7f8e56d1e535]
6: (()+0x5e6d6) [0x7f8e56d1c6d6]
7: (()+0x5e703) [0x7f8e56d1c703]
8: (()+0x5e922) [0x7f8e56d1c922]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x278) [0xba8d78]
10: (SnapMapper::add_oid(hobject_t const&,
std::set<snapid_t, std::less<snapid_t>,
std::allocator<snapid_t> > const&,
MapCacher::Transaction<std::string,
ceph::buffer::list>*)+0x61e) [0x72137e]
11: (PG::update_snap_map(std::vector<pg_log_entry_t,
std::allocator<pg_log_entry_t> > const&,
ObjectStore::Transaction&)+0x402) [0x7d25c2]
12: (PG::append_log(std::vector<pg_log_entry_t,
std::allocator<pg_log_entry_t> > const&, eversion_t,
eversion_t, ObjectStore::Transaction&, bool)+0x4e8) [0x7d2c68]
13: (ReplicatedPG::log_operation(std::vector<pg_log_entry_t,
std::allocator<pg_log_entry_t> > const&,
boost::optional<pg_hit_set_history_t>&, eversion_t
const&, eversion_t const&, bool,
ObjectStore::Transaction*)+0xba) [0x899eca]
14: (ReplicatedBackend::submit_transaction(hobject_t const&,
eversion_t const&, PGBackend::PGTransaction*, eversion_t
const&, eversion_t const&, std::vector<pg_log_entry_t,
std::allocator<pg_log_entry_t> > const&,
boost::optional<pg_hit_set_history_t>&, Context*,
Context*, Context*, unsigned long, osd_reqid_t,
std::tr1::shared_ptr<OpRequest>)+0x77c) [0x9f06cc]
15: (ReplicatedPG::issue_repop(ReplicatedPG::RepGather*)+0x7aa)
[0x8391aa]
16: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0xbdd)
[0x88792d]
17:
(ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x4559)
[0x88cee9]
18:
(ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&,
ThreadPool::TPHandle&)+0x66a) [0x82702a]
19: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
std::tr1::shared_ptr<OpRequest>,
ThreadPool::TPHandle&)+0x3bd) [0x6961dd]
20: (OSD::ShardedOpWQ::_process(unsigned int,
ceph::heartbeat_handle_d*)+0x338) [0x696708]
21: (ShardedThreadPool::shardedthreadpool_worker(unsigned
int)+0x875) [0xb98555]
22: (ShardedThreadPool::WorkThreadSharded::entry()+0x10)
[0xb9a670]
23: (()+0x8182) [0x7f8e57f6c182]
24: (clone()+0x6d) [0x7f8e564d747d]
NOTE: a copy of the executable, or `objdump -rdS
<executable>` is needed to interpret this.
--- begin dump of recent events ---
-7> 2016-03-31 03:07:56.945486 7f8e43829700 1 --
[2a00:c6c0:0:122::105]:6822/11703 <== osd.54
[2a00:c6c0:0:122::105]:6810/3568 7 ==== osd_map(122203..122203 src
has 121489..122203) v3 ==== 222+0+0 (2966331141 0 0) 0x12200d00
con 0x1209d4a0
-6> 2016-03-31 03:07:56.945514 7f8e43829700 3 osd.55
122203 handle_osd_map epochs [122203,122203], i have 122203, src
has [121489,122203]
-5> 2016-03-31 03:07:56.945517 7f8e2d6bd700 1 --
[2a00:c6c0:0:122::105]:6822/11703 <== osd.54
[2a00:c6c0:0:122::105]:6810/3568 8 ====
osd_repop_reply(client.776466.1:190178605 3.117 ondisk, result =
0) v1 ==== 83+0+0 (4008969226 0 0) 0x11e661c0 con 0x1209d4a0
-4> 2016-03-31 03:07:56.945538 7f8e2d6bd700 5 -- op
tracker -- seq: 25, time: 2016-03-31 03:07:56.945488, event:
header_read, op: osd_repop_reply(client.776466.1:190178605 3.117
ondisk, result = 0)
-3> 2016-03-31 03:07:56.945545 7f8e2d6bd700 5 -- op
tracker -- seq: 25, time: 2016-03-31 03:07:56.945489, event:
throttled, op: osd_repop_reply(client.776466.1:190178605 3.117
ondisk, result = 0)
-2> 2016-03-31 03:07:56.945549 7f8e2d6bd700 5 -- op
tracker -- seq: 25, time: 2016-03-31 03:07:56.945512, event:
all_read, op: osd_repop_reply(client.776466.1:190178605 3.117
ondisk, result = 0)
-1> 2016-03-31 03:07:56.945552 7f8e2d6bd700 5 -- op
tracker -- seq: 25, time: 0.000000, event: dispatched, op:
osd_repop_reply(client.776466.1:190178605 3.117 ondisk, result =
0)
0> 2016-03-31 03:07:56.960104 7f8e3680f700 -1 *** Caught
signal (Aborted) **
in thread 7f8e3680f700
ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
1: /usr/bin/ceph-osd() [0xaaff6a]
2: (()+0x10340) [0x7f8e57f74340]
3: (gsignal()+0x39) [0x7f8e56413cc9]
4: (abort()+0x148) [0x7f8e564170d8]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155)
[0x7f8e56d1e535]
6: (()+0x5e6d6) [0x7f8e56d1c6d6]
7: (()+0x5e703) [0x7f8e56d1c703]
8: (()+0x5e922) [0x7f8e56d1c922]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x278) [0xba8d78]
10: (SnapMapper::add_oid(hobject_t const&,
std::set<snapid_t, std::less<snapid_t>,
std::allocator<snapid_t> > const&,
MapCacher::Transaction<std::string,
ceph::buffer::list>*)+0x61e) [0x72137e]
11: (PG::update_snap_map(std::vector<pg_log_entry_t,
std::allocator<pg_log_entry_t> > const&,
ObjectStore::Transaction&)+0x402) [0x7d25c2]
12: (PG::append_log(std::vector<pg_log_entry_t,
std::allocator<pg_log_entry_t> > const&, eversion_t,
eversion_t, ObjectStore::Transaction&, bool)+0x4e8) [0x7d2c68]
13: (ReplicatedPG::log_operation(std::vector<pg_log_entry_t,
std::allocator<pg_log_entry_t> > const&,
boost::optional<pg_hit_set_history_t>&, eversion_t
const&, eversion_t const&, bool,
ObjectStore::Transaction*)+0xba) [0x899eca]
14: (ReplicatedBackend::submit_transaction(hobject_t const&,
eversion_t const&, PGBackend::PGTransaction*, eversion_t
const&, eversion_t const&, std::vector<pg_log_entry_t,
std::allocator<pg_log_entry_t> > const&,
boost::optional<pg_hit_set_history_t>&, Context*,
Context*, Context*, unsigned long, osd_reqid_t,
std::tr1::shared_ptr<OpRequest>)+0x77c) [0x9f06cc]
15: (ReplicatedPG::issue_repop(ReplicatedPG::RepGather*)+0x7aa)
[0x8391aa]
16: (ReplicatedPG::execute_ctx(ReplicatedPG::OpContext*)+0xbdd)
[0x88792d]
17:
(ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0x4559)
[0x88cee9]
18:
(ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&,
ThreadPool::TPHandle&)+0x66a) [0x82702a]
19: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
std::tr1::shared_ptr<OpRequest>,
ThreadPool::TPHandle&)+0x3bd) [0x6961dd]
20: (OSD::ShardedOpWQ::_process(unsigned int,
ceph::heartbeat_handle_d*)+0x338) [0x696708]
21: (ShardedThreadPool::shardedthreadpool_worker(unsigned
int)+0x875) [0xb98555]
22: (ShardedThreadPool::WorkThreadSharded::entry()+0x10)
[0xb9a670]
23: (()+0x8182) [0x7f8e57f6c182]
24: (clone()+0x6d) [0x7f8e564d747d]
NOTE: a copy of the executable, or `objdump -rdS
<executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.55.log
--- end dump of recent events ---
On 03/30/2016 11:36 PM, Mart van
Santen wrote:
Hi there,
With the help of a lot of people we were able to repair the PG and
restored service. We will get back on this later with a full report for
future reference.
Regards,
Mart
On 03/30/2016 08:30 PM, Wido den Hollander wrote:
Hi,
I have an issue with a Ceph cluster which I can't resolve.
Due to OSD failure a PG is incomplete, but I can't query the PG to see what I
can do to fix it.
health HEALTH_WARN
1 pgs incomplete
1 pgs stuck inactive
1 pgs stuck unclean
98 requests are blocked > 32 sec
$ ceph pg 3.117 query
That will hang for ever.
$ ceph pg dump_stuck
pg_stat state up up_primary acting acting_primary
3.117 incomplete [68,55,74] 68 [68,55,74] 68
The primary PG in this case is osd.68 . If I stop the OSD the PG query works,
but it says that bringing osd 68 back online will probably help.
The 98 requests which are blocked are also on osd.68 and they all say:
They all say:
- initiated
- reached_pg
The cluster is running Hammer 0.94.5 in this case.
>From what I know a OSD had a failing disk and was restarted a couple of times
while the disk gave errors. This caused the PG to become incomplete.
I've set debug osd to 20, but I can't really tell what is going wrong on osd.68
which causes it to stall this long.
Any idea what to do here to get this PG up and running again?
Wido
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Goncalo Borges
Research Computing
ARC Centre of Excellence for Particle Physics at the Terascale
School of Physics A28 | University of Sydney, NSW 2006
T: +61 2 93511937
|
