Hi *,
after some crashes we still had to care for some remaining
inconsistancies reported via
ceph -w
and friends.
Well, we traced one of them down via
ceph pg dump
and we picked 79. pg=79.7 and found the corresponding file in the
/var/log/ceph/osd.2.log.
/data/osd4/current/79.7_head/rb.0.0.00000000136c__head_9FB2FA17
and the dup on
/data/osd2/...
Strange though, they had the same checksum but reported a stat-error.
Anyway. Decided to do a:
ceph pg repair 79.7
... byebye ceph-osd on node2!
Here the trace:
=== 8-< ===
2012-03-01 17:49:13.024571 7f3944584700 -- 10.10.10.14:6802/4892 >>
10.10.10.10:6802/19139 pipe(0xfcd2c80 sd=16 pgs=0 cs=0 l=0).connect
protocol version mismatch, my 9 != 0
2012-03-01 17:49:23.674162 7f395001b700 log [ERR] : 79.7 osd.4: soid
9fb2fa17/rb.0.0.00000000136c/headextra attr _, extra attr snapset
2012-03-01 17:49:23.674222 7f395001b700 log [ERR] : 79.7 repair 0
missing, 1 inconsistent objects
*** Caught signal (Aborted) **
in thread 7f395001b700
ceph version 0.42-142-gc9416e6
(commit:c9416e6184905501159e96115f734bdf65a74d28)
1: /usr/bin/ceph-osd() [0x5a6b89]
2: (()+0xeff0) [0x7f3960ca5ff0]
3: (gsignal()+0x35) [0x7f395f2841b5]
4: (abort()+0x180) [0x7f395f286fc0]
5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7f395fb18dc5]
6: (()+0xcb166) [0x7f395fb17166]
7: (()+0xcb193) [0x7f395fb17193]
8: (()+0xcb28e) [0x7f395fb1728e]
9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x13e)
[0x67c5ce]
10: (object_info_t::decode(ceph::buffer::list::iterator&)+0x2c) [0x61663c]
11: (PG::repair_object(hobject_t const&, ScrubMap::object*, int,
int)+0x3be) [0x68d96e]
12: (PG::scrub_finalize()+0x1438) [0x6b8568]
13: (OSD::ScrubFinalizeWQ::_process(PG*)+0xc) [0x588edc]
14: (ThreadPool::worker()+0xa26) [0x5bc426]
15: (ThreadPool::WorkThread::entry()+0xd) [0x585f0d]
16: (()+0x68ca) [0x7f3960c9d8ca]
17: (clone()+0x6d) [0x7f395f32186d]
2012-03-01 17:49:30.017269 7f81b662b780 ceph version 0.42-142-gc9416e6
(commit:c9416e6184905501159e96115f734bdf65a74d28), process ceph-osd, pid
3111
2012-03-01 17:49:30.085426 7f81b662b780 filestore(/data/osd2) mount
FIEMAP ioctl is NOT supported
2012-03-01 17:49:30.085466 7f81b662b780 filestore(/data/osd2) mount did
NOT detect btrfs
2012-03-01 17:49:30.110409 7f81b662b780 filestore(/data/osd2) mount
found snaps <>
2012-03-01 17:49:30.110476 7f81b662b780 filestore(/data/osd2) mount:
enabling WRITEAHEAD journal mode: btrfs not detected
2012-03-01 17:49:31.964977 7f81b662b780 journal _open /dev/sdc1 fd 16:
10737942528 bytes, block size 4096 bytes, directio = 1, aio = 0
2012-03-01 17:49:31.967549 7f81b662b780 journal read_entry 9292222464 :
seq 67841857 11225 bytes
=== 8-< ===
... after some journal-replay things calmed down, but:
2012-03-01 17:58:29.470446 log 2012-03-01 17:58:24.242369 osd.2
10.10.10.14:6801/3111 368 : [WRN] bad locator @56 on object @79 loc @56
op osd_op(client.44350.0:1412387 rb.0.0.00000000136c [write
2465792~49152] 56.9fb2fa17) v4
these type of messages we see ever so often... It corresponds, but in
what way?
Can't we assume, if both snipplets "rb.0.0..." are identical, that
life's good?
We had some other inconsistancies, where we had to delete the whole pool
to get rid of crappy
blocks. The ceph-osd died, too, after doing some
rbd rm <pool>/<image>
the one block in question remained, visable via
rados ls -p <pool>
Any idea, o better clue? ;-)
Kind reg's,
Oliver.
--
Oliver Francke
filoo GmbH
Moltkestraße 25a
33330 Gütersloh
HRB4355 AG Gütersloh
Geschäftsführer: S.Grewing | J.Rehpöhler | C.Kunz
Folgen Sie uns auf Twitter: http://twitter.com/filoogmbh
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
