Hi everyone,
I am facing an OSD in a crash loop and following the troubleshooting
procedure leads me to contacting the mailing list. (As far as I can
tell, this is neither a hardware nor a configuration issue. The system
has been running in this setup for months. Several other OSDs running on
the same host are fine.) Also I have one PG that is currently marked
inconsistent. This PG is not is not mentioned in the log file, though.
The OSD in question crashes about 15 to 20 seconds after starting up.
The logged reasons for the crash are, as far as I can tell:
-1> 2019-02-01 12:22:46.111821 7fe079d53700 -1 rocksdb:
submit_transaction error: Corruption: block checksum mismatch code = 2
Rocksdb transaction:
Put( Prefix = O key =
0x7f80000000000000021600000021213dfffffffffffffffeffffffffffffffff'o'
Value size = 30)
0> 2019-02-01 12:22:46.117761 7fe079d53700 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.9/rpm/el7/BUILD/ceph-12.2.9/src/os/bluestore/BlueStore.cc:
In function 'void BlueStore::_kv_sync_thread()' thread 7fe079d53700 time
2019-02-01 12:22:46.111884
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.9/rpm/el7/BUILD/ceph-12.2.9/src/os/bluestore/BlueStore.cc:
8717: FAILED assert(r == 0)
ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217) luminous
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x110) [0x562af51e5e90]
2: (BlueStore::_kv_sync_thread()+0x3482) [0x562af5090162]
3: (BlueStore::KVSyncThread::entry()+0xd) [0x562af50d701d]
4: (()+0x7e25) [0x7fe089e12e25]
5: (clone()+0x6d) [0x7fe088f03bad]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
I have a full log of the crash cycle available, if it helps.
Is there anything I can do to fix this? Is this a bug that I should
report somewhere else?
David Sieger
--
iTerra GmbH
Böhmestraße 18 Geschäftsführung: Dr. Peter Brodersen
25899 Niebüll Prokurist: Christian Feddersen
Tel. 04661 18540-40 www.iterra-gmbh.de
Rechtsform: Gesellschaft mit begrenzter Haftung, Sitz der Gesellschaft:
Niebüll, Amtsgericht Flensburg HRB 1225 NI
