On 11/29/18 10:45 AM, Mario Giammarco wrote: > I have only that copy, it is a showroom system but someone put a > production vm on it. > I have a feeling this won't be easy to fix or actually fixable: - Compaction error: Corruption: block checksum mismatch - submit_transaction error: Corruption: block checksum mismatch RocksDB got corrupted on that OSD and won't be able to start now. I wouldn't know where to start with this OSD. Wido > Il giorno gio 29 nov 2018 alle ore 10:43 Wido den Hollander > <wido@xxxxxxxx <mailto:wido@xxxxxxxx>> ha scritto: > > > > On 11/29/18 10:28 AM, Mario Giammarco wrote: > > Hello, > > I have a ceph installation in a proxmox cluster. > > Due to a temporary hardware glitch now I get this error on osd startup > > > > -6> 2018-11-26 18:02:33.179327 7fa1d784be00 0 osd.0 1033 > crush map > > has features 1009089991638532096, adjusting msgr requires for > osds > > -5> 2018-11-26 18:02:34.143084 7fa1c33f9700 3 rocksdb: > > > [/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1591] > > Compaction error: Corruption: block checksum mismatch > > -4> 2018-11-26 18:02:34.143123 7fa1c33f9700 4 rocksdb: > (Original Log > > Time 2018/11/26-18:02:34.143021) > > [/build/ceph-12.2.9/src/rocksdb/db/compaction_job.cc:621] > [default] > > compacted to: base level 1 max bytes base268435456 files[17$ > > > > -3> 2018-11-26 18:02:34.143126 7fa1c33f9700 4 rocksdb: > (Original Log > > Time 2018/11/26-18:02:34.143068) EVENT_LOG_v1 {"time_micros": > > 1543251754143044, "job": 3, "event": "compaction_finished", > > "compaction_time_micros": 1997048, "out$ > > -2> 2018-11-26 18:02:34.143152 7fa1c33f9700 2 rocksdb: > > > [/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1275] > > Waiting after background compaction error: Corruption: block > > checksum mismatch, Accumulated background err$ > > -1> 2018-11-26 18:02:34.674171 7fa1c4bfc700 -1 rocksdb: > > submit_transaction error: Corruption: block checksum mismatch > code = > > 2 Rocksdb transaction: > > Delete( Prefix = O key = > > > 0x7f7ffffffffffffffb64000000217363'rub_3.26!='0xfffffffffffffffeffffffffffffffff'o') > > Put( Prefix = S key = 'nid_max' Value size = 8) > > Put( Prefix = S key = 'blobid_max' Value size = 8) > > 0> 2018-11-26 18:02:34.675641 7fa1c4bfc700 -1 > > /build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: In function > 'void > > BlueStore::_kv_sync_thread()' thread 7fa1c4bfc700 time 2018-11-26 > > 18:02:34.674193 > > /build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: 8717: FAILED > > assert(r == 0) > > > > ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217) > > luminous (stable) > > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > > const*)+0x102) [0x55ec83876092] > > 2: (BlueStore::_kv_sync_thread()+0x24b5) [0x55ec836ffb55] > > 3: (BlueStore::KVSyncThread::entry()+0xd) [0x55ec8374040d] > > 4: (()+0x7494) [0x7fa1d5027494] > > 5: (clone()+0x3f) [0x7fa1d4098acf] > > > > > > I have tried to recover it using ceph-bluestore-tool fsck and repair > > DEEP but it says it is ALL ok. > > I see that rocksd ldb tool needs .db files to recover and not a > > partition so I cannot use it. > > I do not understand why I cannot start osd if ceph-bluestore-tools > says > > me I have lost no data. > > Can you help me? > > Why would you try to recover a individual OSD? If all your Placement > Groups are active(+clean) just wipe the OSD and re-deploy it. > > What's the status of your PGs? > > It says there is a checksum error (probably due to the hardware glitch) > so it refuses to start. > > Don't try to outsmart Ceph, let backfill/recovery handle this. Trying to > manually fix this will only make things worse. > > Wido > > > Thanks, > > Mario > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |