The only strange thing is that ceph-bluestore-tool says that repair was done, no errors are found and all is ok.
I ask myself what really does that tool.
Mario
Il giorno gio 29 nov 2018 alle ore 11:03 Wido den Hollander <wido@xxxxxxxx> ha scritto:
On 11/29/18 10:45 AM, Mario Giammarco wrote:
> I have only that copy, it is a showroom system but someone put a
> production vm on it.
>
I have a feeling this won't be easy to fix or actually fixable:
- Compaction error: Corruption: block checksum mismatch
- submit_transaction error: Corruption: block checksum mismatch
RocksDB got corrupted on that OSD and won't be able to start now.
I wouldn't know where to start with this OSD.
Wido
> Il giorno gio 29 nov 2018 alle ore 10:43 Wido den Hollander
> <wido@xxxxxxxx <mailto:wido@xxxxxxxx>> ha scritto:
>
>
>
> On 11/29/18 10:28 AM, Mario Giammarco wrote:
> > Hello,
> > I have a ceph installation in a proxmox cluster.
> > Due to a temporary hardware glitch now I get this error on osd startup
> >
> > -6> 2018-11-26 18:02:33.179327 7fa1d784be00 0 osd.0 1033
> crush map
> > has features 1009089991638532096, adjusting msgr requires for
> osds
> > -5> 2018-11-26 18:02:34.143084 7fa1c33f9700 3 rocksdb:
> >
> [/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1591]
> > Compaction error: Corruption: block checksum mismatch
> > -4> 2018-11-26 18:02:34.143123 7fa1c33f9700 4 rocksdb:
> (Original Log
> > Time 2018/11/26-18:02:34.143021)
> > [/build/ceph-12.2.9/src/rocksdb/db/compaction_job.cc:621]
> [default]
> > compacted to: base level 1 max bytes base268435456 files[17$
> >
> > -3> 2018-11-26 18:02:34.143126 7fa1c33f9700 4 rocksdb:
> (Original Log
> > Time 2018/11/26-18:02:34.143068) EVENT_LOG_v1 {"time_micros":
> > 1543251754143044, "job": 3, "event": "compaction_finished",
> > "compaction_time_micros": 1997048, "out$
> > -2> 2018-11-26 18:02:34.143152 7fa1c33f9700 2 rocksdb:
> >
> [/build/ceph-12.2.9/src/rocksdb/db/db_impl_compaction_flush.cc:1275]
> > Waiting after background compaction error: Corruption: block
> > checksum mismatch, Accumulated background err$
> > -1> 2018-11-26 18:02:34.674171 7fa1c4bfc700 -1 rocksdb:
> > submit_transaction error: Corruption: block checksum mismatch
> code =
> > 2 Rocksdb transaction:
> > Delete( Prefix = O key =
> >
> 0x7f7ffffffffffffffb64000000217363'rub_3.26!='0xfffffffffffffffeffffffffffffffff'o')
> > Put( Prefix = S key = 'nid_max' Value size = 8)
> > Put( Prefix = S key = 'blobid_max' Value size = 8)
> > 0> 2018-11-26 18:02:34.675641 7fa1c4bfc700 -1
> > /build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: In function
> 'void
> > BlueStore::_kv_sync_thread()' thread 7fa1c4bfc700 time 2018-11-26
> > 18:02:34.674193
> > /build/ceph-12.2.9/src/os/bluestore/BlueStore.cc: 8717: FAILED
> > assert(r == 0)
> >
> > ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217)
> > luminous (stable)
> > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> > const*)+0x102) [0x55ec83876092]
> > 2: (BlueStore::_kv_sync_thread()+0x24b5) [0x55ec836ffb55]
> > 3: (BlueStore::KVSyncThread::entry()+0xd) [0x55ec8374040d]
> > 4: (()+0x7494) [0x7fa1d5027494]
> > 5: (clone()+0x3f) [0x7fa1d4098acf]
> >
> >
> > I have tried to recover it using ceph-bluestore-tool fsck and repair
> > DEEP but it says it is ALL ok.
> > I see that rocksd ldb tool needs .db files to recover and not a
> > partition so I cannot use it.
> > I do not understand why I cannot start osd if ceph-bluestore-tools
> says
> > me I have lost no data.
> > Can you help me?
>
> Why would you try to recover a individual OSD? If all your Placement
> Groups are active(+clean) just wipe the OSD and re-deploy it.
>
> What's the status of your PGs?
>
> It says there is a checksum error (probably due to the hardware glitch)
> so it refuses to start.
>
> Don't try to outsmart Ceph, let backfill/recovery handle this. Trying to
> manually fix this will only make things worse.
>
> Wido
>
> > Thanks,
> > Mario
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
