Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
_______________________________________________Hello community,
we have developed a cluster on latest mimic release. We are on quite old hardware, but using Centos7. Monitor, manager and all the same host. Cluster has been running for some week without actual workload. There might have been some sort of power failure (not proved), but at some point monitor node died and won't start anymore. Below is a log from /var/log/messages. What can be done here? Can this be recovered somehow or did we loose everything? All the OSDs seems to be running fine, just that the cluster is not working.
The log is not full, but I think that those line are quite critical..
Jun 27 17:14:06 mds1 ceph-mon: -311> 2019-06-27 17:14:06.169 7f086aa22700 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: expected 3317957558, got 2609532897 in /var/lib/ceph/mon/ceph-mds1/store.db/022334.sst offset 12775887 size 21652 code = 2 Rocksdb transaction:
Jun 27 17:14:06 mds1 ceph-mon: Put( Prefix = p key = 'xos'0x006c6173't_committed' Value size = 8)
Jun 27 17:14:06 mds1 ceph-mon: Put( Prefix = m key = 'nitor_store'0x006c6173't_metadata' Value size = 612)
Jun 27 17:14:06 mds1 ceph-mon: Put( Prefix = l key = 'gm'0x0066756c'l_155850' Value size = 31307)
Jun 27 17:14:06 mds1 ceph-mon: Put( Prefix = l key = 'gm'0x0066756c'l_latest' Value size = 8)
Jun 27 17:14:06 mds1 ceph-mon: Put( Prefix = l key = 'gm'0x00313535'851' Value size = 672)
Jun 27 17:14:06 mds1 ceph-mon: Put( Prefix = l key = 'gm'0x006c6173't_committed' Value size = 8)
Jun 27 17:14:06 mds1 ceph-mon: -311> 2019-06-27 17:14:06.172 7f086aa22700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE
_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/mon/MonitorDBStore.h: In function
'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7f086aa22700 time 2019-06-27 17:14:06.171474
Jun 27 17:14:06 mds1 ceph-mon: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/cento
s7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/mon/MonitorDBStore.h: 311: FAILED assert(0 == "failed to write to db")
Jun 27 17:14:06 mds1 ceph-mon: ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
