Hi,
One OSD in the cluster is down. Tried to restart the service, but its still failing.
I can see the below error in log file. Can this be a hardware issue ?
-------------------------------------
-9> 2017-11-23 09:47:37.768969 7f368686a700 3 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/rocksdb/db/db_impl_compaction_flush.cc:1591] Compaction error: Corruption: block checksum mismatch
-8> 2017-11-23 09:47:37.768980 7f368686a700 4 rocksdb: (Original Log Time 2017/11/23-09:47:37.768936) [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[11 1 0 0 0 0 0] max score 0.00, MB/sec: 2.3 rd, 2.0 wr, level 1, files in(11, 1) out(1) MB in(0.1, 7.8) out(7.0), read-write-amplify(202.0) write-amplify(94.6) Corruption: block checksum mismatch, records in: 42
-7> 2017-11-23 09:47:37.768984 7f368686a700 4 rocksdb: (Original Log Time 2017/11/23-09:47:37.768963) EVENT_LOG_v1 {"time_micros": 1511459257768950, "job": 3, "event": "compaction_finished", "compaction_time_micros": 3667366, "output_level": 1, "num_output_files": 1, "total_output_size": 7317366, "num_input_records": 38738, "num_output_records": 37539, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [11, 1, 0, 0, 0, 0, 0]}
-6> 2017-11-23 09:47:37.768988 7f368686a700 2 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/rocksdb/db/db_impl_compaction_flush.cc:1275] Waiting after background compaction error: Corruption: block checksum mismatch, Accumulated background error counts: 1
-5> 2017-11-23 09:47:38.245022 7f369a708d00 5 osd.6 pg_epoch: 324 pg[3.98s5(unlocked)] enter Initial
-4> 2017-11-23 09:47:38.245256 7f369a708d00 5 osd.6 pg_epoch: 324 pg[3.98s5( empty local-lis/les=323/324 n=0 ec=69/69 lis/c 323/323 les/c/f 324/324/0 323/323/69) [2,11,7,1,0,6,9,3] r=5 lpr=0 crt=0'0 unknown NOTIFY] exit Initial 0.000235 0 0.000000
-3> 2017-11-23 09:47:38.245275 7f369a708d00 5 osd.6 pg_epoch: 324 pg[3.98s5( empty local-lis/les=323/324 n=0 ec=69/69 lis/c 323/323 les/c/f 324/324/0 323/323/69) [2,11,7,1,0,6,9,3] r=5 lpr=0 crt=0'0 unknown NOTIFY] enter Reset
-2> 2017-11-23 09:47:38.245288 7f369a708d00 5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , trimmed_dups: , clear_divergent_priors: 0
-1> 2017-11-23 09:47:38.245355 7f368806d700 -1 rocksdb: submit_transaction error: Corruption: block checksum mismatch code = 2 Rocksdb transaction:
Put( Prefix = M key = 0x000000000000052c'.can_rollback_to' Value size = 12)
Put( Prefix = M key = 0x000000000000052c'.rollback_info_trimmed_to' Value size = 12)
Put( Prefix = O key = 0x8580000000000000031900000021213dfffffffffffffffeffffffffffffffff'o' Value size = 29)
0> 2017-11-23 09:47:38.247357 7f368806d700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_kv_sync_thread()' thread 7f368806d700 time 2017-11-23 09:47:38.245386
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.1/rpm/el7/BUILD/ceph-12.2.1/src/os/bluestore/BlueStore.cc: 8453: FAILED assert(r == 0)
Karun
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
