Hello,
We are getting below trace on failed OSD's . Can you please explain from the below code why this issue happening. We suspect it could be because of underlying HW issue. We can't find anything from the syslogs. All the OSD disk are in healthy condition.
2017-03-31 03:51:28.262403 7f71f3ed5940 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/11.2.0/rpm/el7/BUILD/ceph-11.2.0/src/os/bluestore/KernelDevice.cc: In function 'virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IOContext*, bool)' thread 7f71f3ed5940 time 2017-03-31 03:51:28.256298
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/11.2.0/rpm/el7/BUILD/ceph-11.2.0/src/os/bluestore/KernelDevice.cc: 541: FAILED assert((uint64_t)r == len) =============>>>>>>>>>>>>>>>>>>>>>
ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f71f49b6b35]
2: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::list*, IOContext*, bool)+0x7a0) [0x7f71f47fdf10]
3: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x47a) [0x7f71f47c8d4a]
4: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x36) [0x7f71f47ecd46]
5: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x16) [0x7f71f48f4136]
6: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xd8) [0x7f71f487d658]
7: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x44) [0x7f71f487d734]
8: (rocksdb::log::Reader::ReadRecord(rocksdb::Slice*, std::string*, rocksdb::WALRecoveryMode)+0xe8) [0x7f71f487d9e8]
9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0xdc8) [0x7f71f4837a18]
10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0x866) [0x7f71f4839386]
11: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/11.2.0/rpm/el7/BUILD/ceph-11.2.0/src/os/bluestore/KernelDevice.cc: 541: FAILED assert((uint64_t)r == len) =============>>>>>>>>>>>>>>>>>>>>>
ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f71f49b6b35]
2: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::list*, IOContext*, bool)+0x7a0) [0x7f71f47fdf10]
3: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x47a) [0x7f71f47c8d4a]
4: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x36) [0x7f71f47ecd46]
5: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x16) [0x7f71f48f4136]
6: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xd8) [0x7f71f487d658]
7: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x44) [0x7f71f487d734]
8: (rocksdb::log::Reader::ReadRecord(rocksdb::Slice*, std::string*, rocksdb::WALRecoveryMode)+0xe8) [0x7f71f487d9e8]
9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0xdc8) [0x7f71f4837a18]
10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0x866) [0x7f71f4839386]
11: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std
===
Thanks
Jayaram
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
