Title: Ceph OSD fails to startup with bluefs error
Content:
The crash has happened for three times with the same reason:
direct_read_unaligned ...... error(5) Input/Output err.
while I use ceph-bluestore-tool repair/fsck, it reports:
# ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-1/ --log-level 30
2018-08-17 10:24:42.058156 7f2fa353ed00 -1 bdev(0x55a34bd06600 /var/lib/ceph/osd/ceph-1//block) direct_read_unaligned 0x1c68d500000~43bc error: (5) Input/output error
/build/ceph-12.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read_random(BlueFS::FileReader*, uint64_t, size_t, char*)' thread 7f2fa353ed00 time 2018-08-17 10:24:42.058383
/build/ceph-12.2.6/src/os/bluestore/BlueFS.cc: 916: FAILED assert(r == 0)
ceph version 12.2.6 (488df8a1076c4f5fc5b8d18a90463262c438740f) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x7f2f9a07f76e]
2: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*)+0x708) [0x55a34a1ce248]
3: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x20) [0x55a34a384da0]
4: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x30d) [0x55a34a48d87d]
5: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*, rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockContents*, rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&, rocksdb::PersistentCacheOptions const&)+0x2ae) [0x55a34a4674ee]
6: (()+0x455aa8) [0x55a34a457aa8]
7: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x352) [0x55a34a459f52]
8: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x129) [0x55a34a45a289]
9: (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice const&)+0x89) [0x55a34a463089]
10: (()+0x486366) [0x55a34a488366]
11: (()+0x4869e6) [0x55a34a4889e6]
12: (()+0x4869f8) [0x55a34a4889f8]
13: (rocksdb::MergingIterator::Seek(rocksdb::Slice const&)+0xd7) [0x55a34a46f227]
14: (rocksdb::DBIter::Seek(rocksdb::Slice const&)+0x181) [0x55a34a4fdbd1]
15: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::lower_bound(std::string const&, std::string const&)+0x8e) [0x55a34a2a7b9e]
16: (BitmapFreelistManager::init(unsigned long)+0x1a8) [0x55a34a37df48]
17: (BlueStore::_open_fm(bool)+0xafa) [0x55a34a20877a]
18: (BlueStore::_fsck(bool, bool)+0x3f4) [0x55a34a26e584]
19: (main()+0x1582) [0x55a34a12e462]
20: (__libc_start_main()+0xf5) [0x7f2f988b9f45]
21: (()+0x1c05ef) [0x55a34a1c25ef]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2018-08-17 10:24:42.064170 7f2fa353ed00 -1 /build/ceph-12.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read_random(BlueFS::FileReader*, uint64_t, size_t, char*)' thread 7f2fa353ed00 time 2018-08-17 10:24:42.058383
/build/ceph-12.2.6/src/os/bluestore/BlueFS.cc: 916: FAILED assert(r == 0)
ceph version 12.2.6 (488df8a1076c4f5fc5b8d18a90463262c438740f) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x7f2f9a07f76e]
2: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*)+0x708) [0x55a34a1ce248]
3: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x20) [0x55a34a384da0]
4: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x30d) [0x55a34a48d87d]
5: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*, rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockContents*, rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&, rocksdb::PersistentCacheOptions const&)+0x2ae) [0x55a34a4674ee]
6: (()+0x455aa8) [0x55a34a457aa8]
7: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x352) [0x55a34a459f52]
8: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x129) [0x55a34a45a289]
9: (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice const&)+0x89) [0x55a34a463089]
10: (()+0x486366) [0x55a34a488366]
11: (()+0x4869e6) [0x55a34a4889e6]
12: (()+0x4869f8) [0x55a34a4889f8]
13: (rocksdb::MergingIterator::Seek(rocksdb::Slice const&)+0xd7) [0x55a34a46f227]
14: (rocksdb::DBIter::Seek(rocksdb::Slice const&)+0x181) [0x55a34a4fdbd1]
15: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::lower_bound(std::string const&, std::string const&)+0x8e) [0x55a34a2a7b9e]
16: (BitmapFreelistManager::init(unsigned long)+0x1a8) [0x55a34a37df48]
17: (BlueStore::_open_fm(bool)+0xafa) [0x55a34a20877a]
18: (BlueStore::_fsck(bool, bool)+0x3f4) [0x55a34a26e584]
19: (main()+0x1582) [0x55a34a12e462]
20: (__libc_start_main()+0xf5) [0x7f2f988b9f45]
21: (()+0x1c05ef) [0x55a34a1c25ef]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-1> 2018-08-17 10:24:42.058156 7f2fa353ed00 -1 bdev(0x55a34bd06600 /var/lib/ceph/osd/ceph-1//block) direct_read_unaligned 0x1c68d500000~43bc error: (5) Input/output error
0> 2018-08-17 10:24:42.064170 7f2fa353ed00 -1 /build/ceph-12.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read_random(BlueFS::FileReader*, uint64_t, size_t, char*)' thread 7f2fa353ed00 time 2018-08-17 10:24:42.058383
/build/ceph-12.2.6/src/os/bluestore/BlueFS.cc: 916: FAILED assert(r == 0)
ceph version 12.2.6 (488df8a1076c4f5fc5b8d18a90463262c438740f) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x7f2f9a07f76e]
2: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*)+0x708) [0x55a34a1ce248]
3: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x20) [0x55a34a384da0]
4: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x30d) [0x55a34a48d87d]
5: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*, rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockContents*, rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&, rocksdb::PersistentCacheOptions const&)+0x2ae) [0x55a34a4674ee]
6: (()+0x455aa8) [0x55a34a457aa8]
7: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x352) [0x55a34a459f52]
8: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x129) [0x55a34a45a289]
9: (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice const&)+0x89) [0x55a34a463089]
10: (()+0x486366) [0x55a34a488366]
11: (()+0x4869e6) [0x55a34a4889e6]
12: (()+0x4869f8) [0x55a34a4889f8]
13: (rocksdb::MergingIterator::Seek(rocksdb::Slice const&)+0xd7) [0x55a34a46f227]
14: (rocksdb::DBIter::Seek(rocksdb::Slice const&)+0x181) [0x55a34a4fdbd1]
15: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::lower_bound(std::string const&, std::string const&)+0x8e) [0x55a34a2a7b9e]
16: (BitmapFreelistManager::init(unsigned long)+0x1a8) [0x55a34a37df48]
17: (BlueStore::_open_fm(bool)+0xafa) [0x55a34a20877a]
18: (BlueStore::_fsck(bool, bool)+0x3f4) [0x55a34a26e584]
19: (main()+0x1582) [0x55a34a12e462]
20: (__libc_start_main()+0xf5) [0x7f2f988b9f45]
21: (()+0x1c05ef) [0x55a34a1c25ef]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
*** Caught signal (Aborted) **
in thread 7f2fa353ed00 thread_name:ceph-bluestore-
ceph version 12.2.6 (488df8a1076c4f5fc5b8d18a90463262c438740f) luminous (stable)
1: (()+0x3e15b9) [0x55a34a3e35b9]
2: (()+0x10330) [0x7f2f9989d330]
3: (gsignal()+0x37) [0x7f2f988cec37]
4: (abort()+0x148) [0x7f2f988d2028]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x7f2f9a07f8e0]
6: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*)+0x708) [0x55a34a1ce248]
7: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x20) [0x55a34a384da0]
8: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x30d) [0x55a34a48d87d]
9: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*, rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockContents*, rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&, rocksdb::PersistentCacheOptions const&)+0x2ae) [0x55a34a4674ee]
10: (()+0x455aa8) [0x55a34a457aa8]
11: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x352) [0x55a34a459f52]
12: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x129) [0x55a34a45a289]
13: (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice const&)+0x89) [0x55a34a463089]
14: (()+0x486366) [0x55a34a488366]
15: (()+0x4869e6) [0x55a34a4889e6]
16: (()+0x4869f8) [0x55a34a4889f8]
17: (rocksdb::MergingIterator::Seek(rocksdb::Slice const&)+0xd7) [0x55a34a46f227]
18: (rocksdb::DBIter::Seek(rocksdb::Slice const&)+0x181) [0x55a34a4fdbd1]
19: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::lower_bound(std::string const&, std::string const&)+0x8e) [0x55a34a2a7b9e]
20: (BitmapFreelistManager::init(unsigned long)+0x1a8) [0x55a34a37df48]
21: (BlueStore::_open_fm(bool)+0xafa) [0x55a34a20877a]
22: (BlueStore::_fsck(bool, bool)+0x3f4) [0x55a34a26e584]
23: (main()+0x1582) [0x55a34a12e462]
24: (__libc_start_main()+0xf5) [0x7f2f988b9f45]
25: (()+0x1c05ef) [0x55a34a1c25ef]
2018-08-17 10:24:42.069899 7f2fa353ed00 -1 *** Caught signal (Aborted) **
in thread 7f2fa353ed00 thread_name:ceph-bluestore-
ceph version 12.2.6 (488df8a1076c4f5fc5b8d18a90463262c438740f) luminous (stable)
1: (()+0x3e15b9) [0x55a34a3e35b9]
2: (()+0x10330) [0x7f2f9989d330]
3: (gsignal()+0x37) [0x7f2f988cec37]
4: (abort()+0x148) [0x7f2f988d2028]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x7f2f9a07f8e0]
6: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*)+0x708) [0x55a34a1ce248]
7: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x20) [0x55a34a384da0]
8: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x30d) [0x55a34a48d87d]
9: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*, rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockContents*, rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&, rocksdb::PersistentCacheOptions const&)+0x2ae) [0x55a34a4674ee]
10: (()+0x455aa8) [0x55a34a457aa8]
11: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x352) [0x55a34a459f52]
12: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x129) [0x55a34a45a289]
13: (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice const&)+0x89) [0x55a34a463089]
14: (()+0x486366) [0x55a34a488366]
15: (()+0x4869e6) [0x55a34a4889e6]
16: (()+0x4869f8) [0x55a34a4889f8]
17: (rocksdb::MergingIterator::Seek(rocksdb::Slice const&)+0xd7) [0x55a34a46f227]
18: (rocksdb::DBIter::Seek(rocksdb::Slice const&)+0x181) [0x55a34a4fdbd1]
19: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::lower_bound(std::string const&, std::string const&)+0x8e) [0x55a34a2a7b9e]
20: (BitmapFreelistManager::init(unsigned long)+0x1a8) [0x55a34a37df48]
21: (BlueStore::_open_fm(bool)+0xafa) [0x55a34a20877a]
22: (BlueStore::_fsck(bool, bool)+0x3f4) [0x55a34a26e584]
23: (main()+0x1582) [0x55a34a12e462]
24: (__libc_start_main()+0xf5) [0x7f2f988b9f45]
25: (()+0x1c05ef) [0x55a34a1c25ef]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
0> 2018-08-17 10:24:42.069899 7f2fa353ed00 -1 *** Caught signal (Aborted) **
in thread 7f2fa353ed00 thread_name:ceph-bluestore-
ceph version 12.2.6 (488df8a1076c4f5fc5b8d18a90463262c438740f) luminous (stable)
1: (()+0x3e15b9) [0x55a34a3e35b9]
2: (()+0x10330) [0x7f2f9989d330]
3: (gsignal()+0x37) [0x7f2f988cec37]
4: (abort()+0x148) [0x7f2f988d2028]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x7f2f9a07f8e0]
6: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*)+0x708) [0x55a34a1ce248]
7: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x20) [0x55a34a384da0]
8: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x30d) [0x55a34a48d87d]
9: (rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*, rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockContents*, rocksdb::ImmutableCFOptions const&, bool, rocksdb::Slice const&, rocksdb::PersistentCacheOptions const&)+0x2ae) [0x55a34a4674ee]
10: (()+0x455aa8) [0x55a34a457aa8]
11: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x352) [0x55a34a459f52]
12: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x129) [0x55a34a45a289]
13: (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice const&)+0x89) [0x55a34a463089]
14: (()+0x486366) [0x55a34a488366]
15: (()+0x4869e6) [0x55a34a4889e6]
16: (()+0x4869f8) [0x55a34a4889f8]
17: (rocksdb::MergingIterator::Seek(rocksdb::Slice const&)+0xd7) [0x55a34a46f227]
18: (rocksdb::DBIter::Seek(rocksdb::Slice const&)+0x181) [0x55a34a4fdbd1]
19: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::lower_bound(std::string const&, std::string const&)+0x8e) [0x55a34a2a7b9e]
20: (BitmapFreelistManager::init(unsigned long)+0x1a8) [0x55a34a37df48]
21: (BlueStore::_open_fm(bool)+0xafa) [0x55a34a20877a]
22: (BlueStore::_fsck(bool, bool)+0x3f4) [0x55a34a26e584]
23: (main()+0x1582) [0x55a34a12e462]
24: (__libc_start_main()+0xf5) [0x7f2f988b9f45]
25: (()+0x1c05ef) [0x55a34a1c25ef]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
while I use ceph-bluestore-tool, It reports error after exporting a normal file:
ceph-bluestore-tool bluefs-export --path /var/lib/ceph/osd/ceph-1/ --out /tmp/osd.1
infering bluefs devices from bluestore path
slot 1 /var/lib/ceph/osd/ceph-1//block
db/
db/005827.sst
db/007651.sst
.......
.......
db/020143.sst
/build/ceph-12.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, uint64_t, size_t, ceph::bufferlist*, char*)' thread 7fa25ed5fd00 time 2018-08-17 10:32:00.971631
/build/ceph-12.2.6/src/os/bluestore/BlueFS.cc: 976: FAILED assert(r == 0)
ceph version 12.2.6 (488df8a1076c4f5fc5b8d18a90463262c438740f) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x7fa2558a076e]
2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xca6) [0x55b1f9ac22f6]
3: (main()+0x23a4) [0x55b1f9a1f284]
4: (__libc_start_main()+0xf5) [0x7fa2540daf45]
5: (()+0x1c05ef) [0x55b1f9ab25ef]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2018-08-17 10:32:00.972354 7fa25ed5fd00 -1 /build/ceph-12.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, uint64_t, size_t, ceph::bufferlist*, char*)' thread 7fa25ed5fd00 time 2018-08-17 10:32:00.971631
/build/ceph-12.2.6/src/os/bluestore/BlueFS.cc: 976: FAILED assert(r == 0)
ceph version 12.2.6 (488df8a1076c4f5fc5b8d18a90463262c438740f) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x7fa2558a076e]
2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xca6) [0x55b1f9ac22f6]
3: (main()+0x23a4) [0x55b1f9a1f284]
4: (__libc_start_main()+0xf5) [0x7fa2540daf45]
5: (()+0x1c05ef) [0x55b1f9ab25ef]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
0> 2018-08-17 10:32:00.972354 7fa25ed5fd00 -1 /build/ceph-12.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, uint64_t, size_t, ceph::bufferlist*, char*)' thread 7fa25ed5fd00 time 2018-08-17 10:32:00.971631
/build/ceph-12.2.6/src/os/bluestore/BlueFS.cc: 976: FAILED assert(r == 0)
ceph version 12.2.6 (488df8a1076c4f5fc5b8d18a90463262c438740f) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x7fa2558a076e]
2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xca6) [0x55b1f9ac22f6]
3: (main()+0x23a4) [0x55b1f9a1f284]
4: (__libc_start_main()+0xf5) [0x7fa2540daf45]
5: (()+0x1c05ef) [0x55b1f9ab25ef]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
*** Caught signal (Aborted) **
in thread 7fa25ed5fd00 thread_name:ceph-bluestore-
ceph version 12.2.6 (488df8a1076c4f5fc5b8d18a90463262c438740f) luminous (stable)
1: (()+0x3e15b9) [0x55b1f9cd35b9]
2: (()+0x10330) [0x7fa2550be330]
3: (gsignal()+0x37) [0x7fa2540efc37]
4: (abort()+0x148) [0x7fa2540f3028]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x7fa2558a08e0]
6: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xca6) [0x55b1f9ac22f6]
7: (main()+0x23a4) [0x55b1f9a1f284]
8: (__libc_start_main()+0xf5) [0x7fa2540daf45]
9: (()+0x1c05ef) [0x55b1f9ab25ef]
2018-08-17 10:32:00.973081 7fa25ed5fd00 -1 *** Caught signal (Aborted) **
in thread 7fa25ed5fd00 thread_name:ceph-bluestore-
ceph version 12.2.6 (488df8a1076c4f5fc5b8d18a90463262c438740f) luminous (stable)
1: (()+0x3e15b9) [0x55b1f9cd35b9]
2: (()+0x10330) [0x7fa2550be330]
3: (gsignal()+0x37) [0x7fa2540efc37]
4: (abort()+0x148) [0x7fa2540f3028]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x7fa2558a08e0]
6: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xca6) [0x55b1f9ac22f6]
7: (main()+0x23a4) [0x55b1f9a1f284]
8: (__libc_start_main()+0xf5) [0x7fa2540daf45]
9: (()+0x1c05ef) [0x55b1f9ab25ef]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
0> 2018-08-17 10:32:00.973081 7fa25ed5fd00 -1 *** Caught signal (Aborted) **
in thread 7fa25ed5fd00 thread_name:ceph-bluestore-
ceph version 12.2.6 (488df8a1076c4f5fc5b8d18a90463262c438740f) luminous (stable)
1: (()+0x3e15b9) [0x55b1f9cd35b9]
2: (()+0x10330) [0x7fa2550be330]
3: (gsignal()+0x37) [0x7fa2540efc37]
4: (abort()+0x148) [0x7fa2540f3028]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x7fa2558a08e0]
6: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xca6) [0x55b1f9ac22f6]
7: (main()+0x23a4) [0x55b1f9a1f284]
8: (__libc_start_main()+0xf5) [0x7fa2540daf45]
9: (()+0x1c05ef) [0x55b1f9ab25ef]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
I set up a cluster with 12 PCs, every PC have 3 network interfaces,8GB memory, 1 Intel cort i3-4170 and 1 disk ad system root partion ,two others as ceph bluestore disk.
Here is my ceph cluster configuration:
[global]
osd crush location hook = /usr/bin/calamari-crush-location
fsid = 628e2af3-d0fd-4ad2-ae41-ac3178d37721
mon_initial_members = sd-ceph1, sd-ceph3, sd-ceph5
mon_host = 10.10.10.101,10.10.10.103,10.10.10.105
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd_pool_default_size = 3
public_network = 10.10.10.0/24
cluster_network = 10.10.20.0/24
#osd_pool_default_pg_num = 2048
#osd_pool_default_pgp_num = 2048
osd deep scrub interval = 1209600
osd scrub begin hour = 20
osd scrub end hour = 9
osd max scrubs = 1
#osd scrub during recovery = false
[client]
rbd_cache = true
rbd_cache_size = 134217728
rbd_cache_max_dirty= 10444800
why did the osd crash so frequently. can i fix it whitout rebuilding the osd node.
thanks,
wang yu
发送自我的HTC Phone
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
