After read a lot of documentation I started to try to recover my data doing PG export and import to try get my pool up but all command that I did result a error . I started the work to list all PG to create a simple import/export script . but I can neither list the PGs .
ceph-objectstore-tool --bluestore --op list-pgs --data-path /var/lib/ceph/osd/ceph-6
2019-06-21 20:20:55.054762 7fd42d2dc540 -1 Errors while parsing config file!
2019-06-21 20:20:55.054771 7fd42d2dc540 -1 unexpected character at char 236 of line 39
/home/abuild/rpmbuild/BUILD/ceph-12.2.10-551-gbb089269ea/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, uint64_t, size_t, ceph::bufferlist*, char*)' thread 7fd42d2dc540 time 2019-06-21 20:21:04.136337
/home/abuild/rpmbuild/BUILD/ceph-12.2.10-551-gbb089269ea/src/os/bluestore/BlueFS.cc: 976: FAILED assert(r == 0)
ceph version 12.2.10-551-gbb089269ea (bb089269ea0c1272294c6b9777123ac81662b6d2) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x7fd4239c6a9e]
2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xcb4) [0x55c9beeceef4]
3: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x36) [0x55c9beef1dc6]
4: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x17c) [0x55c9befd7b5c]
5: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xc8) [0x55c9bef4fdf8]
6: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x34) [0x55c9bef4fec4]
7: (rocksdb::log::Reader::ReadRecord(rocksdb::Slice*, std::string*, rocksdb::WALRecoveryMode)+0xe8) [0x55c9bef50168]
8: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0xee8) [0x55c9bef41948]
9: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0x7c6) [0x55c9bef43296]
10: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0xeb3) [0x55c9bef44523]
11: (rocksdb::DB::Open(rocksdb::Options const&, std::string const&, rocksdb::DB**)+0x177) [0x55c9bef45757]
12: (RocksDBStore::do_open(std::ostream&, bool)+0x93b) [0x55c9bee85bfb]
13: (BlueStore::_open_db(bool)+0xd53) [0x55c9bee12273]
14: (BlueStore::_mount(bool)+0x3ea) [0x55c9bee436da]
15: (main()+0x2b96) [0x55c9be8daaa6]
16: (__libc_start_main()+0xf5) [0x7fd4223ab725]
17: (_start()+0x29) [0x55c9be9770f9]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
*** Caught signal (Aborted) **
in thread 7fd42d2dc540 thread_name:ceph-objectstor
ceph version 12.2.10-551-gbb089269ea (bb089269ea0c1272294c6b9777123ac81662b6d2) luminous (stable)
1: (()+0x915959) [0x55c9bef18959]
2: (()+0x10c10) [0x7fd4231e7c10]
3: (gsignal()+0x37) [0x7fd4223bff67]
4: (abort()+0x13a) [0x7fd4223c133a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x7fd4239c6c10]
6: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xcb4) [0x55c9beeceef4]
7: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x36) [0x55c9beef1dc6]
8: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x17c) [0x55c9befd7b5c]
9: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xc8) [0x55c9bef4fdf8]
10: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x34) [0x55c9bef4fec4]
11: (rocksdb::log::Reader::ReadRecord(rocksdb::Slice*, std::string*, rocksdb::WALRecoveryMode)+0xe8) [0x55c9bef50168]
12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0xee8) [0x55c9bef41948]
13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0x7c6) [0x55c9bef43296]
14: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0xeb3) [0x55c9bef44523]
15: (rocksdb::DB::Open(rocksdb::Options const&, std::string const&, rocksdb::DB**)+0x177) [0x55c9bef45757]
16: (RocksDBStore::do_open(std::ostream&, bool)+0x93b) [0x55c9bee85bfb]
17: (BlueStore::_open_db(bool)+0xd53) [0x55c9bee12273]
18: (BlueStore::_mount(bool)+0x3ea) [0x55c9bee436da]
19: (main()+0x2b96) [0x55c9be8daaa6]
20: (__libc_start_main()+0xf5) [0x7fd4223ab725]
21: (_start()+0x29) [0x55c9be9770f9]
Aborted (core dumped)
2019-06-21 20:20:55.054762 7fd42d2dc540 -1 Errors while parsing config file!
2019-06-21 20:20:55.054771 7fd42d2dc540 -1 unexpected character at char 236 of line 39
/home/abuild/rpmbuild/BUILD/ceph-12.2.10-551-gbb089269ea/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, uint64_t, size_t, ceph::bufferlist*, char*)' thread 7fd42d2dc540 time 2019-06-21 20:21:04.136337
/home/abuild/rpmbuild/BUILD/ceph-12.2.10-551-gbb089269ea/src/os/bluestore/BlueFS.cc: 976: FAILED assert(r == 0)
ceph version 12.2.10-551-gbb089269ea (bb089269ea0c1272294c6b9777123ac81662b6d2) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x7fd4239c6a9e]
2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xcb4) [0x55c9beeceef4]
3: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x36) [0x55c9beef1dc6]
4: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x17c) [0x55c9befd7b5c]
5: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xc8) [0x55c9bef4fdf8]
6: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x34) [0x55c9bef4fec4]
7: (rocksdb::log::Reader::ReadRecord(rocksdb::Slice*, std::string*, rocksdb::WALRecoveryMode)+0xe8) [0x55c9bef50168]
8: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0xee8) [0x55c9bef41948]
9: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0x7c6) [0x55c9bef43296]
10: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0xeb3) [0x55c9bef44523]
11: (rocksdb::DB::Open(rocksdb::Options const&, std::string const&, rocksdb::DB**)+0x177) [0x55c9bef45757]
12: (RocksDBStore::do_open(std::ostream&, bool)+0x93b) [0x55c9bee85bfb]
13: (BlueStore::_open_db(bool)+0xd53) [0x55c9bee12273]
14: (BlueStore::_mount(bool)+0x3ea) [0x55c9bee436da]
15: (main()+0x2b96) [0x55c9be8daaa6]
16: (__libc_start_main()+0xf5) [0x7fd4223ab725]
17: (_start()+0x29) [0x55c9be9770f9]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
*** Caught signal (Aborted) **
in thread 7fd42d2dc540 thread_name:ceph-objectstor
ceph version 12.2.10-551-gbb089269ea (bb089269ea0c1272294c6b9777123ac81662b6d2) luminous (stable)
1: (()+0x915959) [0x55c9bef18959]
2: (()+0x10c10) [0x7fd4231e7c10]
3: (gsignal()+0x37) [0x7fd4223bff67]
4: (abort()+0x13a) [0x7fd4223c133a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x7fd4239c6c10]
6: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xcb4) [0x55c9beeceef4]
7: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x36) [0x55c9beef1dc6]
8: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x17c) [0x55c9befd7b5c]
9: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xc8) [0x55c9bef4fdf8]
10: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x34) [0x55c9bef4fec4]
11: (rocksdb::log::Reader::ReadRecord(rocksdb::Slice*, std::string*, rocksdb::WALRecoveryMode)+0xe8) [0x55c9bef50168]
12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0xee8) [0x55c9bef41948]
13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0x7c6) [0x55c9bef43296]
14: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0xeb3) [0x55c9bef44523]
15: (rocksdb::DB::Open(rocksdb::Options const&, std::string const&, rocksdb::DB**)+0x177) [0x55c9bef45757]
16: (RocksDBStore::do_open(std::ostream&, bool)+0x93b) [0x55c9bee85bfb]
17: (BlueStore::_open_db(bool)+0xd53) [0x55c9bee12273]
18: (BlueStore::_mount(bool)+0x3ea) [0x55c9bee436da]
19: (main()+0x2b96) [0x55c9be8daaa6]
20: (__libc_start_main()+0xf5) [0x7fd4223ab725]
21: (_start()+0x29) [0x55c9be9770f9]
Aborted (core dumped)
Em sex, 21 de jun de 2019 às 13:18, Saulo Silva <sauloaugustosilva@xxxxxxxxx> escreveu:
HI Igor ,Here the assert line and the 10 line that goes before and after .-4> 2019-06-21 10:54:45.161493 7f689291ed00 20 bluefs _read left 0x8000 len 0x8000
-3> 2019-06-21 10:54:45.161497 7f689291ed00 20 bluefs _read got 32768
-2> 2019-06-21 10:54:45.161533 7f689291ed00 10 bluefs _read h 0x55bab9dbfc00 0x6d00000~8000 from file(ino 165 size 0xf5775f4 mtime 2019-06-19 02:49:24.017808 bdev 1 allocated f700000 extents [1:0xbfeb00000+100000,1:0xbfec00000+100000,1:0xbfed00000+100000,1:0xbfee00000+100000,1:0xbfef00000+100000,1:0xbff000000+100000,1:0xbff100000+100000,1:0xbff200000+100000,1:0xbff300000+100000,1:0xbff400000+100000,1:0xbff500000+100000,1:0xbff600000+100000,1:0xbff700000+100000,1:0xbff800000+100000,1:0xbff900000+100000,1:0xbffa00000+100000,1:0xbffb00000+100000,1:0xbffc00000+100000,1:0xbffd00000+100000,1:0xbffe00000+100000,1:0xbfff00000+100000,1:0xc00000000+100000,1:0xc00100000+100000,1:0xc00200000+100000,1:0xc00300000+100000,1:0xc00400000+100000,1:0xc00500000+100000,1:0xc00600000+100000,1:0xc00700000+100000,1:0xc00800000+100000,1:0xc00900000+100000,1:0xc00a00000+100000,1:0xc00b00000+100000,1:0xc00c00000+100000,1:0xc00d00000+100000,1:0xc00e00000+100000,1:0xc00f00000+100000,1:0xc01000000+100000,1:0xc01100000+100000,1:0xc01200000+100000,1:0xc01300000+100000,1:0xc01400000+100000,1:0xc01500000+100000,1:0xc01600000+100000,1:0xc01700000+100000,1:0xc01800000+100000,1:0xc01900000+100000,1:0xc01a00000+100000,1:0xc01b00000+100000,1:0xc01c00000+100000,1:0xc01d00000+100000,1:0xc01e00000+100000,1:0xc01f00000+100000,1:0xc02000000+100000,1:0xc02100000+100000,1:0xc02200000+100000,1:0xc02300000+100000,1:0xc02400000+100000,1:0xc02500000+100000,1:0xc02600000+100000,1:0xc02700000+100000,1:0xc02800000+100000,1:0xc02900000+100000,1:0xc02a00000+100000,1:0xc02b00000+100000,1:0xc02c00000+100000,1:0xc02d00000+100000,1:0xc02e00000+100000,1:0xc02f00000+100000,1:0xc03000000+100000,1:0xc03100000+100000,1:0xc03200000+100000,1:0xc03300000+100000,1:0xc03400000+100000,1:0xc03500000+100000,1:0xc03600000+100000,1:0xc03700000+100000,1:0xc03800000+100000,1:0xc03900000+100000,1:0xc03a00000+100000,1:0xc03b00000+100000,1:0xc03c00000+100000,1:0xc03d00000+100000,1:0xc03e00000+100000,1:0xc03f00000+100000,1:0xc04000000+100000,1:0xc04100000+100000,1:0xc04200000+100000,1:0xc04300000+100000,1:0xc04400000+100000,1:0xc04500000+100000,1:0xc04600000+100000,1:0xc04700000+100000,1:0xc04800000+100000,1:0xc04900000+100000,1:0xc04a00000+100000,1:0xc04b00000+100000,1:0xc04c00000+100000,1:0xc04d00000+100000,1:0xc04e00000+100000,1:0xc04f00000+100000,1:0xc05000000+100000,1:0xc05100000+100000,1:0xc05200000+100000,1:0xc05300000+100000,1:0xc05400000+100000,1:0xc05500000+100000,1:0xc05600000+100000,1:0xc05700000+100000,1:0xc05800000+100000,1:0xc05900000+100000,1:0xc05a00000+100000,1:0xc05b00000+100000,1:0xc05c00000+100000,1:0xc05d00000+100000,1:0xc05e00000+100000,1:0xc05f00000+100000,1:0xc06000000+100000,1:0xc06100000+100000,1:0xc06200000+100000,1:0xc06300000+100000,1:0xc06400000+100000,1:0xc06500000+100000,1:0xc06600000+100000,1:0xc06700000+100000,1:0xc06800000+100000,1:0xc06900000+100000,1:0xc06a00000+100000,1:0xc06b00000+100000,1:0xc06c00000+100000,1:0xc06d00000+100000,1:0xc06e00000+100000,1:0xc06f00000+100000,1:0xc07000000+100000,1:0xc07100000+100000,1:0xc07200000+100000,1:0xc07300000+100000,1:0xc07400000+100000,1:0xc07500000+100000,1:0xc07600000+100000,1:0xc07700000+100000,1:0xc07800000+100000,1:0xc07900000+100000,1:0xc07a00000+100000,1:0xc07b00000+100000,1:0xc07c00000+100000,1:0xc07d00000+100000,1:0xc07e00000+100000,1:0xc07f00000+100000,1:0xc08000000+100000,1:0xc08100000+100000,1:0xc08200000+100000,1:0xc08300000+100000,1:0xc08400000+100000,1:0xc08500000+100000,1:0xc08600000+100000,1:0xc08700000+100000,1:0xc08800000+100000,1:0xc08900000+100000,1:0xc08a00000+100000,1:0xc08b00000+100000,1:0xc08c00000+100000,1:0xc08d00000+100000,1:0xc08e00000+100000,1:0xc08f00000+100000,1:0xc09000000+100000,1:0xc09100000+100000,1:0xc09200000+100000,1:0xc09300000+100000,1:0xc09400000+100000,1:0xc09500000+100000,1:0xc09600000+100000,1:0xc09700000+100000,1:0xc09800000+100000,1:0xc09900000+100000,1:0xc09a00000+100000,1:0xc09b00000+100000,1:0xc09c00000+100000,1:0xc09d00000+100000,1:0xc09e00000+100000,1:0xc09f00000+100000,1:0xc0a000000+100000,1:0xc0a100000+100000,1:0xc0a200000+100000,1:0xc0a300000+100000,1:0xc0a400000+100000,1:0xc0a500000+100000,1:0xc0a600000+100000,1:0xc0a700000+100000,1:0xc0a800000+100000,1:0xc0a900000+100000,1:0xc0aa00000+100000,1:0xc0ab00000+100000,1:0xc0ac00000+100000,1:0xc0ad00000+100000,1:0xc0ae00000+100000,1:0xc0af00000+100000,1:0xc0b000000+100000,1:0xc0b100000+100000,1:0xc0b200000+100000,1:0xc0b300000+100000,1:0xc0b400000+100000,1:0xc0b500000+100000,1:0xc0b600000+100000,1:0xc0b700000+100000,1:0xc0b800000+100000,1:0xc0b900000+100000,1:0xc0ba00000+100000,1:0xc0bb00000+100000,1:0xc0bc00000+100000,1:0xc0bd00000+100000,1:0xc0be00000+100000,1:0xc0bf00000+100000,1:0xc0c000000+100000,1:0xc0c100000+100000,1:0xc0c200000+100000,1:0xc0c300000+100000,1:0xc0c400000+100000,1:0xc0c500000+100000,1:0xc0c600000+100000,1:0xc0c700000+100000,1:0xc0c800000+100000,1:0xc0c900000+100000,1:0xc0ca00000+100000,1:0xc0cb00000+100000,1:0xc0cc00000+100000,1:0xc0cd00000+100000,1:0xc0ce00000+100000,1:0xc0cf00000+100000,1:0xc0d000000+100000,1:0xc0d100000+100000,1:0xc0d200000+100000,1:0xc0d300000+100000,1:0xc0d400000+100000,1:0xc0d500000+100000,1:0xc0d600000+100000,1:0xc0d700000+100000,1:0xc0d800000+100000,1:0xc20e00000+100000,1:0xc20f00000+100000,1:0xc21000000+100000,1:0xc21100000+100000,1:0xc21200000+100000,1:0xc21300000+100000,1:0xc21400000+100000,1:0xc21500000+100000,1:0xc21700000+100000])
-1> 2019-06-21 10:54:45.161646 7f689291ed00 20 bluefs _read fetching 0x0~100000 of 1:0xc05800000+100000
0> 2019-06-21 10:54:45.193256 7f689291ed00 -1 /home/abuild/rpmbuild/BUILD/ceph-12.2.10-551-gbb089269ea/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, uint64_t, size_t, ceph::bufferlist*, char*)' thread 7f689291ed00 time 2019-06-21 10:54:45.181081
/home/abuild/rpmbuild/BUILD/ceph-12.2.10-551-gbb089269ea/src/os/bluestore/BlueFS.cc: 976: FAILED assert(r == 0)
ceph version 12.2.10-551-gbb089269ea (bb089269ea0c1272294c6b9777123ac81662b6d2) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x55baaff3172e]
2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xcb4) [0x55baafeaaf44]
3: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x36) [0x55baafecde16]
4: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x17c) [0x55bab028d84c]
5: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xc8) [0x55bab0206628]
6: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x34) [0x55bab02066f4]
--
ceph version 12.2.10-551-gbb089269ea (bb089269ea0c1272294c6b9777123ac81662b6d2) luminous (stable)
1: (()+0xa589a9) [0x55baafef49a9]
2: (()+0x10c10) [0x7f6890c20c10]
3: (gsignal()+0x37) [0x7f688fbe6f67]
4: (abort()+0x13a) [0x7f688fbe833a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x55baaff318a0]
6: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xcb4) [0x55baafeaaf44]
7: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x36) [0x55baafecde16]
8: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x17c) [0x55bab028d84c]
9: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xc8) [0x55bab0206628]
10: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x34) [0x55bab02066f4]
--
ceph version 12.2.10-551-gbb089269ea (bb089269ea0c1272294c6b9777123ac81662b6d2) luminous (stable)
1: (()+0xa589a9) [0x55baafef49a9]
2: (()+0x10c10) [0x7f6890c20c10]
3: (gsignal()+0x37) [0x7f688fbe6f67]
4: (abort()+0x13a) [0x7f688fbe833a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x55baaff318a0]
6: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xcb4) [0x55baafeaaf44]
7: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x36) [0x55baafecde16]
8: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x17c) [0x55bab028d84c]
9: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xc8) [0x55bab0206628]
10: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x34) [0x55bab02066f4]Is that one ? Am I correct ?Best Regards,Saulo Augusto SilvaEm sex, 21 de jun de 2019 às 13:03, Saulo Silva <sauloaugustosilva@xxxxxxxxx> escreveu:Hi Igor ,I am looking at the log and I am not sure what is the exactly line that I shuold send .I tried totail -f /var/log/ceph/ceph-osd.6.log | grep -i assertion -A 5But no valid result returned .What you be the regex to get this specific line ?I also could send the entire log .Best Regards,Saulo Augusto SilvaEm sex, 21 de jun de 2019 às 11:39, Igor Fedotov <ifedotov@xxxxxxx> escreveu:Saulo,
please share a few log lines immediately before the assertion, not the starting ones.
Thanks,
Igor
On 6/21/2019 5:37 PM, Saulo Silva wrote:
Hi Igor , thanks for helping ,
Here part of the log :head ceph-osd.6.log -n80
2019-06-21 10:50:56.090891 7f462db84d00 0 set uid:gid to 167:167 (ceph:ceph)
2019-06-21 10:50:56.090910 7f462db84d00 0 ceph version 12.2.10-551-gbb089269ea (bb089269ea0c1272294c6b9777123ac81662b6d2) luminous (stable), process ceph-osd, pid 18014
2019-06-21 10:50:56.116848 7f462db84d00 0 pidfile_write: ignore empty --pid-file
2019-06-21 10:50:56.127199 7f462db84d00 0 load: jerasure load: lrc load: isa
2019-06-21 10:50:56.435977 7f462db84d00 10 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-6/block
2019-06-21 10:50:56.436187 7f462db84d00 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-6/block size 99.9GiB
2019-06-21 10:50:56.436221 7f462db84d00 1 bluefs mount
2019-06-21 10:50:56.436225 7f462db84d00 10 bluefs _open_super
2019-06-21 10:50:56.436866 7f462db84d00 10 bluefs _open_super superblock 15
2019-06-21 10:50:56.436893 7f462db84d00 10 bluefs _open_super log_fnode file(ino 1 size 0x100000 mtime 0.000000 bdev 0 allocated 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000])
2019-06-21 10:50:56.436904 7f462db84d00 20 bluefs _init_alloc
2019-06-21 10:50:56.436913 7f462db84d00 10 bluefs _replay
2019-06-21 10:50:56.436919 7f462db84d00 10 bluefs _replay log_fnode file(ino 1 size 0x100000 mtime 0.000000 bdev 0 allocated 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000])
2019-06-21 10:50:56.436923 7f462db84d00 10 bluefs _read h 0x5632d7074b80 0x0~1000 from file(ino 1 size 0x100000 mtime 0.000000 bdev 0 allocated 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000])
2019-06-21 10:50:56.436928 7f462db84d00 20 bluefs _read fetching 0x0~100000 of 1:0xc9bf00000+100000
2019-06-21 10:50:56.458689 7f462db84d00 20 bluefs _read left 0x100000 len 0x1000
2019-06-21 10:50:56.458701 7f462db84d00 20 bluefs _read got 4096
2019-06-21 10:50:56.458704 7f462db84d00 20 bluefs _replay need 0x1000 more bytes
2019-06-21 10:50:56.458705 7f462db84d00 10 bluefs _read h 0x5632d7074b80 0x1000~1000 from file(ino 1 size 0x100000 mtime 0.000000 bdev 0 allocated 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000])
2019-06-21 10:50:56.458710 7f462db84d00 20 bluefs _read left 0xff000 len 0x1000
2019-06-21 10:50:56.458711 7f462db84d00 20 bluefs _read got 4096
2019-06-21 10:50:56.458717 7f462db84d00 10 bluefs _replay 0x0: txn(seq 1 len 0x1c73 crc 0xfd2d3aef)
2019-06-21 10:50:56.458719 7f462db84d00 20 bluefs _replay 0x0: op_init
2019-06-21 10:50:56.458721 7f462db84d00 20 bluefs _replay 0x0: op_alloc_add 1:0xbfd100000~ffd00000
2019-06-21 10:50:56.458735 7f462db84d00 20 bluefs _replay 0x0: op_file_update file(ino 1012 size 0x41ef34e mtime 2019-05-22 19:13:16.240226 bdev 1 allocated 4200000 extents [1:0xceda00000+4200000])
2019-06-21 10:50:56.458765 7f462db84d00 20 bluefs _replay 0x0: op_file_update file(ino 167 size 0xf4da0c3 mtime 2019-05-23 13:11:26.154047 bdev 1 allocated f700000 extents [1:0xc0d900000+100000,1:0xc11a00000+100000,1:0xc11b00000+100000,1:0xc11c00000+100000,1:0xc11d00000+100000,1:0xc11e00000+100000,1:0xc11f00000+100000,1:0xc12000000+100000,1:0xc12100000+100000,1:0xc12200000+100000,1:0xc12300000+100000,1:0xc12400000+100000,1:0xc12500000+100000,1:0xc12600000+100000,1:0xc12700000+100000,1:0xc12800000+100000,1:0xc12900000+100000,1:0xc12a00000+100000,1:0xc12b00000+100000,1:0xc12c00000+100000,1:0xc12d00000+100000,1:0xc12e00000+100000,1:0xc12f00000+100000,1:0xc13000000+100000,1:0xc13100000+100000,1:0xc13200000+100000,1:0xc13300000+100000,1:0xc13400000+100000,1:0xc13500000+100000,1:0xc13600000+100000,1:0xc13700000+100000,1:0xc13800000+100000,1:0xc13900000+100000,1:0xc13a00000+100000,1:0xc13b00000+100000,1:0xc13c00000+100000,1:0xc13d00000+100000,1:0xc13e00000+100000,1:0xc13f00000+100000,1:0xc14000000+100000,1:0xc14100000+100000,1:0xc14200000+100000,1:0xc14300000+100000,1:0xc14400000+100000,1:0xc14500000+100000,1:0xc14600000+100000,1:0xc14700000+100000,1:0xc14800000+100000,1:0xc14900000+100000,1:0xc14a00000+100000,1:0xc14b00000+100000,1:0xc14c00000+100000,1:0xc14d00000+100000,1:0xc14e00000+100000,1:0xc14f00000+100000,1:0xc15000000+100000,1:0xc15100000+100000,1:0xc15200000+100000,1:0xc15300000+100000,1:0xc15400000+100000,1:0xc15500000+100000,1:0xc15600000+100000,1:0xc15700000+100000,1:0xc15800000+100000,1:0xc15900000+100000,1:0xc15a00000+100000,1:0xc15b00000+100000,1:0xc15c00000+100000,1:0xc15d00000+100000,1:0xc15e00000+100000,1:0xc15f00000+100000,1:0xc16000000+100000,1:0xc16100000+100000,1:0xc16200000+100000,1:0xc16300000+100000,1:0xc16400000+100000,1:0xc16500000+100000,1:0xc16600000+100000,1:0xc16700000+100000,1:0xc16800000+100000,1:0xc16900000+100000,1:0xc16a00000+100000,1:0xc16b00000+100000,1:0xc16c00000+100000,1:0xc16d00000+100000,1:0xc16e00000+100000,1:0xc16f00000+100000,1:0xc17000000+100000,1:0xc17100000+100000,1:0xc17200000+100000,1:0xc17300000+100000,1:0xc17400000+100000,1:0xc17500000+100000,1:0xc17600000+100000,1:0xc17700000+100000,1:0xc17800000+100000,1:0xc17900000+100000,1:0xc17a00000+100000,1:0xc17b00000+100000,1:0xc17c00000+100000,1:0xc17d00000+100000,1:0xc17e00000+100000,1:0xc17f00000+100000,1:0xc18000000+100000,1:0xc18100000+100000,1:0xc18200000+100000,1:0xc18300000+100000,1:0xc18400000+100000,1:0xc18500000+100000,1:0xc18600000+100000,1:0xc18700000+100000,1:0xc18800000+100000,1:0xc18900000+100000,1:0xc18a00000+100000,1:0xc18b00000+100000,1:0xc18c00000+100000,1:0xc18d00000+100000,1:0xc18e00000+100000,1:0xc18f00000+100000,1:0xc19000000+100000,1:0xc19100000+100000,1:0xc19200000+100000,1:0xc19300000+100000,1:0xc19400000+100000,1:0xc19500000+100000,1:0xc19600000+100000,1:0xc19700000+100000,1:0xc19800000+100000,1:0xc19900000+100000,1:0xc19a00000+100000,1:0xc19b00000+100000,1:0xc19c00000+100000,1:0xc19d00000+100000,1:0xc19e00000+100000,1:0xc19f00000+100000,1:0xc1a000000+100000,1:0xc1a100000+100000,1:0xc1a200000+100000,1:0xc1a300000+100000,1:0xc1a400000+100000,1:0xc1a500000+100000,1:0xc1a600000+100000,1:0xc1a700000+100000,1:0xc1a800000+100000,1:0xc1a900000+100000,1:0xc1aa00000+100000,1:0xc1ab00000+100000,1:0xc1ac00000+100000,1:0xc1ad00000+100000,1:0xc1ae00000+100000,1:0xc1af00000+100000,1:0xc1b000000+100000,1:0xc1b100000+100000,1:0xc1b200000+100000,1:0xc1b300000+100000,1:0xc1b400000+100000,1:0xc1b500000+100000,1:0xc1b600000+100000,1:0xc1b700000+100000,1:0xc1b800000+100000,1:0xc1b900000+100000,1:0xc1ba00000+100000,1:0xc1bb00000+100000,1:0xc1bc00000+100000,1:0xc1bd00000+100000,1:0xc1be00000+100000,1:0xc1bf00000+100000,1:0xc1c000000+100000,1:0xc1c100000+100000,1:0xc1c200000+100000,1:0xc1c300000+100000,1:0xc1c400000+100000,1:0xc1c500000+100000,1:0xc1c600000+100000,1:0xc1c700000+100000,1:0xc1c800000+100000,1:0xc1c900000+100000,1:0xc1ca00000+100000,1:0xc1cb00000+100000,1:0xc1cc00000+100000,1:0xc1cd00000+100000,1:0xc1ce00000+100000,1:0xc1cf00000+100000,1:0xc1d000000+100000,1:0xc1d100000+100000,1:0xc1d200000+100000,1:0xc1d300000+100000,1:0xc1d400000+100000,1:0xc1d500000+100000,1:0xc1d600000+100000,1:0xc1d700000+100000,1:0xc1d800000+100000,1:0xc1d900000+100000,1:0xc1da00000+100000,1:0xc1db00000+100000,1:0xc1dc00000+100000,1:0xc1dd00000+100000,1:0xc1de00000+100000,1:0xc1df00000+100000,1:0xc1e000000+100000,1:0xc1e100000+100000,1:0xc1e200000+100000,1:0xc1e300000+100000,1:0xc1e400000+100000,1:0xc1e500000+100000,1:0xc1e600000+100000,1:0xc1e700000+100000,1:0xc1e800000+100000,1:0xc1e900000+100000,1:0xc1ea00000+100000,1:0xc1eb00000+100000,1:0xc1ec00000+100000,1:0xc1ed00000+100000,1:0xc1ee00000+100000,1:0xc1ef00000+100000,1:0xc1f000000+100000,1:0xc1f100000+100000,1:0xc1f200000+100000,1:0xc1f300000+100000,1:0xc1f400000+100000,1:0xc1f500000+100000,1:0xc1f600000+100000,1:0xc1f700000+100000,1:0xc1f800000+100000,1:0xc1f900000+100000,1:0xc1fa00000+100000,1:0xc1fb00000+100000,1:0xc1fc00000+100000,1:0xc1fd00000+100000,1:0xc1fe00000+100000,1:0xc1ff00000+100000,1:0xc20000000+100000,1:0xc20100000+100000,1:0xc20200000+100000,1:0xc20300000+100000,1:0xc20400000+100000,1:0xc20500000+100000,1:0xc20600000+100000,1:0xc20700000+100000,1:0xc20800000+100000,1:0xc20900000+100000,1:0xc20a00000+100000,1:0xc20b00000+100000,1:0xc20c00000+100000,1:0xc20d00000+100000,1:0xc21600000+100000,1:0xc21800000+100000])
2019-06-21 10:50:56.458874 7f462db84d00 20 bluefs _replay 0x0: op_file_update file(ino 1016 size 0x0 mtime 2019-05-23 13:11:26.173532 bdev 1 allocated 0 extents [])
2019-06-21 10:50:56.458883 7f462db84d00 20 bluefs _replay 0x0: op_file_update file(ino 165 size 0xf4eb0ff mtime 2019-05-23 13:11:26.154138 bdev 1 allocated f700000 extents [1:0xbfeb00000+100000,1:0xbfec00000+100000,1:0xbfed00000+100000,1:0xbfee00000+100000,1:0xbfef00000+100000,1:0xbff000000+100000,1:0xbff100000+100000,1:0xbff200000+100000,1:0xbff300000+100000,1:0xbff400000+100000,1:0xbff500000+100000,1:0xbff600000+100000,1:0xbff700000+100000,1:0xbff800000+100000,1:0xbff900000+100000,1:0xbffa00000+100000,1:0xbffb00000+100000,1:0xbffc00000+100000,1:0xbffd00000+100000,1:0xbffe00000+100000,1:0xbfff00000+100000,1:0xc00000000+100000,1:0xc00100000+100000,1:0xc00200000+100000,1:0xc00300000+100000,1:0xc00400000+100000,1:0xc00500000+100000,1:0xc00600000+100000,1:0xc00700000+100000,1:0xc00800000+100000,1:0xc00900000+100000,1:0xc00a00000+100000,1:0xc00b00000+100000,1:0xc00c00000+100000,1:0xc00d00000+100000,1:0xc00e00000+100000,1:0xc00f00000+100000,1:0xc01000000+100000,1:0xc01100000+100000,1:0xc01200000+100000,1:0xc01300000+100000,1:0xc01400000+100000,1:0xc01500000+100000,1:0xc01600000+100000,1:0xc01700000+100000,1:0xc01800000+100000,1:0xc01900000+100000,1:0xc01a00000+100000,1:0xc01b00000+100000,1:0xc01c00000+100000,1:0xc01d00000+100000,1:0xc01e00000+100000,1:0xc01f00000+100000,1:0xc02000000+100000,1:0xc02100000+100000,1:0xc02200000+100000,1:0xc02300000+100000,1:0xc02400000+100000,1:0xc02500000+100000,1:0xc02600000+100000,1:0xc02700000+100000,1:0xc02800000+100000,1:0xc02900000+100000,1:0xc02a00000+100000,1:0xc02b00000+100000,1:0xc02c00000+100000,1:0xc02d00000+100000,1:0xc02e00000+100000,1:0xc02f00000+100000,1:0xc03000000+100000,1:0xc03100000+100000,1:0xc03200000+100000,1:0xc03300000+100000,1:0xc03400000+100000,1:0xc03500000+100000,1:0xc03600000+100000,1:0xc03700000+100000,1:0xc03800000+100000,1:0xc03900000+100000,1:0xc03a00000+100000,1:0xc03b00000+100000,1:0xc03c00000+100000,1:0xc03d00000+100000,1:0xc03e00000+100000,1:0xc03f00000+100000,1:0xc04000000+100000,1:0xc04100000+100000,1:0xc04200000+100000,1:0xc04300000+100000,1:0xc04400000+100000,1:0xc04500000+100000,1:0xc04600000+100000,1:0xc04700000+100000,1:0xc04800000+100000,1:0xc04900000+100000,1:0xc04a00000+100000,1:0xc04b00000+100000,1:0xc04c00000+100000,1:0xc04d00000+100000,1:0xc04e00000+100000,1:0xc04f00000+100000,1:0xc05000000+100000,1:0xc05100000+100000,1:0xc05200000+100000,1:0xc05300000+100000,1:0xc05400000+100000,1:0xc05500000+100000,1:0xc05600000+100000,1:0xc05700000+100000,1:0xc05800000+100000,1:0xc05900000+100000,1:0xc05a00000+100000,1:0xc05b00000+100000,1:0xc05c00000+100000,1:0xc05d00000+100000,1:0xc05e00000+100000,1:0xc05f00000+100000,1:0xc06000000+100000,1:0xc06100000+100000,1:0xc06200000+100000,1:0xc06300000+100000,1:0xc06400000+100000,1:0xc06500000+100000,1:0xc06600000+100000,1:0xc06700000+100000,1:0xc06800000+100000,1:0xc06900000+100000,1:0xc06a00000+100000,1:0xc06b00000+100000,1:0xc06c00000+100000,1:0xc06d00000+100000,1:0xc06e00000+100000,1:0xc06f00000+100000,1:0xc07000000+100000,1:0xc07100000+100000,1:0xc07200000+100000,1:0xc07300000+100000,1:0xc07400000+100000,1:0xc07500000+100000,1:0xc07600000+100000,1:0xc07700000+100000,1:0xc07800000+100000,1:0xc07900000+100000,1:0xc07a00000+100000,1:0xc07b00000+100000,1:0xc07c00000+100000,1:0xc07d00000+100000,1:0xc07e00000+100000,1:0xc07f00000+100000,1:0xc08000000+100000,1:0xc08100000+100000,1:0xc08200000+100000,1:0xc08300000+100000,1:0xc08400000+100000,1:0xc08500000+100000,1:0xc08600000+100000,1:0xc08700000+100000,1:0xc08800000+100000,1:0xc08900000+100000,1:0xc08a00000+100000,1:0xc08b00000+100000,1:0xc08c00000+100000,1:0xc08d00000+100000,1:0xc08e00000+100000,1:0xc08f00000+100000,1:0xc09000000+100000,1:0xc09100000+100000,1:0xc09200000+100000,1:0xc09300000+100000,1:0xc09400000+100000,1:0xc09500000+100000,1:0xc09600000+100000,1:0xc09700000+100000,1:0xc09800000+100000,1:0xc09900000+100000,1:0xc09a00000+100000,1:0xc09b00000+100000,1:0xc09c00000+100000,1:0xc09d00000+100000,1:0xc09e00000+100000,1:0xc09f00000+100000,1:0xc0a000000+100000,1:0xc0a100000+100000,1:0xc0a200000+100000,1:0xc0a300000+100000,1:0xc0a400000+100000,1:0xc0a500000+100000,1:0xc0a600000+100000,1:0xc0a700000+100000,1:0xc0a800000+100000,1:0xc0a900000+100000,1:0xc0aa00000+100000,1:0xc0ab00000+100000,1:0xc0ac00000+100000,1:0xc0ad00000+100000,1:0xc0ae00000+100000,1:0xc0af00000+100000,1:0xc0b000000+100000,1:0xc0b100000+100000,1:0xc0b200000+100000,1:0xc0b300000+100000,1:0xc0b400000+100000,1:0xc0b500000+100000,1:0xc0b600000+100000,1:0xc0b700000+100000,1:0xc0b800000+100000,1:0xc0b900000+100000,1:0xc0ba00000+100000,1:0xc0bb00000+100000,1:0xc0bc00000+100000,1:0xc0bd00000+100000,1:0xc0be00000+100000,1:0xc0bf00000+100000,1:0xc0c000000+100000,1:0xc0c100000+100000,1:0xc0c200000+100000,1:0xc0c300000+100000,1:0xc0c400000+100000,1:0xc0c500000+100000,1:0xc0c600000+100000,1:0xc0c700000+100000,1:0xc0c800000+100000,1:0xc0c900000+100000,1:0xc0ca00000+100000,1:0xc0cb00000+100000,1:0xc0cc00000+100000,1:0xc0cd00000+100000,1:0xc0ce00000+100000,1:0xc0cf00000+100000,1:0xc0d000000+100000,1:0xc0d100000+100000,1:0xc0d200000+100000,1:0xc0d300000+100000,1:0xc0d400000+100000,1:0xc0d500000+100000,1:0xc0d600000+100000,1:0xc0d700000+100000,1:0xc0d800000+100000,1:0xc20e00000+100000,1:0xc20f00000+100000,1:0xc21000000+100000,1:0xc21100000+100000,1:0xc21200000+100000,1:0xc21300000+100000,1:0xc21400000+100000,1:0xc21500000+100000,1:0xc21700000+100000])
2019-06-21 10:50:56.458949 7f462db84d00 20 bluefs _replay 0x0: op_file_update file(ino 166 size 0x1033 mtime 2019-03-01 11:57:12.151139 bdev 1 allocated 100000 extents [1:0xbfea00000+100000])
2019-06-21 10:50:56.458963 7f462db84d00 20 bluefs _replay 0x0: op_file_update file(ino 5 size 0x25 mtime 2018-11-16 09:47:02.220665 bdev 1 allocated 100000 extents [1:0xbfd700000+100000])
2019-06-21 10:50:56.458970 7f462db84d00 20 bluefs _replay 0x0: op_file_update file(ino 1014 size 0x14e71f1 mtime 2019-05-22 23:10:22.033008 bdev 1 allocated 1500000 extents [1:0xcf5c00000+1500000])
2019-06-21 10:50:56.458975 7f462db84d00 20 bluefs _replay 0x0: op_file_update file(ino 163 size 0x1204c mtime 2019-05-23 07:49:27.039370 bdev 1 allocated 100000 extents [1:0xbfe800000+100000])
2019-06-21 10:50:56.458978 7f462db84d00 20 bluefs _replay 0x0: op_file_update file(ino 2 size 0x0 mtime 2018-11-16 09:47:02.202743 bdev 0 allocated 0 extents [])
2019-06-21 10:50:56.458982 7f462db84d00 20 bluefs _replay 0x0: op_file_update file(ino 1013 size 0x3f3b207 mtime 2019-05-22 19:13:18.907005 bdev 1 allocated 4000000 extents [1:0xcf1c00000+4000000])
2019-06-21 10:50:56.458997 7f462db84d00 20 bluefs _replay 0x0: op_file_update file(ino 146 size 0x1033 mtime 2018-12-28 19:12:38.946579 bdev 1 allocated 100000 extents [1:0xc66200000+100000])
2019-06-21 10:50:56.459003 7f462db84d00 20 bluefs _replay 0x0: op_file_update file(ino 1015 size 0x7ff641 mtime 2019-05-23 07:49:26.959694 bdev 1 allocated 800000 extents [1:0xcf7100000+800000])
2019-06-21 10:50:56.459007 7f462db84d00 20 bluefs _replay 0x0: op_file_update file(ino 164 size 0x10 mtime 2019-03-01 11:57:12.126941 bdev 1 allocated 100000 extents [1:0xbfe900000+100000])
2019-06-21 10:50:56.459013 7f462db84d00 20 bluefs _replay 0x0: op_dir_create db
2019-06-21 10:50:56.459016 7f462db84d00 20 bluefs _replay 0x0: op_dir_link db/001610.sst to 1012
2019-06-21 10:50:56.459020 7f462db84d00 20 bluefs _replay 0x0: op_dir_link db/001611.sst to 1013
2019-06-21 10:50:56.459030 7f462db84d00 20 bluefs _replay 0x0: op_dir_link db/001613.sst to 1014
2019-06-21 10:50:56.459035 7f462db84d00 20 bluefs _replay 0x0: op_dir_link db/001614.log to 165
2019-06-21 10:50:56.459037 7f462db84d00 20 bluefs _replay 0x0: op_dir_link db/001615.sst to 1015
2019-06-21 10:50:56.459039 7f462db84d00 20 bluefs _replay 0x0: op_dir_link db/001616.log to 167
2019-06-21 10:50:56.459041 7f462db84d00 20 bluefs _replay 0x0: op_dir_link db/001617.sst to 1016
2019-06-21 10:50:56.459043 7f462db84d00 20 bluefs _replay 0x0: op_dir_link db/CURRENT to 164
2019-06-21 10:50:56.459044 7f462db84d00 20 bluefs _replay 0x0: op_dir_link db/IDENTITY to 5
2019-06-21 10:50:56.459046 7f462db84d00 20 bluefs _replay 0x0: op_dir_link db/LOCK to 2
2019-06-21 10:50:56.459048 7f462db84d00 20 bluefs _replay 0x0: op_dir_link db/MANIFEST-000148 to 163
2019-06-21 10:50:56.459050 7f462db84d00 20 bluefs _replay 0x0: op_dir_link db/OPTIONS-000120 to 146
2019-06-21 10:50:56.459052 7f462db84d00 20 bluefs _replay 0x0: op_dir_link db/OPTIONS-000151 to 166
2019-06-21 10:50:56.459054 7f462db84d00 20 bluefs _replay 0x0: op_jump seq 332730 offset 0x100000
2019-06-21 10:50:56.459056 7f462db84d00 10 bluefs _read h 0x5632d7074b80 0x2000~fe000 from file(ino 1 size 0x100000 mtime 0.000000 bdev 0 allocated 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000])
2019-06-21 10:50:56.459059 7f462db84d00 20 bluefs _read left 0xfe000 len 0xfe000
2019-06-21 10:50:56.459060 7f462db84d00 20 bluefs _read got 1040384
2019-06-21 10:50:56.459062 7f462db84d00 10 bluefs _read h 0x5632d7074b80 0x100000~1000 from file(ino 1 size 0x100000 mtime 0.000000 bdev 0 allocated 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000])
2019-06-21 10:50:56.459115 7f462db84d00 20 bluefs _read fetching 0x0~100000 of 1:0xc9bb00000+400000
2019-06-21 10:50:56.475215 7f462db84d00 20 bluefs _read left 0x100000 len 0x1000
2019-06-21 10:50:56.475226 7f462db84d00 20 bluefs _read got 4096
2019-06-21 10:50:56.475233 7f462db84d00 10 bluefs _replay 0x100000: txn(seq 332731 len 0x27 crc 0x86332d59)
2019-06-21 10:50:56.475248 7f462db84d00 20 bluefs _replay 0x100000: op_file_update file(ino 1016 size 0x7e2ee4 mtime 2019-05-23 13:11:26.235932 bdev 1 allocated 800000 extents [1:0xc9c000000+800000])
2019-06-21 10:50:56.475258 7f462db84d00 10 bluefs _read h 0x5632d7074b80 0x101000~1000 from file(ino 1 size 0x101000 mtime 0.000000 bdev 0 allocated 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000])
2019-06-21 10:50:56.475262 7f462db84d00 20 bluefs _read left 0xff000 len 0x1000
2019-06-21 10:50:56.475263 7f462db84d00 20 bluefs _read got 4096
2019-06-21 10:50:56.475265 7f462db84d00 10 bluefs _replay 0x101000: txn(seq 332732 len 0x26 crc 0x6ee93342)
2019-06-21 10:50:56.475267 7f462db84d00 20 bluefs _replay 0x101000: op_file_update file(ino 163 size 0x12096 mtime 2019-05-23 13:11:26.314998 bdev 1 allocated 100000 extents [1:0xbfe800000+100000])
2019-06-21 10:50:56.475271 7f462db84d00 10 bluefs _read h 0x5632d7074b80 0x102000~1000 from file(ino 1 size 0x102000 mtime 0.000000 bdev 0 allocated 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000])
2019-06-21 10:50:56.475274 7f462db84d00 20 bluefs _read left 0xfe000 len 0x1000
2019-06-21 10:50:56.475275 7f462db84d00 20 bluefs _read got 4096
2019-06-21 10:50:56.475278 7f462db84d00 10 bluefs _replay 0x102000: txn(seq 332733 len 0xca5 crc 0x8634e0dc)
2019-06-21 10:50:56.475285 7f462db84d00 20 bluefs _replay 0x102000: op_file_update file(ino 167 size 0xf4dd172 mtime 2019-05-23 16:05:23.255290 bdev 1 allocated f700000 extents [1:0xc0d900000+100000,1:0xc11a00000+100000,1:0xc11b00000+100000,1:0xc11c00000+100000,1:0xc11d00000+100000,1:0xc11e00000+100000,1:0xc11f00000+100000,1:0xc12000000+100000,1:0xc12100000+100000,1:0xc12200000+100000,1:0xc12300000+100000,1:0xc12400000+100000,1:0xc12500000+100000,1:0xc12600000+100000,1:0xc12700000+100000,1:0xc12800000+100000,1:0xc12900000+100000,1:0xc12a00000+100000,1:0xc12b00000+100000,1:0xc12c00000+100000,1:0xc12d00000+100000,1:0xc12e00000+100000,1:0xc12f00000+100000,1:0xc13000000+100000,1:0xc13100000+100000,1:0xc13200000+100000,1:0xc13300000+100000,1:0xc13400000+100000,1:0xc13500000+100000,1:0xc13600000+100000,1:0xc13700000+100000,1:0xc13800000+100000,1:0xc13900000+100000,1:0xc13a00000+100000,1:0xc13b00000+100000,1:0xc13c00000+100000,1:0xc13d00000+100000,1:0xc13e00000+100000,1:0xc13f00000+100000,1:0xc14000000+100000,1:0xc14100000+100000,1:0xc14200000+100000,1:0xc14300000+100000,1:0xc14400000+100000,1:0xc14500000+100000,1:0xc14600000+100000,1:0xc14700000+100000,1:0xc14800000+100000,1:0xc14900000+100000,1:0xc14a00000+100000,1:0xc14b00000+100000,1:0xc14c00000+100000,1:0xc14d00000+100000,1:0xc14e00000+100000,1:0xc14f00000+100000,1:0xc15000000+100000,1:0xc15100000+100000,1:0xc15200000+100000,1:0xc15300000+100000,1:0xc15400000+100000,1:0xc15500000+100000,1:0xc15600000+100000,1:0xc15700000+100000,1:0xc15800000+100000,1:0xc15900000+100000,1:0xc15a00000+100000,1:0xc15b00000+100000,1:0xc15c00000+100000,1:0xc15d00000+100000,1:0xc15e00000+100000,1:0xc15f00000+100000,1:0xc16000000+100000,1:0xc16100000+100000,1:0xc16200000+100000,1:0xc16300000+100000,1:0xc16400000+100000,1:0xc16500000+100000,1:0xc16600000+100000,1:0xc16700000+100000,1:0xc16800000+100000,1:0xc16900000+100000,1:0xc16a00000+100000,1:0xc16b00000+100000,1:0xc16c00000+100000,1:0xc16d00000+100000,1:0xc16e00000+100000,1:0xc16f00000+100000,1:0xc17000000+100000,1:0xc17100000+100000,1:0xc17200000+100000,1:0xc17300000+100000,1:0xc17400000+100000,1:0xc17500000+100000,1:0xc17600000+100000,1:0xc17700000+100000,1:0xc17800000+100000,1:0xc17900000+100000,1:0xc17a00000+100000,1:0xc17b00000+100000,1:0xc17c00000+100000,1:0xc17d00000+100000,1:0xc17e00000+100000,1:0xc17f00000+100000,1:0xc18000000+100000,1:0xc18100000+100000,1:0xc18200000+100000,1:0xc18300000+100000,1:0xc18400000+100000,1:0xc18500000+100000,1:0xc18600000+100000,1:0xc18700000+100000,1:0xc18800000+100000,1:0xc18900000+100000,1:0xc18a00000+100000,1:0xc18b00000+100000,1:0xc18c00000+100000,1:0xc18d00000+100000,1:0xc18e00000+100000,1:0xc18f00000+100000,1:0xc19000000+100000,1:0xc19100000+100000,1:0xc19200000+100000,1:0xc19300000+100000,1:0xc19400000+100000,1:0xc19500000+100000,1:0xc19600000+100000,1:0xc19700000+100000,1:0xc19800000+100000,1:0xc19900000+100000,1:0xc19a00000+100000,1:0xc19b00000+100000,1:0xc19c00000+100000,1:0xc19d00000+100000,1:0xc19e00000+100000,1:0xc19f00000+100000,1:0xc1a000000+100000,1:0xc1a100000+100000,1:0xc1a200000+100000,1:0xc1a300000+100000,1:0xc1a400000+100000,1:0xc1a500000+100000,1:0xc1a600000+100000,1:0xc1a700000+100000,1:0xc1a800000+100000,1:0xc1a900000+100000,1:0xc1aa00000+100000,1:0xc1ab00000+100000,1:0xc1ac00000+100000,1:0xc1ad00000+100000,1:0xc1ae00000+100000,1:0xc1af00000+100000,1:0xc1b000000+100000,1:0xc1b100000+100000,1:0xc1b200000+100000,1:0xc1b300000+100000,1:0xc1b400000+100000,1:0xc1b500000+100000,1:0xc1b600000+100000,1:0xc1b700000+100000,1:0xc1b800000+100000,1:0xc1b900000+100000,1:0xc1ba00000+100000,1:0xc1bb00000+100000,1:0xc1bc00000+100000,1:0xc1bd00000+100000,1:0xc1be00000+100000,1:0xc1bf00000+100000,1:0xc1c000000+100000,1:0xc1c100000+100000,1:0xc1c200000+100000,1:0xc1c300000+100000,1:0xc1c400000+100000,1:0xc1c500000+100000,1:0xc1c600000+100000,1:0xc1c700000+100000,1:0xc1c800000+100000,1:0xc1c900000+100000,1:0xc1ca00000+100000,1:0xc1cb00000+100000,1:0xc1cc00000+100000,1:0xc1cd00000+100000,1:0xc1ce00000+100000,1:0xc1cf00000+100000,1:0xc1d000000+100000,1:0xc1d100000+100000,1:0xc1d200000+100000,1:0xc1d300000+100000,1:0xc1d400000+100000,1:0xc1d500000+100000,1:0xc1d600000+100000,1:0xc1d700000+100000,1:0xc1d800000+100000,1:0xc1d900000+100000,1:0xc1da00000+100000,1:0xc1db00000+100000,1:0xc1dc00000+100000,1:0xc1dd00000+100000,1:0xc1de00000+100000,1:0xc1df00000+100000,1:0xc1e000000+100000,1:0xc1e100000+100000,1:0xc1e200000+100000,1:0xc1e300000+100000,1:0xc1e400000+100000,1:0xc1e500000+100000,1:0xc1e600000+100000,1:0xc1e700000+100000,1:0xc1e800000+100000,1:0xc1e900000+100000,1:0xc1ea00000+100000,1:0xc1eb00000+100000,1:0xc1ec00000+100000,1:0xc1ed00000+100000,1:0xc1ee00000+100000,1:0xc1ef00000+100000,1:0xc1f000000+100000,1:0xc1f100000+100000,1:0xc1f200000+100000,1:0xc1f300000+100000,1:0xc1f400000+100000,1:0xc1f500000+100000,1:0xc1f600000+100000,1:0xc1f700000+100000,1:0xc1f800000+100000,1:0xc1f900000+100000,1:0xc1fa00000+100000,1:0xc1fb00000+100000,1:0xc1fc00000+100000,1:0xc1fd00000+100000,1:0xc1fe00000+100000,1:0xc1ff00000+100000,1:0xc20000000+100000,1:0xc20100000+100000,1:0xc20200000+100000,1:0xc20300000+100000,1:0xc20400000+100000,1:0xc20500000+100000,1:0xc20600000+100000,1:0xc20700000+100000,1:0xc20800000+100000,1:0xc20900000+100000,1:0xc20a00000+100000,1:0xc20b00000+100000,1:0xc20c00000+100000,1:0xc20d00000+100000,1:0xc21600000+100000,1:0xc21800000+100000])
2019-06-21 10:50:56.475353 7f462db84d00 10 bluefs _read h 0x5632d7074b80 0x103000~1000 from file(ino 1 size 0x103000 mtime 0.000000 bdev 0 allocated 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000])
2019-06-21 10:50:56.475356 7f462db84d00 20 bluefs _read left 0xfd000 len 0x1000
2019-06-21 10:50:56.475357 7f462db84d00 20 bluefs _read got 4096
2019-06-21 10:50:56.475359 7f462db84d00 10 bluefs _replay 0x103000: txn(seq 332734 len 0xca5 crc 0xa28aacab)
2019-06-21 10:50:56.475366 7f462db84d00 20 bluefs _replay 0x103000: op_file_update file(ino 167 size 0xf4def05 mtime 2019-05-23 16:05:23.261981 bdev 1 allocated f700000 extents [1:0xc0d900000+100000,1:0xc11a00000+100000,1:0xc11b00000+100000,1:0xc11c00000+100000,1:0xc11d00000+100000,1:0xc11e00000+100000,1:0xc11f00000+100000,1:0xc12000000+100000,1:0xc12100000+100000,1:0xc12200000+100000,1:0xc12300000+100000,1:0xc12400000+100000,1:0xc12500000+100000,1:0xc12600000+100000,1:0xc12700000+100000,1:0xc12800000+100000,1:0xc12900000+100000,1:0xc12a00000+100000,1:0xc12b00000+100000,1:0xc12c00000+100000,1:0xc12d00000+100000,1:0xc12e00000+100000,1:0xc12f00000+100000,1:0xc13000000+100000,1:0xc13100000+100000,1:0xc13200000+100000,1:0xc13300000+100000,1:0xc13400000+100000,1:0xc13500000+100000,1:0xc13600000+100000,1:0xc13700000+100000,1:0xc13800000+100000,1:0xc13900000+100000,1:0xc13a00000+100000,1:0xc13b00000+100000,1:0xc13c00000+100000,1:0xc13d00000+100000,1:0xc13e00000+100000,1:0xc13f00000+100000,1:0xc14000000+100000,1:0xc14100000+100000,1:0xc14200000+100000,1:0xc14300000+100000,1:0xc14400000+100000,1:0xc14500000+100000,1:0xc14600000+100000,1:0xc14700000+100000,1:0xc14800000+100000,1:0xc14900000+100000,1:0xc14a00000+100000,1:0xc14b00000+100000,1:0xc14c00000+100000,1:0xc14d00000+100000,1:0xc14e00000+100000,1:0xc14f00000+100000,1:0xc15000000+100000,1:0xc15100000+100000,1:0xc15200000+100000,1:0xc15300000+100000,1:0xc15400000+100000,1:0xc15500000+100000,1:0xc15600000+100000,1:0xc15700000+100000,1:0xc15800000+100000,1:0xc15900000+100000,1:0xc15a00000+100000,1:0xc15b00000+100000,1:0xc15c00000+100000,1:0xc15d00000+100000,1:0xc15e00000+100000,1:0xc15f00000+100000,1:0xc16000000+100000,1:0xc16100000+100000,1:0xc16200000+100000,1:0xc16300000+100000,1:0xc16400000+100000,1:0xc16500000+100000,1:0xc16600000+100000,1:0xc16700000+100000,1:0xc16800000+100000,1:0xc16900000+100000,1:0xc16a00000+100000,1:0xc16b00000+100000,1:0xc16c00000+100000,1:0xc16d00000+100000,1:0xc16e00000+100000,1:0xc16f00000+100000,1:0xc17000000+100000,1:0xc17100000+100000,1:0xc17200000+100000,1:0xc17300000+100000,1:0xc17400000+100000,1:0xc17500000+100000,1:0xc17600000+100000,1:0xc17700000+100000,1:0xc17800000+100000,1:0xc17900000+100000,1:0xc17a00000+100000,1:0xc17b00000+100000,1:0xc17c00000+100000,1:0xc17d00000+100000,1:0xc17e00000+100000,1:0xc17f00000+100000,1:0xc18000000+100000,1:0xc18100000+100000,1:0xc18200000+100000,1:0xc18300000+100000,1:0xc18400000+100000,1:0xc18500000+100000,1:0xc18600000+100000,1:0xc18700000+100000,1:0xc18800000+100000,1:0xc18900000+100000,1:0xc18a00000+100000,1:0xc18b00000+100000,1:0xc18c00000+100000,1:0xc18d00000+100000,1:0xc18e00000+100000,1:0xc18f00000+100000,1:0xc19000000+100000,1:0xc19100000+100000,1:0xc19200000+100000,1:0xc19300000+100000,1:0xc19400000+100000,1:0xc19500000+100000,1:0xc19600000+100000,1:0xc19700000+100000,1:0xc19800000+100000,1:0xc19900000+100000,1:0xc19a00000+100000,1:0xc19b00000+100000,1:0xc19c00000+100000,1:0xc19d00000+100000,1:0xc19e00000+100000,1:0xc19f00000+100000,1:0xc1a000000+100000,1:0xc1a100000+100000,1:0xc1a200000+100000,1:0xc1a300000+100000,1:0xc1a400000+100000,1:0xc1a500000+100000,1:0xc1a600000+100000,1:0xc1a700000+100000,1:0xc1a800000+100000,1:0xc1a900000+100000,1:0xc1aa00000+100000,1:0xc1ab00000+100000,1:0xc1ac00000+100000,1:0xc1ad00000+100000,1:0xc1ae00000+100000,1:0xc1af00000+100000,1:0xc1b000000+100000,1:0xc1b100000+100000,1:0xc1b200000+100000,1:0xc1b300000+100000,1:0xc1b400000+100000,1:0xc1b500000+100000,1:0xc1b600000+100000,1:0xc1b700000+100000,1:0xc1b800000+100000,1:0xc1b900000+100000,1:0xc1ba00000+100000,1:0xc1bb00000+100000,1:0xc1bc00000+100000,1:0xc1bd00000+100000,1:0xc1be00000+100000,1:0xc1bf00000+100000,1:0xc1c000000+100000,1:0xc1c100000+100000,1:0xc1c200000+100000,1:0xc1c300000+100000,1:0xc1c400000+100000,1:0xc1c500000+100000,1:0xc1c600000+100000,1:0xc1c700000+100000,1:0xc1c800000+100000,1:0xc1c900000+100000,1:0xc1ca00000+100000,1:0xc1cb00000+100000,1:0xc1cc00000+100000,1:0xc1cd00000+100000,1:0xc1ce00000+100000,1:0xc1cf00000+100000,1:0xc1d000000+100000,1:0xc1d100000+100000,1:0xc1d200000+100000,1:0xc1d300000+100000,1:0xc1d400000+100000,1:0xc1d500000+100000,1:0xc1d600000+100000,1:0xc1d700000+100000,1:0xc1d800000+100000,1:0xc1d900000+100000,1:0xc1da00000+100000,1:0xc1db00000+100000,1:0xc1dc00000+100000,1:0xc1dd00000+100000,1:0xc1de00000+100000,1:0xc1df00000+100000,1:0xc1e000000+100000,1:0xc1e100000+100000,1:0xc1e200000+100000,1:0xc1e300000+100000,1:0xc1e400000+100000,1:0xc1e500000+100000,1:0xc1e600000+100000,1:0xc1e700000+100000,1:0xc1e800000+100000,1:0xc1e900000+100000,1:0xc1ea00000+100000,1:0xc1eb00000+100000,1:0xc1ec00000+100000,1:0xc1ed00000+100000,1:0xc1ee00000+100000,1:0xc1ef00000+100000,1:0xc1f000000+100000,1:0xc1f100000+100000,1:0xc1f200000+100000,1:0xc1f300000+100000,1:0xc1f400000+100000,1:0xc1f500000+100000,1:0xc1f600000+100000,1:0xc1f700000+100000,1:0xc1f800000+100000,1:0xc1f900000+100000,1:0xc1fa00000+100000,1:0xc1fb00000+100000,1:0xc1fc00000+100000,1:0xc1fd00000+100000,1:0xc1fe00000+100000,1:0xc1ff00000+100000,1:0xc20000000+100000,1:0xc20100000+100000,1:0xc20200000+100000,1:0xc20300000+100000,1:0xc20400000+100000,1:0xc20500000+100000,1:0xc20600000+100000,1:0xc20700000+100000,1:0xc20800000+100000,1:0xc20900000+100000,1:0xc20a00000+100000,1:0xc20b00000+100000,1:0xc20c00000+100000,1:0xc20d00000+100000,1:0xc21600000+100000,1:0xc21800000+100000])
2019-06-21 10:50:56.475433 7f462db84d00 10 bluefs _read h 0x5632d7074b80 0x104000~1000 from file(ino 1 size 0x104000 mtime 0.000000 bdev 0 allocated 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000])
2019-06-21 10:50:56.475436 7f462db84d00 20 bluefs _read left 0xfc000 len 0x1000
2019-06-21 10:50:56.475438 7f462db84d00 20 bluefs _read got 4096
2019-06-21 10:50:56.475440 7f462db84d00 10 bluefs _replay 0x104000: txn(seq 332735 len 0xca5 crc 0x4715a5c6)
The entire file as 17M and I can send with necessary ,
Saulo Augusto Silva
Em sex, 21 de jun de 2019 às 06:42, Igor Fedotov <ifedotov@xxxxxxx> escreveu:
Hi Saulo,
looks like disk I/O error.
Will you set debug_bluefs to 20 and collect the log, then share a few lines prior to the assertion?
Checking smartctl output might be a good idea too.
Thanks,
Igor
On 6/21/2019 11:30 AM, Saulo Silva wrote:
Hi,
After a power failure all OSD´s from a pool are fail with the following error :
-5> 2019-06-20 13:32:58.886299 7f146bcb2d00 4 rocksdb: [/home/abuild/rpmbuild/BUILD/ceph-12.2.12-573-g67074fa839/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-003373 succeeded,manifest_file_number is 3373, next_file_number is 3598, last_sequence is 319489940, log_number is 0,prev_log_number is 0,max_column_family is 0
-4> 2019-06-20 13:32:58.886330 7f146bcb2d00 4 rocksdb: [/home/abuild/rpmbuild/BUILD/ceph-12.2.12-573-g67074fa839/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 3594
-3> 2019-06-20 13:32:58.886401 7f146bcb2d00 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1561048378886391, "job": 1, "event": "recovery_started", "log_files": [3592, 3594]}
-2> 2019-06-20 13:32:58.886407 7f146bcb2d00 4 rocksdb: [/home/abuild/rpmbuild/BUILD/ceph-12.2.12-573-g67074fa839/src/rocksdb/db/db_impl_open.cc:482] Recovering log #3592 mode 0
-1> 2019-06-20 13:33:06.629066 7f146bcb2d00 4 rocksdb: [/home/abuild/rpmbuild/BUILD/ceph-12.2.12-573-g67074fa839/src/rocksdb/db/db_impl_open.cc:482] Recovering log #3594 mode 0
0> 2019-06-20 13:33:10.086512 7f146bcb2d00 -1 /home/abuild/rpmbuild/BUILD/ceph-12.2.12-573-g67074fa839/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, uint64_t, size_t, ceph::bufferlist*, char*)' thread 7f146bcb2d00 time 2019-06-20 13:33:10.073021
/home/abuild/rpmbuild/BUILD/ceph-12.2.12-573-g67074fa839/src/os/bluestore/BlueFS.cc: 996: FAILED assert(r == 0)
All of OSD only read 2 log and return the error .
Is it possible to delete the rocksdb log and start the OSD again?
Best Regards,
Saulo Augusto Silva
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
