Hi,
I have a 3 node cluster of mimic with 9 osds (3 osds on each node).
I use this cluster to test integration of an application with S3 api.
The problem is that after a few days all OSD starts filling up with
bluestore logs and goes down and out one by one!
I cannot stop the logs and I cannot find the setting to fix this leakage,
this should be a leakage in logs because it's not logical to fill up all
OSD with bluefs logs.
This is an example of logs which is being repeated in bluestore logs:
[root@server2 ~]# ceph-bluestore-tool --command bluefs-log-dump --path
/var/lib/ceph/osd/ceph-5
.
.
. 0x40d000: op_file_update file(ino 30 size 0x7a713 mtime 2020-04-18
15:54:29.056488 bdev 1 allocated 80000 extents
[1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000])
0x40e000: txn(seq 1156100 len 0x78 crc 0x3e1c626f)
0x40e000: op_file_update file(ino 30 size 0x7a72a mtime 2020-04-18
15:54:30.057828 bdev 1 allocated 80000 extents
[1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000])
0x40f000: txn(seq 1156101 len 0x78 crc 0xc1f9ec5f)
0x40f000: op_file_update file(ino 30 size 0x7a741 mtime 2020-04-18
15:54:31.059252 bdev 1 allocated 80000 extents
[1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000])
*** Caught signal (Segmentation fault) **
in thread 7f4108cfe600 thread_name:ceph-bluestore-
ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic
(stable)
1: (()+0xf5f0) [0x7f40fd47f5f0]
2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned
long, unsigned long, ceph::buffer::list*, char*)+0x3bf) [0x55603a1ebb3f]
3: (BlueFS::_replay(bool, bool)+0x214) [0x55603a1f6654]
4: (BlueFS::log_dump(CephContext*, std::string const&,
std::vector<std::string, std::allocator<std::string> > const&)+0x3b)
[0x55603a1fad3b]
5: (log_dump(CephContext*, std::string const&, std::vector<std::string,
std::allocator<std::string> > const&)+0x64) [0x55603a1da764]
6: (main()+0x2f3e) [0x55603a105e3e]
7: (__libc_start_main()+0xf5) [0x7f40fbe51505]
8: (()+0x23115f) [0x55603a1d915f]
2020-04-18 21:08:25.410 7f4108cfe600 -1 *** Caught signal (Segmentation
fault) **
in thread 7f4108cfe600 thread_name:ceph-bluestore-
.
.
.
And this is the output of daemonperf for one of remaining (filling up!)
OSDs:
[root@server2 ~]# ceph daemonperf osd.5
------bluefs------- ------------bluestore-------------
----------osd-----------
jlen j wal sst |fl_l k_l io_l th_l s_l c_l r_l |ops wr rd l
rop |
81M 0 0 0 | 0 0 0 0 0 0 0 | 0 0 0 0
0
81M 4.0k 1.8k 0 | 0 0 0 0 0 0 0 | 0 0 0 0
0
81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0
0
81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0
0
81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0
0
^C[root@server2 ~]#
And this is the enospc logs when trying to start OSD:
[root@server2 ~]# journalctl -u -f ceph-osd@4.service
Failed to add match 'ceph-osd@4.service': Invalid argument
Failed to add filters: Invalid argument
[root@server2 ~]# journalctl -f -u ceph-osd@4.service
-- Logs begin at Sat 2020-04-18 15:38:15 +0430. --
Apr 18 21:18:22 server2 ceph-osd[17485]: 22: (()+0x378c10) [0x557348ebac10]
Apr 18 21:18:22 server2 ceph-osd[17485]: NOTE: a copy of the executable, or
`objdump -rdS <executable>` is needed to interpret this.
Apr 18 21:18:22 server2 systemd[1]: ceph-osd@4.service: main process
exited, code=killed, status=6/ABRT
Apr 18 21:18:22 server2 systemd[1]: Unit ceph-osd@4.service entered failed
state.
Apr 18 21:18:22 server2 systemd[1]: ceph-osd@4.service failed.
Apr 18 21:18:42 server2 systemd[1]: ceph-osd@4.service holdoff time over,
scheduling restart.
Apr 18 21:18:42 server2 systemd[1]: Stopped Ceph object storage daemon
osd.4.
Apr 18 21:18:42 server2 systemd[1]: Starting Ceph object storage daemon
osd.4...
Apr 18 21:18:42 server2 systemd[1]: Started Ceph object storage daemon
osd.4.
Apr 18 21:18:43 server2 ceph-osd[17777]: starting osd.4 at - osd_data
/var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal
Apr 18 21:24:37 server2 ceph-osd[17777]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289
Apr 18 21:24:37 server2 ceph-osd[17777]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
1704: FAILED assert(0 == "bluefs enospc")
Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090
7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne
Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090
7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length:
0x794de9
Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
(5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x14b) [0x7f43c074987b]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07]
Apr 18 21:24:37 server2 ceph-osd[17777]: 3:
(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
long)+0x1ac6) [0x55ad52029266]
Apr 18 21:24:37 server2 ceph-osd[17777]: 4:
(BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
Apr 18 21:24:37 server2 ceph-osd[17777]: 5:
(rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
(rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
(rocksdb::BuildTable(std::string const&, rocksdb::Env*,
rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::TableCache*,
rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
rocksdb::InternalKeyComparator const&,
std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
unsigned int, std::string const&, std::vector<unsigned long,
std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
rocksdb::TableProperties*, int, unsigned long, unsigned long,
rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
(rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
[0x55ad520b2dcb]
Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
bool)+0xa59) [0x55ad520b3d09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
bool)+0x689) [0x55ad520b4ab9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
(rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
[0x55ad520b62e2]
Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
(RocksDBStore::do_open(std::ostream&, bool,
std::vector<KeyValueDB::ColumnFamily,
std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool,
bool)+0xcf4) [0x55ad51f527a4]
Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool,
bool)+0x4e9) [0x55ad51f828f9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339)
[0x55ad51b1cf09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2)
[0x55ad51a00d52]
Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5)
[0x7f43bc6c7505]
Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10]
Apr 18 21:24:37 server2 ceph-osd[17777]: *** Caught signal (Aborted) **
Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80
thread_name:ceph-osd
Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.094
7f43c936bb80 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289
Apr 18 21:24:37 server2 ceph-osd[17777]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
1704: FAILED assert(0 == "bluefs enospc")
Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
(5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x14b) [0x7f43c074987b]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07]
Apr 18 21:24:37 server2 ceph-osd[17777]: 3:
(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
long)+0x1ac6) [0x55ad52029266]
Apr 18 21:24:37 server2 ceph-osd[17777]: 4:
(BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
Apr 18 21:24:37 server2 ceph-osd[17777]: 5:
(rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
(rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
(rocksdb::BuildTable(std::string const&, rocksdb::Env*,
rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::TableCache*,
rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
rocksdb::InternalKeyComparator const&,
std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
unsigned int, std::string const&, std::vector<unsigned long,
std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
rocksdb::TableProperties*, int, unsigned long, unsigned long,
rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
(rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
[0x55ad520b2dcb]
Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
bool)+0xa59) [0x55ad520b3d09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
bool)+0x689) [0x55ad520b4ab9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
(rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
[0x55ad520b62e2]
Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
(RocksDBStore::do_open(std::ostream&, bool,
std::vector<KeyValueDB::ColumnFamily,
std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool,
bool)+0xcf4) [0x55ad51f527a4]
Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool,
bool)+0x4e9) [0x55ad51f828f9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339)
[0x55ad51b1cf09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2)
[0x55ad51a00d52]
Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5)
[0x7f43bc6c7505]
Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10]
Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
(5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37)
[0x7f43bc6db337]
Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28]
Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x248) [0x7f43c0749978]
Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07]
Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
long)+0x1ac6) [0x55ad52029266]
Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
(BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
(rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
(rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
(rocksdb::BuildTable(std::string const&, rocksdb::Env*,
rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::TableCache*,
rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
rocksdb::InternalKeyComparator const&,
std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
unsigned int, std::string const&, std::vector<unsigned long,
std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
rocksdb::TableProperties*, int, unsigned long, unsigned long,
rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
(rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
[0x55ad520b2dcb]
Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
bool)+0xa59) [0x55ad520b3d09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 14:
(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
bool)+0x689) [0x55ad520b4ab9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 15:
(rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
[0x55ad520b62e2]
Apr 18 21:24:37 server2 ceph-osd[17777]: 16:
(RocksDBStore::do_open(std::ostream&, bool,
std::vector<KeyValueDB::ColumnFamily,
std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool,
bool)+0xcf4) [0x55ad51f527a4]
Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool,
bool)+0x4e9) [0x55ad51f828f9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339)
[0x55ad51b1cf09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2)
[0x55ad51a00d52]
Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5)
[0x7f43bc6c7505]
Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.099
7f43c936bb80 -1 *** Caught signal (Aborted) **
Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80
thread_name:ceph-osd
Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
(5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37)
[0x7f43bc6db337]
Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28]
Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x248) [0x7f43c0749978]
Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07]
Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
long)+0x1ac6) [0x55ad52029266]
Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
(BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
(rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
(rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
(rocksdb::BuildTable(std::string const&, rocksdb::Env*,
rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::TableCache*,
rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
rocksdb::InternalKeyComparator const&,
std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
unsigned int, std::string const&, std::vector<unsigned long,
std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
rocksdb::TableProperties*, int, unsigned long, unsigned long,
rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
(rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
[0x55ad520b2dcb]
Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
bool)+0xa59) [0x55ad520b3d09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 14:
(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
bool)+0x689) [0x55ad520b4ab9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 15:
(rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
[0x55ad520b62e2]
Apr 18 21:24:37 server2 ceph-osd[17777]: 16:
(RocksDBStore::do_open(std::ostream&, bool,
std::vector<KeyValueDB::ColumnFamily,
std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool,
bool)+0xcf4) [0x55ad51f527a4]
Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool,
bool)+0x4e9) [0x55ad51f828f9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339)
[0x55ad51b1cf09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2)
[0x55ad51a00d52]
Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5)
[0x7f43bc6c7505]
Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10]
Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or
`objdump -rdS <executable>` is needed to interpret this.
Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090
7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne
Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090
7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length:
0x794de9
Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094
7f43c936bb80 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289
Apr 18 21:24:37 server2 ceph-osd[17777]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
1704: FAILED assert(0 == "bluefs enospc")
Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
(5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x14b) [0x7f43c074987b]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07]
Apr 18 21:24:37 server2 ceph-osd[17777]: 3:
(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
long)+0x1ac6) [0x55ad52029266]
Apr 18 21:24:37 server2 ceph-osd[17777]: 4:
(BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
Apr 18 21:24:37 server2 ceph-osd[17777]: 5:
(rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
(rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
(rocksdb::BuildTable(std::string const&, rocksdb::Env*,
rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::TableCache*,
rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
rocksdb::InternalKeyComparator const&,
std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
unsigned int, std::string const&, std::vector<unsigned long,
std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
rocksdb::TableProperties*, int, unsigned long, unsigned long,
rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
(rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
[0x55ad520b2dcb]
Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
bool)+0xa59) [0x55ad520b3d09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
bool)+0x689) [0x55ad520b4ab9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
(rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
[0x55ad520b62e2]
Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
(RocksDBStore::do_open(std::ostream&, bool,
std::vector<KeyValueDB::ColumnFamily,
std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool,
bool)+0xcf4) [0x55ad51f527a4]
Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool,
bool)+0x4e9) [0x55ad51f828f9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339)
[0x55ad51b1cf09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2)
[0x55ad51a00d52]
Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5)
[0x7f43bc6c7505]
Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10]
Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099
7f43c936bb80 -1 *** Caught signal (Aborted) **
Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80
thread_name:ceph-osd
Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
(5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37)
[0x7f43bc6db337]
Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28]
Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x248) [0x7f43c0749978]
Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07]
Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
long)+0x1ac6) [0x55ad52029266]
Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
(BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
(rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
(rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
(rocksdb::BuildTable(std::string const&, rocksdb::Env*,
rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::TableCache*,
rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
rocksdb::InternalKeyComparator const&,
std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
unsigned int, std::string const&, std::vector<unsigned long,
std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
rocksdb::TableProperties*, int, unsigned long, unsigned long,
rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
(rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
[0x55ad520b2dcb]
Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
bool)+0xa59) [0x55ad520b3d09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 14:
(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
bool)+0x689) [0x55ad520b4ab9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 15:
(rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
[0x55ad520b62e2]
Apr 18 21:24:37 server2 ceph-osd[17777]: 16:
(RocksDBStore::do_open(std::ostream&, bool,
std::vector<KeyValueDB::ColumnFamily,
std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool,
bool)+0xcf4) [0x55ad51f527a4]
Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool,
bool)+0x4e9) [0x55ad51f828f9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339)
[0x55ad51b1cf09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2)
[0x55ad51a00d52]
Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5)
[0x7f43bc6c7505]
Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10]
Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or
`objdump -rdS <executable>` is needed to interpret this.
Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090
7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne
Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090
7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length:
0x794de9
Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094
7f43c936bb80 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289
Apr 18 21:24:37 server2 ceph-osd[17777]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
1704: FAILED assert(0 == "bluefs enospc")
Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
(5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x14b) [0x7f43c074987b]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07]
Apr 18 21:24:37 server2 ceph-osd[17777]: 3:
(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
long)+0x1ac6) [0x55ad52029266]
Apr 18 21:24:37 server2 ceph-osd[17777]: 4:
(BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
Apr 18 21:24:37 server2 ceph-osd[17777]: 5:
(rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
(rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
(rocksdb::BuildTable(std::string const&, rocksdb::Env*,
rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::TableCache*,
rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
rocksdb::InternalKeyComparator const&,
std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
unsigned int, std::string const&, std::vector<unsigned long,
std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
rocksdb::TableProperties*, int, unsigned long, unsigned long,
rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
(rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
[0x55ad520b2dcb]
Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
bool)+0xa59) [0x55ad520b3d09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
bool)+0x689) [0x55ad520b4ab9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
(rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
[0x55ad520b62e2]
Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
(RocksDBStore::do_open(std::ostream&, bool,
std::vector<KeyValueDB::ColumnFamily,
std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool,
bool)+0xcf4) [0x55ad51f527a4]
Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool,
bool)+0x4e9) [0x55ad51f828f9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339)
[0x55ad51b1cf09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2)
[0x55ad51a00d52]
Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5)
[0x7f43bc6c7505]
Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10]
Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099
7f43c936bb80 -1 *** Caught signal (Aborted) **
Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80
thread_name:ceph-osd
Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
(5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37)
[0x7f43bc6db337]
Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28]
Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x248) [0x7f43c0749978]
Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07]
Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
long)+0x1ac6) [0x55ad52029266]
Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
(BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
(rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
(rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
(rocksdb::BuildTable(std::string const&, rocksdb::Env*,
rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::TableCache*,
rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
rocksdb::InternalKeyComparator const&,
std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
unsigned int, std::string const&, std::vector<unsigned long,
std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
rocksdb::TableProperties*, int, unsigned long, unsigned long,
rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
(rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
[0x55ad520b2dcb]
Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
bool)+0xa59) [0x55ad520b3d09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 14:
(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
bool)+0x689) [0x55ad520b4ab9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 15:
(rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
[0x55ad520b62e2]
Apr 18 21:24:37 server2 ceph-osd[17777]: 16:
(RocksDBStore::do_open(std::ostream&, bool,
std::vector<KeyValueDB::ColumnFamily,
std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool,
bool)+0xcf4) [0x55ad51f527a4]
Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool,
bool)+0x4e9) [0x55ad51f828f9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339)
[0x55ad51b1cf09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2)
[0x55ad51a00d52]
Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5)
[0x7f43bc6c7505]
Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10]
Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or
`objdump -rdS <executable>` is needed to interpret this.
Apr 18 21:24:37 server2 systemd[1]: ceph-osd@4.service: main process
exited, code=killed, status=6/ABRT
Apr 18 21:24:37 server2 systemd[1]: Unit ceph-osd@4.service entered failed
state.
Apr 18 21:24:37 server2 systemd[1]: ceph-osd@4.service failed.
Apr 18 21:24:57 server2 systemd[1]: ceph-osd@4.service holdoff time over,
scheduling restart.
Apr 18 21:24:57 server2 systemd[1]: Stopped Ceph object storage daemon
osd.4.
Apr 18 21:24:57 server2 systemd[1]: Starting Ceph object storage daemon
osd.4...
Apr 18 21:24:57 server2 systemd[1]: Started Ceph object storage daemon
osd.4.
and some more data:
[root@server1 ~]# ceph df detail
GLOBAL:
SIZE AVAIL RAW USED %RAW USED OBJECTS
38 GiB 35 GiB 2.6 GiB 6.83 303
POOLS:
NAME ID QUOTA OBJECTS QUOTA BYTES
USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE
RAW USED
.rgw.root 18 N/A N/A
1.1 KiB 0 50 GiB 4 4 63 B 4
B 1.1 KiB
default.rgw.meta 19 N/A N/A
4.1 KiB 0 50 GiB 22 22 1.5 KiB 182
B 9.0 KiB
default.rgw.log 20 N/A N/A
0 B 0 50 GiB 107 107 307 KiB 205
KiB 0 B
default.rgw.control 21 N/A N/A
0 B 0 50 GiB 5 5 0 B 0
B 0 B
default.rgw.buckets.index 22 N/A N/A
0 B 0 50 GiB 5 5 501 B 60
B 0 B
default.rgw.buckets.data 23 N/A N/A
377 MiB 0.39 50 GiB 160 160 385 B 1.1
KiB 603 MiB
[root@server1 ~]#
[root@server1 ~]# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE
VAR PGS TYPE NAME
-1 0.16727 - 0 B 0 B 0 B 0 B 0 B 0 B 0
0 - root default
-3 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0
0 - host server1
0 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0
0 0 osd.0
1 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0
0 0 osd.1
2 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0
0 0 osd.2
-5 0.05576 - 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 0
0 - host server2
3 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0
0 0 osd.3
4 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0
0 0 osd.4
5 hdd 0.01859 1.00000 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 7.11
1.04 99 osd.5
-7 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0
0 - host server3
6 hdd 0.01859 1.00000 19 GiB 1.2 GiB 249 MiB 3 KiB 1024 MiB 18 GiB 6.55
0.96 78 osd.6
7 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0
0 0 osd.7
8 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0
0 0 osd.8
TOTAL 38 GiB 2.6 GiB 610 MiB 6 KiB 2.0 GiB 35 GiB 6.83
MIN/MAX VAR: 0/1.04 STDDEV: 5.58
[root@server1 ~]#
I'm kind of newbie to ceph, so any help or hint would be appreciated.
Did I hit a bug or something is wrong with my configuration?
Thanks a lot,
Khodayar
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
