I’ve also had this problem. In my case it was because I’d added a number of OSDs recently and the cluster was spending a lot of time rebalancing. Is there a reason it might be doing that in yours? > On Apr 18, 2020, at 12:59 PM, Khodayar Doustar <doustar@xxxxxxxxxxxx> wrote: > > Hi, > > I have a 3 node cluster of mimic with 9 osds (3 osds on each node). > I use this cluster to test integration of an application with S3 api. > > The problem is that after a few days all OSD starts filling up with > bluestore logs and goes down and out one by one! > I cannot stop the logs and I cannot find the setting to fix this leakage, > this should be a leakage in logs because it's not logical to fill up all > OSD with bluefs logs. > > This is an example of logs which is being repeated in bluestore logs: > > [root@server2 ~]# ceph-bluestore-tool --command bluefs-log-dump --path > /var/lib/ceph/osd/ceph-5 > . > . > . 0x40d000: op_file_update file(ino 30 size 0x7a713 mtime 2020-04-18 > 15:54:29.056488 bdev 1 allocated 80000 extents > [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) > 0x40e000: txn(seq 1156100 len 0x78 crc 0x3e1c626f) > 0x40e000: op_file_update file(ino 30 size 0x7a72a mtime 2020-04-18 > 15:54:30.057828 bdev 1 allocated 80000 extents > [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) > 0x40f000: txn(seq 1156101 len 0x78 crc 0xc1f9ec5f) > 0x40f000: op_file_update file(ino 30 size 0x7a741 mtime 2020-04-18 > 15:54:31.059252 bdev 1 allocated 80000 extents > [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) > *** Caught signal (Segmentation fault) ** > in thread 7f4108cfe600 thread_name:ceph-bluestore- > ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic > (stable) > 1: (()+0xf5f0) [0x7f40fd47f5f0] > 2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned > long, unsigned long, ceph::buffer::list*, char*)+0x3bf) [0x55603a1ebb3f] > 3: (BlueFS::_replay(bool, bool)+0x214) [0x55603a1f6654] > 4: (BlueFS::log_dump(CephContext*, std::string const&, > std::vector<std::string, std::allocator<std::string> > const&)+0x3b) > [0x55603a1fad3b] > 5: (log_dump(CephContext*, std::string const&, std::vector<std::string, > std::allocator<std::string> > const&)+0x64) [0x55603a1da764] > 6: (main()+0x2f3e) [0x55603a105e3e] > 7: (__libc_start_main()+0xf5) [0x7f40fbe51505] > 8: (()+0x23115f) [0x55603a1d915f] > 2020-04-18 21:08:25.410 7f4108cfe600 -1 *** Caught signal (Segmentation > fault) ** > in thread 7f4108cfe600 thread_name:ceph-bluestore- > . > . > . > > > And this is the output of daemonperf for one of remaining (filling up!) > OSDs: > > [root@server2 ~]# ceph daemonperf osd.5 > ------bluefs------- ------------bluestore------------- > ----------osd----------- > jlen j wal sst |fl_l k_l io_l th_l s_l c_l r_l |ops wr rd l > rop | > 81M 0 0 0 | 0 0 0 0 0 0 0 | 0 0 0 0 > 0 > 81M 4.0k 1.8k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 > 0 > 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 > 0 > 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 > 0 > 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 > 0 > ^C[root@server2 ~]# > > And this is the enospc logs when trying to start OSD: > > [root@server2 ~]# journalctl -u -f ceph-osd@4.service > Failed to add match 'ceph-osd@4.service': Invalid argument > Failed to add filters: Invalid argument > [root@server2 ~]# journalctl -f -u ceph-osd@4.service > -- Logs begin at Sat 2020-04-18 15:38:15 +0430. -- > Apr 18 21:18:22 server2 ceph-osd[17485]: 22: (()+0x378c10) [0x557348ebac10] > Apr 18 21:18:22 server2 ceph-osd[17485]: NOTE: a copy of the executable, or > `objdump -rdS <executable>` is needed to interpret this. > Apr 18 21:18:22 server2 systemd[1]: ceph-osd@4.service: main process > exited, code=killed, status=6/ABRT > Apr 18 21:18:22 server2 systemd[1]: Unit ceph-osd@4.service entered failed > state. > Apr 18 21:18:22 server2 systemd[1]: ceph-osd@4.service failed. > Apr 18 21:18:42 server2 systemd[1]: ceph-osd@4.service holdoff time over, > scheduling restart. > Apr 18 21:18:42 server2 systemd[1]: Stopped Ceph object storage daemon > osd.4. > Apr 18 21:18:42 server2 systemd[1]: Starting Ceph object storage daemon > osd.4... > Apr 18 21:18:42 server2 systemd[1]: Started Ceph object storage daemon > osd.4. > Apr 18 21:18:43 server2 ceph-osd[17777]: starting osd.4 at - osd_data > /var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal > Apr 18 21:24:37 server2 ceph-osd[17777]: > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: > In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, > uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 > Apr 18 21:24:37 server2 ceph-osd[17777]: > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: > 1704: FAILED assert(0 == "bluefs enospc") > Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090 > 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne > Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090 > 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: > 0x794de9 > Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 > (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] > Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] > Apr 18 21:24:37 server2 ceph-osd[17777]: 3: > (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned > long)+0x1ac6) [0x55ad52029266] > Apr 18 21:24:37 server2 ceph-osd[17777]: 4: > (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] > Apr 18 21:24:37 server2 ceph-osd[17777]: 5: > (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] > Apr 18 21:24:37 server2 ceph-osd[17777]: 6: > (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] > Apr 18 21:24:37 server2 ceph-osd[17777]: 7: > (rocksdb::BuildTable(std::string const&, rocksdb::Env*, > rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, > rocksdb::EnvOptions const&, rocksdb::TableCache*, > rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, > std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, > rocksdb::InternalKeyComparator const&, > std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, > unsigned int, std::string const&, std::vector<unsigned long, > std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, > rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, > rocksdb::TableProperties*, int, unsigned long, unsigned long, > rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] > Apr 18 21:24:37 server2 ceph-osd[17777]: 8: > (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, > rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] > Apr 18 21:24:37 server2 ceph-osd[17777]: 9: > (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, > std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) > [0x55ad520b2dcb] > Apr 18 21:24:37 server2 ceph-osd[17777]: 10: > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, > bool)+0xa59) [0x55ad520b3d09] > Apr 18 21:24:37 server2 ceph-osd[17777]: 11: > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, > std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > std::vector<rocksdb::ColumnFamilyHandle*, > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, > bool)+0x689) [0x55ad520b4ab9] > Apr 18 21:24:37 server2 ceph-osd[17777]: 12: > (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, > std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > std::vector<rocksdb::ColumnFamilyHandle*, > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) > [0x55ad520b62e2] > Apr 18 21:24:37 server2 ceph-osd[17777]: 13: > (RocksDBStore::do_open(std::ostream&, bool, > std::vector<KeyValueDB::ColumnFamily, > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] > Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, > bool)+0xcf4) [0x55ad51f527a4] > Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, > bool)+0x4e9) [0x55ad51f828f9] > Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) > [0x55ad51b1cf09] > Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) > [0x55ad51a00d52] > Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) > [0x7f43bc6c7505] > Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] > Apr 18 21:24:37 server2 ceph-osd[17777]: *** Caught signal (Aborted) ** > Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 > thread_name:ceph-osd > Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.094 > 7f43c936bb80 -1 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: > In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, > uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 > Apr 18 21:24:37 server2 ceph-osd[17777]: > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: > 1704: FAILED assert(0 == "bluefs enospc") > Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 > (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] > Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] > Apr 18 21:24:37 server2 ceph-osd[17777]: 3: > (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned > long)+0x1ac6) [0x55ad52029266] > Apr 18 21:24:37 server2 ceph-osd[17777]: 4: > (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] > Apr 18 21:24:37 server2 ceph-osd[17777]: 5: > (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] > Apr 18 21:24:37 server2 ceph-osd[17777]: 6: > (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] > Apr 18 21:24:37 server2 ceph-osd[17777]: 7: > (rocksdb::BuildTable(std::string const&, rocksdb::Env*, > rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, > rocksdb::EnvOptions const&, rocksdb::TableCache*, > rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, > std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, > rocksdb::InternalKeyComparator const&, > std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, > unsigned int, std::string const&, std::vector<unsigned long, > std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, > rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, > rocksdb::TableProperties*, int, unsigned long, unsigned long, > rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] > Apr 18 21:24:37 server2 ceph-osd[17777]: 8: > (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, > rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] > Apr 18 21:24:37 server2 ceph-osd[17777]: 9: > (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, > std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) > [0x55ad520b2dcb] > Apr 18 21:24:37 server2 ceph-osd[17777]: 10: > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, > bool)+0xa59) [0x55ad520b3d09] > Apr 18 21:24:37 server2 ceph-osd[17777]: 11: > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, > std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > std::vector<rocksdb::ColumnFamilyHandle*, > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, > bool)+0x689) [0x55ad520b4ab9] > Apr 18 21:24:37 server2 ceph-osd[17777]: 12: > (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, > std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > std::vector<rocksdb::ColumnFamilyHandle*, > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) > [0x55ad520b62e2] > Apr 18 21:24:37 server2 ceph-osd[17777]: 13: > (RocksDBStore::do_open(std::ostream&, bool, > std::vector<KeyValueDB::ColumnFamily, > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] > Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, > bool)+0xcf4) [0x55ad51f527a4] > Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, > bool)+0x4e9) [0x55ad51f828f9] > Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) > [0x55ad51b1cf09] > Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) > [0x55ad51a00d52] > Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) > [0x7f43bc6c7505] > Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] > Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 > (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] > Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) > [0x7f43bc6db337] > Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] > Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x248) [0x7f43c0749978] > Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] > Apr 18 21:24:37 server2 ceph-osd[17777]: 6: > (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned > long)+0x1ac6) [0x55ad52029266] > Apr 18 21:24:37 server2 ceph-osd[17777]: 7: > (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] > Apr 18 21:24:37 server2 ceph-osd[17777]: 8: > (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] > Apr 18 21:24:37 server2 ceph-osd[17777]: 9: > (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] > Apr 18 21:24:37 server2 ceph-osd[17777]: 10: > (rocksdb::BuildTable(std::string const&, rocksdb::Env*, > rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, > rocksdb::EnvOptions const&, rocksdb::TableCache*, > rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, > std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, > rocksdb::InternalKeyComparator const&, > std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, > unsigned int, std::string const&, std::vector<unsigned long, > std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, > rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, > rocksdb::TableProperties*, int, unsigned long, unsigned long, > rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] > Apr 18 21:24:37 server2 ceph-osd[17777]: 11: > (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, > rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] > Apr 18 21:24:37 server2 ceph-osd[17777]: 12: > (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, > std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) > [0x55ad520b2dcb] > Apr 18 21:24:37 server2 ceph-osd[17777]: 13: > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, > bool)+0xa59) [0x55ad520b3d09] > Apr 18 21:24:37 server2 ceph-osd[17777]: 14: > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, > std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > std::vector<rocksdb::ColumnFamilyHandle*, > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, > bool)+0x689) [0x55ad520b4ab9] > Apr 18 21:24:37 server2 ceph-osd[17777]: 15: > (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, > std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > std::vector<rocksdb::ColumnFamilyHandle*, > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) > [0x55ad520b62e2] > Apr 18 21:24:37 server2 ceph-osd[17777]: 16: > (RocksDBStore::do_open(std::ostream&, bool, > std::vector<KeyValueDB::ColumnFamily, > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] > Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, > bool)+0xcf4) [0x55ad51f527a4] > Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, > bool)+0x4e9) [0x55ad51f828f9] > Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) > [0x55ad51b1cf09] > Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) > [0x55ad51a00d52] > Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) > [0x7f43bc6c7505] > Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] > Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.099 > 7f43c936bb80 -1 *** Caught signal (Aborted) ** > Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 > thread_name:ceph-osd > Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 > (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] > Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) > [0x7f43bc6db337] > Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] > Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x248) [0x7f43c0749978] > Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] > Apr 18 21:24:37 server2 ceph-osd[17777]: 6: > (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned > long)+0x1ac6) [0x55ad52029266] > Apr 18 21:24:37 server2 ceph-osd[17777]: 7: > (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] > Apr 18 21:24:37 server2 ceph-osd[17777]: 8: > (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] > Apr 18 21:24:37 server2 ceph-osd[17777]: 9: > (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] > Apr 18 21:24:37 server2 ceph-osd[17777]: 10: > (rocksdb::BuildTable(std::string const&, rocksdb::Env*, > rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, > rocksdb::EnvOptions const&, rocksdb::TableCache*, > rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, > std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, > rocksdb::InternalKeyComparator const&, > std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, > unsigned int, std::string const&, std::vector<unsigned long, > std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, > rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, > rocksdb::TableProperties*, int, unsigned long, unsigned long, > rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] > Apr 18 21:24:37 server2 ceph-osd[17777]: 11: > (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, > rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] > Apr 18 21:24:37 server2 ceph-osd[17777]: 12: > (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, > std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) > [0x55ad520b2dcb] > Apr 18 21:24:37 server2 ceph-osd[17777]: 13: > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, > bool)+0xa59) [0x55ad520b3d09] > Apr 18 21:24:37 server2 ceph-osd[17777]: 14: > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, > std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > std::vector<rocksdb::ColumnFamilyHandle*, > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, > bool)+0x689) [0x55ad520b4ab9] > Apr 18 21:24:37 server2 ceph-osd[17777]: 15: > (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, > std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > std::vector<rocksdb::ColumnFamilyHandle*, > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) > [0x55ad520b62e2] > Apr 18 21:24:37 server2 ceph-osd[17777]: 16: > (RocksDBStore::do_open(std::ostream&, bool, > std::vector<KeyValueDB::ColumnFamily, > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] > Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, > bool)+0xcf4) [0x55ad51f527a4] > Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, > bool)+0x4e9) [0x55ad51f828f9] > Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) > [0x55ad51b1cf09] > Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) > [0x55ad51a00d52] > Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) > [0x7f43bc6c7505] > Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] > Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or > `objdump -rdS <executable>` is needed to interpret this. > Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 > 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne > Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 > 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: > 0x794de9 > Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094 > 7f43c936bb80 -1 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: > In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, > uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 > Apr 18 21:24:37 server2 ceph-osd[17777]: > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: > 1704: FAILED assert(0 == "bluefs enospc") > Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 > (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] > Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] > Apr 18 21:24:37 server2 ceph-osd[17777]: 3: > (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned > long)+0x1ac6) [0x55ad52029266] > Apr 18 21:24:37 server2 ceph-osd[17777]: 4: > (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] > Apr 18 21:24:37 server2 ceph-osd[17777]: 5: > (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] > Apr 18 21:24:37 server2 ceph-osd[17777]: 6: > (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] > Apr 18 21:24:37 server2 ceph-osd[17777]: 7: > (rocksdb::BuildTable(std::string const&, rocksdb::Env*, > rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, > rocksdb::EnvOptions const&, rocksdb::TableCache*, > rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, > std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, > rocksdb::InternalKeyComparator const&, > std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, > unsigned int, std::string const&, std::vector<unsigned long, > std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, > rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, > rocksdb::TableProperties*, int, unsigned long, unsigned long, > rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] > Apr 18 21:24:37 server2 ceph-osd[17777]: 8: > (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, > rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] > Apr 18 21:24:37 server2 ceph-osd[17777]: 9: > (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, > std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) > [0x55ad520b2dcb] > Apr 18 21:24:37 server2 ceph-osd[17777]: 10: > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, > bool)+0xa59) [0x55ad520b3d09] > Apr 18 21:24:37 server2 ceph-osd[17777]: 11: > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, > std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > std::vector<rocksdb::ColumnFamilyHandle*, > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, > bool)+0x689) [0x55ad520b4ab9] > Apr 18 21:24:37 server2 ceph-osd[17777]: 12: > (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, > std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > std::vector<rocksdb::ColumnFamilyHandle*, > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) > [0x55ad520b62e2] > Apr 18 21:24:37 server2 ceph-osd[17777]: 13: > (RocksDBStore::do_open(std::ostream&, bool, > std::vector<KeyValueDB::ColumnFamily, > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] > Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, > bool)+0xcf4) [0x55ad51f527a4] > Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, > bool)+0x4e9) [0x55ad51f828f9] > Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) > [0x55ad51b1cf09] > Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) > [0x55ad51a00d52] > Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) > [0x7f43bc6c7505] > Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] > Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099 > 7f43c936bb80 -1 *** Caught signal (Aborted) ** > Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 > thread_name:ceph-osd > Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 > (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] > Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) > [0x7f43bc6db337] > Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] > Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x248) [0x7f43c0749978] > Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] > Apr 18 21:24:37 server2 ceph-osd[17777]: 6: > (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned > long)+0x1ac6) [0x55ad52029266] > Apr 18 21:24:37 server2 ceph-osd[17777]: 7: > (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] > Apr 18 21:24:37 server2 ceph-osd[17777]: 8: > (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] > Apr 18 21:24:37 server2 ceph-osd[17777]: 9: > (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] > Apr 18 21:24:37 server2 ceph-osd[17777]: 10: > (rocksdb::BuildTable(std::string const&, rocksdb::Env*, > rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, > rocksdb::EnvOptions const&, rocksdb::TableCache*, > rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, > std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, > rocksdb::InternalKeyComparator const&, > std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, > unsigned int, std::string const&, std::vector<unsigned long, > std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, > rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, > rocksdb::TableProperties*, int, unsigned long, unsigned long, > rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] > Apr 18 21:24:37 server2 ceph-osd[17777]: 11: > (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, > rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] > Apr 18 21:24:37 server2 ceph-osd[17777]: 12: > (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, > std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) > [0x55ad520b2dcb] > Apr 18 21:24:37 server2 ceph-osd[17777]: 13: > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, > bool)+0xa59) [0x55ad520b3d09] > Apr 18 21:24:37 server2 ceph-osd[17777]: 14: > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, > std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > std::vector<rocksdb::ColumnFamilyHandle*, > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, > bool)+0x689) [0x55ad520b4ab9] > Apr 18 21:24:37 server2 ceph-osd[17777]: 15: > (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, > std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > std::vector<rocksdb::ColumnFamilyHandle*, > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) > [0x55ad520b62e2] > Apr 18 21:24:37 server2 ceph-osd[17777]: 16: > (RocksDBStore::do_open(std::ostream&, bool, > std::vector<KeyValueDB::ColumnFamily, > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] > Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, > bool)+0xcf4) [0x55ad51f527a4] > Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, > bool)+0x4e9) [0x55ad51f828f9] > Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) > [0x55ad51b1cf09] > Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) > [0x55ad51a00d52] > Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) > [0x7f43bc6c7505] > Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] > Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or > `objdump -rdS <executable>` is needed to interpret this. > Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 > 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne > Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 > 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: > 0x794de9 > Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094 > 7f43c936bb80 -1 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: > In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, > uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 > Apr 18 21:24:37 server2 ceph-osd[17777]: > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: > 1704: FAILED assert(0 == "bluefs enospc") > Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 > (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] > Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] > Apr 18 21:24:37 server2 ceph-osd[17777]: 3: > (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned > long)+0x1ac6) [0x55ad52029266] > Apr 18 21:24:37 server2 ceph-osd[17777]: 4: > (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] > Apr 18 21:24:37 server2 ceph-osd[17777]: 5: > (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] > Apr 18 21:24:37 server2 ceph-osd[17777]: 6: > (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] > Apr 18 21:24:37 server2 ceph-osd[17777]: 7: > (rocksdb::BuildTable(std::string const&, rocksdb::Env*, > rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, > rocksdb::EnvOptions const&, rocksdb::TableCache*, > rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, > std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, > rocksdb::InternalKeyComparator const&, > std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, > unsigned int, std::string const&, std::vector<unsigned long, > std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, > rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, > rocksdb::TableProperties*, int, unsigned long, unsigned long, > rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] > Apr 18 21:24:37 server2 ceph-osd[17777]: 8: > (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, > rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] > Apr 18 21:24:37 server2 ceph-osd[17777]: 9: > (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, > std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) > [0x55ad520b2dcb] > Apr 18 21:24:37 server2 ceph-osd[17777]: 10: > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, > bool)+0xa59) [0x55ad520b3d09] > Apr 18 21:24:37 server2 ceph-osd[17777]: 11: > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, > std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > std::vector<rocksdb::ColumnFamilyHandle*, > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, > bool)+0x689) [0x55ad520b4ab9] > Apr 18 21:24:37 server2 ceph-osd[17777]: 12: > (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, > std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > std::vector<rocksdb::ColumnFamilyHandle*, > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) > [0x55ad520b62e2] > Apr 18 21:24:37 server2 ceph-osd[17777]: 13: > (RocksDBStore::do_open(std::ostream&, bool, > std::vector<KeyValueDB::ColumnFamily, > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] > Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, > bool)+0xcf4) [0x55ad51f527a4] > Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, > bool)+0x4e9) [0x55ad51f828f9] > Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) > [0x55ad51b1cf09] > Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) > [0x55ad51a00d52] > Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) > [0x7f43bc6c7505] > Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] > Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099 > 7f43c936bb80 -1 *** Caught signal (Aborted) ** > Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 > thread_name:ceph-osd > Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 > (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] > Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) > [0x7f43bc6db337] > Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] > Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x248) [0x7f43c0749978] > Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] > Apr 18 21:24:37 server2 ceph-osd[17777]: 6: > (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned > long)+0x1ac6) [0x55ad52029266] > Apr 18 21:24:37 server2 ceph-osd[17777]: 7: > (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] > Apr 18 21:24:37 server2 ceph-osd[17777]: 8: > (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] > Apr 18 21:24:37 server2 ceph-osd[17777]: 9: > (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] > Apr 18 21:24:37 server2 ceph-osd[17777]: 10: > (rocksdb::BuildTable(std::string const&, rocksdb::Env*, > rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, > rocksdb::EnvOptions const&, rocksdb::TableCache*, > rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, > std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, > rocksdb::InternalKeyComparator const&, > std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, > unsigned int, std::string const&, std::vector<unsigned long, > std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, > rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, > rocksdb::TableProperties*, int, unsigned long, unsigned long, > rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] > Apr 18 21:24:37 server2 ceph-osd[17777]: 11: > (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, > rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] > Apr 18 21:24:37 server2 ceph-osd[17777]: 12: > (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, > std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) > [0x55ad520b2dcb] > Apr 18 21:24:37 server2 ceph-osd[17777]: 13: > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, > bool)+0xa59) [0x55ad520b3d09] > Apr 18 21:24:37 server2 ceph-osd[17777]: 14: > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, > std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > std::vector<rocksdb::ColumnFamilyHandle*, > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, > bool)+0x689) [0x55ad520b4ab9] > Apr 18 21:24:37 server2 ceph-osd[17777]: 15: > (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, > std::vector<rocksdb::ColumnFamilyDescriptor, > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > std::vector<rocksdb::ColumnFamilyHandle*, > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) > [0x55ad520b62e2] > Apr 18 21:24:37 server2 ceph-osd[17777]: 16: > (RocksDBStore::do_open(std::ostream&, bool, > std::vector<KeyValueDB::ColumnFamily, > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] > Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, > bool)+0xcf4) [0x55ad51f527a4] > Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, > bool)+0x4e9) [0x55ad51f828f9] > Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) > [0x55ad51b1cf09] > Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) > [0x55ad51a00d52] > Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) > [0x7f43bc6c7505] > Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] > Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or > `objdump -rdS <executable>` is needed to interpret this. > Apr 18 21:24:37 server2 systemd[1]: ceph-osd@4.service: main process > exited, code=killed, status=6/ABRT > Apr 18 21:24:37 server2 systemd[1]: Unit ceph-osd@4.service entered failed > state. > Apr 18 21:24:37 server2 systemd[1]: ceph-osd@4.service failed. > Apr 18 21:24:57 server2 systemd[1]: ceph-osd@4.service holdoff time over, > scheduling restart. > Apr 18 21:24:57 server2 systemd[1]: Stopped Ceph object storage daemon > osd.4. > Apr 18 21:24:57 server2 systemd[1]: Starting Ceph object storage daemon > osd.4... > Apr 18 21:24:57 server2 systemd[1]: Started Ceph object storage daemon > osd.4. > > > > > > and some more data: > > [root@server1 ~]# ceph df detail > GLOBAL: > SIZE AVAIL RAW USED %RAW USED OBJECTS > 38 GiB 35 GiB 2.6 GiB 6.83 303 > POOLS: > NAME ID QUOTA OBJECTS QUOTA BYTES > USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE > RAW USED > .rgw.root 18 N/A N/A > 1.1 KiB 0 50 GiB 4 4 63 B 4 > B 1.1 KiB > default.rgw.meta 19 N/A N/A > 4.1 KiB 0 50 GiB 22 22 1.5 KiB 182 > B 9.0 KiB > default.rgw.log 20 N/A N/A > 0 B 0 50 GiB 107 107 307 KiB 205 > KiB 0 B > default.rgw.control 21 N/A N/A > 0 B 0 50 GiB 5 5 0 B 0 > B 0 B > default.rgw.buckets.index 22 N/A N/A > 0 B 0 50 GiB 5 5 501 B 60 > B 0 B > default.rgw.buckets.data 23 N/A N/A > 377 MiB 0.39 50 GiB 160 160 385 B 1.1 > KiB 603 MiB > [root@server1 ~]# > > [root@server1 ~]# ceph osd df tree > ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE > VAR PGS TYPE NAME > -1 0.16727 - 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 - root default > -3 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 - host server1 > 0 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 0 osd.0 > 1 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 0 osd.1 > 2 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 0 osd.2 > -5 0.05576 - 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 0 > 0 - host server2 > 3 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 0 osd.3 > 4 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 0 osd.4 > 5 hdd 0.01859 1.00000 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 7.11 > 1.04 99 osd.5 > -7 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 - host server3 > 6 hdd 0.01859 1.00000 19 GiB 1.2 GiB 249 MiB 3 KiB 1024 MiB 18 GiB 6.55 > 0.96 78 osd.6 > 7 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 0 osd.7 > 8 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 > 0 0 osd.8 > TOTAL 38 GiB 2.6 GiB 610 MiB 6 KiB 2.0 GiB 35 GiB 6.83 > > MIN/MAX VAR: 0/1.04 STDDEV: 5.58 > [root@server1 ~]# > > > I'm kind of newbie to ceph, so any help or hint would be appreciated. > Did I hit a bug or something is wrong with my configuration? > > Thanks a lot, > Khodayar > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |