No, it's a fresh cluster and it has only 500MB of data! On Sun, Apr 19, 2020 at 1:13 AM Jarett DeAngelis <jarett@xxxxxxxxxxxx> wrote: > I’ve also had this problem. In my case it was because I’d added a number > of OSDs recently and the cluster was spending a lot of time rebalancing. Is > there a reason it might be doing that in yours? > > > On Apr 18, 2020, at 12:59 PM, Khodayar Doustar <doustar@xxxxxxxxxxxx> > wrote: > > > > Hi, > > > > I have a 3 node cluster of mimic with 9 osds (3 osds on each node). > > I use this cluster to test integration of an application with S3 api. > > > > The problem is that after a few days all OSD starts filling up with > > bluestore logs and goes down and out one by one! > > I cannot stop the logs and I cannot find the setting to fix this leakage, > > this should be a leakage in logs because it's not logical to fill up all > > OSD with bluefs logs. > > > > This is an example of logs which is being repeated in bluestore logs: > > > > [root@server2 ~]# ceph-bluestore-tool --command bluefs-log-dump --path > > /var/lib/ceph/osd/ceph-5 > > . > > . > > . 0x40d000: op_file_update file(ino 30 size 0x7a713 mtime 2020-04-18 > > 15:54:29.056488 bdev 1 allocated 80000 extents > > > [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) > > 0x40e000: txn(seq 1156100 len 0x78 crc 0x3e1c626f) > > 0x40e000: op_file_update file(ino 30 size 0x7a72a mtime 2020-04-18 > > 15:54:30.057828 bdev 1 allocated 80000 extents > > > [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) > > 0x40f000: txn(seq 1156101 len 0x78 crc 0xc1f9ec5f) > > 0x40f000: op_file_update file(ino 30 size 0x7a741 mtime 2020-04-18 > > 15:54:31.059252 bdev 1 allocated 80000 extents > > > [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) > > *** Caught signal (Segmentation fault) ** > > in thread 7f4108cfe600 thread_name:ceph-bluestore- > > ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic > > (stable) > > 1: (()+0xf5f0) [0x7f40fd47f5f0] > > 2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, > unsigned > > long, unsigned long, ceph::buffer::list*, char*)+0x3bf) [0x55603a1ebb3f] > > 3: (BlueFS::_replay(bool, bool)+0x214) [0x55603a1f6654] > > 4: (BlueFS::log_dump(CephContext*, std::string const&, > > std::vector<std::string, std::allocator<std::string> > const&)+0x3b) > > [0x55603a1fad3b] > > 5: (log_dump(CephContext*, std::string const&, std::vector<std::string, > > std::allocator<std::string> > const&)+0x64) [0x55603a1da764] > > 6: (main()+0x2f3e) [0x55603a105e3e] > > 7: (__libc_start_main()+0xf5) [0x7f40fbe51505] > > 8: (()+0x23115f) [0x55603a1d915f] > > 2020-04-18 21:08:25.410 7f4108cfe600 -1 *** Caught signal (Segmentation > > fault) ** > > in thread 7f4108cfe600 thread_name:ceph-bluestore- > > . > > . > > . > > > > > > And this is the output of daemonperf for one of remaining (filling up!) > > OSDs: > > > > [root@server2 ~]# ceph daemonperf osd.5 > > ------bluefs------- ------------bluestore------------- > > ----------osd----------- > > jlen j wal sst |fl_l k_l io_l th_l s_l c_l r_l |ops wr rd l > > rop | > > 81M 0 0 0 | 0 0 0 0 0 0 0 | 0 0 0 0 > > 0 > > 81M 4.0k 1.8k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 > > 0 > > 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 > > 0 > > 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 > > 0 > > 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 > > 0 > > ^C[root@server2 ~]# > > > > And this is the enospc logs when trying to start OSD: > > > > [root@server2 ~]# journalctl -u -f ceph-osd@4.service > > Failed to add match 'ceph-osd@4.service': Invalid argument > > Failed to add filters: Invalid argument > > [root@server2 ~]# journalctl -f -u ceph-osd@4.service > > -- Logs begin at Sat 2020-04-18 15:38:15 +0430. -- > > Apr 18 21:18:22 server2 ceph-osd[17485]: 22: (()+0x378c10) > [0x557348ebac10] > > Apr 18 21:18:22 server2 ceph-osd[17485]: NOTE: a copy of the executable, > or > > `objdump -rdS <executable>` is needed to interpret this. > > Apr 18 21:18:22 server2 systemd[1]: ceph-osd@4.service: main process > > exited, code=killed, status=6/ABRT > > Apr 18 21:18:22 server2 systemd[1]: Unit ceph-osd@4.service entered > failed > > state. > > Apr 18 21:18:22 server2 systemd[1]: ceph-osd@4.service failed. > > Apr 18 21:18:42 server2 systemd[1]: ceph-osd@4.service holdoff time > over, > > scheduling restart. > > Apr 18 21:18:42 server2 systemd[1]: Stopped Ceph object storage daemon > > osd.4. > > Apr 18 21:18:42 server2 systemd[1]: Starting Ceph object storage daemon > > osd.4... > > Apr 18 21:18:42 server2 systemd[1]: Started Ceph object storage daemon > > osd.4. > > Apr 18 21:18:43 server2 ceph-osd[17777]: starting osd.4 at - osd_data > > /var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal > > Apr 18 21:24:37 server2 ceph-osd[17777]: > > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: > > In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, > > uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 > > Apr 18 21:24:37 server2 ceph-osd[17777]: > > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: > > 1704: FAILED assert(0 == "bluefs enospc") > > Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090 > > 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne > > Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090 > > 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: > > 0x794de9 > > Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 > > (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > > Apr 18 21:24:37 server2 ceph-osd[17777]: 1: > (ceph::__ceph_assert_fail(char > > const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) > [0x7f43c0749a07] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 3: > > (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned > > long)+0x1ac6) [0x55ad52029266] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 4: > > (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 5: > > (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 6: > > (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 7: > > (rocksdb::BuildTable(std::string const&, rocksdb::Env*, > > rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, > > rocksdb::EnvOptions const&, rocksdb::TableCache*, > > rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, > > std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, > > rocksdb::InternalKeyComparator const&, > > std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, > > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, > > unsigned int, std::string const&, std::vector<unsigned long, > > std::allocator<unsigned long> >, unsigned long, > rocksdb::SnapshotChecker*, > > rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, > > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, > > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, > > rocksdb::TableProperties*, int, unsigned long, unsigned long, > > rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 8: > > (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, > > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, > > rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 9: > > (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, > > std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) > > [0x55ad520b2dcb] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 10: > > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, > > bool)+0xa59) [0x55ad520b3d09] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 11: > > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, > > std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > > std::vector<rocksdb::ColumnFamilyHandle*, > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, > > bool)+0x689) [0x55ad520b4ab9] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 12: > > (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, > > std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > > std::vector<rocksdb::ColumnFamilyHandle*, > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) > > [0x55ad520b62e2] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 13: > > (RocksDBStore::do_open(std::ostream&, bool, > > std::vector<KeyValueDB::ColumnFamily, > > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) > [0x55ad51fc65de] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, > > bool)+0xcf4) [0x55ad51f527a4] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, > > bool)+0x4e9) [0x55ad51f828f9] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) > > [0x55ad51b1cf09] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) > > [0x55ad51a00d52] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) > > [0x7f43bc6c7505] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) > [0x55ad51ad8c10] > > Apr 18 21:24:37 server2 ceph-osd[17777]: *** Caught signal (Aborted) ** > > Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 > > thread_name:ceph-osd > > Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.094 > > 7f43c936bb80 -1 > > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: > > In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, > > uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 > > Apr 18 21:24:37 server2 ceph-osd[17777]: > > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: > > 1704: FAILED assert(0 == "bluefs enospc") > > Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 > > (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > > Apr 18 21:24:37 server2 ceph-osd[17777]: 1: > (ceph::__ceph_assert_fail(char > > const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) > [0x7f43c0749a07] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 3: > > (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned > > long)+0x1ac6) [0x55ad52029266] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 4: > > (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 5: > > (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 6: > > (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 7: > > (rocksdb::BuildTable(std::string const&, rocksdb::Env*, > > rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, > > rocksdb::EnvOptions const&, rocksdb::TableCache*, > > rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, > > std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, > > rocksdb::InternalKeyComparator const&, > > std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, > > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, > > unsigned int, std::string const&, std::vector<unsigned long, > > std::allocator<unsigned long> >, unsigned long, > rocksdb::SnapshotChecker*, > > rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, > > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, > > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, > > rocksdb::TableProperties*, int, unsigned long, unsigned long, > > rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 8: > > (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, > > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, > > rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 9: > > (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, > > std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) > > [0x55ad520b2dcb] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 10: > > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, > > bool)+0xa59) [0x55ad520b3d09] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 11: > > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, > > std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > > std::vector<rocksdb::ColumnFamilyHandle*, > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, > > bool)+0x689) [0x55ad520b4ab9] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 12: > > (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, > > std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > > std::vector<rocksdb::ColumnFamilyHandle*, > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) > > [0x55ad520b62e2] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 13: > > (RocksDBStore::do_open(std::ostream&, bool, > > std::vector<KeyValueDB::ColumnFamily, > > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) > [0x55ad51fc65de] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, > > bool)+0xcf4) [0x55ad51f527a4] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, > > bool)+0x4e9) [0x55ad51f828f9] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) > > [0x55ad51b1cf09] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) > > [0x55ad51a00d52] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) > > [0x7f43bc6c7505] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) > [0x55ad51ad8c10] > > Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 > > (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > > Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) > > [0x7f43bc6db337] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) > [0x7f43bc6dca28] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 4: > (ceph::__ceph_assert_fail(char > > const*, char const*, int, char const*)+0x248) [0x7f43c0749978] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) > [0x7f43c0749a07] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 6: > > (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned > > long)+0x1ac6) [0x55ad52029266] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 7: > > (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 8: > > (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 9: > > (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 10: > > (rocksdb::BuildTable(std::string const&, rocksdb::Env*, > > rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, > > rocksdb::EnvOptions const&, rocksdb::TableCache*, > > rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, > > std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, > > rocksdb::InternalKeyComparator const&, > > std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, > > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, > > unsigned int, std::string const&, std::vector<unsigned long, > > std::allocator<unsigned long> >, unsigned long, > rocksdb::SnapshotChecker*, > > rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, > > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, > > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, > > rocksdb::TableProperties*, int, unsigned long, unsigned long, > > rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 11: > > (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, > > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, > > rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 12: > > (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, > > std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) > > [0x55ad520b2dcb] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 13: > > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, > > bool)+0xa59) [0x55ad520b3d09] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 14: > > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, > > std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > > std::vector<rocksdb::ColumnFamilyHandle*, > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, > > bool)+0x689) [0x55ad520b4ab9] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 15: > > (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, > > std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > > std::vector<rocksdb::ColumnFamilyHandle*, > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) > > [0x55ad520b62e2] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 16: > > (RocksDBStore::do_open(std::ostream&, bool, > > std::vector<KeyValueDB::ColumnFamily, > > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) > [0x55ad51fc65de] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, > > bool)+0xcf4) [0x55ad51f527a4] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, > > bool)+0x4e9) [0x55ad51f828f9] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) > > [0x55ad51b1cf09] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) > > [0x55ad51a00d52] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) > > [0x7f43bc6c7505] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) > [0x55ad51ad8c10] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.099 > > 7f43c936bb80 -1 *** Caught signal (Aborted) ** > > Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 > > thread_name:ceph-osd > > Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 > > (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > > Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) > > [0x7f43bc6db337] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) > [0x7f43bc6dca28] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 4: > (ceph::__ceph_assert_fail(char > > const*, char const*, int, char const*)+0x248) [0x7f43c0749978] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) > [0x7f43c0749a07] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 6: > > (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned > > long)+0x1ac6) [0x55ad52029266] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 7: > > (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 8: > > (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 9: > > (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 10: > > (rocksdb::BuildTable(std::string const&, rocksdb::Env*, > > rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, > > rocksdb::EnvOptions const&, rocksdb::TableCache*, > > rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, > > std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, > > rocksdb::InternalKeyComparator const&, > > std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, > > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, > > unsigned int, std::string const&, std::vector<unsigned long, > > std::allocator<unsigned long> >, unsigned long, > rocksdb::SnapshotChecker*, > > rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, > > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, > > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, > > rocksdb::TableProperties*, int, unsigned long, unsigned long, > > rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 11: > > (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, > > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, > > rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 12: > > (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, > > std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) > > [0x55ad520b2dcb] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 13: > > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, > > bool)+0xa59) [0x55ad520b3d09] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 14: > > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, > > std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > > std::vector<rocksdb::ColumnFamilyHandle*, > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, > > bool)+0x689) [0x55ad520b4ab9] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 15: > > (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, > > std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > > std::vector<rocksdb::ColumnFamilyHandle*, > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) > > [0x55ad520b62e2] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 16: > > (RocksDBStore::do_open(std::ostream&, bool, > > std::vector<KeyValueDB::ColumnFamily, > > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) > [0x55ad51fc65de] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, > > bool)+0xcf4) [0x55ad51f527a4] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, > > bool)+0x4e9) [0x55ad51f828f9] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) > > [0x55ad51b1cf09] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) > > [0x55ad51a00d52] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) > > [0x7f43bc6c7505] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) > [0x55ad51ad8c10] > > Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, > or > > `objdump -rdS <executable>` is needed to interpret this. > > Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 > > 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne > > Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 > > 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: > > 0x794de9 > > Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094 > > 7f43c936bb80 -1 > > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: > > In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, > > uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 > > Apr 18 21:24:37 server2 ceph-osd[17777]: > > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: > > 1704: FAILED assert(0 == "bluefs enospc") > > Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 > > (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > > Apr 18 21:24:37 server2 ceph-osd[17777]: 1: > (ceph::__ceph_assert_fail(char > > const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) > [0x7f43c0749a07] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 3: > > (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned > > long)+0x1ac6) [0x55ad52029266] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 4: > > (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 5: > > (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 6: > > (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 7: > > (rocksdb::BuildTable(std::string const&, rocksdb::Env*, > > rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, > > rocksdb::EnvOptions const&, rocksdb::TableCache*, > > rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, > > std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, > > rocksdb::InternalKeyComparator const&, > > std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, > > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, > > unsigned int, std::string const&, std::vector<unsigned long, > > std::allocator<unsigned long> >, unsigned long, > rocksdb::SnapshotChecker*, > > rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, > > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, > > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, > > rocksdb::TableProperties*, int, unsigned long, unsigned long, > > rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 8: > > (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, > > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, > > rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 9: > > (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, > > std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) > > [0x55ad520b2dcb] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 10: > > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, > > bool)+0xa59) [0x55ad520b3d09] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 11: > > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, > > std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > > std::vector<rocksdb::ColumnFamilyHandle*, > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, > > bool)+0x689) [0x55ad520b4ab9] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 12: > > (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, > > std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > > std::vector<rocksdb::ColumnFamilyHandle*, > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) > > [0x55ad520b62e2] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 13: > > (RocksDBStore::do_open(std::ostream&, bool, > > std::vector<KeyValueDB::ColumnFamily, > > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) > [0x55ad51fc65de] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, > > bool)+0xcf4) [0x55ad51f527a4] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, > > bool)+0x4e9) [0x55ad51f828f9] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) > > [0x55ad51b1cf09] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) > > [0x55ad51a00d52] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) > > [0x7f43bc6c7505] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) > [0x55ad51ad8c10] > > Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099 > > 7f43c936bb80 -1 *** Caught signal (Aborted) ** > > Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 > > thread_name:ceph-osd > > Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 > > (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > > Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) > > [0x7f43bc6db337] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) > [0x7f43bc6dca28] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 4: > (ceph::__ceph_assert_fail(char > > const*, char const*, int, char const*)+0x248) [0x7f43c0749978] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) > [0x7f43c0749a07] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 6: > > (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned > > long)+0x1ac6) [0x55ad52029266] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 7: > > (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 8: > > (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 9: > > (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 10: > > (rocksdb::BuildTable(std::string const&, rocksdb::Env*, > > rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, > > rocksdb::EnvOptions const&, rocksdb::TableCache*, > > rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, > > std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, > > rocksdb::InternalKeyComparator const&, > > std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, > > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, > > unsigned int, std::string const&, std::vector<unsigned long, > > std::allocator<unsigned long> >, unsigned long, > rocksdb::SnapshotChecker*, > > rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, > > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, > > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, > > rocksdb::TableProperties*, int, unsigned long, unsigned long, > > rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 11: > > (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, > > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, > > rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 12: > > (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, > > std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) > > [0x55ad520b2dcb] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 13: > > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, > > bool)+0xa59) [0x55ad520b3d09] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 14: > > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, > > std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > > std::vector<rocksdb::ColumnFamilyHandle*, > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, > > bool)+0x689) [0x55ad520b4ab9] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 15: > > (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, > > std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > > std::vector<rocksdb::ColumnFamilyHandle*, > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) > > [0x55ad520b62e2] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 16: > > (RocksDBStore::do_open(std::ostream&, bool, > > std::vector<KeyValueDB::ColumnFamily, > > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) > [0x55ad51fc65de] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, > > bool)+0xcf4) [0x55ad51f527a4] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, > > bool)+0x4e9) [0x55ad51f828f9] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) > > [0x55ad51b1cf09] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) > > [0x55ad51a00d52] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) > > [0x7f43bc6c7505] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) > [0x55ad51ad8c10] > > Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, > or > > `objdump -rdS <executable>` is needed to interpret this. > > Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 > > 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne > > Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 > > 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: > > 0x794de9 > > Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094 > > 7f43c936bb80 -1 > > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: > > In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, > > uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 > > Apr 18 21:24:37 server2 ceph-osd[17777]: > > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: > > 1704: FAILED assert(0 == "bluefs enospc") > > Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 > > (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > > Apr 18 21:24:37 server2 ceph-osd[17777]: 1: > (ceph::__ceph_assert_fail(char > > const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) > [0x7f43c0749a07] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 3: > > (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned > > long)+0x1ac6) [0x55ad52029266] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 4: > > (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 5: > > (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 6: > > (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 7: > > (rocksdb::BuildTable(std::string const&, rocksdb::Env*, > > rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, > > rocksdb::EnvOptions const&, rocksdb::TableCache*, > > rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, > > std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, > > rocksdb::InternalKeyComparator const&, > > std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, > > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, > > unsigned int, std::string const&, std::vector<unsigned long, > > std::allocator<unsigned long> >, unsigned long, > rocksdb::SnapshotChecker*, > > rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, > > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, > > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, > > rocksdb::TableProperties*, int, unsigned long, unsigned long, > > rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 8: > > (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, > > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, > > rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 9: > > (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, > > std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) > > [0x55ad520b2dcb] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 10: > > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, > > bool)+0xa59) [0x55ad520b3d09] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 11: > > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, > > std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > > std::vector<rocksdb::ColumnFamilyHandle*, > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, > > bool)+0x689) [0x55ad520b4ab9] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 12: > > (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, > > std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > > std::vector<rocksdb::ColumnFamilyHandle*, > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) > > [0x55ad520b62e2] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 13: > > (RocksDBStore::do_open(std::ostream&, bool, > > std::vector<KeyValueDB::ColumnFamily, > > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) > [0x55ad51fc65de] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, > > bool)+0xcf4) [0x55ad51f527a4] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, > > bool)+0x4e9) [0x55ad51f828f9] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) > > [0x55ad51b1cf09] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) > > [0x55ad51a00d52] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) > > [0x7f43bc6c7505] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) > [0x55ad51ad8c10] > > Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099 > > 7f43c936bb80 -1 *** Caught signal (Aborted) ** > > Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 > > thread_name:ceph-osd > > Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 > > (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > > Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) > > [0x7f43bc6db337] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) > [0x7f43bc6dca28] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 4: > (ceph::__ceph_assert_fail(char > > const*, char const*, int, char const*)+0x248) [0x7f43c0749978] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) > [0x7f43c0749a07] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 6: > > (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned > > long)+0x1ac6) [0x55ad52029266] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 7: > > (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 8: > > (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 9: > > (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 10: > > (rocksdb::BuildTable(std::string const&, rocksdb::Env*, > > rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, > > rocksdb::EnvOptions const&, rocksdb::TableCache*, > > rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, > > std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, > > rocksdb::InternalKeyComparator const&, > > std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > > std::default_delete<rocksdb::IntTblPropCollectorFactory> >, > > std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, > > std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, > > unsigned int, std::string const&, std::vector<unsigned long, > > std::allocator<unsigned long> >, unsigned long, > rocksdb::SnapshotChecker*, > > rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, > > rocksdb::InternalStats*, rocksdb::TableFileCreationReason, > > rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, > > rocksdb::TableProperties*, int, unsigned long, unsigned long, > > rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 11: > > (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, > > rocksdb::ColumnFamilyData*, rocksdb::MemTable*, > > rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 12: > > (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, > > std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) > > [0x55ad520b2dcb] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 13: > > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, > > bool)+0xa59) [0x55ad520b3d09] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 14: > > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, > > std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > > std::vector<rocksdb::ColumnFamilyHandle*, > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, > > bool)+0x689) [0x55ad520b4ab9] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 15: > > (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, > > std::vector<rocksdb::ColumnFamilyDescriptor, > > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, > > std::vector<rocksdb::ColumnFamilyHandle*, > > std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) > > [0x55ad520b62e2] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 16: > > (RocksDBStore::do_open(std::ostream&, bool, > > std::vector<KeyValueDB::ColumnFamily, > > std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) > [0x55ad51fc65de] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, > > bool)+0xcf4) [0x55ad51f527a4] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, > > bool)+0x4e9) [0x55ad51f828f9] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) > > [0x55ad51b1cf09] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) > > [0x55ad51a00d52] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) > > [0x7f43bc6c7505] > > Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) > [0x55ad51ad8c10] > > Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, > or > > `objdump -rdS <executable>` is needed to interpret this. > > Apr 18 21:24:37 server2 systemd[1]: ceph-osd@4.service: main process > > exited, code=killed, status=6/ABRT > > Apr 18 21:24:37 server2 systemd[1]: Unit ceph-osd@4.service entered > failed > > state. > > Apr 18 21:24:37 server2 systemd[1]: ceph-osd@4.service failed. > > Apr 18 21:24:57 server2 systemd[1]: ceph-osd@4.service holdoff time > over, > > scheduling restart. > > Apr 18 21:24:57 server2 systemd[1]: Stopped Ceph object storage daemon > > osd.4. > > Apr 18 21:24:57 server2 systemd[1]: Starting Ceph object storage daemon > > osd.4... > > Apr 18 21:24:57 server2 systemd[1]: Started Ceph object storage daemon > > osd.4. > > > > > > > > > > > > and some more data: > > > > [root@server1 ~]# ceph df detail > > GLOBAL: > > SIZE AVAIL RAW USED %RAW USED OBJECTS > > 38 GiB 35 GiB 2.6 GiB 6.83 303 > > POOLS: > > NAME ID QUOTA OBJECTS QUOTA BYTES > > USED %USED MAX AVAIL OBJECTS DIRTY READ > WRITE > > RAW USED > > .rgw.root 18 N/A N/A > > 1.1 KiB 0 50 GiB 4 4 63 B > 4 > > B 1.1 KiB > > default.rgw.meta 19 N/A N/A > > 4.1 KiB 0 50 GiB 22 22 1.5 KiB > 182 > > B 9.0 KiB > > default.rgw.log 20 N/A N/A > > 0 B 0 50 GiB 107 107 307 KiB 205 > > KiB 0 B > > default.rgw.control 21 N/A N/A > > 0 B 0 50 GiB 5 5 0 B > 0 > > B 0 B > > default.rgw.buckets.index 22 N/A N/A > > 0 B 0 50 GiB 5 5 501 B > 60 > > B 0 B > > default.rgw.buckets.data 23 N/A N/A > > 377 MiB 0.39 50 GiB 160 160 385 B 1.1 > > KiB 603 MiB > > [root@server1 ~]# > > > > [root@server1 ~]# ceph osd df tree > > ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL > %USE > > VAR PGS TYPE NAME > > -1 0.16727 - 0 B 0 B 0 B 0 B 0 B 0 B > 0 > > 0 - root default > > -3 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B > 0 > > 0 - host server1 > > 0 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B > 0 > > 0 0 osd.0 > > 1 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B > 0 > > 0 0 osd.1 > > 2 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B > 0 > > 0 0 osd.2 > > -5 0.05576 - 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB > 0 > > 0 - host server2 > > 3 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B > 0 > > 0 0 osd.3 > > 4 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B > 0 > > 0 0 osd.4 > > 5 hdd 0.01859 1.00000 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB > 7.11 > > 1.04 99 osd.5 > > -7 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B > 0 > > 0 - host server3 > > 6 hdd 0.01859 1.00000 19 GiB 1.2 GiB 249 MiB 3 KiB 1024 MiB 18 GiB > 6.55 > > 0.96 78 osd.6 > > 7 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B > 0 > > 0 0 osd.7 > > 8 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B > 0 > > 0 0 osd.8 > > TOTAL 38 GiB 2.6 GiB 610 MiB 6 KiB 2.0 GiB 35 GiB > 6.83 > > > > MIN/MAX VAR: 0/1.04 STDDEV: 5.58 > > [root@server1 ~]# > > > > > > I'm kind of newbie to ceph, so any help or hint would be appreciated. > > Did I hit a bug or something is wrong with my configuration? > > > > Thanks a lot, > > Khodayar > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |