Re: OSDs get full with bluestore logs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I’ve also had this problem. In my case it was because I’d added a number of OSDs recently and the cluster was spending a lot of time rebalancing. Is there a reason it might be doing that in yours?

> On Apr 18, 2020, at 12:59 PM, Khodayar Doustar <doustar@xxxxxxxxxxxx> wrote:
> 
> Hi,
> 
> I have a 3 node cluster of mimic with 9 osds (3 osds on each node).
> I use this cluster to test integration of an application with S3 api.
> 
> The problem is that after a few days all OSD starts filling up with
> bluestore logs and goes down and out one by one!
> I cannot stop the logs and I cannot find the setting to fix this leakage,
> this should be a leakage in logs because it's not logical to fill up all
> OSD with bluefs logs.
> 
> This is an example of logs which is being repeated in bluestore logs:
> 
> [root@server2 ~]# ceph-bluestore-tool --command bluefs-log-dump --path
> /var/lib/ceph/osd/ceph-5
> .
> .
> . 0x40d000:  op_file_update  file(ino 30 size 0x7a713 mtime 2020-04-18
> 15:54:29.056488 bdev 1 allocated 80000 extents
> [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000])
> 0x40e000: txn(seq 1156100 len 0x78 crc 0x3e1c626f)
> 0x40e000:  op_file_update  file(ino 30 size 0x7a72a mtime 2020-04-18
> 15:54:30.057828 bdev 1 allocated 80000 extents
> [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000])
> 0x40f000: txn(seq 1156101 len 0x78 crc 0xc1f9ec5f)
> 0x40f000:  op_file_update  file(ino 30 size 0x7a741 mtime 2020-04-18
> 15:54:31.059252 bdev 1 allocated 80000 extents
> [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000])
> *** Caught signal (Segmentation fault) **
> in thread 7f4108cfe600 thread_name:ceph-bluestore-
> ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic
> (stable)
> 1: (()+0xf5f0) [0x7f40fd47f5f0]
> 2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned
> long, unsigned long, ceph::buffer::list*, char*)+0x3bf) [0x55603a1ebb3f]
> 3: (BlueFS::_replay(bool, bool)+0x214) [0x55603a1f6654]
> 4: (BlueFS::log_dump(CephContext*, std::string const&,
> std::vector<std::string, std::allocator<std::string> > const&)+0x3b)
> [0x55603a1fad3b]
> 5: (log_dump(CephContext*, std::string const&, std::vector<std::string,
> std::allocator<std::string> > const&)+0x64) [0x55603a1da764]
> 6: (main()+0x2f3e) [0x55603a105e3e]
> 7: (__libc_start_main()+0xf5) [0x7f40fbe51505]
> 8: (()+0x23115f) [0x55603a1d915f]
> 2020-04-18 21:08:25.410 7f4108cfe600 -1 *** Caught signal (Segmentation
> fault) **
> in thread 7f4108cfe600 thread_name:ceph-bluestore-
> .
> .
> .
> 
> 
> And this is the output of daemonperf for one of remaining (filling up!)
> OSDs:
> 
> [root@server2 ~]# ceph daemonperf osd.5
> ------bluefs------- ------------bluestore-------------
> ----------osd-----------
> jlen j    wal  sst |fl_l k_l  io_l th_l s_l  c_l  r_l |ops  wr   rd   l
> rop |
> 81M   0    0    0 |  0    0    0    0    0    0    0 |  0    0    0    0
> 0
> 81M 4.0k 1.8k   0 |  0    0    0    0    0    0    0 |  0    0    0    0
> 0
> 81M 4.0k 1.9k   0 |  0    0    0    0    0    0    0 |  0    0    0    0
> 0
> 81M 4.0k 1.9k   0 |  0    0    0    0    0    0    0 |  0    0    0    0
> 0
> 81M 4.0k 1.9k   0 |  0    0    0    0    0    0    0 |  0    0    0    0
> 0
> ^C[root@server2 ~]#
> 
> And this is the enospc logs when trying to start OSD:
> 
> [root@server2 ~]# journalctl -u -f ceph-osd@4.service
> Failed to add match 'ceph-osd@4.service': Invalid argument
> Failed to add filters: Invalid argument
> [root@server2 ~]# journalctl -f -u ceph-osd@4.service
> -- Logs begin at Sat 2020-04-18 15:38:15 +0430. --
> Apr 18 21:18:22 server2 ceph-osd[17485]: 22: (()+0x378c10) [0x557348ebac10]
> Apr 18 21:18:22 server2 ceph-osd[17485]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Apr 18 21:18:22 server2 systemd[1]: ceph-osd@4.service: main process
> exited, code=killed, status=6/ABRT
> Apr 18 21:18:22 server2 systemd[1]: Unit ceph-osd@4.service entered failed
> state.
> Apr 18 21:18:22 server2 systemd[1]: ceph-osd@4.service failed.
> Apr 18 21:18:42 server2 systemd[1]: ceph-osd@4.service holdoff time over,
> scheduling restart.
> Apr 18 21:18:42 server2 systemd[1]: Stopped Ceph object storage daemon
> osd.4.
> Apr 18 21:18:42 server2 systemd[1]: Starting Ceph object storage daemon
> osd.4...
> Apr 18 21:18:42 server2 systemd[1]: Started Ceph object storage daemon
> osd.4.
> Apr 18 21:18:43 server2 ceph-osd[17777]: starting osd.4 at - osd_data
> /var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal
> Apr 18 21:24:37 server2 ceph-osd[17777]:
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
> In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
> uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289
> Apr 18 21:24:37 server2 ceph-osd[17777]:
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
> 1704: FAILED assert(0 == "bluefs enospc")
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090
> 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090
> 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length:
> 0x794de9
> Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
> (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
> Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x14b) [0x7f43c074987b]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 3:
> (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x1ac6) [0x55ad52029266]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 4:
> (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 5:
> (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
> (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
> (rocksdb::BuildTable(std::string const&, rocksdb::Env*,
> rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
> rocksdb::EnvOptions const&, rocksdb::TableCache*,
> rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
> std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
> rocksdb::InternalKeyComparator const&,
> std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
> unsigned int, std::string const&, std::vector<unsigned long,
> std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
> rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
> rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
> rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
> rocksdb::TableProperties*, int, unsigned long, unsigned long,
> rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
> (rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
> rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
> rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
> std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
> [0x55ad520b2dcb]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
> bool)+0xa59) [0x55ad520b3d09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
> bool)+0x689) [0x55ad520b4ab9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
> (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
> [0x55ad520b62e2]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
> (RocksDBStore::do_open(std::ostream&, bool,
> std::vector<KeyValueDB::ColumnFamily,
> std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool,
> bool)+0xcf4) [0x55ad51f527a4]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool,
> bool)+0x4e9) [0x55ad51f828f9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339)
> [0x55ad51b1cf09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2)
> [0x55ad51a00d52]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5)
> [0x7f43bc6c7505]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10]
> Apr 18 21:24:37 server2 ceph-osd[17777]: *** Caught signal (Aborted) **
> Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80
> thread_name:ceph-osd
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.094
> 7f43c936bb80 -1
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
> In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
> uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289
> Apr 18 21:24:37 server2 ceph-osd[17777]:
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
> 1704: FAILED assert(0 == "bluefs enospc")
> Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
> (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
> Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x14b) [0x7f43c074987b]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 3:
> (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x1ac6) [0x55ad52029266]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 4:
> (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 5:
> (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
> (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
> (rocksdb::BuildTable(std::string const&, rocksdb::Env*,
> rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
> rocksdb::EnvOptions const&, rocksdb::TableCache*,
> rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
> std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
> rocksdb::InternalKeyComparator const&,
> std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
> unsigned int, std::string const&, std::vector<unsigned long,
> std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
> rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
> rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
> rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
> rocksdb::TableProperties*, int, unsigned long, unsigned long,
> rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
> (rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
> rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
> rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
> std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
> [0x55ad520b2dcb]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
> bool)+0xa59) [0x55ad520b3d09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
> bool)+0x689) [0x55ad520b4ab9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
> (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
> [0x55ad520b62e2]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
> (RocksDBStore::do_open(std::ostream&, bool,
> std::vector<KeyValueDB::ColumnFamily,
> std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool,
> bool)+0xcf4) [0x55ad51f527a4]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool,
> bool)+0x4e9) [0x55ad51f828f9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339)
> [0x55ad51b1cf09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2)
> [0x55ad51a00d52]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5)
> [0x7f43bc6c7505]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10]
> Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
> (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
> Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37)
> [0x7f43bc6db337]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x248) [0x7f43c0749978]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
> (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x1ac6) [0x55ad52029266]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
> (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
> (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
> (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
> (rocksdb::BuildTable(std::string const&, rocksdb::Env*,
> rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
> rocksdb::EnvOptions const&, rocksdb::TableCache*,
> rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
> std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
> rocksdb::InternalKeyComparator const&,
> std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
> unsigned int, std::string const&, std::vector<unsigned long,
> std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
> rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
> rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
> rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
> rocksdb::TableProperties*, int, unsigned long, unsigned long,
> rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
> (rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
> rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
> rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
> std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
> [0x55ad520b2dcb]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
> bool)+0xa59) [0x55ad520b3d09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 14:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
> bool)+0x689) [0x55ad520b4ab9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 15:
> (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
> [0x55ad520b62e2]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 16:
> (RocksDBStore::do_open(std::ostream&, bool,
> std::vector<KeyValueDB::ColumnFamily,
> std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool,
> bool)+0xcf4) [0x55ad51f527a4]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool,
> bool)+0x4e9) [0x55ad51f828f9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339)
> [0x55ad51b1cf09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2)
> [0x55ad51a00d52]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5)
> [0x7f43bc6c7505]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.099
> 7f43c936bb80 -1 *** Caught signal (Aborted) **
> Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80
> thread_name:ceph-osd
> Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
> (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
> Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37)
> [0x7f43bc6db337]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x248) [0x7f43c0749978]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
> (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x1ac6) [0x55ad52029266]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
> (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
> (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
> (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
> (rocksdb::BuildTable(std::string const&, rocksdb::Env*,
> rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
> rocksdb::EnvOptions const&, rocksdb::TableCache*,
> rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
> std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
> rocksdb::InternalKeyComparator const&,
> std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
> unsigned int, std::string const&, std::vector<unsigned long,
> std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
> rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
> rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
> rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
> rocksdb::TableProperties*, int, unsigned long, unsigned long,
> rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
> (rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
> rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
> rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
> std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
> [0x55ad520b2dcb]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
> bool)+0xa59) [0x55ad520b3d09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 14:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
> bool)+0x689) [0x55ad520b4ab9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 15:
> (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
> [0x55ad520b62e2]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 16:
> (RocksDBStore::do_open(std::ostream&, bool,
> std::vector<KeyValueDB::ColumnFamily,
> std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool,
> bool)+0xcf4) [0x55ad51f527a4]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool,
> bool)+0x4e9) [0x55ad51f828f9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339)
> [0x55ad51b1cf09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2)
> [0x55ad51a00d52]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5)
> [0x7f43bc6c7505]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10]
> Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090
> 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne
> Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090
> 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length:
> 0x794de9
> Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094
> 7f43c936bb80 -1
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
> In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
> uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289
> Apr 18 21:24:37 server2 ceph-osd[17777]:
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
> 1704: FAILED assert(0 == "bluefs enospc")
> Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
> (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
> Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x14b) [0x7f43c074987b]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 3:
> (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x1ac6) [0x55ad52029266]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 4:
> (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 5:
> (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
> (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
> (rocksdb::BuildTable(std::string const&, rocksdb::Env*,
> rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
> rocksdb::EnvOptions const&, rocksdb::TableCache*,
> rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
> std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
> rocksdb::InternalKeyComparator const&,
> std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
> unsigned int, std::string const&, std::vector<unsigned long,
> std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
> rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
> rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
> rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
> rocksdb::TableProperties*, int, unsigned long, unsigned long,
> rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
> (rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
> rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
> rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
> std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
> [0x55ad520b2dcb]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
> bool)+0xa59) [0x55ad520b3d09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
> bool)+0x689) [0x55ad520b4ab9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
> (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
> [0x55ad520b62e2]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
> (RocksDBStore::do_open(std::ostream&, bool,
> std::vector<KeyValueDB::ColumnFamily,
> std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool,
> bool)+0xcf4) [0x55ad51f527a4]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool,
> bool)+0x4e9) [0x55ad51f828f9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339)
> [0x55ad51b1cf09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2)
> [0x55ad51a00d52]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5)
> [0x7f43bc6c7505]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10]
> Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099
> 7f43c936bb80 -1 *** Caught signal (Aborted) **
> Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80
> thread_name:ceph-osd
> Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
> (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
> Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37)
> [0x7f43bc6db337]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x248) [0x7f43c0749978]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
> (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x1ac6) [0x55ad52029266]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
> (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
> (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
> (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
> (rocksdb::BuildTable(std::string const&, rocksdb::Env*,
> rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
> rocksdb::EnvOptions const&, rocksdb::TableCache*,
> rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
> std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
> rocksdb::InternalKeyComparator const&,
> std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
> unsigned int, std::string const&, std::vector<unsigned long,
> std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
> rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
> rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
> rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
> rocksdb::TableProperties*, int, unsigned long, unsigned long,
> rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
> (rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
> rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
> rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
> std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
> [0x55ad520b2dcb]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
> bool)+0xa59) [0x55ad520b3d09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 14:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
> bool)+0x689) [0x55ad520b4ab9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 15:
> (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
> [0x55ad520b62e2]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 16:
> (RocksDBStore::do_open(std::ostream&, bool,
> std::vector<KeyValueDB::ColumnFamily,
> std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool,
> bool)+0xcf4) [0x55ad51f527a4]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool,
> bool)+0x4e9) [0x55ad51f828f9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339)
> [0x55ad51b1cf09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2)
> [0x55ad51a00d52]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5)
> [0x7f43bc6c7505]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10]
> Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090
> 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne
> Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090
> 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length:
> 0x794de9
> Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094
> 7f43c936bb80 -1
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
> In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
> uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289
> Apr 18 21:24:37 server2 ceph-osd[17777]:
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
> 1704: FAILED assert(0 == "bluefs enospc")
> Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
> (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
> Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x14b) [0x7f43c074987b]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 3:
> (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x1ac6) [0x55ad52029266]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 4:
> (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 5:
> (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
> (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
> (rocksdb::BuildTable(std::string const&, rocksdb::Env*,
> rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
> rocksdb::EnvOptions const&, rocksdb::TableCache*,
> rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
> std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
> rocksdb::InternalKeyComparator const&,
> std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
> unsigned int, std::string const&, std::vector<unsigned long,
> std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
> rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
> rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
> rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
> rocksdb::TableProperties*, int, unsigned long, unsigned long,
> rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
> (rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
> rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
> rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
> std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
> [0x55ad520b2dcb]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
> bool)+0xa59) [0x55ad520b3d09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
> bool)+0x689) [0x55ad520b4ab9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
> (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
> [0x55ad520b62e2]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
> (RocksDBStore::do_open(std::ostream&, bool,
> std::vector<KeyValueDB::ColumnFamily,
> std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool,
> bool)+0xcf4) [0x55ad51f527a4]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool,
> bool)+0x4e9) [0x55ad51f828f9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339)
> [0x55ad51b1cf09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2)
> [0x55ad51a00d52]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5)
> [0x7f43bc6c7505]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10]
> Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099
> 7f43c936bb80 -1 *** Caught signal (Aborted) **
> Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80
> thread_name:ceph-osd
> Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
> (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
> Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37)
> [0x7f43bc6db337]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x248) [0x7f43c0749978]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
> (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x1ac6) [0x55ad52029266]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
> (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
> (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
> (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
> (rocksdb::BuildTable(std::string const&, rocksdb::Env*,
> rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
> rocksdb::EnvOptions const&, rocksdb::TableCache*,
> rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
> std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
> rocksdb::InternalKeyComparator const&,
> std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
> unsigned int, std::string const&, std::vector<unsigned long,
> std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
> rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
> rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
> rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
> rocksdb::TableProperties*, int, unsigned long, unsigned long,
> rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
> (rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
> rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
> rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
> std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
> [0x55ad520b2dcb]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
> bool)+0xa59) [0x55ad520b3d09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 14:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
> bool)+0x689) [0x55ad520b4ab9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 15:
> (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
> [0x55ad520b62e2]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 16:
> (RocksDBStore::do_open(std::ostream&, bool,
> std::vector<KeyValueDB::ColumnFamily,
> std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool,
> bool)+0xcf4) [0x55ad51f527a4]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool,
> bool)+0x4e9) [0x55ad51f828f9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339)
> [0x55ad51b1cf09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2)
> [0x55ad51a00d52]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5)
> [0x7f43bc6c7505]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10]
> Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Apr 18 21:24:37 server2 systemd[1]: ceph-osd@4.service: main process
> exited, code=killed, status=6/ABRT
> Apr 18 21:24:37 server2 systemd[1]: Unit ceph-osd@4.service entered failed
> state.
> Apr 18 21:24:37 server2 systemd[1]: ceph-osd@4.service failed.
> Apr 18 21:24:57 server2 systemd[1]: ceph-osd@4.service holdoff time over,
> scheduling restart.
> Apr 18 21:24:57 server2 systemd[1]: Stopped Ceph object storage daemon
> osd.4.
> Apr 18 21:24:57 server2 systemd[1]: Starting Ceph object storage daemon
> osd.4...
> Apr 18 21:24:57 server2 systemd[1]: Started Ceph object storage daemon
> osd.4.
> 
> 
> 
> 
> 
> and some more data:
> 
> [root@server1 ~]# ceph df detail
> GLOBAL:
>    SIZE       AVAIL      RAW USED     %RAW USED     OBJECTS
>    38 GiB     35 GiB      2.6 GiB          6.83        303
> POOLS:
>    NAME                          ID     QUOTA OBJECTS     QUOTA BYTES
> USED        %USED     MAX AVAIL     OBJECTS     DIRTY     READ        WRITE
>      RAW USED
>    .rgw.root                     18     N/A               N/A
> 1.1 KiB         0        50 GiB           4        4         63 B         4
> B      1.1 KiB
>    default.rgw.meta              19     N/A               N/A
> 4.1 KiB         0        50 GiB          22       22      1.5 KiB       182
> B      9.0 KiB
>    default.rgw.log               20     N/A               N/A
>    0 B         0        50 GiB         107      107      307 KiB     205
> KiB          0 B
>    default.rgw.control           21     N/A               N/A
>    0 B         0        50 GiB           5        5          0 B         0
> B          0 B
>    default.rgw.buckets.index     22     N/A               N/A
>    0 B         0        50 GiB           5        5        501 B        60
> B          0 B
>    default.rgw.buckets.data      23     N/A               N/A
> 377 MiB      0.39        50 GiB         160      160        385 B     1.1
> KiB      603 MiB
> [root@server1 ~]#
> 
> [root@server1 ~]# ceph osd df tree
> ID CLASS WEIGHT  REWEIGHT SIZE   USE     DATA    OMAP  META     AVAIL  %USE
> VAR  PGS TYPE NAME
> -1       0.16727        -    0 B     0 B     0 B   0 B      0 B    0 B    0
>   0   - root default
> -3       0.05576        -    0 B     0 B     0 B   0 B      0 B    0 B    0
>   0   -     host server1
> 0   hdd 0.01859  1.00000    0 B     0 B     0 B   0 B      0 B    0 B    0
>   0   0         osd.0
> 1   hdd 0.01859        0    0 B     0 B     0 B   0 B      0 B    0 B    0
>   0   0         osd.1
> 2   hdd 0.01859        0    0 B     0 B     0 B   0 B      0 B    0 B    0
>   0   0         osd.2
> -5       0.05576        - 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB    0
>   0   -     host server2
> 3   hdd 0.01859  1.00000    0 B     0 B     0 B   0 B      0 B    0 B    0
>   0   0         osd.3
> 4   hdd 0.01859        0    0 B     0 B     0 B   0 B      0 B    0 B    0
>   0   0         osd.4
> 5   hdd 0.01859  1.00000 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 7.11
> 1.04  99         osd.5
> -7       0.05576        -    0 B     0 B     0 B   0 B      0 B    0 B    0
>   0   -     host server3
> 6   hdd 0.01859  1.00000 19 GiB 1.2 GiB 249 MiB 3 KiB 1024 MiB 18 GiB 6.55
> 0.96  78         osd.6
> 7   hdd 0.01859  1.00000    0 B     0 B     0 B   0 B      0 B    0 B    0
>   0   0         osd.7
> 8   hdd 0.01859  1.00000    0 B     0 B     0 B   0 B      0 B    0 B    0
>   0   0         osd.8
>                    TOTAL 38 GiB 2.6 GiB 610 MiB 6 KiB  2.0 GiB 35 GiB 6.83
> 
> MIN/MAX VAR: 0/1.04  STDDEV: 5.58
> [root@server1 ~]#
> 
> 
> I'm kind of newbie to ceph, so any help or hint would be appreciated.
> Did I hit a bug or something is wrong with my configuration?
> 
> Thanks a lot,
> Khodayar
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx





[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux