Let me see if I have the approach right'ish: scrounge some more disk for the servers with full/down OSDs. partition the new disks into LVs for each downed OSD. Attach as a lvm new-db to the downed OSDs. Restart the OSDs. Profit. Is that about right? On Tue, Nov 26, 2024 at 11:28 AM Igor Fedotov <igor.fedotov@xxxxxxxx> wrote: > Well, so there is a single shared volume (disk) per OSD, right? > > If so one can add dedicated DB volume to such an OSD - one done OSD will > have two underlying devices: main(which is original shared disk) and new > dedicated DB ones. And hence this will effectively provide additional > space for BlueFS/RocksDB and permit OSD to start up. > > I'm not aware of all the details how to do that with cephadm (or whatever > RH uses) but on bare metal setup this could be achieved by issuing > 'ceph-volume lvm new-db' command which will attach new LV (provided by > user) to specific OSD. > > > Thanks, > > Igor > > > On 26.11.2024 19:16, John Jasen wrote: > > They're all bluefs_single_shared_device, if I understand your question. > There's no room left on the devices to expand. > > We started at quincy with this cluster, and didn't vary too much from the > Redhat Ceph storage 6 documentation for setting it up. > > > On Tue, Nov 26, 2024 at 4:48 AM Igor Fedotov <igor.fedotov@xxxxxxxx> > wrote: > >> Hi John, >> >> you haven't described your OSD volume configuration but you might want >> to try adding standalone DB volume if OSD uses LVM and has single main >> device only. >> >> 'ceph-volume lvm new-db' command is the preferred way of doing that, see >> >> https://docs.ceph.com/en/quincy/ceph-volume/lvm/newdb/ >> >> >> Thanks, >> >> Igor >> >> On 25.11.2024 21:37, John Jasen wrote: >> > Ceph version 17.2.6 >> > >> > After a power loss event affecting my ceph cluster, I've been putting >> > humpty dumpty back together since. >> > >> > One problem I face is that with objects degraded, rebalancing doesn't >> run >> > -- and this resulted in several of my fast OSDs filling up. >> > >> > I have 8 OSDs currently down, 100% full (exceeding all the full ratio >> > settings on by default or I toggled to try and keep it together), and >> when >> > I try to restart them, they fail out. Is there any way to bring these >> back >> > from the dead? >> > >> > Here's some interesting output from journalctl -xeu on the failed OSD: >> > >> > ceph-osd[2383080]: bluestore::NCB::__restore_allocator::No Valid >> allocation >> > info on disk (empty file) >> > ceph-osd[2383080]: bluestore(/var/lib/ceph/osd/ceph-242) >> > _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from >> ONodes >> > (might take a while) ... >> > >> > ceph-osd[2389725]: bluefs _allocate allocation failed, needed 0x3000 >> > >> > ceph-6ab85342-53d6-11ee-88a7-e43d1a153e91-osd-242[2389718]: -2> >> > 2024-11-25T18:31:42.070+0000 7f0adfdef540 -1 bluefs _flush_range_F >> > allocated: 0x0 offset: 0x0 length: 0x230f >> > ceph-osd[2389725]: bluefs _flush_range_F allocated: 0x0 offset: 0x0 >> length: >> > 0x230f >> > >> > Followed quickly by an abort: >> > >> > >> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.6/rpm/el8/BUILD/ceph-17.2.6/src/os/bluestore/BlueFS.cc: >> > In funct> >> > >> > >> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.6/rpm/el8/BUILD/ceph-17.2.6/src/os/bluestore/BlueFS.cc: >> > 3380: ce> >> > >> > ceph >> version >> > 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) >> > 1: >> > (ceph::__ceph_abort(char const*, int, char const*, >> > std::__cxx11::basic_string<char, std::char_traits<char>, >> > std::allocator<char> > const&)+0xd7) [0x559bf4361d2f] >> > 2: >> > (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned >> > long)+0x7a9) [0x559bf4b225f9] >> > 3: >> > (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa2) >> [0x559bf4b22812] >> > 4: >> > (BlueFS::fsync(BlueFS::FileWriter*)+0x8e) [0x559bf4b40c3e] >> > 5: >> > (BlueRocksWritableFile::Sync()+0x19) [0x559bf4b51ed9] >> > 6: >> > (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, >> > rocksdb::IODebugContext*)+0x22) [0x559bf507fbd2] >> > 7: >> > (rocksdb::WritableFileWriter::SyncInternal(bool)+0x5aa) [0x559bf51a880a] >> > 8: >> > (rocksdb::WritableFileWriter::Sync(bool)+0x100) [0x559bf51aa0a0] >> > 9: >> > (rocksdb::SyncManifest(rocksdb::Env*, rocksdb::ImmutableDBOptions >> const*, >> > rocksdb::WritableFileWriter*)+0x10b) [0x559bf51a3bfb] >> > 10: >> > >> (rocksdb::VersionSet::ProcessManifestWrites(std::deque<rocksdb::VersionSet::ManifestWriter, >> > std::allocator<rocksdb::VersionSet::ManifestWriter> >&, >> > rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, rocks> >> > 11: >> > >> (rocksdb::VersionSet::LogAndApply(rocksdb::autovector<rocksdb::ColumnFamilyData*, >> > 8ul> const&, rocksdb::autovector<rocksdb::MutableCFOptions const*, 8ul> >> > const&, rocksdb::autovector<rocksdb::autovector<rocksdb::> >> > 12: >> > (rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*, >> > rocksdb::MutableCFOptions const&, rocksdb::VersionEdit*, >> > rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, >> > rocksdb::ColumnFamilyOptions const> >> > 13: >> > (rocksdb::DBImpl::DeleteUnreferencedSstFiles()+0xa30) [0x559bf50bd250] >> > 14: >> > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, >> > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, >> bool, >> > unsigned long*)+0x13f1) [0x559bf50d3f21] >> > 15: >> > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, >> > std::__cxx11::basic_string<char, std::char_traits<char>, >> > std::allocator<char> > const&, >> std::vector<rocksdb::ColumnFamilyDescriptor, >> > std::allocator<rocksdb::Colu> >> > 16: >> > (rocksdb::DB::Open(rocksdb::DBOptions const&, >> > std::__cxx11::basic_string<char, std::char_traits<char>, >> > std::allocator<char> > const&, >> std::vector<rocksdb::ColumnFamilyDescriptor, >> > std::allocator<rocksdb::ColumnFa> >> > 17: >> > (RocksDBStore::do_open(std::ostream&, bool, bool, >> > std::__cxx11::basic_string<char, std::char_traits<char>, >> > std::allocator<char> > const&)+0x77a) [0x559bf503766a] >> > 18: >> > (BlueStore::_open_db(bool, bool, bool)+0xbb4) [0x559bf4a4bff4] >> > 19: >> > (BlueStore::_open_db_and_around(bool, bool)+0x500) [0x559bf4a766e0] >> > 20: >> > (BlueStore::_mount()+0x396) [0x559bf4a795d6] >> > 21: >> > (OSD::init()+0x556) [0x559bf44a0eb6] >> > 22: main() >> > 23: >> > __libc_start_main() >> > 24: >> _start() >> > >> > *** Caught signal (Aborted) ** >> > in thread >> > 7f0adfdef540 thread_name:ceph-osd >> > >> > ceph >> version >> > 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) >> > 1: >> > /lib64/libpthread.so.0(+0x12cf0) [0x7f0addff1cf0] >> > 2: >> gsignal() >> > 3: abort() >> > 4: >> > (ceph::__ceph_abort(char const*, int, char const*, >> > std::__cxx11::basic_string<char, std::char_traits<char>, >> > std::allocator<char> > const&)+0x197) [0x559bf4361def] >> > 5: >> > (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned >> > long)+0x7a9) [0x559bf4b225f9] >> > 6: >> > (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa2) >> [0x559bf4b22812] >> > 7: >> > (BlueFS::fsync(BlueFS::FileWriter*)+0x8e) [0x559bf4b40c3e] >> > 8: >> > (BlueRocksWritableFile::Sync()+0x19) [0x559bf4b51ed9] >> > 9: >> > (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, >> > rocksdb::IODebugContext*)+0x22) [0x559bf507fbd2] >> > 10: >> > (rocksdb::WritableFileWriter::SyncInternal(bool)+0x5aa) [0x559bf51a880a] >> > 11: >> > (rocksdb::WritableFileWriter::Sync(bool)+0x100) [0x559bf51aa0a0] >> > 12: >> > (rocksdb::SyncManifest(rocksdb::Env*, rocksdb::ImmutableDBOptions >> const*, >> > rocksdb::WritableFileWriter*)+0x10b) [0x559bf51a3bfb] >> > 13: >> > >> (rocksdb::VersionSet::ProcessManifestWrites(std::deque<rocksdb::VersionSet::ManifestWriter, >> > std::allocator<rocksdb::VersionSet::ManifestWriter> >&, >> > rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, rocks> >> > 14: >> > >> (rocksdb::VersionSet::LogAndApply(rocksdb::autovector<rocksdb::ColumnFamilyData*, >> > 8ul> const&, rocksdb::autovector<rocksdb::MutableCFOptions const*, 8ul> >> > const&, rocksdb::autovector<rocksdb::autovector<rocksdb::> >> > 15: >> > (rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*, >> > rocksdb::MutableCFOptions const&, rocksdb::VersionEdit*, >> > rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, >> > rocksdb::ColumnFamilyOptions const> >> > 16: >> > (rocksdb::DBImpl::DeleteUnreferencedSstFiles()+0xa30) [0x559bf50bd250] >> > 17: >> > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, >> > std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, >> bool, >> > unsigned long*)+0x13f1) [0x559bf50d3f21] >> > 18: >> > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, >> > std::__cxx11::basic_string<char, std::char_traits<char>, >> > std::allocator<char> > const&, >> std::vector<rocksdb::ColumnFamilyDescriptor, >> > std::allocator<rocksdb::Colu> >> > 19: >> > (rocksdb::DB::Open(rocksdb::DBOptions const&, >> > std::__cxx11::basic_string<char, std::char_traits<char>, >> > std::allocator<char> > const&, >> std::vector<rocksdb::ColumnFamilyDescriptor, >> > std::allocator<rocksdb::ColumnFa> >> > 20: >> > (RocksDBStore::do_open(std::ostream&, bool, bool, >> > std::__cxx11::basic_string<char, std::char_traits<char>, >> > std::allocator<char> > const&)+0x77a) [0x559bf503766a] >> > 21: >> > (BlueStore::_open_db(bool, bool, bool)+0xbb4) [0x559bf4a4bff4] >> > 22: >> > (BlueStore::_open_db_and_around(bool, bool)+0x500) [0x559bf4a766e0] >> > 23: >> > (BlueStore::_mount()+0x396) [0x559bf4a795d6] >> > 24: >> > (OSD::init()+0x556) [0x559bf44a0eb6] >> > 25: main() >> > 26: >> > __libc_start_main() >> > 27: >> _start() >> > NOTE: a >> copy >> > of the executable, or `objdump -rdS <executable>` is needed to interpret >> > this. >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users@xxxxxxx >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |