They're all bluefs_single_shared_device, if I understand your
question. There's no room left on the devices to expand.
We started at quincy with this cluster, and didn't vary too much from
the Redhat Ceph storage 6 documentation for setting it up.
On Tue, Nov 26, 2024 at 4:48 AM Igor Fedotov <igor.fedotov@xxxxxxxx>
wrote:
Hi John,
you haven't described your OSD volume configuration but you might
want
to try adding standalone DB volume if OSD uses LVM and has single
main
device only.
'ceph-volume lvm new-db' command is the preferred way of doing
that, see
https://docs.ceph.com/en/quincy/ceph-volume/lvm/newdb/
Thanks,
Igor
On 25.11.2024 21:37, John Jasen wrote:
> Ceph version 17.2.6
>
> After a power loss event affecting my ceph cluster, I've been
putting
> humpty dumpty back together since.
>
> One problem I face is that with objects degraded, rebalancing
doesn't run
> -- and this resulted in several of my fast OSDs filling up.
>
> I have 8 OSDs currently down, 100% full (exceeding all the full
ratio
> settings on by default or I toggled to try and keep it
together), and when
> I try to restart them, they fail out. Is there any way to bring
these back
> from the dead?
>
> Here's some interesting output from journalctl -xeu on the
failed OSD:
>
> ceph-osd[2383080]: bluestore::NCB::__restore_allocator::No Valid
allocation
> info on disk (empty file)
> ceph-osd[2383080]: bluestore(/var/lib/ceph/osd/ceph-242)
> _init_alloc::NCB::restore_allocator() failed! Run Full Recovery
from ONodes
> (might take a while) ...
>
> ceph-osd[2389725]: bluefs _allocate allocation failed, needed 0x3000
>
> ceph-6ab85342-53d6-11ee-88a7-e43d1a153e91-osd-242[2389718]: -2>
> 2024-11-25T18:31:42.070+0000 7f0adfdef540 -1 bluefs _flush_range_F
> allocated: 0x0 offset: 0x0 length: 0x230f
> ceph-osd[2389725]: bluefs _flush_range_F allocated: 0x0 offset:
0x0 length:
> 0x230f
>
> Followed quickly by an abort:
>
>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.6/rpm/el8/BUILD/ceph-17.2.6/src/os/bluestore/BlueFS.cc:
> In funct>
>
>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.6/rpm/el8/BUILD/ceph-17.2.6/src/os/bluestore/BlueFS.cc:
> 3380: ce>
>
> ceph version
> 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
> 1:
> (ceph::__ceph_abort(char const*, int, char const*,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&)+0xd7) [0x559bf4361d2f]
> 2:
> (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x7a9) [0x559bf4b225f9]
> 3:
> (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa2)
[0x559bf4b22812]
> 4:
> (BlueFS::fsync(BlueFS::FileWriter*)+0x8e) [0x559bf4b40c3e]
> 5:
> (BlueRocksWritableFile::Sync()+0x19) [0x559bf4b51ed9]
> 6:
> (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&,
> rocksdb::IODebugContext*)+0x22) [0x559bf507fbd2]
> 7:
> (rocksdb::WritableFileWriter::SyncInternal(bool)+0x5aa)
[0x559bf51a880a]
> 8:
> (rocksdb::WritableFileWriter::Sync(bool)+0x100) [0x559bf51aa0a0]
> 9:
> (rocksdb::SyncManifest(rocksdb::Env*,
rocksdb::ImmutableDBOptions const*,
> rocksdb::WritableFileWriter*)+0x10b) [0x559bf51a3bfb]
> 10:
>
(rocksdb::VersionSet::ProcessManifestWrites(std::deque<rocksdb::VersionSet::ManifestWriter,
> std::allocator<rocksdb::VersionSet::ManifestWriter> >&,
> rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, rocks>
> 11:
>
(rocksdb::VersionSet::LogAndApply(rocksdb::autovector<rocksdb::ColumnFamilyData*,
> 8ul> const&, rocksdb::autovector<rocksdb::MutableCFOptions
const*, 8ul>
> const&, rocksdb::autovector<rocksdb::autovector<rocksdb::>
> 12:
> (rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*,
> rocksdb::MutableCFOptions const&, rocksdb::VersionEdit*,
> rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool,
> rocksdb::ColumnFamilyOptions const>
> 13:
> (rocksdb::DBImpl::DeleteUnreferencedSstFiles()+0xa30)
[0x559bf50bd250]
> 14:
>
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool,
bool, bool,
> unsigned long*)+0x13f1) [0x559bf50d3f21]
> 15:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::Colu>
> 16:
> (rocksdb::DB::Open(rocksdb::DBOptions const&,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFa>
> 17:
> (RocksDBStore::do_open(std::ostream&, bool, bool,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&)+0x77a) [0x559bf503766a]
> 18:
> (BlueStore::_open_db(bool, bool, bool)+0xbb4) [0x559bf4a4bff4]
> 19:
> (BlueStore::_open_db_and_around(bool, bool)+0x500) [0x559bf4a766e0]
> 20:
> (BlueStore::_mount()+0x396) [0x559bf4a795d6]
> 21:
> (OSD::init()+0x556) [0x559bf44a0eb6]
> 22: main()
> 23:
> __libc_start_main()
> 24: _start()
>
> *** Caught signal (Aborted) **
> in thread
> 7f0adfdef540 thread_name:ceph-osd
>
> ceph version
> 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
> 1:
> /lib64/libpthread.so.0(+0x12cf0) [0x7f0addff1cf0]
> 2: gsignal()
> 3: abort()
> 4:
> (ceph::__ceph_abort(char const*, int, char const*,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&)+0x197) [0x559bf4361def]
> 5:
> (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x7a9) [0x559bf4b225f9]
> 6:
> (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa2)
[0x559bf4b22812]
> 7:
> (BlueFS::fsync(BlueFS::FileWriter*)+0x8e) [0x559bf4b40c3e]
> 8:
> (BlueRocksWritableFile::Sync()+0x19) [0x559bf4b51ed9]
> 9:
> (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&,
> rocksdb::IODebugContext*)+0x22) [0x559bf507fbd2]
> 10:
> (rocksdb::WritableFileWriter::SyncInternal(bool)+0x5aa)
[0x559bf51a880a]
> 11:
> (rocksdb::WritableFileWriter::Sync(bool)+0x100) [0x559bf51aa0a0]
> 12:
> (rocksdb::SyncManifest(rocksdb::Env*,
rocksdb::ImmutableDBOptions const*,
> rocksdb::WritableFileWriter*)+0x10b) [0x559bf51a3bfb]
> 13:
>
(rocksdb::VersionSet::ProcessManifestWrites(std::deque<rocksdb::VersionSet::ManifestWriter,
> std::allocator<rocksdb::VersionSet::ManifestWriter> >&,
> rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool, rocks>
> 14:
>
(rocksdb::VersionSet::LogAndApply(rocksdb::autovector<rocksdb::ColumnFamilyData*,
> 8ul> const&, rocksdb::autovector<rocksdb::MutableCFOptions
const*, 8ul>
> const&, rocksdb::autovector<rocksdb::autovector<rocksdb::>
> 15:
> (rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*,
> rocksdb::MutableCFOptions const&, rocksdb::VersionEdit*,
> rocksdb::InstrumentedMutex*, rocksdb::FSDirectory*, bool,
> rocksdb::ColumnFamilyOptions const>
> 16:
> (rocksdb::DBImpl::DeleteUnreferencedSstFiles()+0xa30)
[0x559bf50bd250]
> 17:
>
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool,
bool, bool,
> unsigned long*)+0x13f1) [0x559bf50d3f21]
> 18:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::Colu>
> 19:
> (rocksdb::DB::Open(rocksdb::DBOptions const&,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFa>
> 20:
> (RocksDBStore::do_open(std::ostream&, bool, bool,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&)+0x77a) [0x559bf503766a]
> 21:
> (BlueStore::_open_db(bool, bool, bool)+0xbb4) [0x559bf4a4bff4]
> 22:
> (BlueStore::_open_db_and_around(bool, bool)+0x500) [0x559bf4a766e0]
> 23:
> (BlueStore::_mount()+0x396) [0x559bf4a795d6]
> 24:
> (OSD::init()+0x556) [0x559bf44a0eb6]
> 25: main()
> 26:
> __libc_start_main()
> 27: _start()
> NOTE: a copy
> of the executable, or `objdump -rdS <executable>` is needed to
interpret
> this.
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
