Hi Martin, Thanks for sharing your thoughts. I was wondering how database would handle low disk space when receiving new requests. Maybe it's not much different than what you've described below. Cheers, Frédéric. ----- Le 29 Nov 24, à 17:46, Martin Konold <martin.konold@xxxxxxxxxx> a écrit : > Hi, > the traditional solution is to deny anything but deletions and either write the > transaction log to another device or even Filesystem or add support deletions > without a transaction log together with a force switch. > Regards > --martin > Am 29.11.2024 12:03 schrieb Igor Fedotov <igor.fedotov@xxxxxxxx>: >> Hi Frederic, >> >My question was more about why bluefs would still fail to allocate 4k >> chunks after being allowed to do so by >> https://tracker.ceph.com/issues/53466 (John's case with v17.2.6 actually) >> My hypothesis is that it's facing real "no-space" case not the one I >> explained above. But again - we haven't performed thorough analysis of >> John's case, so we are just speculating... >> >Is BlueFS aware of the remaining space and maybe using some sort of >> reserved blocks/chunks like other filesystems to handle full/near null >> situations ? >> What remaining space are you talking here - if the one consisting of >> small (<64K) extents only then BlueFS hopefully uses it since v17.2.6. >> Reserving more spare space at Bluefs on mkfs? Previously at Bluestore >> team we had discussions on something like that to permit easier recovery >> from "no-space" cases. No final solution has been taken yet and in fact >> this provides no complete fix for the issue anyway - that spare space >> might end at some point as well... >> >If so, then it should never crash, right? >> What we have here is not a real crash (although looks like that) - it's >> an expected assertion. We just don't have good enough [automatic] >> scenario to exit from this state. Hence "crying" about that aloud. >> The problem is that by its design RocksDB has to write out some data >> (either due to the need for internal maintenance or to fulfill client >> requests) on any update access. So at some point we have no space to >> record such a transaction. How one can proceed in that case - refuse to >> execute it and return an error? OK, but what's next? Any followup data >> removal would need some transactions to be recorded in RocksDB as well. >> And DB/FS would need more space for that. Use some reserved spare space? >> But that's conceptually similar to stalling OSD writes at some free >> space threshold we already have at OSD level - as you can see this >> protection doesn't work from time to time. >> So IMO it's better polish existing protection means then. >> Not to mention - I'm pretty sure that someone would abuse additional >> spare space mechanics if any and finally face the same problem at some >> point. Thus triggering another iteration of it.. ;) >> Thanks, >> Igor >> On 29.11.2024 10:20, Frédéric Nass wrote: >> > Hi Igor, >> > Thank you for taking the time to explains the fragmentation issue. I >> > had figured out the most part of it by reading the tracker and the PR >> > but it's always clearer when you explain it. >> > My question was more about why bluefs would still fail to allocate 4k >> > chunks after being allowed to do so by >> > https://tracker.ceph.com/issues/53466 (John's case with v17.2.6 actually) >> > Is BlueFS aware of the remaining space and maybe using some sort of >> > reserved blocks/chunks like other filesystems to handle full/near null >> > situations ? If so, then it should never crash, right? >> > Like other filesystems don't crash, drives's firmwares dont crash, etc. >> > Thanks, >> > Frédéric. >> > ----- Le 28 Nov 24, à 12:52, Igor Fedotov <igor.fedotov@xxxxxxxx> a >> > écrit : >> > Hi Frederic, >> > here is an overview of the case when BlueFS ıs unable to allocate >> > more space at main/shared device albeıt free space is available. >> > Below I'm talking about stuff exısted before fıxıng >> > https://tracker.ceph.com/issues/53466. >> > First of al - BlueFS's minimal allocation unit for shared device >> > was bluefs_shared_alloc_size (=64K by default). Which means that >> > it was unable to use e.g. 2x32K or 16x4K chunks when it needed >> > additional 64K bytes. >> > Secondly - sometimes RocksDB performs recovery - and some other >> > maintenance tasks that require space allocation - on startup. >> > Which evidently triggers allocation of N*64K chunks from shared >> > device. >> > Thirdly - a while ago we switched to 4K chunk allocations for user >> > data (please not confuse with BlueFS allocation). Which >> > potentially could result ın specific free space fragmentation >> > pattern when there ıs limited (or even empty) set of long (>=64K) >> > chunks free. Still technically having enough free space available. >> > E.g. free extent list could look like (off~len, both in hex): >> > 0x0~1000, 0x2000~1000, 0x4000~2000, 0x10000~4000, 0x2000~1000, etc... >> > In that case original BlueFS allocator implementation was unable >> > to locate more free space which in turn was effectively breaking >> > both RockDB and OSD boot up. >> > One should realize that the above free space fragmentation depends >> > on a bunch of factors, none of which is absolutely dominating: >> > 1. how user write/remove objects >> > 2. how allocator seeks for free space >> > 3. how much free space is available >> > So we don't have full control on 1. and 3. and have limited >> > opportunities in tuning 2. >> > Small device sizes and high space utilization severely increase >> > the probability for the issue to happen but theoretically even a >> > large disk with mediocre utilization could reach "bad" state over >> > time if used (by both clients and allocator) >> > "improperly/inefficiently". Hence tuning thresholds can reduce the >> > issue's probability to occur (at cost of additional spare space >> > waste) but it isn't a silver bullet. >> > https://tracker.ceph.com/issues/53466 fixes (or rather works >> > around) the issue by allowing BlueFS to use 4K extents. Plus we're >> > working on making better resulting free space fragmentation on >> > aged OSDs by improving allocation strategies, e.g. see : >> > - https://github.com/ceph/ceph/pull/52489 >> > - https://github.com/ceph/ceph/pull/57789 >> > - https://github.com/ceph/ceph/pull/60870 >> > Hope this is helpful. >> > Thanks, >> > Igor >> > On 27.11.2024 16:31, Frédéric Nass wrote: >> > ----- Le 27 Nov 24, à 10:19, Igor Fedotov >> > <igor.fedotov@xxxxxxxx> <mailto:igor.fedotov@xxxxxxxx> a écrit : >> > Hi Istvan, >> > first of all let me make a remark that we don't know why >> > BlueStore is out of space at John's cluster. >> > It's just an unconfirmed hypothesis from Frederic that >> > it's caused by high fragmentation and BlueFS'es inability >> > to use chunks smaller than 64K. In fact fragmentation >> > issue is fixed since 17.2.6 so I doubt that's the problem. >> > Hi Igor, >> > I wasn't actually pointing this as the root cause (since >> > John's already using 17.2.6) but more to explain the context, >> > but while we're at it... >> > Could you elaborate on circumstances that could prevent BlueFS >> > from being able to allocate chunks in collocated OSDs >> > scenario? Does this ability depend on near/full thresholds >> > being reached or not? If so then icreasing these thresholds by >> > 1-2% may help avoiding the crash, no? >> > Also, if BlueFS is aware of these thresholds, shouldn't an >> > OSDs be able to start and live without crashing even when it's >> > full and simply (maybe easier said than done...) refuse any >> > I/Os? Sorry for the noob questions. :-) >> > This topic is particularly important when using NVMe drives as >> > 'collocated' OSDs, expecially since they often host critical >> > metadata pools (cephfs, rgw index). >> > Cheers, >> > Frédéric. >> > Thanks, >> > Igor >> > On 27.11.2024 4:01, Szabo, Istvan (Agoda) wrote: >> > Hi, >> > This issue should not happen anymore from 17.2.8 am I >> > correct? In this version all the fragmentation issue >> > should have gone even with collocated wal+db+block. >> > ------------------------------------------------------------------------ >> > *From:* Frédéric Nass <frederic.nass@xxxxxxxxxxxxxxxx> >> > <mailto:frederic.nass@xxxxxxxxxxxxxxxx> >> > *Sent:* Wednesday, November 27, 2024 6:12:46 AM >> > *To:* John Jasen <jjasen@xxxxxxxxx> >> > <mailto:jjasen@xxxxxxxxx> >> > *Cc:* Igor Fedotov <igor.fedotov@xxxxxxxx> >> > <mailto:igor.fedotov@xxxxxxxx>; ceph-users >> > <ceph-users@xxxxxxx> <mailto:ceph-users@xxxxxxx> >> > *Subject:* Re: down OSDs, Bluestore out >> > of space, unable to restart >> > Email received from the internet. If in doubt, don't >> > click any link nor open any attachment ! >> > ________________________________ >> > Hi John, >> > That's about right. Two potential solutions exist: >> > 1. Adding a new drive to the server and sharing it for >> > RocksDB metadata, or >> > 2. Repurposing one of the failed OSDs for the same >> > purpose (if adding more drives isn't feasible). >> > Igor's post #6 [1] explains the challenges with >> > co-located OSDs (DB+WAL+data on the same device) when >> > they run out of space, where significant fragmentation >> > occurs and BlueFS and BlueStore block sizes are >> > misaligned. The solution (included in 17.2.6) was to >> > allow BlueFS to allocate 4k extents when it couldn't >> > find 64k contiguous extents. However, it seems that >> > even with this fix, these OSDs still can't boot up. >> > Therefore, the recommendation is to extend the RocksDB >> > volume to another device as a temporary workaround. >> > Before proceeding, I recommend checking the failed >> > OSDs' bluefs_shared_alloc_size value. If it's 64k, you >> > might want to try lowering this to 32k or even 4k, as >> > some users reported [2] that reducing this value >> > helped failed OSDs boot up and remain stable for a >> > period of time. Might be worth checking and trying. >> > Regards, >> > Frédéric. >> > [1] https://tracker.ceph.com/issues/53466#note-6 >> > [2] >> > https://github.com/rook/rook/issues/9885#issuecomment-1761076861"; >> > <https://github.com/rook/rook/issues/9885#issuecomment-1761076861";> >> > ________________________________ >> > De : John Jasen <jjasen@xxxxxxxxx> >> > <mailto:jjasen@xxxxxxxxx> >> > Envoyé : mardi 26 novembre 2024 18:50 >> > À : Igor Fedotov >> > Cc: ceph-users >> > Objet : Re: down OSDs, Bluestore out of >> > space, unable to restart >> > Let me see if I have the approach right'ish: >> > scrounge some more disk for the servers with full/down >> > OSDs. >> > partition the new disks into LVs for each downed OSD. >> > Attach as a lvm new-db to the downed OSDs. >> > Restart the OSDs. >> > Profit. >> > Is that about right? >> > On Tue, Nov 26, 2024 at 11:28 AM Igor Fedotov >> > <igor.fedotov@xxxxxxxx> <mailto:igor.fedotov@xxxxxxxx> >> > wrote: >> > > Well, so there is a single shared volume (disk) per >> > OSD, right? >> > > If so one can add dedicated DB volume to such an OSD >> > - one done OSD will >> > > have two underlying devices: main(which is original >> > shared disk) and new >> > > dedicated DB ones. And hence this will effectively >> > provide additional >> > > space for BlueFS/RocksDB and permit OSD to start up. >> > > I'm not aware of all the details how to do that with >> > cephadm (or whatever >> > > RH uses) but on bare metal setup this could be >> > achieved by issuing >> > > 'ceph-volume lvm new-db' command which will attach >> > new LV (provided by >> > > user) to specific OSD. >> > > Thanks, >> > > Igor >> > > On 26.11.2024 19:16, John Jasen wrote: >> > > They're all bluefs_single_shared_device, if I >> > understand your question. >> > > There's no room left on the devices to expand. >> > > We started at quincy with this cluster, and didn't >> > vary too much from the >> > > Redhat Ceph storage 6 documentation for setting it up. >> > > On Tue, Nov 26, 2024 at 4:48 AM Igor Fedotov >> > <igor.fedotov@xxxxxxxx> <mailto:igor.fedotov@xxxxxxxx> >> > > wrote: >> > >> Hi John, >> > >> you haven't described your OSD volume configuration >> > but you might want >> > >> to try adding standalone DB volume if OSD uses LVM >> > and has single main >> > >> device only. >> > >> 'ceph-volume lvm new-db' command is the preferred >> > way of doing that, see >> > >> https://docs.ceph.com/en/quincy/ceph-volume/lvm/newdb/ >> > >> Thanks, >> > >> Igor >> > >> On 25.11.2024 21:37, John Jasen wrote: >> > >> > Ceph version 17.2.6 >> > >> > After a power loss event affecting my ceph >> > cluster, I've been putting >> > >> > humpty dumpty back together since. >> > >> > One problem I face is that with objects degraded, >> > rebalancing doesn't >> > >> run >> > >> > -- and this resulted in several of my fast OSDs >> > filling up. >> > >> > I have 8 OSDs currently down, 100% full >> > (exceeding all the full ratio >> > >> > settings on by default or I toggled to try and >> > keep it together), and >> > >> when >> > >> > I try to restart them, they fail out. Is there >> > any way to bring these >> > >> back >> > >> > from the dead? >> > >> > Here's some interesting output from journalctl >> > -xeu on the failed OSD: >> > >> > ceph-osd[2383080]: >> > bluestore::NCB::__restore_allocator::No Valid >> > >> allocation >> > >> > info on disk (empty file) >> > >> > ceph-osd[2383080]: >> > bluestore(/var/lib/ceph/osd/ceph-242) >> > >> > _init_alloc::NCB::restore_allocator() failed! Run >> > Full Recovery from >> > >> ONodes >> > >> > (might take a while) ... >> > >> > ceph-osd[2389725]: bluefs _allocate allocation >> > failed, needed 0x3000 >> > ceph-6ab85342-53d6-11ee-88a7-e43d1a153e91-osd-242[2389718]: >> > -2> >> > >> > 2024-11-25T18:31:42.070+0000 7f0adfdef540 -1 >> > bluefs _flush_range_F >> > >> > allocated: 0x0 offset: 0x0 length: 0x230f >> > >> > ceph-osd[2389725]: bluefs _flush_range_F >> > allocated: 0x0 offset: 0x0 >> > >> length: >> > >> > 0x230f >> > >> > Followed quickly by an abort: >> > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.6/rpm/el8/BUILD/ceph-17.2.6/src/os/bluestore/BlueFS.cc: >> > >> > In funct> >> > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.6/rpm/el8/BUILD/ceph-17.2.6/src/os/bluestore/BlueFS.cc: >> > >> > 3380: ce> >> > >> > ceph >> > >> version >> > >> > 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) >> > quincy (stable) >> > >> > 1: >> > >> > (ceph::__ceph_abort(char const*, int, char const*, >> > >> > std::__cxx11::basic_string<char, >> > std::char_traits<char>, >> > >> > std::allocator<char> > const&)+0xd7) [0x559bf4361d2f] >> > >> > 2: >> > >> > (BlueFS::_flush_range_F(BlueFS::FileWriter*, >> > unsigned long, unsigned >> > >> > long)+0x7a9) [0x559bf4b225f9] >> > >> > 3: >> > >> > (BlueFS::_flush_F(BlueFS::FileWriter*, bool, >> > bool*)+0xa2) >> > >> [0x559bf4b22812] >> > >> > 4: >> > >> > (BlueFS::fsync(BlueFS::FileWriter*)+0x8e) >> > [0x559bf4b40c3e] >> > >> > 5: >> > >> > (BlueRocksWritableFile::Sync()+0x19) [0x559bf4b51ed9] >> > >> > 6: >> > (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions >> > const&, >> > >> > rocksdb::IODebugContext*)+0x22) [0x559bf507fbd2] >> > >> > 7: >> > (rocksdb::WritableFileWriter::SyncInternal(bool)+0x5aa) >> > [0x559bf51a880a] >> > >> > 8: >> > >> > (rocksdb::WritableFileWriter::Sync(bool)+0x100) >> > [0x559bf51aa0a0] >> > >> > 9: >> > >> > (rocksdb::SyncManifest(rocksdb::Env*, >> > rocksdb::ImmutableDBOptions >> > >> const*, >> > >> > rocksdb::WritableFileWriter*)+0x10b) [0x559bf51a3bfb] >> > >> > 10: >> > (rocksdb::VersionSet::ProcessManifestWrites(std::deque<rocksdb::VersionSet::ManifestWriter, >> > std::allocator<rocksdb::VersionSet::ManifestWriter> >&, >> > >> > rocksdb::InstrumentedMutex*, >> > rocksdb::FSDirectory*, bool, rocks> >> > >> > 11: >> > (rocksdb::VersionSet::LogAndApply(rocksdb::autovector<rocksdb::ColumnFamilyData*, >> > >> > 8ul> const&, >> > rocksdb::autovector<rocksdb::MutableCFOptions const*, 8ul> >> > >> > const&, >> > rocksdb::autovector<rocksdb::autovector<rocksdb::> >> > >> > 12: >> > (rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*, >> > >> > rocksdb::MutableCFOptions const&, >> > rocksdb::VersionEdit*, >> > >> > rocksdb::InstrumentedMutex*, >> > rocksdb::FSDirectory*, bool, >> > >> > rocksdb::ColumnFamilyOptions const> >> > >> > 13: >> > (rocksdb::DBImpl::DeleteUnreferencedSstFiles()+0xa30) >> > [0x559bf50bd250] >> > >> > 14: >> > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, >> > >> > std::allocator<rocksdb::ColumnFamilyDescriptor> > >> > const&, bool, bool, >> > >> bool, >> > >> > unsigned long*)+0x13f1) [0x559bf50d3f21] >> > >> > 15: >> > >> > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, >> > >> > std::__cxx11::basic_string<char, >> > std::char_traits<char>, >> > >> > std::allocator<char> > const&, >> > >> std::vector<rocksdb::ColumnFamilyDescriptor, >> > >> > std::allocator<rocksdb::Colu> >> > >> > 16: >> > >> > (rocksdb::DB::Open(rocksdb::DBOptions const&, >> > >> > std::__cxx11::basic_string<char, >> > std::char_traits<char>, >> > >> > std::allocator<char> > const&, >> > >> std::vector<rocksdb::ColumnFamilyDescriptor, >> > >> > std::allocator<rocksdb::ColumnFa> >> > >> > 17: >> > >> > (RocksDBStore::do_open(std::ostream&, bool, bool, >> > >> > std::__cxx11::basic_string<char, >> > std::char_traits<char>, >> > >> > std::allocator<char> > const&)+0x77a) >> > [0x559bf503766a] >> > >> > 18: >> > >> > (BlueStore::_open_db(bool, bool, bool)+0xbb4) >> > [0x559bf4a4bff4] >> > >> > 19: >> > >> > (BlueStore::_open_db_and_around(bool, >> > bool)+0x500) [0x559bf4a766e0] >> > >> > 20: >> > >> > (BlueStore::_mount()+0x396) [0x559bf4a795d6] >> > >> > 21: >> > >> > (OSD::init()+0x556) [0x559bf44a0eb6] >> > >> > 22: main() >> > >> > 23: >> > >> > __libc_start_main() >> > >> > 24: >> > >> _start() >> > >> > *** Caught signal (Aborted) ** >> > >> > in thread >> > >> > 7f0adfdef540 thread_name:ceph-osd >> > >> > ceph >> > >> version >> > >> > 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) >> > quincy (stable) >> > >> > 1: >> > >> > /lib64/libpthread.so.0(+0x12cf0) [0x7f0addff1cf0] >> > >> > 2: >> > >> gsignal() >> > >> > 3: abort() >> > >> > 4: >> > >> > (ceph::__ceph_abort(char const*, int, char const*, >> > >> > std::__cxx11::basic_string<char, >> > std::char_traits<char>, >> > >> > std::allocator<char> > const&)+0x197) >> > [0x559bf4361def] >> > >> > 5: >> > >> > (BlueFS::_flush_range_F(BlueFS::FileWriter*, >> > unsigned long, unsigned >> > >> > long)+0x7a9) [0x559bf4b225f9] >> > >> > 6: >> > >> > (BlueFS::_flush_F(BlueFS::FileWriter*, bool, >> > bool*)+0xa2) >> > >> [0x559bf4b22812] >> > >> > 7: >> > >> > (BlueFS::fsync(BlueFS::FileWriter*)+0x8e) >> > [0x559bf4b40c3e] >> > >> > 8: >> > >> > (BlueRocksWritableFile::Sync()+0x19) [0x559bf4b51ed9] >> > >> > 9: >> > (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions >> > const&, >> > >> > rocksdb::IODebugContext*)+0x22) [0x559bf507fbd2] >> > >> > 10: >> > (rocksdb::WritableFileWriter::SyncInternal(bool)+0x5aa) >> > [0x559bf51a880a] >> > >> > 11: >> > >> > (rocksdb::WritableFileWriter::Sync(bool)+0x100) >> > [0x559bf51aa0a0] >> > >> > 12: >> > >> > (rocksdb::SyncManifest(rocksdb::Env*, >> > rocksdb::ImmutableDBOptions >> > >> const*, >> > >> > rocksdb::WritableFileWriter*)+0x10b) [0x559bf51a3bfb] >> > >> > 13: >> > (rocksdb::VersionSet::ProcessManifestWrites(std::deque<rocksdb::VersionSet::ManifestWriter, >> > std::allocator<rocksdb::VersionSet::ManifestWriter> >&, >> > >> > rocksdb::InstrumentedMutex*, >> > rocksdb::FSDirectory*, bool, rocks> >> > >> > 14: >> > (rocksdb::VersionSet::LogAndApply(rocksdb::autovector<rocksdb::ColumnFamilyData*, >> > >> > 8ul> const&, >> > rocksdb::autovector<rocksdb::MutableCFOptions const*, 8ul> >> > >> > const&, >> > rocksdb::autovector<rocksdb::autovector<rocksdb::> >> > >> > 15: >> > (rocksdb::VersionSet::LogAndApply(rocksdb::ColumnFamilyData*, >> > >> > rocksdb::MutableCFOptions const&, >> > rocksdb::VersionEdit*, >> > >> > rocksdb::InstrumentedMutex*, >> > rocksdb::FSDirectory*, bool, >> > >> > rocksdb::ColumnFamilyOptions const> >> > >> > 16: >> > (rocksdb::DBImpl::DeleteUnreferencedSstFiles()+0xa30) >> > [0x559bf50bd250] >> > >> > 17: >> > (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, >> > >> > std::allocator<rocksdb::ColumnFamilyDescriptor> > >> > const&, bool, bool, >> > >> bool, >> > >> > unsigned long*)+0x13f1) [0x559bf50d3f21] >> > >> > 18: >> > >> > (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, >> > >> > std::__cxx11::basic_string<char, >> > std::char_traits<char>, >> > >> > std::allocator<char> > const&, >> > >> std::vector<rocksdb::ColumnFamilyDescriptor, >> > >> > std::allocator<rocksdb::Colu> >> > >> > 19: >> > >> > (rocksdb::DB::Open(rocksdb::DBOptions const&, >> > >> > std::__cxx11::basic_string<char, >> > std::char_traits<char>, >> > >> > std::allocator<char> > const&, >> > >> std::vector<rocksdb::ColumnFamilyDescriptor, >> > >> > std::allocator<rocksdb::ColumnFa> >> > >> > 20: >> > >> > (RocksDBStore::do_open(std::ostream&, bool, bool, >> > >> > std::__cxx11::basic_string<char, >> > std::char_traits<char>, >> > >> > std::allocator<char> > const&)+0x77a) >> > [0x559bf503766a] >> > >> > 21: >> > >> > (BlueStore::_open_db(bool, bool, bool)+0xbb4) >> > [0x559bf4a4bff4] >> > >> > 22: >> > >> > (BlueStore::_open_db_and_around(bool, >> > bool)+0x500) [0x559bf4a766e0] >> > >> > 23: >> > >> > (BlueStore::_mount()+0x396) [0x559bf4a795d6] >> > >> > 24: >> > >> > (OSD::init()+0x556) [0x559bf44a0eb6] >> > >> > 25: main() >> > >> > 26: >> > >> > __libc_start_main() >> > >> > 27: >> > >> _start() >> > >> > NOTE: a >> > >> copy >> > >> > of the executable, or `objdump -rdS <executable>` >> > is needed to interpret >> > >> > this. >> > >> > _______________________________________________ >> > >> > ceph-users mailing list -- ceph-users@xxxxxxx >> > >> > To unsubscribe send an email to >> > ceph-users-leave@xxxxxxx >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users@xxxxxxx >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users@xxxxxxx >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > ------------------------------------------------------------------------ >> > This message is confidential and is for the >> > sole use of the intended recipient(s). It >> > may also be privileged or otherwise >> > protected by copyright or other legal rules. >> > If you have received it by mistake please >> > let us know by reply email and delete it >> > from your system. It is prohibited to copy >> > this message or disclose its content to >> > anyone. Any confidentiality or privilege is >> > not waived or lost by any mistaken delivery >> > or unauthorized disclosure of the message. >> > All messages sent to and from Agoda may be >> > monitored to ensure compliance with company >> > policies, to protect the company's interests >> > and to remove potential malware. Electronic >> > messages may be intercepted, amended, lost >> > or deleted, or contain viruses. >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |