The majority of the pools have ‘replicated size 3 min_size 2’.
I do see a few pools such as .rgw.control and a few others have ‘replicated size 3 min_size 1’.
I am not using erasure encoding and none of the pools are set to ‘replicated size 3 min_size 3’.
Thank you,
Shain
From: Anthony D'Atri <anthony.datri@xxxxxxxxx>
Date: Sunday, October 13, 2024 at 11:29 AM
To: Shain Miley <SMiley@xxxxxxx>
Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx>
Subject: Re: Reduced data availability: 3 pgs inactive, 3 pgs down
!-------------------------------------------------------------------|
External Email - Use Caution
|-------------------------------------------------------------------!
When you get the cluster healthy, redeploy those Filestore OSDs as BlueStore. Not before.
Does you r pool have size=3, min_size=3? Is this a replicated pool? Or EC 2,1?
Don’t mark lost, there are things we can do. I don’t want to suggest anything until you share the above info.
> On Oct 13, 2024, at 10:00 AM, Shain Miley <SMiley@xxxxxxx> wrote:
>
> Hello,
>
> I am seeing the following information after reviewing ‘ceph health detail’:
>
> [WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs inactive, 3 pgs down
>
> pg 0.1a is down, acting [234,35]
>
> pg 0.20 is down, acting [226,267]
>
> pg 0.2f is down, acting [227,161]
>
>
> When I query each of those pgs I see the following message on each of them:
>
> "peering_blocked_by": [
>
> {
>
> "osd": 233,
>
> "current_lost_at": 0,
>
> "comment": "starting or marking this osd lost may let us proceed"
>
> }
>
>
> Osd.233 crashed a while ago and when I try to start it the log shows some sort of issue with the filesystem:
>
>
> ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)
>
> 1: (()+0x12980) [0x7f2779617980]
>
> 2: (gsignal()+0xc7) [0x7f27782c9fb7]
>
> 3: (abort()+0x141) [0x7f27782cb921]
>
> 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b2) [0x556ebe773ddf]
>
> 5: (FileStore::_do_transaction(ceph::os::Transaction&, unsigned long, int, ThreadPool::TPHandle*, char const*)+0x62b3) [0x556ebebe2753]
>
> 6: (FileStore::_do_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, unsigned long, ThreadPool::TPHandle*, char const*)+0x48) [0x556ebebe3f38]
>
> 7: (JournalingObjectStore::journal_replay(unsigned long)+0x105a) [0x556ebebfc56a]
>
> 8: (FileStore::mount()+0x438a) [0x556ebebda82a]
>
> 9: (OSD::init()+0x4d1) [0x556ebe80fdc1]
>
> 10: (main()+0x3f8c) [0x556ebe77ad2c]
>
> 11: (__libc_start_main()+0xe7) [0x7f27782acbf7]
>
> 12: (_start()+0x2a) [0x556ebe78fc4a]
>
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>
>
>
>
>
> At this point I am thinking about either running an xfs repair on osd.233 and trying to see if I can get it back up (once the pgs are healthy again I would likey zap/readd or replace the drive).
>
>
>
> Another option it sounds like is to mark the osd as lost.
>
>
>
> I am just looking for advice on what exactly I should do next to try to minimize the chances of any data loss.
>
> Here is the query output for each of those pgs:
> https://urldefense.com/v3/__https://pastebin.com/YbfnpZGC__;!!Iwwt!XTVUuKiQDmZ8ZXQP-pHoxFFWYAIntSqVBuXcigFVVWbYMtpJTcQeg4BzgQQxWSAhs1BKujMNHx4rDIhAStU$<https://urldefense.com/v3/__https:/pastebin.com/YbfnpZGC__;!!Iwwt!XTVUuKiQDmZ8ZXQP-pHoxFFWYAIntSqVBuXcigFVVWbYMtpJTcQeg4BzgQQxWSAhs1BKujMNHx4rDIhAStU$>
>
>
>
> Thank you,
>
> Shain
>
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
