Hm, yes it does [and I was wondering why loopbacks were showing up suddenly
in the logs]. This wasn't happening with 14.2.16 so what's changed about
how we specify stuff?
This might correlate with the other person on the IRC list who has problems
with 14.2.18 and their OSDs deciding they don't work sometimes until they
forcibly restart their network links...
Sam
On Mon, 22 Mar 2021 at 17:20, Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote:
> What's with the OSDs having loopback addresses? E.g. v2:
> 127.0.0.1:6881/17664667,v1:127.0.0.1:6882/17664667
>
> Does `ceph osd dump` show those same loopback addresses for each OSD?
>
> This sounds familiar... I'm trying to find the recent ticket.
>
> .. dan
>
>
> On Mon, Mar 22, 2021, 6:07 PM Sam Skipsey <aoanla@xxxxxxxxx> wrote:
>
>> hi Dan:
>>
>> So, unsetting nodown results in... almost all of the OSDs being marked
>> down. (231 down out of 328).
>> Checking the actual OSD services, most of them were actually up and
>> active on the nodes, even when the mons had marked them down.
>> (On a few nodes, the down services corresponded to OSDs that had been
>> flapping - but increasing osd_max_markdown locally to keep them up despite
>> the previous flapping, and restarting the services... didn't help.)
>>
>> In fact, starting up the few OSD services which had actually stopped,
>> resulted in a different set of OSDs being marked down, and some others
>> coming up.
>> We currently have a sort of "rolling OSD outness" passing through the
>> cluster - there's always ~230 OSDs marked down now, but which ones those
>> are changes (we've had everything from 1 HOST down to 4 HOSTS down over the
>> past 14 minutes as things fluctuate.
>>
>> A log from one of the "down" OSDs [which is actually running, and on the
>> same host as OSDs which are marked up] shows this worrying snippet
>>
>> 2021-03-22 17:01:45.298 7f6c9c883700 1 osd.127 253515 is_healthy false
>> -- only 0/10 up peers (less than 33%)
>> 2021-03-22 17:01:45.298 7f6c9c883700 1 osd.127 253515 not healthy;
>> waiting to boot
>> 2021-03-22 17:01:46.340 7f6c9c883700 1 osd.127 253515 is_healthy false
>> -- only 0/10 up peers (less than 33%)
>> 2021-03-22 17:01:46.340 7f6c9c883700 1 osd.127 253515 not healthy;
>> waiting to boot
>> 2021-03-22 17:01:47.376 7f6c9c883700 1 osd.127 253515 is_healthy false
>> -- only 0/10 up peers (less than 33%)
>> 2021-03-22 17:01:47.376 7f6c9c883700 1 osd.127 253515 not healthy;
>> waiting to boot
>> 2021-03-22 17:01:48.395 7f6c9c883700 1 osd.127 253515 is_healthy false
>> -- only 0/10 up peers (less than 33%)
>> 2021-03-22 17:01:48.395 7f6c9c883700 1 osd.127 253515 not healthy;
>> waiting to boot
>> 2021-03-22 17:01:49.407 7f6c9c883700 1 osd.127 253515 is_healthy false
>> -- only 0/10 up peers (less than 33%)
>> 2021-03-22 17:01:49.407 7f6c9c883700 1 osd.127 253515 not healthy;
>> waiting to boot
>> 2021-03-22 17:01:50.400 7f6c9c883700 1 osd.127 253515 is_healthy false
>> -- only 0/10 up peers (less than 33%)
>> 2021-03-22 17:01:50.400 7f6c9c883700 1 osd.127 253515 not healthy;
>> waiting to boot
>> 2021-03-22 17:01:50.922 7f6c9f088700 -1 --2- 10.1.50.21:0/23673 >> [v2:
>> 127.0.0.1:6881/17664667,v1:127.0.0.1:6882/17664667] conn(0x56010903e400
>> 0x56011a71fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0
>> tx=0)._handle_peer_banner peer [v2:
>> 127.0.0.1:6881/17664667,v1:127.0.0.1:6882/17664667] is using msgr V1
>> protocol
>> 2021-03-22 17:01:50.922 7f6c9f889700 -1 --2- 10.1.50.21:0/23673 >> [v2:
>> 127.0.0.1:6821/13015214,v1:127.0.0.1:6831/13015214] conn(0x5600df434000
>> 0x56011718e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0
>> tx=0)._handle_peer_banner peer [v2:
>> 127.0.0.1:6821/13015214,v1:127.0.0.1:6831/13015214] is using msgr V1
>> protocol
>> 2021-03-22 17:01:50.922 7f6ca008a700 -1 --2- 10.1.50.21:0/23673 >> [v2:
>> 127.0.0.1:6826/11091658,v1:127.0.0.1:6828/11091658] conn(0x5600f85ed800
>> 0x560109df2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0
>> tx=0)._handle_peer_banner peer [v2:
>> 127.0.0.1:6826/11091658,v1:127.0.0.1:6828/11091658] is using msgr V1
>> protocol
>> 2021-03-22 17:01:50.922 7f6ca008a700 -1 --2- 10.1.50.21:0/23673 >> [v2:
>> 127.0.0.1:6859/2683393,v1:127.0.0.1:6862/2683393] conn(0x5600f22ea000
>> 0x560117182300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0
>> tx=0)._handle_peer_banner peer [v2:
>> 127.0.0.1:6859/2683393,v1:127.0.0.1:6862/2683393] is using msgr V1
>> protocol
>> 2021-03-22 17:01:50.922 7f6ca008a700 -1 --2- 10.1.50.21:0/23673 >> [v2:
>> 127.0.0.1:6901/15090566,v1:127.0.0.1:6907/15090566] conn(0x5600df435c00
>> 0x560139370300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0
>> tx=0)._handle_peer_banner peer [v2:
>> 127.0.0.1:6901/15090566,v1:127.0.0.1:6907/15090566] is using msgr V1
>> protocol
>> 2021-03-22 17:01:51.377 7f6c9c883700 1 osd.127 253515 is_healthy false
>> -- only 0/10 up peers (less than 33%)
>> 2021-03-22 17:01:51.377 7f6c9c883700 1 osd.127 253515 not healthy;
>> waiting to boot
>> 2021-03-22 17:01:52.370 7f6c9c883700 1 osd.127 253515 is_healthy false
>> -- only 0/10 up peers (less than 33%)
>> 2021-03-22 17:01:52.370 7f6c9c883700 1 osd.127 253515 not healthy;
>> waiting to boot
>> 2021-03-22 17:01:53.377 7f6c9c883700 1 osd.127 253515 is_healthy false
>> -- only 0/10 up peers (less than 33%)
>> 2021-03-22 17:01:53.377 7f6c9c883700 1 osd.127 253515 not healthy;
>> waiting to boot
>> 2021-03-22 17:01:54.385 7f6c9c883700 1 osd.127 253515 is_healthy false
>> -- only 0/10 up peers (less than 33%)
>> 2021-03-22 17:01:54.385 7f6c9c883700 1 osd.127 253515 not healthy;
>> waiting to boot
>> 2021-03-22 17:01:55.385 7f6c9c883700 1 osd.127 253515 is_healthy false
>> -- only 0/10 up peers (less than 33%)
>> 2021-03-22 17:01:55.385 7f6c9c883700 1 osd.127 253515 not healthy;
>> waiting to boot
>> 2021-03-22 17:01:56.362 7f6c9c883700 1 osd.127 253515 is_healthy false
>> -- only 0/10 up peers (less than 33%)
>> 2021-03-22 17:01:56.362 7f6c9c883700 1 osd.127 253515 not healthy;
>> waiting to boot
>> 2021-03-22 17:01:57.324 7f6c9c883700 1 osd.127 253515 is_healthy false
>> -- only 0/10 up peers (less than 33%)
>> 2021-03-22 17:01:57.324 7f6c9c883700 1 osd.127 253515 not healthy;
>> waiting to boot
>>
>>
>>
>> Any suggestions?
>>
>> Sam
>>
>> P.S. an example ceph status as it is now [with everything now on 14.2.18,
>> since we had to restart osds anyway]:
>>
>> cluster:
>> id: a1148af2-6eaf-4486-a27e-a05a78c2b378
>> health: HEALTH_WARN
>> pauserd,pausewr,noout,nobackfill,norebalance flag(s) set
>> 230 osds down
>> 4 hosts (80 osds) down
>> Reduced data availability: 2048 pgs inactive
>> 8 slow ops, oldest one blocked for 901 sec, mon.cephs01 has
>> slow ops
>>
>> services:
>> mon: 3 daemons, quorum cephs01,cephs02,cephs03 (age 2h)
>> mgr: cephs01(active, since 77m)
>> osd: 329 osds: 98 up (since 4s), 328 in (since 4d)
>> flags pauserd,pausewr,noout,nobackfill,norebalance
>>
>> data:
>> pools: 3 pools, 2048 pgs
>> objects: 0 objects, 0 B
>> usage: 0 B used, 0 B / 0 B avail
>> pgs: 100.000% pgs unknown
>> 2048 unknown
>>
>>
>>
>> On Mon, 22 Mar 2021 at 14:57, Dan van der Ster <dan@xxxxxxxxxxxxxx>
>> wrote:
>>
>>> Hi,
>>>
>>> I would unset nodown (hiding osd failures) and norecover (blcoking PGs
>>> from recovering degraded objects), then start starting osds.
>>> As soon as you have some osd logs reporting some failures, then share
>>> those...
>>>
>>> - Dan
>>>
>>> On Mon, Mar 22, 2021 at 3:49 PM Sam Skipsey <aoanla@xxxxxxxxx> wrote:
>>> >
>>> > So, we started the mons and mgr up again, and here's the relevant
>>> logs, including also ceph versions. We've also turned off all of the
>>> firewalls on all of the nodes so we know that there can't be network issues
>>> [and, indeed, all of our management of the OSDs happens via logins from the
>>> service nodes or to each other]
>>> >
>>> > > ceph status
>>> >
>>> >
>>> > cluster:
>>> > id: a1148af2-6eaf-4486-a27e-a05a78c2b378
>>> > health: HEALTH_WARN
>>> >
>>> pauserd,pausewr,nodown,noout,nobackfill,norebalance,norecover flag(s) set
>>> > 1 nearfull osd(s)
>>> > 3 pool(s) nearfull
>>> > Reduced data availability: 2048 pgs inactive
>>> > mons cephs01,cephs02,cephs03 are using a lot of disk space
>>> >
>>> > services:
>>> > mon: 3 daemons, quorum cephs01,cephs02,cephs03 (age 61s)
>>> > mgr: cephs01(active, since 76s)
>>> > osd: 329 osds: 329 up (since 63s), 328 in (since 4d); 466 remapped
>>> pgs
>>> > flags
>>> pauserd,pausewr,nodown,noout,nobackfill,norebalance,norecover
>>> >
>>> > data:
>>> > pools: 3 pools, 2048 pgs
>>> > objects: 0 objects, 0 B
>>> > usage: 0 B used, 0 B / 0 B avail
>>> > pgs: 100.000% pgs unknown
>>> > 2048 unknown
>>> >
>>> >
>>> > > ceph health detail
>>> >
>>> > HEALTH_WARN
>>> pauserd,pausewr,nodown,noout,nobackfill,norebalance,norecover flag(s) set;
>>> 1 nearfull osd(s); 3 pool(s) nearfull; Reduced data availability: 2048 pgs
>>> inactive; mons cephs01,cephs02,cephs03 are using a lot of disk space
>>> > OSDMAP_FLAGS
>>> pauserd,pausewr,nodown,noout,nobackfill,norebalance,norecover flag(s) set
>>> > OSD_NEARFULL 1 nearfull osd(s)
>>> > osd.63 is near full
>>> > POOL_NEARFULL 3 pool(s) nearfull
>>> > pool 'dteam' is nearfull
>>> > pool 'atlas' is nearfull
>>> > pool 'atlas-localgroup' is nearfull
>>> > PG_AVAILABILITY Reduced data availability: 2048 pgs inactive
>>> > pg 13.1ef is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 13.1f0 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 13.1f1 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 13.1f2 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 13.1f3 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 13.1f4 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 13.1f5 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 13.1f6 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 13.1f7 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 13.1f8 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 13.1f9 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 13.1fa is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 13.1fb is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 13.1fc is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 13.1fd is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 13.1fe is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 13.1ff is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1ec is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1f0 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1f1 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1f2 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1f3 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1f4 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1f5 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1f6 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1f7 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1f8 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1f9 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1fa is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1fb is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1fc is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1fd is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1fe is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 14.1ff is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1ed is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1f0 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1f1 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1f2 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1f3 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1f4 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1f5 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1f6 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1f7 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1f8 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1f9 is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1fa is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1fb is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1fc is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1fd is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1fe is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > pg 15.1ff is stuck inactive for 89.322981, current state unknown,
>>> last acting []
>>> > MON_DISK_BIG mons cephs01,cephs02,cephs03 are using a lot of disk space
>>> > mon.cephs01 is 96 GiB >= mon_data_size_warn (15 GiB)
>>> > mon.cephs02 is 96 GiB >= mon_data_size_warn (15 GiB)
>>> > mon.cephs03 is 96 GiB >= mon_data_size_warn (15 GiB)
>>> >
>>> >
>>> > > ceph versions
>>> >
>>> > {
>>> > "mon": {
>>> > "ceph version 14.2.18
>>> (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3
>>> > },
>>> > "mgr": {
>>> > "ceph version 14.2.18
>>> (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 1
>>> > },
>>> > "osd": {
>>> > "ceph version 14.2.10
>>> (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable)": 1,
>>> > "ceph version 14.2.15
>>> (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 188,
>>> > "ceph version 14.2.16
>>> (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)": 18,
>>> > "ceph version 14.2.18
>>> (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 122
>>> > },
>>> >
>>> >
>>> > >>>>>>
>>> >
>>> > As a note, the log where the mgr explodes (which precipitated all of
>>> this) definitely shows the problem occurring on the 12th [when 14.2.17
>>> dropped], but things didn't "break" until we tried upgrading OSDs to
>>> 14.2.18...
>>> >
>>> >
>>> > Sam
>>> >
>>> >
>>> > On Mon, 22 Mar 2021 at 12:20, Sam Skipsey <aoanla@xxxxxxxxx> wrote:
>>> >>
>>> >> Hi Dan:
>>> >>
>>> >> Thanks for the reply - at present, our mons and mgrs are off [because
>>> of the unsustainable nature of the filesystem usage]. We'll try putting
>>> them on again for long enough to get "ceph status" out of them, but because
>>> the mgr was unable to actually talk to anything, and reply at that point.
>>> >>
>>> >> (And thanks for the link to the bug tracker - I guess this mismatch
>>> of expectations is why the devs are so keen to move to containerised
>>> deployments where there is no co-location of different types of server, as
>>> it means they don't need to worry as much about the assumptions about when
>>> it's okay to restart a service on package update. Disappointing that it
>>> seems stale after 2 years...)
>>> >>
>>> >> Sam
>>> >>
>>> >>
>>> >>
>>> >> On Mon, 22 Mar 2021 at 12:11, Dan van der Ster <dan@xxxxxxxxxxxxxx>
>>> wrote:
>>> >>>
>>> >>> Hi Sam,
>>> >>>
>>> >>> The daemons restart (for *some* releases) because of this:
>>> >>> https://tracker.ceph.com/issues/21672
>>> >>> In short, if the selinux module changes, and if you have selinux
>>> >>> enabled, then midway through yum update, there will be a systemctl
>>> >>> restart ceph.target issued.
>>> >>>
>>> >>> For the rest -- I think you should focus on getting the PGs all
>>> >>> active+clean as soon as possible, because the degraded and remapped
>>> >>> states are what leads to mon / osdmap growth.
>>> >>> This kind of scenario is why we wrote this tool:
>>> >>>
>>> https://github.com/cernceph/ceph-scripts/blob/master/tools/upmap/upmap-remapped.py
>>> >>> It will use pg-upmap-items to force the PGs to the OSDs where they
>>> are
>>> >>> currently residing.
>>> >>>
>>> >>> But there is some clarification needed before you go ahead with that.
>>> >>> Could you share `ceph status`, `ceph health detail`?
>>> >>>
>>> >>> Cheers, Dan
>>> >>>
>>> >>>
>>> >>> On Mon, Mar 22, 2021 at 12:05 PM Sam Skipsey <aoanla@xxxxxxxxx>
>>> wrote:
>>> >>> >
>>> >>> > Hi everyone:
>>> >>> >
>>> >>> > I posted to the list on Friday morning (UK time), but apparently
>>> my email
>>> >>> > is still in moderation (I have an email from the list bot telling
>>> me that
>>> >>> > it's held for moderation but no updates).
>>> >>> >
>>> >>> > Since this is a bit urgent - we have ~3PB of storage offline - I'm
>>> posting
>>> >>> > again.
>>> >>> >
>>> >>> > To save retyping the whole thing, I will direct you to a copy of
>>> the email
>>> >>> > I wrote on Friday:
>>> >>> >
>>> >>> > http://aoanla.pythonanywhere.com/Logs/EmailToCephUsers.txt
>>> >>> >
>>> >>> > (Since that was sent, we did successfully add big SSDs to the MON
>>> hosts so
>>> >>> > they don't fill up their disks with store.db s).
>>> >>> >
>>> >>> > I would appreciate any advice - assuming this also doesn't get
>>> stuck in
>>> >>> > moderation queues.
>>> >>> >
>>> >>> > --
>>> >>> > Sam Skipsey (he/him, they/them)
>>> >>> > _______________________________________________
>>> >>> > ceph-users mailing list -- ceph-users@xxxxxxx
>>> >>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Sam Skipsey (he/him, they/them)
>>> >>
>>> >>
>>> >
>>> >
>>> > --
>>> > Sam Skipsey (he/him, they/them)
>>> >
>>> >
>>>
>>
>>
>> --
>> Sam Skipsey (he/him, they/them)
>>
>>
>>
--
Sam Skipsey (he/him, they/them)
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
