Re: Unexpected recovering after nautilus 14.2.7 -> 14.2.8

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

before I ran the update to 14.2.8 I checked that the state was healthy
with all OSDs up and in. I still have the command history I typed
visible in my kde terminal buffer and there I see that after the update
but before the reboot I ran a ceph -s and there were 144 osd's up and in
the state was HEALTH_OK.

Could it be of interest that the node rebooted was a monitor node and
should mon_osd_down_out_interval at least in theory have prevented what
happened to my cluster?

Thanks
Rainer

Am 05.03.20 um 09:49 schrieb Dan van der Ster:
> Did you have `144 total, 144 up, 144 in` also before the upgrade?
> If an osd was out, then you upgraded/restarted and it went back in, it
> would trigger data movement.
> (I usually set noin before an upgrade).
> 
> -- dan
> 
> On Thu, Mar 5, 2020 at 9:46 AM Rainer Krienke <krienke@xxxxxxxxxxxxxx> wrote:
>>
>> I found some information in ceph.log that might help to find out what
>> happened. node2  was the one I rebooted:
>>
>> 2020-03-05 07:24:29.844953 osd.45 (osd.45) 483 : cluster [DBG] 36.323
>> scrub starts
>> 2020-03-05 07:24:33.552221 osd.45 (osd.45) 484 : cluster [DBG] 36.323
>> scrub ok
>> 2020-03-05 07:24:38.948404 mon.node2 (mon.0) 692706 : cluster [DBG]
>> osdmap e31855: 144 total, 144 up, 144 in
>> 2020-03-05 07:24:39.969404 mon.node2 (mon.0) 692713 : cluster [DBG]
>> osdmap e31856: 144 total, 144 up, 144 in
>> 2020-03-05 07:24:39.979238 mon.node2 (mon.0) 692714 : cluster [WRN]
>> Health check failed: 1 pools have many more objects per pg than average
>> (MANY_OBJECTS_PER_PG)
>> 2020-03-05 07:24:40.533392 mon.node2 (mon.0) 692717 : cluster [DBG]
>> osdmap e31857: 144 total, 144 up, 144 in
>> 2020-03-05 07:24:41.550395 mon.node2 (mon.0) 692728 : cluster [DBG]
>> osdmap e31858: 144 total, 144 up, 144 in
>> 2020-03-05 07:24:41.598004 osd.127 (osd.127) 691 : cluster [DBG]
>> 36.3eds0 starting backfill to osd.18(4) from (0'0,0'0] MAX to 31854'297918
>> 2020-03-05 07:24:41.619293 osd.127 (osd.127) 692 : cluster [DBG]
>> 36.3eds0 starting backfill to osd.49(5) from (0'0,0'0] MAX to 31854'297918
>> 2020-03-05 07:24:41.631869 osd.127 (osd.127) 693 : cluster [DBG]
>> 36.3eds0 starting backfill to osd.65(2) from (0'0,0'0] MAX to 31854'297918
>> 2020-03-05 07:24:41.644089 osd.127 (osd.127) 694 : cluster [DBG]
>> 36.3eds0 starting backfill to osd.97(3) from (0'0,0'0] MAX to 31854'297918
>> 2020-03-05 07:24:41.656223 osd.127 (osd.127) 695 : cluster [DBG]
>> 36.3eds0 starting backfill to osd.122(0) from (0'0,0'0] MAX to 31854'297918
>> 2020-03-05 07:24:41.669265 osd.127 (osd.127) 696 : cluster [DBG]
>> 36.3eds0 starting backfill to osd.134(1) from (0'0,0'0] MAX to 31854'297918
>> 2020-03-05 07:24:41.582485 osd.69 (osd.69) 549 : cluster [DBG] 36.3fes0
>> starting backfill to osd.13(1) from (0'0,0'0] MAX to 31854'280018
>> 2020-03-05 07:24:41.590541 osd.5 (osd.5) 349 : cluster [DBG] 36.3f2s0
>> starting backfill to osd.10(0) from (0'0,0'0] MAX to 31854'331157
>> 2020-03-05 07:24:41.596496 osd.69 (osd.69) 550 : cluster [DBG] 36.3fes0
>> starting backfill to osd.25(5) from (0'0,0'0] MAX to 31854'280018
>> 2020-03-05 07:24:41.601781 osd.86 (osd.86) 457 : cluster [DBG] 36.3ees0
>> starting backfill to osd.10(4) from (0'0,0'0] MAX to 31854'511090
>> 2020-03-05 07:24:41.603864 osd.69 (osd.69) 551 : cluster [DBG] 36.3fes0
>> starting backfill to osd.58(2) from (0'0,0'0] MAX to 31854'280018
>> 2020-03-05 07:24:41.610409 osd.69 (osd.69) 552 : cluster [DBG] 36.3fes0
>> starting backfill to osd.78(3) from (0'0,0'0] MAX to 31854'280018
>> 2020-03-05 07:24:41.614494 osd.5 (osd.5) 350 : cluster [DBG] 36.3f2s0
>> starting backfill to osd.41(1) from (0'0,0'0] MAX to 31854'331157
>> 2020-03-05 07:24:41.617208 osd.69 (osd.69) 553 : cluster [DBG] 36.3fes0
>> starting backfill to osd.99(0) from (0'0,0'0] MAX to 31854'280018
>> 2020-03-05 07:24:41.622645 osd.86 (osd.86) 458 : cluster [DBG] 36.3ees0
>> starting backfill to osd.48(5) from (0'0,0'0] MAX to 31854'511090
>> 2020-03-05 07:24:41.624049 osd.69 (osd.69) 554 : cluster [DBG] 36.3fes0
>> starting backfill to osd.121(4) from (0'0,0'0] MAX to 31854'280018
>> 2020-03-05 07:24:41.625556 osd.5 (osd.5) 351 : cluster [DBG] 36.3f2s0
>> starting backfill to osd.61(3) from (0'0,0'0] MAX to 31854'331157
>> 2020-03-05 07:24:41.631348 osd.86 (osd.86) 459 : cluster [DBG] 36.3ees0
>> starting backfill to osd.78(3) from (0'0,0'0] MAX to 31854'511090
>> 2020-03-05 07:24:41.634572 osd.5 (osd.5) 352 : cluster [DBG] 36.3f2s0
>> starting backfill to osd.71(4) from (0'0,0'0] MAX to 31854'331157
>> 2020-03-05 07:24:41.641651 osd.86 (osd.86) 460 : cluster [DBG] 36.3ees0
>> starting backfill to osd.90(0) from (0'0,0'0] MAX to 31854'511090
>> 2020-03-05 07:24:41.644983 osd.5 (osd.5) 353 : cluster [DBG] 36.3f2s0
>> starting backfill to osd.122(5) from (0'0,0'0] MAX to 31854'331157
>> 2020-03-05 07:24:41.649661 osd.86 (osd.86) 461 : cluster [DBG] 36.3ees0
>> starting backfill to osd.118(2) from (0'0,0'0] MAX to 31854'511090
>> 2020-03-05 07:24:41.652407 osd.5 (osd.5) 354 : cluster [DBG] 36.3f2s0
>> starting backfill to osd.131(2) from (0'0,0'0] MAX to 31854'331157
>> 2020-03-05 07:24:41.659823 osd.86 (osd.86) 462 : cluster [DBG] 36.3ees0
>> starting backfill to osd.139(1) from (0'0,0'0] MAX to 31854'511090
>> 2020-03-05 07:24:42.055680 mon.node2 (mon.0) 692729 : cluster [INF]
>> osd.23 marked itself down
>> 2020-03-05 07:24:42.055765 mon.node2 (mon.0) 692730 : cluster [INF]
>> osd.18 marked itself down
>> 2020-03-05 07:24:42.055919 mon.node2 (mon.0) 692731 : cluster [INF]
>> osd.21 marked itself down
>> 2020-03-05 07:24:42.056002 mon.node2 (mon.0) 692732 : cluster [INF]
>> osd.24 marked itself down
>> 2020-03-05 07:24:42.056250 mon.node2 (mon.0) 692733 : cluster [INF]
>> osd.17 marked itself down
>> 2020-03-05 07:24:42.058049 mon.node2 (mon.0) 692734 : cluster [INF]
>> osd.16 marked itself down
>> 2020-03-05 07:24:42.064002 mon.node2 (mon.0) 692735 : cluster [INF]
>> osd.31 marked itself down
>> 2020-03-05 07:24:42.069635 mon.node2 (mon.0) 692736 : cluster [INF]
>> osd.26 marked itself down
>> 2020-03-05 07:24:42.075325 mon.node2 (mon.0) 692737 : cluster [INF]
>> osd.29 marked itself down
>> 2020-03-05 07:24:42.080842 mon.node2 (mon.0) 692738 : cluster [INF]
>> osd.19 marked itself down
>> 2020-03-05 07:24:42.086368 mon.node2 (mon.0) 692739 : cluster [INF]
>> osd.22 marked itself down
>> 2020-03-05 07:24:42.091810 mon.node2 (mon.0) 692740 : cluster [INF]
>> osd.28 marked itself down
>> 2020-03-05 07:24:42.125240 mon.node2 (mon.0) 692741 : cluster [INF]
>> osd.30 marked itself down
>> 2020-03-05 07:24:42.125318 mon.node2 (mon.0) 692742 : cluster [INF]
>> osd.27 marked itself down
>> 2020-03-05 07:24:42.177279 mon.node2 (mon.0) 692743 : cluster [INF]
>> osd.20 marked itself down
>> 2020-03-05 07:24:42.189747 mon.node2 (mon.0) 692744 : cluster [INF]
>> osd.25 marked itself down
>> 2020-03-05 07:24:42.567690 mon.node2 (mon.0) 692745 : cluster [WRN]
>> Health check failed: 16 osds down (OSD_DOWN)
>> 2020-03-05 07:24:42.567743 mon.node2 (mon.0) 692746 : cluster [WRN]
>> Health check failed: 1 host (16 osds) down (OSD_HOST_DOWN)
>> 2020-03-05 07:24:42.673270 mon.node2 (mon.0) 692747 : cluster [DBG]
>> osdmap e31859: 144 total, 128 up, 144 in
>> 2020-03-05 07:24:41.577509 osd.122 (osd.122) 633 : cluster [DBG]
>> 36.3f0s0 starting backfill to osd.15(0) from (0'0,0'0] MAX to 31854'314030
>> 2020-03-05 07:24:41.588537 osd.94 (osd.94) 501 : cluster [DBG] 36.3eas0
>> starting backfill to osd.0(0) from (0'0,0'0] MAX to 31854'307657
>> 2020-03-05 07:24:41.593430 osd.114 (osd.114) 633 : cluster [DBG]
>> 36.3f3s0 starting backfill to osd.4(3) from (0'0,0'0] MAX to 31854'313629
>> 2020-03-05 07:24:41.593977 osd.122 (osd.122) 634 : cluster [DBG]
>> 36.3f0s0 starting backfill to osd.25(3) from (0'0,0'0] MAX to 31854'314030
>> 2020-03-05 07:24:41.595369 osd.126 (osd.126) 559 : cluster [DBG]
>> 36.3e7s0 starting backfill to osd.7(3) from (0'0,0'0] MAX to 31854'275181
>> 2020-03-05 07:24:41.598564 osd.85 (osd.85) 473 : cluster [DBG] 36.3f5s0
>> starting backfill to osd.3(5) from (0'0,0'0] MAX to 31854'313436
>> ....
>>
>> Am 05.03.20 um 08:58 schrieb Rainer Krienke:
>>> Hello,
>>>
>>> at the moment my ceph is still working but in a degraded state after I
>>> upgraded one (in 9) hosts from 14.2.7 to 14.2.8 and rebooting this host
>>> (node2, one  monitor in 3) after the upgrade.
>>>
>>> Usually before rebooting I set
>>>
>>>    ceph osd set noout
>>>    ceph osd set nobackfill
>>>    ceph osd set norecover
>>>
>>> before rebooting, but I fogot this time. After having realized my error
>>> I thought, ok I forgot to set the flags but I configured
>>> mon_osd_down_out_interval to 900sec:
>>>
>>> # ceph config get mon.mon_osd_down_out_interval
>>> WHO    MASK LEVEL    OPTION                    VALUE RO
>>> mon         advanced mon_osd_down_out_interval 900
>>>
>>> The reboot took 5min so I expected nothing to happen. But it did and now
>>> I do not understand why and if there are more timeout values I
>>> could/should set to avoid this happening again if I ever should again
>>> forget to set the noout , nobackfill, norecover flags prior to a reboot?
>>
>> --
>> Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
>> 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312
>> Web: http://userpages.uni-koblenz.de/~krienke
>> PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@xxxxxxx
>> To unsubscribe send an email to ceph-users-leave@xxxxxxx


-- 
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312
Web: http://userpages.uni-koblenz.de/~krienke
PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux