After reducing the recovery parameter values did not change much.
There are a lot of OSD still marked down.
I don't know what I need to do after this point.
[osd]
osd recovery op priority = 63
osd client op priority = 1
osd recovery max active = 1
osd max scrubs = 1
ceph -s
cluster:
id: 89569e73-eb89-41a4-9fc9-d2a5ec5f4106
health: HEALTH_ERR
42 osds down
1 host (6 osds) down
61/8948582 objects unfound (0.001%)
Reduced data availability: 3837 pgs inactive, 1822 pgs
down, 1900 pgs peering, 6 pgs stale
Possible data damage: 18 pgs recovery_unfound
Degraded data redundancy: 457246/17897164 objects degraded
(2.555%), 213 pgs degraded, 209 pgs undersized
2554 slow requests are blocked > 32 sec
3273 slow ops, oldest one blocked for 1453 sec, daemons
[osd.0,osd.1,osd.10,osd.100,osd.101,osd.102,osd.103,osd.104,osd.105,osd.106]...
have slow ops.
services:
mon: 3 daemons, quorum SRV-SEKUARK3,SRV-SBKUARK2,SRV-SBKUARK3
mgr: SRV-SBKUARK2(active), standbys: SRV-SEKUARK2, SRV-SEKUARK3,
SRV-SEKUARK4
osd: 168 osds: 118 up, 160 in
data:
pools: 1 pools, 4096 pgs
objects: 8.95 M objects, 17 TiB
usage: 33 TiB used, 553 TiB / 586 TiB avail
pgs: 93.677% pgs not active
457246/17897164 objects degraded (2.555%)
61/8948582 objects unfound (0.001%)
1676 down
1372 peering
528 stale+peering
164 active+undersized+degraded
145 stale+down
73 activating
40 active+clean
29 stale+activating
17 active+recovery_unfound+undersized+degraded
16 stale+active+clean
16 stale+active+undersized+degraded
9 activating+undersized+degraded
3 active+recovery_wait+degraded
2 activating+undersized
2 activating+degraded
1 creating+down
1 stale+active+recovery_unfound+undersized+degraded
1 stale+active+clean+scrubbing+deep
1 stale+active+recovery_wait+degraded
ceph -w: https://paste.ubuntu.com/p/WZ2YqzS86S/
ceph health detail: https://paste.ubuntu.com/p/8w7Jpms8fj/
by morphin <morphinwithyou@xxxxxxxxx>, 25 Eyl 2018 Sal, 14:32
tarihinde şunu yazdı:
The config didnt work. Because increasing the number faced with
more OSD Drops.
bhfs -s
cluster:
id: 89569e73-eb89-41a4-9fc9-d2a5ec5f4106
health: HEALTH_ERR
norebalance,norecover flag(s) set
1 osds down
17/8839434 objects unfound (0.000%)
Reduced data availability: 3578 pgs inactive, 861 pgs
down, 1928 pgs peering, 11 pgs stale
Degraded data redundancy: 44853/17678868 objects degraded
(0.254%), 221 pgs degraded, 20 pgs undersized
610 slow requests are blocked > 32 sec
3996 stuck requests are blocked > 4096 sec
6076 slow ops, oldest one blocked for 4129 sec, daemons
[osd.0,osd.1,osd.10,osd.100,osd.101,osd.102,osd.103,osd.104,osd.105,osd.106]...
have slow ops.
services:
mon: 3 daemons, quorum SRV-SEKUARK3,SRV-SBKUARK2,SRV-SBKUARK3
mgr: SRV-SBKUARK2(active), standbys: SRV-SEKUARK2, SRV-SEKUARK3
osd: 168 osds: 128 up, 129 in; 2 remapped pgs
flags norebalance,norecover
data:
pools: 1 pools, 4096 pgs
objects: 8.84 M objects, 17 TiB
usage: 26 TiB used, 450 TiB / 477 TiB avail
pgs: 0.024% pgs unknown
89.160% pgs not active
44853/17678868 objects degraded (0.254%)
17/8839434 objects unfound (0.000%)
1612 peering
720 down
583 activating
319 stale+peering
255 active+clean
157 stale+activating
108 stale+down
95 activating+degraded
84 stale+active+clean
50 active+recovery_wait+degraded
29 creating+down
23 stale+activating+degraded
18 stale+active+recovery_wait+degraded
14 active+undersized+degraded
12 active+recovering+degraded
4 stale+creating+down
3 stale+active+recovering+degraded
3 stale+active+undersized+degraded
2 stale
1 active+recovery_wait+undersized+degraded
1 active+clean+scrubbing+deep
1 unknown
1 active+undersized+degraded+remapped+backfilling
1 active+recovering+undersized+degraded
I guess OSD down and drop issue increases the recovery time. So I
decided to try with decreasing recovery parameters for less load on
cluster.
I have Nvme and SAS disks. Servers are powerfull enough. Network is 4x10Gb.
I dont think my cluster is a bad shape. Because I have datacenter
redundancy (14 servers + 14 servers). The crashed 7 servers are on
only datacenter A. And it took only a few minutes to back online. Also
2 of them is monitors and cluster I/O should be suspended so there
should be less data difference.
On the other hand I dont understand the burden of recovery. I have
faced many recoverys but none of the stopped my cluster working. This
recovery burden is so high that it didnt stop for hours. I wish I
could just decrease the recovery speed and continue to serve my VMs.
Is the change of recovery load some what different than mimic?
Luminous was pretty fine indeed.
by morphin <morphinwithyou@xxxxxxxxx>, 25 Eyl 2018 Sal, 13:57
tarihinde şunu yazdı:
>
> Thank you for answer
>
> What do you think the conf for speed the recover?
>
> [osd]
> osd recovery op priority = 63
> osd client op priority = 1
> osd recovery max active = 16
> osd max scrubs = 16
> <admin@xxxxxxxxxxxxxxx> adresine sahip kullanıcı 25 Eyl 2018 Sal,
> 13:37 tarihinde şunu yazdı:
> >
> > Just let it recover.
> >
> > data:
> > pools: 1 pools, 4096 pgs
> > objects: 8.95 M objects, 17 TiB
> > usage: 34 TiB used, 577 TiB / 611 TiB avail
> > pgs: 94.873% pgs not active
> > 48475/17901254 objects degraded (0.271%)
> > 1/8950627 objects unfound (0.000%)
> > 2631 peering
> > 637 activating
> > 562 down
> > 159 active+clean
> > 44 activating+degraded
> > 30 active+recovery_wait+degraded
> > 12 activating+undersized+degraded
> > 10 active+recovering+degraded
> > 10 active+undersized+degraded
> > 1 active+clean+scrubbing+deep
> >
> > You've got deep scrubbed PGs which put considerable IO load on OSDs.
> >
> >
> > September 25, 2018 1:23 PM, "by morphin"
<morphinwithyou@xxxxxxxxx> wrote:
> >
> >
> > > What should I do now?
> > >
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
