Re: [Octopus] OSD overloading

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Are you sure your not being hit by:



ceph config set osd bluestore_fsck_quick_fix_on_mount false @ https://docs.ceph.com/docs/master/releases/octopus/

Have all your OSD's successfully completed the fsck?



Reasons I say that is I can see "20 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats"





---- On Thu, 09 Apr 2020 02:15:02 +0800 Jack <mailto:ceph@xxxxxxxxxxxxxx> wrote ----



Just to confirm this does not get better: 
 
root@backup1:~# ceph status 
 cluster: 
 id:     9cd41f0f-936d-4b59-8e5d-9b679dae9140 
 health: HEALTH_WARN 
 20 OSD(s) reporting legacy (not per-pool) BlueStore omap 
usage stats 
 4/50952060 objects unfound (0.000%) 
 nobackfill,norecover,noscrub,nodeep-scrub flag(s) set 
 1 osds down 
 3 nearfull osd(s) 
 Reduced data availability: 826 pgs inactive, 616 pgs down, 
185 pgs peering, 158 pgs stale 
 Low space hindering backfill (add storage if this doesn't 
resolve itself): 93 pgs backfill_toofull 
 Degraded data redundancy: 13285415/101904120 objects 
degraded (13.037%), 706 pgs degraded, 696 pgs undersized 
 989 pgs not deep-scrubbed in time 
 378 pgs not scrubbed in time 
 10 pool(s) nearfull 
 2216 slow ops, oldest one blocked for 13905 sec, daemons 
[osd.1,osd.11,osd.20,osd.24,osd.25,osd.29,osd.31,osd.37,osd.4,osd.5]... 
have slow ops. 
 
 services: 
 mon: 1 daemons, quorum backup1 (age 8d) 
 mgr: backup1(active, since 8d) 
 osd: 37 osds: 26 up (since 9m), 27 in (since 2h); 626 remapped pgs 
 flags nobackfill,norecover,noscrub,nodeep-scrub 
 rgw: 1 daemon active (backup1.odiso.net) 
 
 task status: 
 
 data: 
 pools:   10 pools, 2785 pgs 
 objects: 50.95M objects, 92 TiB 
 usage:   121 TiB used, 39 TiB / 160 TiB avail 
 pgs:     29.659% pgs not active 
 13285415/101904120 objects degraded (13.037%) 
 433992/101904120 objects misplaced (0.426%) 
 4/50952060 objects unfound (0.000%) 
 840 active+clean+snaptrim_wait 
 536 down 
 490 active+undersized+degraded+remapped+backfilling 
 326 active+clean 
 113 peering 
 88  active+undersized+degraded 
 83  active+undersized+degraded+remapped+backfill_toofull 
 79  stale+down 
 63  stale+peering 
 51  active+clean+snaptrim 
 24  activating 
 22  active+recovering+degraded 
 19  active+remapped+backfilling 
 13  stale+active+undersized+degraded 
 9   remapped+peering 
 9   active+undersized+remapped+backfilling 
 9 
active+undersized+degraded+remapped+backfill_wait+backfill_toofull 
 2   stale+active+clean+snaptrim 
 2   active+undersized 
 1   stale+active+clean+snaptrim_wait 
 1   active+remapped+backfill_toofull 
 1   active+clean+snaptrim_wait+laggy 
 1   active+recovering+undersized+remapped 
 1   down+remapped 
 1   activating+undersized+degraded+remapped 
 1   active+recovering+laggy 
 
On 4/8/20 3:27 PM, Jack wrote: 
> The CPU is used by userspace, not kernelspace 
> 
> Here is the perf top, see attachment 
> 
> Rocksdb eats everything :/ 
> 
> 
> On 4/8/20 3:14 PM, Paul Emmerich wrote: 
>> What's the CPU busy with while spinning at 100%? 
>> 
>> Check "perf top" for a quick overview 
>> 
>> 
>> Paul 
>> 
> 
> 
> _______________________________________________ 
> ceph-users mailing list -- mailto:ceph-users@xxxxxxx 
> To unsubscribe send an email to mailto:ceph-users-leave@xxxxxxx 
> 
_______________________________________________ 
ceph-users mailing list -- mailto:ceph-users@xxxxxxx 
To unsubscribe send an email to mailto:ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux