On 7/25/19 7:49 AM, Sangwhan Moon wrote:
> Hello,
>
> Original Message:
>>
>>
>> On 7/25/19 6:49 AM, Sangwhan Moon wrote:
>>> Hello,
>>>
>>> I've inherited a Ceph cluster from someone who has left zero documentation or any handover. A couple days ago it decided to show the entire company what it is capable of..
>>>
>>> The health report looks like this:
>>>
>>> [root@host mnt]# ceph -s
>>> cluster:
>>> id: 809718aa-3eac-4664-b8fa-38c46cdbfdab
>>> health: HEALTH_ERR
>>> 1 MDSs report damaged metadata
>>> 1 MDSs are read only
>>> 2 MDSs report slow requests
>>> 6 MDSs behind on trimming
>>> Reduced data availability: 2 pgs stale
>>> Degraded data redundancy: 2593/186803520 objects degraded (0.001%), 2 pgs degraded, 2 pgs undersized
>>> 1 slow requests are blocked > 32 sec. Implicated osds
>>> 716 stuck requests are blocked > 4096 sec. Implicated osds 25,31,38\
>>
>> I would start here:
>>
>>>
>>> services:
>>> mon: 3 daemons, quorum f,rook-ceph-mon2,rook-ceph-mon0
>>> mgr: a(active)
>>> mds: ceph-fs-2/2/2 up odd-fs-2/2/2 up {[ceph-fs:0]=ceph-fs-5b997cbf7b-5tjwh=up:active,[ceph-fs:1]=ceph-fs-5b997cbf
>>> 7b-nstqz=up:active,[user-fs:0]=odd-fs-5668c75f9f-hflps=up:active,[user-fs:1]=odd-fs-5668c75f9f-jf59x=up:active}, 4 up:sta
>>> ndby-replay
>>> osd: 39 osds: 39 up, 38 in
>>>
>>> data:
>>> pools: 5 pools, 706 pgs
>>> objects: 91212k objects, 4415 GB
>>> usage: 10415 GB used, 13024 GB / 23439 GB avail
>>> pgs: 2593/186803520 objects degraded (0.001%)
>>> 703 active+clean
>>> 2 stale+active+undersized+degraded
>>
>> This is a problem! Can you check:
>>
>> $ ceph pg dump_stuck
>>
>> The PGs will start with a number like 8.1a where '8' it the pool ID.
>>
>> Then check:
>>
>> $ ceph df
>>
>> To which pools to those PGs belong?
>>
>> Then check:
>>
>> $ ceph pg <PGID> query
>>
>> And the bottom somewhere should show why these PGs are not active. You
>> might even want to try a restart of these OSDs involved with those two PGs.
>
> Thanks a lot for the suggestions - I just checked and it says that the problematic PGs are 4.4f and 4.59 - but querying those seem result in the following error:
>
> Error ENOENT: i don't have pgid 4.4f
>
> (same applies for 4.59 - they do seem to show up in "ceph pg ls" though.)
>
> In ceph pg ls, it shows that for these PGs UP, UP_PRIMARY ACTING, ACTING_PRIMARY all only have one OSD associated with it. (24, 13 - although both the PG ID mentioned above and these numbers probably don't help much with the diagnosis) Should restarting be a safe thing to try first?
>
> ceph health detail says the following:
>
> MDS_DAMAGE 1 MDSs report damaged metadata
> mdsceph-fs-5b997cbf7b-5tjwh(mds.0): Metadata damage detected
> MDS_READ_ONLY 1 MDSs are read only
> mdsceph-fs-5b997cbf7b-5tjwh(mds.0): MDS in read-only mode
> MDS_SLOW_REQUEST 2 MDSs report slow requests
> mdsuser-fs-5668c75f9f-hflps(mds.0): 3 slow requests are blocked > 30 sec
> mdsuser-fs-5668c75f9f-jf59x(mds.1): 980 slow requests are blocked > 30 sec
> MDS_TRIM 6 MDSs behind on trimming
> mdsuser-fs-5668c75f9f-hflps(mds.0): Behind on trimming (342/128) max_segments: 128, num_segments: 342
> mdsuser-fs-5668c75f9f-jf59x(mds.1): Behind on trimming (461/128) max_segments: 128, num_segments: 461
> mdsuser-fs-5668c75f9f-h8p2t(mds.0): Behind on trimming (342/128) max_segments: 128, num_segments: 342
> mdsuser-fs-5668c75f9f-7gs67(mds.1): Behind on trimming (461/128) max_segments: 128, num_segments: 461
> mdsceph-fs-5b997cbf7b-5tjwh(mds.0): Behind on trimming (386/128) max_segments: 128, num_segments: 386
> mdsceph-fs-5b997cbf7b-hmrxr(mds.0): Behind on trimming (386/128) max_segments: 128, num_segments: 386
> PG_AVAILABILITY Reduced data availability: 2 pgs stale
> pg 4.4f is stuck stale for 171783.855465, current state stale+active+undersized+degraded, last acting [24]
> pg 4.59 is stuck stale for 171751.961506, current state stale+active+undersized+degraded, last acting [13]
> PG_DEGRADED Degraded data redundancy: 2593/186805106 objects degraded (0.001%), 2 pgs degraded, 2 pgs undersized
> pg 4.4f is stuck undersized for 171797.245359, current state stale+active+undersized+degraded, last acting [24]> pg 4.59 is stuck undersized for 171797.257707, current state
stale+active+undersized+degraded, last acting [13]
So where are osd.24 and osd.13?
To which pool do these PGs belong?
But these PGs are probably the root-cause of all the issues you are seeing.
Wido
> REQUEST_SLOW 3 slow requests are blocked > 32 sec. Implicated osds
> 3 ops are blocked > 2097.15 sec
> REQUEST_STUCK 717 stuck requests are blocked > 4096 sec. Implicated osds 25,31,38
> 286 ops are blocked > 268435 sec
> 211 ops are blocked > 134218 sec
> 5 ops are blocked > 67108.9 sec
> 2 ops are blocked > 33554.4 sec
> 134 ops are blocked > 16777.2 sec
> 79 ops are blocked > 8388.61 sec
> osds 25,31,38 have stuck requests > 268435 sec
>
> Cheers,
> Sangwhan
>
>>
>> Wido
>>
>>> 1 active+clean+scrubbing+deep
>>>
>>> io:
>>> client: 168 kB/s rd, 6336 B/s wr, 10 op/s rd, 1 op/s wr
>>>
>>> The offending broken MDS entry (damaged metadata) seems to be this:
>>>
>>> mds.ceph-fs-5b997cbf7b-5tjwh: [
>>> {
>>> "damage_type": "dir_frag",
>>> "id": 1190692215,
>>> "ino": 2199023258131,
>>> "frag": "*",
>>> "path": "/f/01/59"
>>> }
>>> ]
>>>
>>> Is there any idea how I can diagnose and find out what is wrong? For the other issues I'm not even sure what/where I need to look into.
>>>
>>> Cheers,
>>> Sangwhan
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@xxxxxxxxxxxxxx
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
