Re: outdated mds slow requests

[Eugen Block <eblock@xxxxxx>]

Hi,

is this still an issue? If so, I would try to either evict the client  
via admin socket:

ceph tell mds.5 client evict [<filters>...] --- Evict client  
session(s) based on a filter

alternatively locally on the MDS:
cephadm enter mds.<MDS>
ceph daemon mds.<MDS> client evict <client>

or restart the MDS which should also clear the client, I believe.

Zitat von Ben <ruidong.gao@xxxxxxxxx>:

> Hi,
> It is running 17.2.5. there are slow requests warnings in cluster log.
>
> ceph tell mds.5 dump_ops_in_flight,
> get the following.
>
> These look like outdated and clients were k8s pods. There are warning of
> the kind in other mds as well. How could they be cleaned from warnings
> safely?
>
> Many thanks.
>
> {
> "ops": [
> {
> "description": "peer_request(mds.3:5311742.0 authpin)",
> "initiated_at": "2023-09-14T12:25:43.092201+0000",
> "age": 926013.05098558997,
> "duration": 926013.051015759,
> "type_data": {
> "flag_point": "dispatched",
> "reqid": "mds.3:5311742",
> "op_type": "peer_request",
> "leader_info": {
> "leader": "3"
> },
> "request_info": {
> "attempt": 0,
> "op_type": "authpin",
> "lock_type": 0,
> "object_info": "0x60001205d6d.head",
> "srcdnpath": "",
> "destdnpath": "",
> "witnesses": "",
> "has_inode_export": false,
> "inode_export_v": 0,
> "op_stamp": "0.000000"
> },
> "events": [
> {
> "time": "2023-09-14T12:25:43.092201+0000",
> "event": "initiated"
> },
> {
> "time": "2023-09-14T12:25:43.092202+0000",
> "event": "throttled"
> },
> {
> "time": "2023-09-14T12:25:43.092201+0000",
> "event": "header_read"
> },
> {
> "time": "2023-09-14T12:25:43.092207+0000",
> "event": "all_read"
> },
> {
> "time": "2023-09-14T12:25:43.092218+0000",
> "event": "dispatched"
> }
> ]
> }
> },
> {
> "description": "peer_request(mds.3:5311743.0 authpin)",
> "initiated_at": "2023-09-14T12:25:43.092371+0000",
> "age": 926013.05081614305,
> "duration": 926013.05089185503,
> "type_data": {
> "flag_point": "dispatched",
> "reqid": "mds.3:5311743",
> "op_type": "peer_request",
> "leader_info": {
> "leader": "3"
> },
> "request_info": {
> "attempt": 0,
> "op_type": "authpin",
> "lock_type": 0,
> "object_info": "0x60001205d6d.head",
> "srcdnpath": "",
> "destdnpath": "",
> "witnesses": "",
> "has_inode_export": false,
> "inode_export_v": 0,
> "op_stamp": "0.000000"
> },
> "events": [
> {
> "time": "2023-09-14T12:25:43.092371+0000",
> "event": "initiated"
> },
> {
> "time": "2023-09-14T12:25:43.092371+0000",
> "event": "throttled"
> },
> {
> "time": "2023-09-14T12:25:43.092371+0000",
> "event": "header_read"
> },
> {
> "time": "2023-09-14T12:25:43.092374+0000",
> "event": "all_read"
> },
> {
> "time": "2023-09-14T12:25:43.092381+0000",
> "event": "dispatched"
> }
> ]
> }
> },
> {
> "description": "peer_request(mds.4:4503615.0 authpin)",
> "initiated_at": "2023-09-14T13:40:25.150040+0000",
> "age": 921530.99314722,
> "duration": 921530.99326053297,
> "type_data": {
> "flag_point": "dispatched",
> "reqid": "mds.4:4503615",
> "op_type": "peer_request",
> "leader_info": {
> "leader": "4"
> },
> "request_info": {
> "attempt": 0,
> "op_type": "authpin",
> "lock_type": 0,
> "object_info": "0x60001205c4f.head",
> "srcdnpath": "",
> "destdnpath": "",
> "witnesses": "",
> "has_inode_export": false,
> "inode_export_v": 0,
> "op_stamp": "0.000000"
> },
> "events": [
> {
> "time": "2023-09-14T13:40:25.150040+0000",
> "event": "initiated"
> },
> {
> "time": "2023-09-14T13:40:25.150040+0000",
> "event": "throttled"
> },
> {
> "time": "2023-09-14T13:40:25.150040+0000",
> "event": "header_read"
> },
> {
> "time": "2023-09-14T13:40:25.150045+0000",
> "event": "all_read"
> },
> {
> "time": "2023-09-14T13:40:25.150053+0000",
> "event": "dispatched"
> }
> ]
> }
> },
> {
> "description": "client_request(client.460983:5731820 getattr pAsLsXsFs
> #0x60001205c4f 2023-09-14T13:40:25.144336+0000 caller_uid=0,
> caller_gid=0{})",
> "initiated_at": "2023-09-14T13:40:25.150176+0000",
> "age": 921530.99301089498,
> "duration": 921530.99316312897,
> "type_data": {
> "flag_point": "failed to authpin, inode is being exported",
> "reqid": "client.460983:5731820",
> "op_type": "client_request",
> "client_info": {
> "client": "client.460983",
> "tid": 5731820
> },
> "events": [
> {
> "time": "2023-09-14T13:40:25.150176+0000",
> "event": "initiated"
> },
> {
> "time": "2023-09-14T13:40:25.150177+0000",
> "event": "throttled"
> },
> {
> "time": "2023-09-14T13:40:25.150176+0000",
> "event": "header_read"
> },
> {
> "time": "2023-09-14T13:40:25.150180+0000",
> "event": "all_read"
> },
> {
> "time": "2023-09-14T13:40:25.150186+0000",
> "event": "dispatched"
> },
> {
> "time": "2023-09-14T13:40:25.150195+0000",
> "event": "failed to authpin, inode is being exported"
> }
> ]
> }
> }
> ],
> "num_ops": 4
> }
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx


_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx