No, I don't. I guess I could pin a large part of the tree, if that's something that's likely to help. On Wed, 21 Apr 2021 at 21:02, Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote: > You don't pin subtrees ? > I would guess that something in the workload changed and it's triggering a > particularly bad behavior in the md balancer. > Increase debug_mds gradually on both mds's; hopefully that gives a hint as > to what it's doing. > > .. dan > > > On Wed, Apr 21, 2021, 8:48 PM Flemming Frandsen <dren.dk@xxxxxxxxx> wrote: > >> Not as of yet, it's steadily getting further behind. >> >> We're now up to 6797 segments and there's still the same 14 long-running >> operations that are all "cleaned up request". >> >> Something is blocking trimming, normally I'd follow the advice of >> restarting the mds: >> https://docs.ceph.com/en/latest/cephfs/troubleshooting/#slow-requests-mds >> >> ... but I just did that and replaying 1200 segments took two hours, so if >> the same happens again, then I'm looking at 12 hours of downtime, which >> would be very unpopular. >> >> >> >> On Wed, 21 Apr 2021 at 20:26, Dan van der Ster <dan@xxxxxxxxxxxxxx> >> wrote: >> >>> Did this eventually clear? >>> We had something like this happen once when we changed an md export pin >>> for a very top level directory from mds.3 to mds.0. This triggered so much >>> subtree export work that it took something like 30 minutes to complete. In >>> our case the md segments kept growing into a few 10k, iirc. As soon as the >>> exports completed the md log trimmed quickly. >>> >>> .. Dan >>> >>> >>> >>> On Wed, Apr 21, 2021, 7:38 PM Flemming Frandsen <dren.dk@xxxxxxxxx> >>> wrote: >>> >>>> I've gone through the clients mentioned by the ops in flight and none of >>>> them are connected any more. >>>> >>>> The number of segments that the MDS is behind on is rising steadily and >>>> the >>>> ops_in_flight remain, this feels a lot like a catastrophe brewing. >>>> >>>> The documentation suggests trying to restart the MDS server, but the >>>> last >>>> time I did replay took two hours before any cephfs worked again, so I'd >>>> rather not risk that, if I can help it. >>>> >>>> Any hints are appreciated. >>>> >>>> # ceph health detail >>>> HEALTH_WARN 1 MDSs report slow metadata IOs; 1 MDSs report slow >>>> requests; 1 >>>> MDSs behind on trimming >>>> MDS_SLOW_METADATA_IO 1 MDSs report slow metadata IOs >>>> mds.dalmore(mds.0): 1 slow metadata IOs are blocked > 30 secs, oldest >>>> blocked for 6046 secs >>>> MDS_SLOW_REQUEST 1 MDSs report slow requests >>>> mds.dalmore(mds.0): 14 slow requests are blocked > 30 secs >>>> MDS_TRIM 1 MDSs behind on trimming >>>> mds.dalmore(mds.0): Behind on trimming (4515/128) max_segments: 128, >>>> num_segments: 4515 >>>> >>>> >>>> >>>> On Wed, 21 Apr 2021 at 19:09, Flemming Frandsen <dren.dk@xxxxxxxxx> >>>> wrote: >>>> >>>> > I've just spent a couple of hours waiting for an MDS server to replay >>>> a >>>> > journal that it was behind on and it seems to be getting worse. >>>> > >>>> > The system is not terribly busy, but there are 14 ops in flight that >>>> are >>>> > very old and do not seem to go away on their own. >>>> > >>>> > Is there anything I can do to unwedge the mds server? >>>> > >>>> > >>>> > root@dalmore:~# ceph health detail >>>> > HEALTH_WARN 1 MDSs report slow metadata IOs; 1 MDSs report slow >>>> requests; >>>> > 1 MDSs behind on trimming >>>> > MDS_SLOW_METADATA_IO 1 MDSs report slow metadata IOs >>>> > mds.dalmore(mds.0): 1 slow metadata IOs are blocked > 30 secs, >>>> oldest >>>> > blocked for 4107 secs >>>> > MDS_SLOW_REQUEST 1 MDSs report slow requests >>>> > mds.dalmore(mds.0): 14 slow requests are blocked > 30 secs >>>> > MDS_TRIM 1 MDSs behind on trimming >>>> > mds.dalmore(mds.0): Behind on trimming (3443/128) max_segments: >>>> 128, >>>> > num_segments: 3443 >>>> > >>>> > >>>> > root@dalmore:~# ceph daemon mds.dalmore dump_ops_in_flight >>>> > { >>>> > "ops": [ >>>> > { >>>> > "description": "client_request(client.9801215:348401522 >>>> readdir >>>> > #0x2001fba1c83 2021-04-21 15:46:30.820531 caller_uid=1000, >>>> > caller_gid=1000{})", >>>> > "initiated_at": "2021-04-21 15:46:30.822084", >>>> > "age": 4020.3439449050002, >>>> > "duration": 4020.343998116, >>>> > "type_data": { >>>> > "flag_point": "cleaned up request", >>>> > "reqid": "client.9801215:348401522", >>>> > "op_type": "client_request", >>>> > "client_info": { >>>> > "client": "client.9801215", >>>> > "tid": 348401522 >>>> > }, >>>> > "events": [ >>>> > { >>>> > "time": "2021-04-21 15:46:30.822084", >>>> > "event": "initiated" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:46:30.822084", >>>> > "event": "header_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:46:30.822090", >>>> > "event": "throttled" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:46:30.822108", >>>> > "event": "all_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:46:30.966336", >>>> > "event": "dispatched" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:46:30.966419", >>>> > "event": "acquired locks" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084378", >>>> > "event": "killing request" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084438", >>>> > "event": "cleaned up request" >>>> > } >>>> > ] >>>> > } >>>> > }, >>>> > { >>>> > "description": "client_request(client.9801215:348454506 >>>> > setfilelock rule 1, type 4, owner 14336643358275911908, pid 32085, >>>> start 0, >>>> > length 0, wait 0 #0x1001e694623 20 >>>> > 21-04-21 15:55:50.829175 caller_uid=1000, caller_gid=1000{})", >>>> > "initiated_at": "2021-04-21 15:55:50.832432", >>>> > "age": 3460.3335972099999, >>>> > "duration": 3460.333751787, >>>> > "type_data": { >>>> > "flag_point": "cleaned up request", >>>> > "reqid": "client.9801215:348454506", >>>> > "op_type": "client_request", >>>> > "client_info": { >>>> > "client": "client.9801215", >>>> > "tid": 348454506 >>>> > }, >>>> > "events": [ >>>> > { >>>> > "time": "2021-04-21 15:55:50.832432", >>>> > "event": "initiated" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:55:50.832432", >>>> > "event": "header_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:55:50.832439", >>>> > "event": "throttled" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:55:50.832465", >>>> > "event": "all_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:55:50.832608", >>>> > "event": "dispatched" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084448", >>>> > "event": "killing request" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084451", >>>> > "event": "cleaned up request" >>>> > } >>>> > ] >>>> > } >>>> > }, >>>> > { >>>> > "description": "client_request(client.9143865:2383945773 >>>> > getattr pAsLsXsFs #0x10004f6a16b 2021-04-21 16:35:30.075110 >>>> caller_uid=0, >>>> > caller_gid=0{})", >>>> > "initiated_at": "2021-04-21 16:35:30.077995", >>>> > "age": 1081.0880337159999, >>>> > "duration": 1081.0882681850001, >>>> > "type_data": { >>>> > "flag_point": "failed to authpin, subtree is being >>>> > exported", >>>> > "reqid": "client.9143865:2383945773", >>>> > "op_type": "client_request", >>>> > "client_info": { >>>> > "client": "client.9143865", >>>> > "tid": 2383945773 >>>> > }, >>>> > "events": [ >>>> > { >>>> > "time": "2021-04-21 16:35:30.077995", >>>> > "event": "initiated" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:35:30.077995", >>>> > "event": "header_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:35:30.077999", >>>> > "event": "throttled" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:35:30.078018", >>>> > "event": "all_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:35:30.078091", >>>> > "event": "dispatched" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:35:30.078156", >>>> > "event": "failed to authpin, subtree is being >>>> > exported" >>>> > } >>>> > ] >>>> > } >>>> > }, >>>> > { >>>> > "description": "client_request(client.9801215:348417261 >>>> lookup >>>> > #0x10004cd0675/tasks 2021-04-21 15:49:09.248696 caller_uid=1000, >>>> > caller_gid=1000{})", >>>> > "initiated_at": "2021-04-21 15:49:09.252834", >>>> > "age": 3861.9131950870001, >>>> > "duration": 3861.9135065690002, >>>> > "type_data": { >>>> > "flag_point": "cleaned up request", >>>> > "reqid": "client.9801215:348417261", >>>> > "op_type": "client_request", >>>> > "client_info": { >>>> > "client": "client.9801215", >>>> > "tid": 348417261 >>>> > }, >>>> > "events": [ >>>> > { >>>> > "time": "2021-04-21 15:49:09.252834", >>>> > "event": "initiated" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:49:09.252834", >>>> > "event": "header_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:49:09.252839", >>>> > "event": "throttled" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:49:09.252856", >>>> > "event": "all_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:49:09.252982", >>>> > "event": "dispatched" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:49:09.253070", >>>> > "event": "failed to authpin, subtree is being >>>> > exported" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084439", >>>> > "event": "killing request" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084444", >>>> > "event": "cleaned up request" >>>> > } >>>> > ] >>>> > } >>>> > }, >>>> > { >>>> > "description": "client_request(client.9143865:2383945793 >>>> > getattr pAsLsXsFs #0x10004f6a16b 2021-04-21 16:40:33.868658 >>>> caller_uid=0, >>>> > caller_gid=0{})", >>>> > "initiated_at": "2021-04-21 16:40:33.871565", >>>> > "age": 777.29446343799998, >>>> > "duration": 777.29486169500001, >>>> > "type_data": { >>>> > "flag_point": "failed to authpin, subtree is being >>>> > exported", >>>> > "reqid": "client.9143865:2383945793", >>>> > "op_type": "client_request", >>>> > "client_info": { >>>> > "client": "client.9143865", >>>> > "tid": 2383945793 >>>> > }, >>>> > "events": [ >>>> > { >>>> > "time": "2021-04-21 16:40:33.871565", >>>> > "event": "initiated" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:40:33.871565", >>>> > "event": "header_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:40:33.871572", >>>> > "event": "throttled" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:40:33.871617", >>>> > "event": "all_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:40:33.871699", >>>> > "event": "dispatched" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:40:33.871909", >>>> > "event": "failed to authpin, subtree is being >>>> > exported" >>>> > } >>>> > ] >>>> > } >>>> > }, >>>> > { >>>> > "description": "client_request(client.9801215:348424659 >>>> lookup >>>> > #0x10004cd0675/slaves 2021-04-21 15:50:01.592755 caller_uid=1000, >>>> > caller_gid=1000{})", >>>> > "initiated_at": "2021-04-21 15:50:01.595195", >>>> > "age": 3809.5708340870001, >>>> > "duration": 3809.5713271260001, >>>> > "type_data": { >>>> > "flag_point": "cleaned up request", >>>> > "reqid": "client.9801215:348424659", >>>> > "op_type": "client_request", >>>> > "client_info": { >>>> > "client": "client.9801215", >>>> > "tid": 348424659 >>>> > }, >>>> > "events": [ >>>> > { >>>> > "time": "2021-04-21 15:50:01.595195", >>>> > "event": "initiated" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:50:01.595195", >>>> > "event": "header_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:50:01.595198", >>>> > "event": "throttled" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:50:01.595210", >>>> > "event": "all_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:50:01.595307", >>>> > "event": "dispatched" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:50:01.595389", >>>> > "event": "failed to authpin, subtree is being >>>> > exported" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084444", >>>> > "event": "killing request" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084448", >>>> > "event": "cleaned up request" >>>> > } >>>> > ] >>>> > } >>>> > }, >>>> > { >>>> > "description": "client_request(client.9801215:348456034 >>>> > setfilelock rule 1, type 4, owner 14336643483181153700, pid 32085, >>>> start 0, >>>> > length 0, wait 0 #0x1001e694623 20 >>>> > 21-04-21 15:56:20.849213 caller_uid=1000, caller_gid=1000{})", >>>> > "initiated_at": "2021-04-21 15:56:20.851803", >>>> > "age": 3430.3142257680001, >>>> > "duration": 3430.3148090620002, >>>> > "type_data": { >>>> > "flag_point": "cleaned up request", >>>> > "reqid": "client.9801215:348456034", >>>> > "op_type": "client_request", >>>> > "client_info": { >>>> > "client": "client.9801215", >>>> > "tid": 348456034 >>>> > }, >>>> > "events": [ >>>> > { >>>> > "time": "2021-04-21 15:56:20.851803", >>>> > "event": "initiated" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:20.851803", >>>> > "event": "header_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:20.851815", >>>> > "event": "throttled" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:20.851853", >>>> > "event": "all_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:20.851983", >>>> > "event": "dispatched" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084452", >>>> > "event": "killing request" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084454", >>>> > "event": "cleaned up request" >>>> > } >>>> > ] >>>> > } >>>> > }, >>>> > { >>>> > "description": "client_request(client.9801215:348456035 >>>> lookup >>>> > #0x10004cd0675/tasks 2021-04-21 15:56:20.849213 caller_uid=1000, >>>> > caller_gid=1000{})", >>>> > "initiated_at": "2021-04-21 15:56:20.851899", >>>> > "age": 3430.314129722, >>>> > "duration": 3430.3147965909998, >>>> > "type_data": { >>>> > "flag_point": "cleaned up request", >>>> > "reqid": "client.9801215:348456035", >>>> > "op_type": "client_request", >>>> > "client_info": { >>>> > "client": "client.9801215", >>>> > "tid": 348456035 >>>> > }, >>>> > "events": [ >>>> > { >>>> > "time": "2021-04-21 15:56:20.851899", >>>> > "event": "initiated" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:20.851899", >>>> > "event": "header_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:20.851900", >>>> > "event": "throttled" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:20.851908", >>>> > "event": "all_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:20.852165", >>>> > "event": "dispatched" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:20.852254", >>>> > "event": "failed to authpin, subtree is being >>>> > exported" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084455", >>>> > "event": "killing request" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084457", >>>> > "event": "cleaned up request" >>>> > } >>>> > ] >>>> > } >>>> > }, >>>> > { >>>> > "description": "client_request(client.9801215:348456036 >>>> lookup >>>> > #0x10004cd0675/slaves 2021-04-21 15:56:20.849213 caller_uid=1000, >>>> > caller_gid=1000{})", >>>> > "initiated_at": "2021-04-21 15:56:20.879287", >>>> > "age": 3430.2867415639998, >>>> > "duration": 3430.287496596, >>>> > "type_data": { >>>> > "flag_point": "cleaned up request", >>>> > "reqid": "client.9801215:348456036", >>>> > "op_type": "client_request", >>>> > "client_info": { >>>> > "client": "client.9801215", >>>> > "tid": 348456036 >>>> > }, >>>> > "events": [ >>>> > { >>>> > "time": "2021-04-21 15:56:20.879287", >>>> > "event": "initiated" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:20.879287", >>>> > "event": "header_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:20.879292", >>>> > "event": "throttled" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:20.879306", >>>> > "event": "all_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:20.879408", >>>> > "event": "dispatched" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:20.879502", >>>> > "event": "failed to authpin, subtree is being >>>> > exported" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084458", >>>> > "event": "killing request" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084460", >>>> > "event": "cleaned up request" >>>> > } >>>> > ] >>>> > } >>>> > }, >>>> > { >>>> > "description": "client_request(client.9801215:348456042 >>>> lookup >>>> > #0x10004cd0675/jul 2021-04-21 15:58:53.729411 caller_uid=1000, >>>> > caller_gid=1000{})", >>>> > "initiated_at": "2021-04-21 15:58:53.740318", >>>> > "age": 3277.4257106780001, >>>> > "duration": 3277.4265688219998, >>>> > "type_data": { >>>> > "flag_point": "cleaned up request", >>>> > "reqid": "client.9801215:348456042", >>>> > "op_type": "client_request", >>>> > "client_info": { >>>> > "client": "client.9801215", >>>> > "tid": 348456042 >>>> > }, >>>> > "events": [ >>>> > { >>>> > "time": "2021-04-21 15:58:53.740318", >>>> > "event": "initiated" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:58:53.740318", >>>> > "event": "header_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:58:53.740324", >>>> > "event": "throttled" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:58:53.740345", >>>> > "event": "all_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:58:53.743424", >>>> > "event": "dispatched" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:58:53.750527", >>>> > "event": "failed to authpin, subtree is being >>>> > exported" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084465", >>>> > "event": "killing request" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084468", >>>> > "event": "cleaned up request" >>>> > } >>>> > ] >>>> > } >>>> > }, >>>> > { >>>> > "description": "client_request(client.9143865:2383914038 >>>> > getattr pAsLsXsFs #0x10004f6a16b 2021-04-21 16:29:35.332632 >>>> caller_uid=903, >>>> > caller_gid=1003{})", >>>> > "initiated_at": "2021-04-21 16:29:35.333838", >>>> > "age": 1435.8321910940001, >>>> > "duration": 1435.833136707, >>>> > "type_data": { >>>> > "flag_point": "failed to authpin, subtree is being >>>> > exported", >>>> > "reqid": "client.9143865:2383914038", >>>> > "op_type": "client_request", >>>> > "client_info": { >>>> > "client": "client.9143865", >>>> > "tid": 2383914038 >>>> > }, >>>> > "events": [ >>>> > { >>>> > "time": "2021-04-21 16:29:35.333838", >>>> > "event": "initiated" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:29:35.333838", >>>> > "event": "header_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:29:35.333841", >>>> > "event": "throttled" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:29:35.333854", >>>> > "event": "all_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:29:35.335013", >>>> > "event": "dispatched" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:29:35.335091", >>>> > "event": "failed to authpin, subtree is being >>>> > exported" >>>> > } >>>> > ] >>>> > } >>>> > }, >>>> > { >>>> > "description": "client_request(client.10439445:7 lookup >>>> > #0x10004cd0675/jul 2021-04-21 16:26:17.334272 caller_uid=1000, >>>> > caller_gid=1000{})", >>>> > "initiated_at": "2021-04-21 16:26:17.352448", >>>> > "age": 1633.813581226, >>>> > "duration": 1633.8146036109999, >>>> > "type_data": { >>>> > "flag_point": "cleaned up request", >>>> > "reqid": "client.10439445:7", >>>> > "op_type": "client_request", >>>> > "client_info": { >>>> > "client": "client.10439445", >>>> > "tid": 7 >>>> > }, >>>> > "events": [ >>>> > { >>>> > "time": "2021-04-21 16:26:17.352448", >>>> > "event": "initiated" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:26:17.352448", >>>> > "event": "header_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:26:17.352450", >>>> > "event": "throttled" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:26:17.352460", >>>> > "event": "all_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:26:17.356901", >>>> > "event": "dispatched" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:26:17.357337", >>>> > "event": "failed to authpin, subtree is being >>>> > exported" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:38:08.054109", >>>> > "event": "killing request" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:38:08.054120", >>>> > "event": "cleaned up request" >>>> > } >>>> > ] >>>> > } >>>> > }, >>>> > { >>>> > "description": "client_request(client.9801215:348456041 >>>> lookup >>>> > #0x10004cd0675/jul 2021-04-21 15:56:23.089216 caller_uid=1000, >>>> > caller_gid=1000{})", >>>> > "initiated_at": "2021-04-21 15:56:23.091531", >>>> > "age": 3428.0744974170002, >>>> > "duration": 3428.0756079930002, >>>> > "type_data": { >>>> > "flag_point": "cleaned up request", >>>> > "reqid": "client.9801215:348456041", >>>> > "op_type": "client_request", >>>> > "client_info": { >>>> > "client": "client.9801215", >>>> > "tid": 348456041 >>>> > }, >>>> > "events": [ >>>> > { >>>> > "time": "2021-04-21 15:56:23.091531", >>>> > "event": "initiated" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:23.091531", >>>> > "event": "header_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:23.091536", >>>> > "event": "throttled" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:23.091555", >>>> > "event": "all_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:23.091660", >>>> > "event": "dispatched" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 15:56:23.091763", >>>> > "event": "failed to authpin, subtree is being >>>> > exported" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084461", >>>> > "event": "killing request" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:01:40.084465", >>>> > "event": "cleaned up request" >>>> > } >>>> > ] >>>> > } >>>> > }, >>>> > { >>>> > "description": "client_request(client.9143865:2383885743 >>>> > getattr pAsLsXsFs #0x10004f6a16b 2021-04-21 16:26:36.713371 >>>> caller_uid=903, >>>> > caller_gid=1003{})", >>>> > "initiated_at": "2021-04-21 16:26:36.716418", >>>> > "age": 1614.4496113350001, >>>> > "duration": 1614.4508114350001, >>>> > "type_data": { >>>> > "flag_point": "failed to authpin, subtree is being >>>> > exported", >>>> > "reqid": "client.9143865:2383885743", >>>> > "op_type": "client_request", >>>> > "client_info": { >>>> > "client": "client.9143865", >>>> > "tid": 2383885743 >>>> > }, >>>> > "events": [ >>>> > { >>>> > "time": "2021-04-21 16:26:36.716418", >>>> > "event": "initiated" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:26:36.716418", >>>> > "event": "header_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:26:36.716421", >>>> > "event": "throttled" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:26:36.716435", >>>> > "event": "all_read" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:26:36.716526", >>>> > "event": "dispatched" >>>> > }, >>>> > { >>>> > "time": "2021-04-21 16:26:36.716632", >>>> > "event": "failed to authpin, subtree is being >>>> > exported" >>>> > } >>>> > ] >>>> > } >>>> > } >>>> > ], >>>> > "num_ops": 14 >>>> > } >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > -- >>>> > Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ >>>> > >>>> >>>> >>>> -- >>>> Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>>> >>> >> >> -- >> Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ >> > -- Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |