No no pinning now won't help anything... I was asking to understand if it's likely there is balancing happening actively now. If you don't pin, then it's likely. Try the debug logs. And check the exports using something like : ceph daemon mds.b get subtrees | jq '.[] | [.dir.path, .auth_first, .export_pin]' Dan On Wed, Apr 21, 2021, 9:05 PM Flemming Frandsen <dren.dk@xxxxxxxxx> wrote: > No, I don't. > > I guess I could pin a large part of the tree, if that's something that's > likely to help. > > On Wed, 21 Apr 2021 at 21:02, Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote: > >> You don't pin subtrees ? >> I would guess that something in the workload changed and it's triggering >> a particularly bad behavior in the md balancer. >> Increase debug_mds gradually on both mds's; hopefully that gives a hint >> as to what it's doing. >> >> .. dan >> >> >> On Wed, Apr 21, 2021, 8:48 PM Flemming Frandsen <dren.dk@xxxxxxxxx> >> wrote: >> >>> Not as of yet, it's steadily getting further behind. >>> >>> We're now up to 6797 segments and there's still the same 14 long-running >>> operations that are all "cleaned up request". >>> >>> Something is blocking trimming, normally I'd follow the advice of >>> restarting the mds: >>> https://docs.ceph.com/en/latest/cephfs/troubleshooting/#slow-requests-mds >>> >>> ... but I just did that and replaying 1200 segments took two hours, so >>> if the same happens again, then I'm looking at 12 hours of downtime, which >>> would be very unpopular. >>> >>> >>> >>> On Wed, 21 Apr 2021 at 20:26, Dan van der Ster <dan@xxxxxxxxxxxxxx> >>> wrote: >>> >>>> Did this eventually clear? >>>> We had something like this happen once when we changed an md export pin >>>> for a very top level directory from mds.3 to mds.0. This triggered so much >>>> subtree export work that it took something like 30 minutes to complete. In >>>> our case the md segments kept growing into a few 10k, iirc. As soon as the >>>> exports completed the md log trimmed quickly. >>>> >>>> .. Dan >>>> >>>> >>>> >>>> On Wed, Apr 21, 2021, 7:38 PM Flemming Frandsen <dren.dk@xxxxxxxxx> >>>> wrote: >>>> >>>>> I've gone through the clients mentioned by the ops in flight and none >>>>> of >>>>> them are connected any more. >>>>> >>>>> The number of segments that the MDS is behind on is rising steadily >>>>> and the >>>>> ops_in_flight remain, this feels a lot like a catastrophe brewing. >>>>> >>>>> The documentation suggests trying to restart the MDS server, but the >>>>> last >>>>> time I did replay took two hours before any cephfs worked again, so I'd >>>>> rather not risk that, if I can help it. >>>>> >>>>> Any hints are appreciated. >>>>> >>>>> # ceph health detail >>>>> HEALTH_WARN 1 MDSs report slow metadata IOs; 1 MDSs report slow >>>>> requests; 1 >>>>> MDSs behind on trimming >>>>> MDS_SLOW_METADATA_IO 1 MDSs report slow metadata IOs >>>>> mds.dalmore(mds.0): 1 slow metadata IOs are blocked > 30 secs, >>>>> oldest >>>>> blocked for 6046 secs >>>>> MDS_SLOW_REQUEST 1 MDSs report slow requests >>>>> mds.dalmore(mds.0): 14 slow requests are blocked > 30 secs >>>>> MDS_TRIM 1 MDSs behind on trimming >>>>> mds.dalmore(mds.0): Behind on trimming (4515/128) max_segments: 128, >>>>> num_segments: 4515 >>>>> >>>>> >>>>> >>>>> On Wed, 21 Apr 2021 at 19:09, Flemming Frandsen <dren.dk@xxxxxxxxx> >>>>> wrote: >>>>> >>>>> > I've just spent a couple of hours waiting for an MDS server to >>>>> replay a >>>>> > journal that it was behind on and it seems to be getting worse. >>>>> > >>>>> > The system is not terribly busy, but there are 14 ops in flight that >>>>> are >>>>> > very old and do not seem to go away on their own. >>>>> > >>>>> > Is there anything I can do to unwedge the mds server? >>>>> > >>>>> > >>>>> > root@dalmore:~# ceph health detail >>>>> > HEALTH_WARN 1 MDSs report slow metadata IOs; 1 MDSs report slow >>>>> requests; >>>>> > 1 MDSs behind on trimming >>>>> > MDS_SLOW_METADATA_IO 1 MDSs report slow metadata IOs >>>>> > mds.dalmore(mds.0): 1 slow metadata IOs are blocked > 30 secs, >>>>> oldest >>>>> > blocked for 4107 secs >>>>> > MDS_SLOW_REQUEST 1 MDSs report slow requests >>>>> > mds.dalmore(mds.0): 14 slow requests are blocked > 30 secs >>>>> > MDS_TRIM 1 MDSs behind on trimming >>>>> > mds.dalmore(mds.0): Behind on trimming (3443/128) max_segments: >>>>> 128, >>>>> > num_segments: 3443 >>>>> > >>>>> > >>>>> > root@dalmore:~# ceph daemon mds.dalmore dump_ops_in_flight >>>>> > { >>>>> > "ops": [ >>>>> > { >>>>> > "description": "client_request(client.9801215:348401522 >>>>> readdir >>>>> > #0x2001fba1c83 2021-04-21 15:46:30.820531 caller_uid=1000, >>>>> > caller_gid=1000{})", >>>>> > "initiated_at": "2021-04-21 15:46:30.822084", >>>>> > "age": 4020.3439449050002, >>>>> > "duration": 4020.343998116, >>>>> > "type_data": { >>>>> > "flag_point": "cleaned up request", >>>>> > "reqid": "client.9801215:348401522", >>>>> > "op_type": "client_request", >>>>> > "client_info": { >>>>> > "client": "client.9801215", >>>>> > "tid": 348401522 >>>>> > }, >>>>> > "events": [ >>>>> > { >>>>> > "time": "2021-04-21 15:46:30.822084", >>>>> > "event": "initiated" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:46:30.822084", >>>>> > "event": "header_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:46:30.822090", >>>>> > "event": "throttled" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:46:30.822108", >>>>> > "event": "all_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:46:30.966336", >>>>> > "event": "dispatched" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:46:30.966419", >>>>> > "event": "acquired locks" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084378", >>>>> > "event": "killing request" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084438", >>>>> > "event": "cleaned up request" >>>>> > } >>>>> > ] >>>>> > } >>>>> > }, >>>>> > { >>>>> > "description": "client_request(client.9801215:348454506 >>>>> > setfilelock rule 1, type 4, owner 14336643358275911908, pid 32085, >>>>> start 0, >>>>> > length 0, wait 0 #0x1001e694623 20 >>>>> > 21-04-21 15:55:50.829175 caller_uid=1000, caller_gid=1000{})", >>>>> > "initiated_at": "2021-04-21 15:55:50.832432", >>>>> > "age": 3460.3335972099999, >>>>> > "duration": 3460.333751787, >>>>> > "type_data": { >>>>> > "flag_point": "cleaned up request", >>>>> > "reqid": "client.9801215:348454506", >>>>> > "op_type": "client_request", >>>>> > "client_info": { >>>>> > "client": "client.9801215", >>>>> > "tid": 348454506 >>>>> > }, >>>>> > "events": [ >>>>> > { >>>>> > "time": "2021-04-21 15:55:50.832432", >>>>> > "event": "initiated" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:55:50.832432", >>>>> > "event": "header_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:55:50.832439", >>>>> > "event": "throttled" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:55:50.832465", >>>>> > "event": "all_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:55:50.832608", >>>>> > "event": "dispatched" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084448", >>>>> > "event": "killing request" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084451", >>>>> > "event": "cleaned up request" >>>>> > } >>>>> > ] >>>>> > } >>>>> > }, >>>>> > { >>>>> > "description": "client_request(client.9143865:2383945773 >>>>> > getattr pAsLsXsFs #0x10004f6a16b 2021-04-21 16:35:30.075110 >>>>> caller_uid=0, >>>>> > caller_gid=0{})", >>>>> > "initiated_at": "2021-04-21 16:35:30.077995", >>>>> > "age": 1081.0880337159999, >>>>> > "duration": 1081.0882681850001, >>>>> > "type_data": { >>>>> > "flag_point": "failed to authpin, subtree is being >>>>> > exported", >>>>> > "reqid": "client.9143865:2383945773", >>>>> > "op_type": "client_request", >>>>> > "client_info": { >>>>> > "client": "client.9143865", >>>>> > "tid": 2383945773 >>>>> > }, >>>>> > "events": [ >>>>> > { >>>>> > "time": "2021-04-21 16:35:30.077995", >>>>> > "event": "initiated" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:35:30.077995", >>>>> > "event": "header_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:35:30.077999", >>>>> > "event": "throttled" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:35:30.078018", >>>>> > "event": "all_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:35:30.078091", >>>>> > "event": "dispatched" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:35:30.078156", >>>>> > "event": "failed to authpin, subtree is being >>>>> > exported" >>>>> > } >>>>> > ] >>>>> > } >>>>> > }, >>>>> > { >>>>> > "description": "client_request(client.9801215:348417261 >>>>> lookup >>>>> > #0x10004cd0675/tasks 2021-04-21 15:49:09.248696 caller_uid=1000, >>>>> > caller_gid=1000{})", >>>>> > "initiated_at": "2021-04-21 15:49:09.252834", >>>>> > "age": 3861.9131950870001, >>>>> > "duration": 3861.9135065690002, >>>>> > "type_data": { >>>>> > "flag_point": "cleaned up request", >>>>> > "reqid": "client.9801215:348417261", >>>>> > "op_type": "client_request", >>>>> > "client_info": { >>>>> > "client": "client.9801215", >>>>> > "tid": 348417261 >>>>> > }, >>>>> > "events": [ >>>>> > { >>>>> > "time": "2021-04-21 15:49:09.252834", >>>>> > "event": "initiated" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:49:09.252834", >>>>> > "event": "header_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:49:09.252839", >>>>> > "event": "throttled" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:49:09.252856", >>>>> > "event": "all_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:49:09.252982", >>>>> > "event": "dispatched" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:49:09.253070", >>>>> > "event": "failed to authpin, subtree is being >>>>> > exported" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084439", >>>>> > "event": "killing request" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084444", >>>>> > "event": "cleaned up request" >>>>> > } >>>>> > ] >>>>> > } >>>>> > }, >>>>> > { >>>>> > "description": "client_request(client.9143865:2383945793 >>>>> > getattr pAsLsXsFs #0x10004f6a16b 2021-04-21 16:40:33.868658 >>>>> caller_uid=0, >>>>> > caller_gid=0{})", >>>>> > "initiated_at": "2021-04-21 16:40:33.871565", >>>>> > "age": 777.29446343799998, >>>>> > "duration": 777.29486169500001, >>>>> > "type_data": { >>>>> > "flag_point": "failed to authpin, subtree is being >>>>> > exported", >>>>> > "reqid": "client.9143865:2383945793", >>>>> > "op_type": "client_request", >>>>> > "client_info": { >>>>> > "client": "client.9143865", >>>>> > "tid": 2383945793 >>>>> > }, >>>>> > "events": [ >>>>> > { >>>>> > "time": "2021-04-21 16:40:33.871565", >>>>> > "event": "initiated" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:40:33.871565", >>>>> > "event": "header_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:40:33.871572", >>>>> > "event": "throttled" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:40:33.871617", >>>>> > "event": "all_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:40:33.871699", >>>>> > "event": "dispatched" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:40:33.871909", >>>>> > "event": "failed to authpin, subtree is being >>>>> > exported" >>>>> > } >>>>> > ] >>>>> > } >>>>> > }, >>>>> > { >>>>> > "description": "client_request(client.9801215:348424659 >>>>> lookup >>>>> > #0x10004cd0675/slaves 2021-04-21 15:50:01.592755 caller_uid=1000, >>>>> > caller_gid=1000{})", >>>>> > "initiated_at": "2021-04-21 15:50:01.595195", >>>>> > "age": 3809.5708340870001, >>>>> > "duration": 3809.5713271260001, >>>>> > "type_data": { >>>>> > "flag_point": "cleaned up request", >>>>> > "reqid": "client.9801215:348424659", >>>>> > "op_type": "client_request", >>>>> > "client_info": { >>>>> > "client": "client.9801215", >>>>> > "tid": 348424659 >>>>> > }, >>>>> > "events": [ >>>>> > { >>>>> > "time": "2021-04-21 15:50:01.595195", >>>>> > "event": "initiated" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:50:01.595195", >>>>> > "event": "header_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:50:01.595198", >>>>> > "event": "throttled" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:50:01.595210", >>>>> > "event": "all_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:50:01.595307", >>>>> > "event": "dispatched" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:50:01.595389", >>>>> > "event": "failed to authpin, subtree is being >>>>> > exported" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084444", >>>>> > "event": "killing request" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084448", >>>>> > "event": "cleaned up request" >>>>> > } >>>>> > ] >>>>> > } >>>>> > }, >>>>> > { >>>>> > "description": "client_request(client.9801215:348456034 >>>>> > setfilelock rule 1, type 4, owner 14336643483181153700, pid 32085, >>>>> start 0, >>>>> > length 0, wait 0 #0x1001e694623 20 >>>>> > 21-04-21 15:56:20.849213 caller_uid=1000, caller_gid=1000{})", >>>>> > "initiated_at": "2021-04-21 15:56:20.851803", >>>>> > "age": 3430.3142257680001, >>>>> > "duration": 3430.3148090620002, >>>>> > "type_data": { >>>>> > "flag_point": "cleaned up request", >>>>> > "reqid": "client.9801215:348456034", >>>>> > "op_type": "client_request", >>>>> > "client_info": { >>>>> > "client": "client.9801215", >>>>> > "tid": 348456034 >>>>> > }, >>>>> > "events": [ >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.851803", >>>>> > "event": "initiated" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.851803", >>>>> > "event": "header_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.851815", >>>>> > "event": "throttled" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.851853", >>>>> > "event": "all_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.851983", >>>>> > "event": "dispatched" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084452", >>>>> > "event": "killing request" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084454", >>>>> > "event": "cleaned up request" >>>>> > } >>>>> > ] >>>>> > } >>>>> > }, >>>>> > { >>>>> > "description": "client_request(client.9801215:348456035 >>>>> lookup >>>>> > #0x10004cd0675/tasks 2021-04-21 15:56:20.849213 caller_uid=1000, >>>>> > caller_gid=1000{})", >>>>> > "initiated_at": "2021-04-21 15:56:20.851899", >>>>> > "age": 3430.314129722, >>>>> > "duration": 3430.3147965909998, >>>>> > "type_data": { >>>>> > "flag_point": "cleaned up request", >>>>> > "reqid": "client.9801215:348456035", >>>>> > "op_type": "client_request", >>>>> > "client_info": { >>>>> > "client": "client.9801215", >>>>> > "tid": 348456035 >>>>> > }, >>>>> > "events": [ >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.851899", >>>>> > "event": "initiated" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.851899", >>>>> > "event": "header_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.851900", >>>>> > "event": "throttled" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.851908", >>>>> > "event": "all_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.852165", >>>>> > "event": "dispatched" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.852254", >>>>> > "event": "failed to authpin, subtree is being >>>>> > exported" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084455", >>>>> > "event": "killing request" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084457", >>>>> > "event": "cleaned up request" >>>>> > } >>>>> > ] >>>>> > } >>>>> > }, >>>>> > { >>>>> > "description": "client_request(client.9801215:348456036 >>>>> lookup >>>>> > #0x10004cd0675/slaves 2021-04-21 15:56:20.849213 caller_uid=1000, >>>>> > caller_gid=1000{})", >>>>> > "initiated_at": "2021-04-21 15:56:20.879287", >>>>> > "age": 3430.2867415639998, >>>>> > "duration": 3430.287496596, >>>>> > "type_data": { >>>>> > "flag_point": "cleaned up request", >>>>> > "reqid": "client.9801215:348456036", >>>>> > "op_type": "client_request", >>>>> > "client_info": { >>>>> > "client": "client.9801215", >>>>> > "tid": 348456036 >>>>> > }, >>>>> > "events": [ >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.879287", >>>>> > "event": "initiated" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.879287", >>>>> > "event": "header_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.879292", >>>>> > "event": "throttled" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.879306", >>>>> > "event": "all_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.879408", >>>>> > "event": "dispatched" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:20.879502", >>>>> > "event": "failed to authpin, subtree is being >>>>> > exported" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084458", >>>>> > "event": "killing request" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084460", >>>>> > "event": "cleaned up request" >>>>> > } >>>>> > ] >>>>> > } >>>>> > }, >>>>> > { >>>>> > "description": "client_request(client.9801215:348456042 >>>>> lookup >>>>> > #0x10004cd0675/jul 2021-04-21 15:58:53.729411 caller_uid=1000, >>>>> > caller_gid=1000{})", >>>>> > "initiated_at": "2021-04-21 15:58:53.740318", >>>>> > "age": 3277.4257106780001, >>>>> > "duration": 3277.4265688219998, >>>>> > "type_data": { >>>>> > "flag_point": "cleaned up request", >>>>> > "reqid": "client.9801215:348456042", >>>>> > "op_type": "client_request", >>>>> > "client_info": { >>>>> > "client": "client.9801215", >>>>> > "tid": 348456042 >>>>> > }, >>>>> > "events": [ >>>>> > { >>>>> > "time": "2021-04-21 15:58:53.740318", >>>>> > "event": "initiated" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:58:53.740318", >>>>> > "event": "header_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:58:53.740324", >>>>> > "event": "throttled" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:58:53.740345", >>>>> > "event": "all_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:58:53.743424", >>>>> > "event": "dispatched" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:58:53.750527", >>>>> > "event": "failed to authpin, subtree is being >>>>> > exported" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084465", >>>>> > "event": "killing request" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084468", >>>>> > "event": "cleaned up request" >>>>> > } >>>>> > ] >>>>> > } >>>>> > }, >>>>> > { >>>>> > "description": "client_request(client.9143865:2383914038 >>>>> > getattr pAsLsXsFs #0x10004f6a16b 2021-04-21 16:29:35.332632 >>>>> caller_uid=903, >>>>> > caller_gid=1003{})", >>>>> > "initiated_at": "2021-04-21 16:29:35.333838", >>>>> > "age": 1435.8321910940001, >>>>> > "duration": 1435.833136707, >>>>> > "type_data": { >>>>> > "flag_point": "failed to authpin, subtree is being >>>>> > exported", >>>>> > "reqid": "client.9143865:2383914038", >>>>> > "op_type": "client_request", >>>>> > "client_info": { >>>>> > "client": "client.9143865", >>>>> > "tid": 2383914038 >>>>> > }, >>>>> > "events": [ >>>>> > { >>>>> > "time": "2021-04-21 16:29:35.333838", >>>>> > "event": "initiated" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:29:35.333838", >>>>> > "event": "header_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:29:35.333841", >>>>> > "event": "throttled" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:29:35.333854", >>>>> > "event": "all_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:29:35.335013", >>>>> > "event": "dispatched" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:29:35.335091", >>>>> > "event": "failed to authpin, subtree is being >>>>> > exported" >>>>> > } >>>>> > ] >>>>> > } >>>>> > }, >>>>> > { >>>>> > "description": "client_request(client.10439445:7 lookup >>>>> > #0x10004cd0675/jul 2021-04-21 16:26:17.334272 caller_uid=1000, >>>>> > caller_gid=1000{})", >>>>> > "initiated_at": "2021-04-21 16:26:17.352448", >>>>> > "age": 1633.813581226, >>>>> > "duration": 1633.8146036109999, >>>>> > "type_data": { >>>>> > "flag_point": "cleaned up request", >>>>> > "reqid": "client.10439445:7", >>>>> > "op_type": "client_request", >>>>> > "client_info": { >>>>> > "client": "client.10439445", >>>>> > "tid": 7 >>>>> > }, >>>>> > "events": [ >>>>> > { >>>>> > "time": "2021-04-21 16:26:17.352448", >>>>> > "event": "initiated" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:26:17.352448", >>>>> > "event": "header_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:26:17.352450", >>>>> > "event": "throttled" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:26:17.352460", >>>>> > "event": "all_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:26:17.356901", >>>>> > "event": "dispatched" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:26:17.357337", >>>>> > "event": "failed to authpin, subtree is being >>>>> > exported" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:38:08.054109", >>>>> > "event": "killing request" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:38:08.054120", >>>>> > "event": "cleaned up request" >>>>> > } >>>>> > ] >>>>> > } >>>>> > }, >>>>> > { >>>>> > "description": "client_request(client.9801215:348456041 >>>>> lookup >>>>> > #0x10004cd0675/jul 2021-04-21 15:56:23.089216 caller_uid=1000, >>>>> > caller_gid=1000{})", >>>>> > "initiated_at": "2021-04-21 15:56:23.091531", >>>>> > "age": 3428.0744974170002, >>>>> > "duration": 3428.0756079930002, >>>>> > "type_data": { >>>>> > "flag_point": "cleaned up request", >>>>> > "reqid": "client.9801215:348456041", >>>>> > "op_type": "client_request", >>>>> > "client_info": { >>>>> > "client": "client.9801215", >>>>> > "tid": 348456041 >>>>> > }, >>>>> > "events": [ >>>>> > { >>>>> > "time": "2021-04-21 15:56:23.091531", >>>>> > "event": "initiated" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:23.091531", >>>>> > "event": "header_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:23.091536", >>>>> > "event": "throttled" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:23.091555", >>>>> > "event": "all_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:23.091660", >>>>> > "event": "dispatched" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 15:56:23.091763", >>>>> > "event": "failed to authpin, subtree is being >>>>> > exported" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084461", >>>>> > "event": "killing request" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:01:40.084465", >>>>> > "event": "cleaned up request" >>>>> > } >>>>> > ] >>>>> > } >>>>> > }, >>>>> > { >>>>> > "description": "client_request(client.9143865:2383885743 >>>>> > getattr pAsLsXsFs #0x10004f6a16b 2021-04-21 16:26:36.713371 >>>>> caller_uid=903, >>>>> > caller_gid=1003{})", >>>>> > "initiated_at": "2021-04-21 16:26:36.716418", >>>>> > "age": 1614.4496113350001, >>>>> > "duration": 1614.4508114350001, >>>>> > "type_data": { >>>>> > "flag_point": "failed to authpin, subtree is being >>>>> > exported", >>>>> > "reqid": "client.9143865:2383885743", >>>>> > "op_type": "client_request", >>>>> > "client_info": { >>>>> > "client": "client.9143865", >>>>> > "tid": 2383885743 >>>>> > }, >>>>> > "events": [ >>>>> > { >>>>> > "time": "2021-04-21 16:26:36.716418", >>>>> > "event": "initiated" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:26:36.716418", >>>>> > "event": "header_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:26:36.716421", >>>>> > "event": "throttled" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:26:36.716435", >>>>> > "event": "all_read" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:26:36.716526", >>>>> > "event": "dispatched" >>>>> > }, >>>>> > { >>>>> > "time": "2021-04-21 16:26:36.716632", >>>>> > "event": "failed to authpin, subtree is being >>>>> > exported" >>>>> > } >>>>> > ] >>>>> > } >>>>> > } >>>>> > ], >>>>> > "num_ops": 14 >>>>> > } >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > -- >>>>> > Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ >>>>> > >>>>> >>>>> >>>>> -- >>>>> Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>>>> >>>> >>> >>> -- >>> Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ >>> >> > > -- > Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |