An MDS-wide lock is acquired before the cache dump is done.
After the dump is complete, the lock is released.
So, the MDS freezing temporarily during the cache dump is expected.
On Fri, May 26, 2023 at 12:51 PM Emmanuel Jaep <emmanuel.jaep@xxxxxxxxx>
wrote:
> Hi Milind,
>
> I finally managed to dump the cache and find the file.
> It generated a 1.5 GB file with about 7 Mio lines. It's kind of hard to
> know what is out of the ordinary…
>
> Furthermore, I noticed that dumping the cache was actually stopping the
> MDS. Is it a normal behavior?
>
> Best,
>
> Emmanuel
>
> On Thu, May 25, 2023 at 1:19 PM Milind Changire <mchangir@xxxxxxxxxx>
> wrote:
>
>> try the command with the --id argument:
>>
>> # ceph --id admin --cluster floki daemon mds.icadmin011 dump cache
>> /tmp/dump.txt
>>
>> I presume that your keyring has an appropriate entry for the client.admin
>> user
>>
>>
>> On Wed, May 24, 2023 at 5:10 PM Emmanuel Jaep <emmanuel.jaep@xxxxxxxxx>
>> wrote:
>>
>>> Absolutely! :-)
>>>
>>> root@icadmin011:/tmp# ceph --cluster floki daemon mds.icadmin011 dump
>>> cache /tmp/dump.txt
>>> root@icadmin011:/tmp# ll
>>> total 48
>>> drwxrwxrwt 12 root root 4096 May 24 13:23 ./
>>> drwxr-xr-x 18 root root 4096 Jun 9 2022 ../
>>> drwxrwxrwt 2 root root 4096 May 4 12:43 .ICE-unix/
>>> drwxrwxrwt 2 root root 4096 May 4 12:43 .Test-unix/
>>> drwxrwxrwt 2 root root 4096 May 4 12:43 .X11-unix/
>>> drwxrwxrwt 2 root root 4096 May 4 12:43 .XIM-unix/
>>> drwxrwxrwt 2 root root 4096 May 4 12:43 .font-unix/
>>> drwx------ 2 root root 4096 May 24 13:23 ssh-Sl5AiotnXp/
>>> drwx------ 3 root root 4096 May 8 13:26
>>> 'systemd-private-18c17b770fc24c48a0507b8faa1c0ec2-ceph-mds@icadmin011.service-SGZrKf
>>> '/
>>> drwx------ 3 root root 4096 May 4 12:43
>>> systemd-private-18c17b770fc24c48a0507b8faa1c0ec2-systemd-logind.service-uU1GAi/
>>> drwx------ 3 root root 4096 May 4 12:43
>>> systemd-private-18c17b770fc24c48a0507b8faa1c0ec2-systemd-resolved.service-KYHd7f/
>>> drwx------ 3 root root 4096 May 4 12:43
>>> systemd-private-18c17b770fc24c48a0507b8faa1c0ec2-systemd-timesyncd.service-1Qtj5i/
>>>
>>> On Wed, May 24, 2023 at 1:17 PM Milind Changire <mchangir@xxxxxxxxxx>
>>> wrote:
>>>
>>>> I hope the daemon mds.icadmin011 is running on the same machine that
>>>> you are looking for /tmp/dump.txt, since the file is created on the system
>>>> which has that daemon running.
>>>>
>>>>
>>>> On Wed, May 24, 2023 at 2:16 PM Emmanuel Jaep <emmanuel.jaep@xxxxxxxxx>
>>>> wrote:
>>>>
>>>>> Hi Milind,
>>>>>
>>>>> you are absolutely right.
>>>>>
>>>>> The dump_ops_in_flight is giving a good hint about what's happening:
>>>>> {
>>>>> "ops": [
>>>>> {
>>>>> "description": "internal op exportdir:mds.5:975673",
>>>>> "initiated_at": "2023-05-23T17:49:53.030611+0200",
>>>>> "age": 60596.355186077999,
>>>>> "duration": 60596.355234167997,
>>>>> "type_data": {
>>>>> "flag_point": "failed to wrlock, waiting",
>>>>> "reqid": "mds.5:975673",
>>>>> "op_type": "internal_op",
>>>>> "internal_op": 5377,
>>>>> "op_name": "exportdir",
>>>>> "events": [
>>>>> {
>>>>> "time": "2023-05-23T17:49:53.030611+0200",
>>>>> "event": "initiated"
>>>>> },
>>>>> {
>>>>> "time": "2023-05-23T17:49:53.030611+0200",
>>>>> "event": "throttled"
>>>>> },
>>>>> {
>>>>> "time": "2023-05-23T17:49:53.030611+0200",
>>>>> "event": "header_read"
>>>>> },
>>>>> {
>>>>> "time": "2023-05-23T17:49:53.030611+0200",
>>>>> "event": "all_read"
>>>>> },
>>>>> {
>>>>> "time": "2023-05-23T17:49:53.030611+0200",
>>>>> "event": "dispatched"
>>>>> },
>>>>> {
>>>>> "time": "2023-05-23T17:49:53.030657+0200",
>>>>> "event": "requesting remote authpins"
>>>>> },
>>>>> {
>>>>> "time": "2023-05-23T17:49:53.050253+0200",
>>>>> "event": "failed to wrlock, waiting"
>>>>> }
>>>>> ]
>>>>> }
>>>>> }
>>>>> ],
>>>>> "num_ops": 1
>>>>> }
>>>>>
>>>>> However, the dump cache does not seem to produce an output:
>>>>> root@icadmin011:~# ceph --cluster floki daemon mds.icadmin011 dump
>>>>> cache /tmp/dump.txt
>>>>> root@icadmin011:~# ls /tmp
>>>>> ssh-cHvP3iF611
>>>>>
>>>>> systemd-private-18c17b770fc24c48a0507b8faa1c0ec2-ceph-mds@icadmin011.service-SGZrKf
>>>>>
>>>>> systemd-private-18c17b770fc24c48a0507b8faa1c0ec2-systemd-logind.service-uU1GAi
>>>>>
>>>>> systemd-private-18c17b770fc24c48a0507b8faa1c0ec2-systemd-resolved.service-KYHd7f
>>>>>
>>>>> systemd-private-18c17b770fc24c48a0507b8faa1c0ec2-systemd-timesyncd.service-1Qtj5i
>>>>>
>>>>> Do you have any hint?
>>>>>
>>>>> Best,
>>>>>
>>>>> Emmanuel
>>>>>
>>>>> On Wed, May 24, 2023 at 10:30 AM Milind Changire <mchangir@xxxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>>> Emmanuel,
>>>>>> You probably missed the "daemon" keyword after the "ceph" command
>>>>>> name.
>>>>>> Here's the docs for pacific:
>>>>>> https://docs.ceph.com/en/pacific/cephfs/troubleshooting/
>>>>>>
>>>>>> So, your command should've been:
>>>>>> # ceph daemon mds.icadmin011 dump cache /tmp/dump.txt
>>>>>>
>>>>>> You could also dump the ops in flight with:
>>>>>> # ceph daemon mds.icadmin011 dump_ops_in_flight
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, May 24, 2023 at 1:38 PM Emmanuel Jaep <
>>>>>> emmanuel.jaep@xxxxxxxxx>
>>>>>> wrote:
>>>>>>
>>>>>> > Hi,
>>>>>> >
>>>>>> > we are running a cephfs cluster with the following version:
>>>>>> > ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17)
>>>>>> pacific
>>>>>> > (stable)
>>>>>> >
>>>>>> > Several MDSs are reporting slow requests:
>>>>>> > HEALTH_WARN 4 MDSs report slow requests
>>>>>> > [WRN] MDS_SLOW_REQUEST: 4 MDSs report slow requests
>>>>>> > mds.icadmin011(mds.5): 1 slow requests are blocked > 30 secs
>>>>>> > mds.icadmin015(mds.6): 2 slow requests are blocked > 30 secs
>>>>>> > mds.icadmin006(mds.4): 8 slow requests are blocked > 30 secs
>>>>>> > mds.icadmin007(mds.2): 2 slow requests are blocked > 30 secs
>>>>>> >
>>>>>> > According to Quincy's documentation (
>>>>>> > https://docs.ceph.com/en/quincy/cephfs/troubleshooting/), this can
>>>>>> be
>>>>>> > investigated by issuing:
>>>>>> > ceph mds.icadmin011 dump cache /tmp/dump.txt
>>>>>> >
>>>>>> > Unfortunately, this command fails:
>>>>>> > no valid command found; 10 closest matches:
>>>>>> > pg stat
>>>>>> > pg getmap
>>>>>> > pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...]
>>>>>> > pg dump_json [all|summary|sum|pools|osds|pgs...]
>>>>>> > pg dump_pools_json
>>>>>> > pg ls-by-pool <poolstr> [<states>...]
>>>>>> > pg ls-by-primary <id|osd.id> [<pool:int>] [<states>...]
>>>>>> > pg ls-by-osd <id|osd.id> [<pool:int>] [<states>...]
>>>>>> > pg ls [<pool:int>] [<states>...]
>>>>>> > pg dump_stuck [inactive|unclean|stale|undersized|degraded...]
>>>>>> > [<threshold:int>]
>>>>>> > Error EINVAL: invalid command
>>>>>> >
>>>>>> >
>>>>>> > I imagine that it is related to the fact that we are running the
>>>>>> Pacific
>>>>>> > version and not the Quincy version.
>>>>>> >
>>>>>> > When looking at the Pacific's documentation (
>>>>>> > https://docs.ceph.com/en/pacific/cephfs/health-messages/), I
>>>>>> should:
>>>>>> > > Use the ops admin socket command to list outstanding metadata
>>>>>> operations.
>>>>>> >
>>>>>> > Unfortunately, I fail to really understand what I'm supposed to do.
>>>>>> Can
>>>>>> > someone give a pointer?
>>>>>> >
>>>>>> > Best,
>>>>>> >
>>>>>> > Emmanuel
>>>>>> > _______________________________________________
>>>>>> > ceph-users mailing list -- ceph-users@xxxxxxx
>>>>>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>>>>> >
>>>>>> >
>>>>>>
>>>>>> --
>>>>>> Milind
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Milind
>>>>
>>>>
>>
>> --
>> Milind
>>
>>
--
Milind
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
