Troubleshooting "N slow requests are blocked > 30 secs" on Pacific

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

we are running a cephfs cluster with the following version:
ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific
(stable)

Several MDSs are reporting slow requests:
HEALTH_WARN 4 MDSs report slow requests
[WRN] MDS_SLOW_REQUEST: 4 MDSs report slow requests
    mds.icadmin011(mds.5): 1 slow requests are blocked > 30 secs
    mds.icadmin015(mds.6): 2 slow requests are blocked > 30 secs
    mds.icadmin006(mds.4): 8 slow requests are blocked > 30 secs
    mds.icadmin007(mds.2): 2 slow requests are blocked > 30 secs

According to Quincy's documentation (
https://docs.ceph.com/en/quincy/cephfs/troubleshooting/), this can be
investigated by issuing:
ceph mds.icadmin011 dump cache /tmp/dump.txt

Unfortunately, this command fails:
no valid command found; 10 closest matches:
pg stat
pg getmap
pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...]
pg dump_json [all|summary|sum|pools|osds|pgs...]
pg dump_pools_json
pg ls-by-pool <poolstr> [<states>...]
pg ls-by-primary <id|osd.id> [<pool:int>] [<states>...]
pg ls-by-osd <id|osd.id> [<pool:int>] [<states>...]
pg ls [<pool:int>] [<states>...]
pg dump_stuck [inactive|unclean|stale|undersized|degraded...]
[<threshold:int>]
Error EINVAL: invalid command


I imagine that it is related to the fact that we are running the Pacific
version and not the Quincy version.

When looking at the Pacific's documentation (
https://docs.ceph.com/en/pacific/cephfs/health-messages/), I should:
> Use the ops admin socket command to list outstanding metadata operations.

Unfortunately, I fail to really understand what I'm supposed to do. Can
someone give a pointer?

Best,

Emmanuel
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux