Hi guys, As a distributed filesystem, all clients of CephFS share the whole cluster's resources, for example, IOPS, throughput. In some cases, resources will be occupied by some clients. So QoS for CephFS is needed in most cases. I have made two kinds of design, as follows: 1. all clients use the same QoS setting, just as the implementation in this PR. (PR: https://github.com/ceph/ceph/pull/29266). Maybe there are multiple mount points, if we limit the total IO, the number of total mount points is also limited. So in my implementation, the total IO & BPS is not limited. 2. all clients share a specific QoS setting. I think there are two kinds of use cases in detail. (not implemented) 2.1 setting a total limit, all clients limited by the average: total_limit/clients_num. 2.2 setting a total limit, the mds decide the client's limitation by their historical IO&BPS. Based on the token bucket algorithm, I implement QoS for CephFS. The basic idea is as follows: Set QoS info as one of the dir's xattrs; All clients can access the same dirs with the same QoS setting. Similar to the Quota's config flow. when the MDS receives the QoS setting, it'll also broadcast the message to all clients. We can change the limit online. And we will config QoS as follows, it supports {limit/burst}{iops/bps/read_iops/read_bps/write_iops/write_bps} configure setting, some examples: setfattr -n ceph.qos.limit.iops -v 200 /mnt/cephfs/testdirs/ setfattr -n ceph.qos.burst.read_bps -v 200 /mnt/cephfs/testdirs/ getfattr -n ceph.qos.limit.iops /mnt/cephfs/testdirs/ getfattr -n ceph.qos /mnt/cephfs/testdirs/ But, there is also a big problem. For the bps{bps/write_bps/read_bps} setting, if the bps is lower than the request's block size, the client will be blocked until it gets enough token. Any suggestion will be appreciated, thanks! PR: https://github.com/ceph/ceph/pull/29266 And I am finishing the QoS implementation in Cephfs-kernel client and will post the code when I finish it. _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
