> On Jan 10, 2025, at 7:46 AM, Olaf Seibert <o.seibert@xxxxxxxxxxxx> wrote: > > Hi! I am trying to find if Ceph has any QoS settings that apply per-client. I would like to be able, say, to have different QoS settings for RBD clients named "nova" and "cinder" and different again from an RGW client named "enduser". > > While researching this, I found amongst other things, the following references, which do not appear to be what I want: > > - https://docs.ceph.com/en/reef/dev/osd_internals/mclock_wpq_cmp_study/ and https://docs.ceph.com/en/reef/rados/configuration/osd-config-ref/#dmclock-qos . This appears to be instead about the balancing of user-iops against ceph-internal iops, not balancing clients against each other. Yes, mclock is within the cluster. > (aside: somebody should rewrite this paragraph, I find it totally incomprehensible: https://docs.ceph.com/en/reef/rados/configuration/osd-config-ref/#subtleties-of-mclock ) If I could understand mblock I’d rewrite it myself :-/. Please do enter a tracker ticket. > - https://docs.ceph.com/en/latest/rbd/rbd-config-ref/#qos-settings and https://tracker.ceph.com/issues/36191 This seems to be about setting limits on the RBD pools as a whole, or on separate images inside them. However separate clients each still get to use up to the limits. That link https://docs.ceph.com/en/reef/rbd/rbd-config-ref/#qos-settings does have a section that describes per-image (volume) settings, which you should be able to enforce on the OpenStack side. OpenStack / libvirt do have their own IOPS and throughput throttles you can find in their docs. This is important to do so that instances don’t DoS each other or the entire Nova node. Be sure to increase the system-wide file limit on those nodes, to like 4 million. Each RBD attachment needs two sockets to *each* OSD node for each attachment. For RGW you can do rate limiting: https://docs.ceph.com/en/latest/radosgw/adminops/#rate-limit Which I *think* is per-RGW granularity. > Is there anything else that I have not found so far but which is about balancing individual clients across all services? Remember that a CephFS mount, an RBD attachment, and an RGW session are three different clients from Ceph’s perspective. If you actually mean rate limiting > > > -- > Olaf Seibert > Site Reliability Engineer > > SysEleven GmbH > Boxhagener Straße 80 > 10245 Berlin > > T +49 30 233 2012 0 > F +49 30 616 7555 0 > > https://www.syseleven.de > https://www.linkedin.com/company/syseleven-gmbh/ > > Current system status always at: > https://www.syseleven-status.net/ > > Company headquarters: Berlin > Registered court: AG Berlin Charlottenburg, HRB 108571 Berlin > Managing directors: Andreas Hermann, Jens Ihlenfeld, Norbert Müller, Jens Plogsties > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |