We are asking the Ceph community to provide their thoughts on this draft proposal for expanding the librados API with calls that would allow clients to specify QoS (quality of service) parameters for their operations. We have an on-going effort to provide Ceph users with more options to manage QoS. With the release of Luminous we introduced access to a prototype of the mclock QoS algorithm for queuing operations by class of operation and either differentiating clients or treating them as a unit. Although not yet integrated, the library we're using supports dmClock, a distributed version of mClock. Both are documented in _mClock: Handling Throughput Variability for Hypervisor IO Scheduling_ by Gulati, Merchant, and Varman 2010. In order to offer greater flexibility, we'd like to move forward with providing clients with the ability to use different QoS parameters. We are keeping our options open w.r.t. the ultimate set of algorithm(s) we'll use. The mClock/dmClock algorithm allows a "client", which we can interpret broadly, to set a minimum ops/sec (reservation) and a maximum ops/sec (limit). Furthermore a "client" can also define a weight (a.k.a. priority), which is a scalar value to determine relative weighting. We think reservation, limit, and weight are sufficiently generic that we'd be able to use or adapt them other QoS algorithms we may try or use in the future. [To give you a sense of how broadly we can interpret "client", we currently have code that interprets classes of operations (e.g., background replication or background snap-trimming) as a client.] == Units == One key difference we're considering, however, is changing the unit that reservations and limits are expressed in from ops/sec to something more appropriate for Ceph. Operations have payloads of different sizes and will therefore take different amounts of time, and that should be factored in. We might refer to this as the "cost" of the operation. And the cost is not linear with the size of the payload. For example, a write of 4 MB might only take 20 times as long as a write of 4 KB even though the sizes differ by a factor of 1000. Using cost would allow us to, for example, achieve a fairer prioritization of a client doing many small writes against a client that's doing a few larger writes. One proposed formula to translate one op into cost would be something along the lines of: cost_units = a + b * log(payload_size) where a and b would have to be chosen or tuned based on the storage back-end. And that gets us to the units for defining reservation and limit -- cost_units per unit of time. Typically these are floating point values, however we do not use floating point types in librados calls because qemu, when calling into librbd, does not save and restore the cpu's floating point mode. There are two ways of getting appropriate ranges of values given that we need to use integral types for cost_units per unit of time. One is a large time unit in the denominator, such as minutes or even hours. That would leave us with cost_units per minute. We are unsure that the strange unit is the best approach and your feedback would be appreciated. A standard alternative would be to use a standard time unit, such as seconds, but integers as fixed-point values. So a floating-point value in cost_units per second would be multiplied by, say, 1000 and rounded to get the corresponding integer value. == librados Additions == The basic idea is that one would be able to create (and destroy) qos profiles and then associate a profile with an ioctx. Ops on the ioctx would use the qos profile associated with it. typedef void* rados_qos_profile_t; // opaque // parameters uint64_t in cost_units per time unit as discussed above profile1 = rados_qos_profile_create(reservation, weight, limit); rados_ioctx_set_qos_profile(ioctx3, profile1); ... // ops to ioctx3 would now use the specified profile ... // use the profile just for a particular operation rados_write_op_set_qos_prefile(op1, profile1); rados_ioctx_set_qos_profile(ioctx3, NULL); // set to default profile rados_qos_profile_destroy(profile1); == MOSDOp and MOSDOpReply Changes == Because the qos_profile would be managed by the ioctx, MOSDOps sent via that ioctx would include the reservation, weight, and limit. At this point we think this would be better than keeping the profiles on the back-end, although it increases the MOSDOp data structure by about 128 bits. The MOSDOp type already contains dmclock's delta and rho parameters and MOSDOpReply already contains the dmclock phase indicator due to prior work. Given that we're moving towards using cost_unit per time_unit rather than ops per sec, perhaps we should also include the calculated cost in the MOSDOpReply. == Conclusion == So that's what we're thinking about and your own thoughts and feedback would be appreciated. Thanks! -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html