> Il giorno 15 lug 2021, alle ore 15:30, Jan Kara <jack@xxxxxxx> ha scritto: > > Hello! > Hi! > Here is the second revision of my patches to fix how bfq weights apply on > cgroup throughput. I don't remember whether I replied to your first version. Anyway, thanks for this important contribution. > This version has only one change fixing how we compute > number of tags that should be available to a cgroup. Previous version didn't > combine weights at several levels correctly for deeper hierarchies. It is > somewhat unfortunate that for really deep cgroup hierarchies we would now do > memory allocation inside bfq_limit_depth(). I have an idea how we could avoid > that if the rest of the approach proves OK so don't concentrate too much on > that detail please. > > Changes since v1: > * Fixed computation of appropriate proportion of scheduler tags for a cgroup > to work with deeper cgroup hierarchies. > > Original cover letter: > > I was looking into why cgroup weights do not have any measurable impact on > writeback throughput from different cgroups. This actually a regression from > CFQ where things work more or less OK and weights have roughly the impact they > should. The problem can be reproduced e.g. by running the following easy fio > job in two cgroups with different weight: > > [writer] > directory=/mnt/repro/ > numjobs=1 > rw=write > size=8g > time_based > runtime=30 > ramp_time=10 > blocksize=1m > direct=0 > ioengine=sync > > I can observe there's no significat difference in the amount of data written > from different cgroups despite their weights are in say 1:3 ratio. > > After some debugging I've understood the dynamics of the system. There are two > issues: > > 1) The amount of scheduler tags needs to be significantly larger than the > amount of device tags. Otherwise there are not enough requests waiting in BFQ > to be dispatched to the device and thus there's nothing to schedule on. > Before discussing your patches in detail, I need a little help on this point. You state that the number of scheduler tags must be larger than the number of device tags. So, I expected some of your patches to address somehow this issue, e.g., by increasing the number of scheduler tags. Yet I have not found such a change. Did I miss something? Thanks, Paolo > 2) Even with enough scheduler tags, writers from two cgroups eventually start > contending on scheduler tag allocation. These are served on first come first > served basis so writers from both cgroups feed requests into bfq with > approximately the same speed. Since bfq prefers IO from heavier cgroup, that is > submitted and completed faster and eventually we end up in a situation when > there's no IO from the heavier cgroup in bfq and all scheduler tags are > consumed by requests from the lighter cgroup. At that point bfq just dispatches > lots of the IO from the lighter cgroup since there's no contender for disk > throughput. As a result observed throughput for both cgroups are the same. > > This series fixes this problem by accounting how many scheduler tags are > allocated for each cgroup and if a cgroup has more tags allocated than its > fair share (based on weights) in its service tree, we heavily limit scheduler > tag bitmap depth for it so that it is not be able to starve other cgroups from > scheduler tags. > > What do people think about this? > > Honza > > Previous versions: > Link: http://lore.kernel.org/r/20210712171146.12231-1-jack@xxxxxxx # v1
