On Mon, Aug 05, 2019 at 06:32:51PM +0000, Chris Mason wrote: > On 2 Aug 2019, at 19:28, Dave Chinner wrote: > > > On Fri, Aug 02, 2019 at 02:11:53PM +0000, Chris Mason wrote: > >> On 1 Aug 2019, at 19:58, Dave Chinner wrote: > >> I can't really see bio->b_ioprio working without the rest of the IO > >> controller logic creating a sensible system, > > > > That's exactly the problem we need to solve. The current situation > > is ... untenable. Regardless of whether the io.latency controller > > works well, the fact is that the wbt subsystem is active on -all- > > configurations and the way it "prioritises" is completely broken. > > Completely broken is probably a little strong. Before wbt, it was > impossible to do buffered IO without periodically saturating the drive > in unexpected ways. We've got a lot of data showing it helping, and > it's pretty easy to setup a new A/B experiment to demonstrate it's > usefulness in current kernels. But that doesn't mean it's perfect. I'm not arguing that wbt is useless, I'm just saying that it's design w.r.t. IO prioritisation is fundamentally broken. Using request types to try to infer priority just doesn't work, as I've been trying to explain. > >> framework to define weights etc. My question is if it's worth trying > >> inside of the wbt code, or if we should just let the metadata go > >> through. > > > > As I said, that doesn't solve the problem. We /want/ critical > > journal IO to have higher priority that background metadata > > writeback. Just ignoring REQ_META doesn't help us there - it just > > moves the priority inversion to blocking on request queue tags. > > Does XFS background metadata IO ever get waited on by critical journal > threads? No. Background writeback (which, with this series, is the only way metadata gets written in XFS) is almost entirely non-blocking until IO submission occurs. It will force the log if pinned items are prevents the log tail from moving (hence blocking on log IO) but largely it doesn't block on anything except IO submission. The only thing that blocks on journal IO is CIL flushing and, subsequently, anything that is waiting on a journal flush to complete. CIL flushing happens in it's own workqueue, so it doesn't block anything directly. The only operations that wait for log IO require items to be stable in the journal (e.g. fsync()). Starting a transactional change may block on metadata writeback. If there isn't space in the log for the new transaction, it will kick and wait for background metadata writeback to make progress and push the tail of the log forwards. And this may wait on journal IO if pinned items need to be flushed to the log before writeback can occur. This is the way we prevent transactions requiring journal IO to blocking on metadata writeback to make progress - we don't allow a transaction to start until it is guaranteed that it can complete without requiring journal IO to flush other metadata to the journal. That way there is always space available in the log for all pending journal IO to complete with a dependency no metadata writeback making progress. This "block on metadata writeback at transaction start" design means data writeback can block on metadata writeback because we do allocation transactions in the IO path. Which means data IO can block behind metadata IO, which can block behind log IO, and that largely defines the IO heirarchy in XFS. Hence the IO priority order is very clear in XFS - it was designed this way because you can't support things like guaranteed rate IO storage applications (one of the prime use cases XFS was originally designed for) without having a clear model for avoiding priority inversions between data, metadata and the journal. I'm not guessing about any of this - I know how all this is supposed to work because I spent years at SGI working with people far smarter than me supporting real-time IO applications working along with real-time IO schedulers in a real time kernel (i.e. Irix). I don't make this stuff up for fun or to argue, I say stuff because I know how it's supposed to work. And, FWIW, Irix also had a block layer writeback throttling mechanism to prevent bulk data writeback from thrashing disks and starving higher priority IO. It was also fully IO priority aware - this stuff isn't rocket science, and Linux is not the first OS to ever implement this sort of functionality. Linux was not my first rodeo.... > My understanding is that all of the filesystems do this from > time to time. Without a way to bump the priority of throttled > background metadata IO, I can't see how to avoid prio inversions without > running background metadata at the same prio as all of the critical > journal IO. Perhaps you just haven't thought about it enough. :) > > Core infrastructure needs to work without cgroups being configured > > to confine everything in userspace to "safe" bounds, and right now > > just running things in the root cgroup doesn't appear to work very > > well at all. > > I'm not disagreeing with this part, my real point is there isn't a > single answer. It's possible for swap to be critical to the running of > the box in some workloads, and totally unimportant in others. Sure, but that only indicates that we need to be able to adjust the priority of IO within certain bounds. The problem is right now is that the default behaviour is pretty nasty and core functionality is non-functional. It doesn't matter if swap priority is adjustable or not, users should not have to tune the kernel to use an esoteric cgroup configuration in order for the kernel to function correctly out of the box. I'm not sure when we lost sight of the fact we need to make the default configurations work correctly first, and only then do we worry about how tunable somethign is when the default behaviour has been proven to be insufficient. Hiding bad behaviour behind custom cgroup configuration does nobody any favours. Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx
