On Mon, Apr 27, 2009 at 11:53:39AM +0200, Jens Axboe wrote: > > I'm kind of curious why you implemented things in this way, though. > > Is there a reason why the bosting is happening deep in the guts of the > > cfq code, instead of in blk-core.c when the submission of the block > > I/O request is processed? > > You would need to implement a lot more logic in the block layer to > handle it there, as it stands it's basically a scheduler decision. So > the positioning is right imho, the placement of fs hooks is probably > mostly crap and could do with some work. The question is whether you see this in terms of a scheduler decision or in terms of an I/O priority issue. At the moment I agree it's a scheduler decision (which to be honest is implemented in somewhat of a hacky way --- which I suspect won't bother you since, you yourself called it "half-assed" :-) which happens to be implemented in the I/O scheduler. I tend to think of it more as an I/O priority issue, and specifically, as you put it, an priority inversion issue, but much of that is no doubt influenced by how I did the patches to reduce the fsync() latencies in ext3 and ext4. And indeed the get_fs_excl()/put_fs_excl() paradigm doesn't really work well for ext3/ext4 since all of the work which grabs a filesystem-wide "exclusive lock" is done in a separate process, kjournald. Hence with the exception of freeze and unfreeze --- and while this might be considered irresponsible for a system administrator to freeze a filesystem in a ionice'd process, I could imagine a badly written backup script which created a snapshot while being ionice'd --- ext3/4 can't really very profitably use get_fs_excl()/put_fs_excl(). Maybe ext3/ext4 are a special case, but perhaps we should nevertheless ask some fundamental design questions about the get/put_fs_excl() interface. *) Most filesystems will go to great lengths to avoid having any kind of fs-wide "exclusive lock", simply because of the disastrous performance impacts. This is *why* in ext3/ext4, we try to do most of the commit work in the context of another process, and normally usually we let other filesystem operations run in the "current transaction" while we let the "committing transaction" complete. If you have too many programs running fsync() this tends to screw things up, but that's a separate question. So in practice, there really shouldn't be that many "fs-wide" locks. On the other hand, there can be more subtle forms of I/O priority inversion; suppose a low priority process has grabbed a mutex which protects a directory, and a high (I/O) priority process needs access to the same directory. Do we care about trying to solve that issue? *) Do we only want to support instances where the fs-wide resource is held in kernel-space only, or do we want to support things like the FREEZE ioctl, where the filesystem has been frozen --- the very definition of an I/O wide resource? (I would argue no, for simplicity's sake but document the fact that the well-written program using the FREEZE ioctl should strongly consider bumping up its I/O and possibly CPU priority levels to minimize the impact on the rest of the system. Since the FREEZE ioctl requires root privileges, it's fair to assume a certain amount of competence by the users of this interface.) If the answer to this question is no, then we can add warning/debugging code which warns if the filesystem ever tries returning to userspace with an elevated get_fs_excl() count. *) Do we only care about processes whose I/O priority is below the default? (i.e., either in the idle class, or in a low-priority best efforts class) What if the concern is a real-time process which is being blocked by a default I/O priority process taking its time while holding some fs-wide resource? If the answer to the previous question is no, it becomes more reasonable to consider bump the submission priority of the process in question to the highest priority "best efforts" level. After all, if this truly is a "filesystem-wide" resource, then no one is going to make forward progress relating to this block device unless and until the filesystem-wide lock is resolved. Also, if we don't allow this situation to return to userspace, presumably the kernel-code involved will only be writing to the block-device in question. (This might not be entirely true if in the case of the sendfile(2) syscall, but currently we can only read from filesystems with sendfile, and so presumably a filesystem would never call get_fs_excl why servicing a sendfile request.) *) Is implementing the bulk of this in the cfq scheduler really the best place to do this? To explore something completely different, what if the filesystem simply explicitly set I/O priority levels in its block I/O submissions, and provided optional callback functions which could be used by the page writeback routines to determine the appropriate I/O priority level that should be used given a particular filesystem and inode number. (That actually could be used to provide another cool function --- we could expose to userspace the concept that particular inode should always have its I/O go out with a higher priority, perhaps via chattr flag.) Basically, the argument here is that we already have the appropriate mechanism for ordering I/O requests, which is I/O priority mechanism, and the policy really needs to be set by the filesystem --- and it might be far more than just "do we have a filesystem-wide exclusive lock" or not. What do other filesystem developers think? - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
