Polled IO is only allowed for conditions where task completion is safe
anyway, so we can always complete it inline. This cannot easily be
checked with a submission side flag, as the block layer may clear the
polled flag and turn it into a regular IO instead. Hence we need to
check this at completion time. If REQ_POLLED is still set, then we know
that this IO was successfully polled, and is completing in task context.
Reviewed-by: Darrick J. Wong <djwong@xxxxxxxxxx>
Reviewed-by: Christoph Hellwig <hch@xxxxxx>
Signed-off-by: Jens Axboe <axboe@xxxxxxxxx>
---
fs/iomap/direct-io.c | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index 17b695b0e9d6..2b453e12dc16 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -173,9 +173,19 @@ void iomap_dio_bio_end_io(struct bio *bio)
}
/*
- * Flagged with IOMAP_DIO_INLINE_COMP, we can complete it inline
+ * Flagged with IOMAP_DIO_INLINE_COMP, we can complete it inline.
+ * Ditto for polled requests - if the flag is still at completion
+ * time, then we know the request was actually polled and completion
+ * is called from the task itself. This is why we need to check it
+ * here rather than flag it at issue time.
*/
- if (dio->flags & IOMAP_DIO_INLINE_COMP) {
+ if ((dio->flags & IOMAP_DIO_INLINE_COMP) || (bio->bi_opf & REQ_POLLED)) {
+ /*
+ * For polled IO, we need to clear ->private as it points to
+ * the bio being polled for. The completion side uses it to
+ * know if a given request has been found yet or not. For
+ * non-polled IO, ->private isn't applicable.
+ */
WRITE_ONCE(iocb->private, NULL);
iomap_dio_complete_work(&dio->aio.work);
goto release_bio;
--
2.40.1
