This is a note to let you know that I've just added the patch titled io-wq: serialize hash clear with wakeup to the 5.14-stable tree which can be found at: http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary The filename of the patch is: io-wq-serialize-hash-clear-with-wakeup.patch and it can be found in the queue-5.14 subdirectory. If you, or anyone else, feels it should not be added to the stable tree, please let <stable@xxxxxxxxxxxxxxx> know about it. >From foo@baz Mon Nov 15 03:27:04 PM CET 2021 From: Jens Axboe <axboe@xxxxxxxxx> Date: Sun, 14 Nov 2021 07:36:47 -0700 Subject: io-wq: serialize hash clear with wakeup From: Jens Axboe <axboe@xxxxxxxxx> commit d3e3c102d107bb84251455a298cf475f24bab995 upstream. We need to ensure that we serialize the stalled and hash bits with the wait_queue wait handler, or we could be racing with someone modifying the hashed state after we find it busy, but before we then give up and wait for it to be cleared. This can cause random delays or stalls when handling buffered writes for many files, where some of these files cause hash collisions between the worker threads. Cc: stable@xxxxxxxxxxxxxxx Reported-by: Daniel Black <daniel@xxxxxxxxxxx> Fixes: e941894eae31 ("io-wq: make buffered file write hashed work map per-ctx") Signed-off-by: Jens Axboe <axboe@xxxxxxxxx> Signed-off-by: Greg Kroah-Hartman <gregkh@xxxxxxxxxxxxxxxxxxx> --- fs/io-wq.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) --- a/fs/io-wq.c +++ b/fs/io-wq.c @@ -401,9 +401,10 @@ static inline unsigned int io_get_work_h return work->flags >> IO_WQ_HASH_SHIFT; } -static void io_wait_on_hash(struct io_wqe *wqe, unsigned int hash) +static bool io_wait_on_hash(struct io_wqe *wqe, unsigned int hash) { struct io_wq *wq = wqe->wq; + bool ret = false; spin_lock_irq(&wq->hash->wait.lock); if (list_empty(&wqe->wait.entry)) { @@ -411,9 +412,11 @@ static void io_wait_on_hash(struct io_wq if (!test_bit(hash, &wq->hash->map)) { __set_current_state(TASK_RUNNING); list_del_init(&wqe->wait.entry); + ret = true; } } spin_unlock_irq(&wq->hash->wait.lock); + return ret; } /* @@ -474,14 +477,21 @@ static struct io_wq_work *io_get_next_wo } if (stall_hash != -1U) { + bool unstalled; + /* * Set this before dropping the lock to avoid racing with new * work being added and clearing the stalled bit. */ wqe->flags |= IO_WQE_FLAG_STALLED; raw_spin_unlock(&wqe->lock); - io_wait_on_hash(wqe, stall_hash); + unstalled = io_wait_on_hash(wqe, stall_hash); raw_spin_lock(&wqe->lock); + if (unstalled) { + wqe->flags &= ~IO_WQE_FLAG_STALLED; + if (wq_has_sleeper(&wqe->wq->hash->wait)) + wake_up(&wqe->wq->hash->wait); + } } return NULL; @@ -562,11 +572,14 @@ get_next: io_wqe_enqueue(wqe, linked); if (hash != -1U && !next_hashed) { + /* serialize hash clear with wake_up() */ + spin_lock_irq(&wq->hash->wait.lock); clear_bit(hash, &wq->hash->map); + wqe->flags &= ~IO_WQE_FLAG_STALLED; + spin_unlock_irq(&wq->hash->wait.lock); if (wq_has_sleeper(&wq->hash->wait)) wake_up(&wq->hash->wait); raw_spin_lock_irq(&wqe->lock); - wqe->flags &= ~IO_WQE_FLAG_STALLED; /* skip unnecessary unlock-lock wqe->lock */ if (!work) goto get_next; Patches currently in stable-queue which might be from axboe@xxxxxxxxx are queue-5.14/floppy-fix-add_disk-assumption-on-exit-due-to-new-de.patch queue-5.14/block-ataflop-more-blk-mq-refactoring-fixes.patch queue-5.14/nbd-fix-use-after-free-in-pid_show.patch queue-5.14/floppy-use-blk_cleanup_disk.patch queue-5.14/block-ataflop-use-the-blk_cleanup_disk-helper.patch queue-5.14/md-update-superblock-after-changing-rdev-flags-in-st.patch queue-5.14/alsa-hda-reduce-udelay-at-skl-position-reporting.patch queue-5.14/block-ataflop-add-registration-bool-before-calling-d.patch queue-5.14/md-raid1-only-allocate-write-behind-bio-for-writemostly-device.patch queue-5.14/ataflop-remove-ataflop_probe_lock-mutex.patch queue-5.14/block-remove-inaccurate-requeue-check.patch queue-5.14/block-ataflop-provide-a-helper-for-cleanup-up-an-ata.patch queue-5.14/floppy-fix-calling-platform_device_unregister-on-inv.patch queue-5.14/io-wq-ensure-that-hash-wait-lock-is-irq-disabling.patch queue-5.14/block-schedule-queue-restart-after-blk_sts_zone_reso.patch queue-5.14/io-wq-fix-queue-stalling-race.patch queue-5.14/block-ataflop-fix-breakage-introduced-at-blk-mq-refa.patch queue-5.14/io-wq-serialize-hash-clear-with-wakeup.patch queue-5.14/nvdimm-btt-do-not-call-del_gendisk-if-not-needed.patch queue-5.14/blk-cgroup-synchronize-blkg-creation-against-policy-.patch queue-5.14/nbd-fix-possible-overflow-for-first_minor-in-nbd_dev.patch queue-5.14/nbd-fix-max-value-for-first_minor.patch queue-5.14/block-bump-max-plugged-deferred-size-from-16-to-32.patch