This is a note to let you know that I've just added the patch titled task_work: Use TIF_NOTIFY_SIGNAL if available to the 5.10-stable tree which can be found at: http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary The filename of the patch is: task_work-use-tif_notify_signal-if-available.patch and it can be found in the queue-5.10 subdirectory. If you, or anyone else, feels it should not be added to the stable tree, please let <stable@xxxxxxxxxxxxxxx> know about it. >From fd7614eb55ed1cd0017baea8a7621586d44089be Mon Sep 17 00:00:00 2001 From: Jens Axboe <axboe@xxxxxxxxx> Date: Mon, 26 Oct 2020 14:32:30 -0600 Subject: task_work: Use TIF_NOTIFY_SIGNAL if available From: Jens Axboe <axboe@xxxxxxxxx> [ Upstream commit 114518eb6430b832d2f9f5a008043b913ccf0e24 ] If the arch supports TIF_NOTIFY_SIGNAL, then use that for TWA_SIGNAL as it's more efficient than using the signal delivery method. This is especially true on threaded applications, where ->sighand is shared across threads, but it's also lighter weight on non-shared cases. io_uring is a heavy consumer of TWA_SIGNAL based task_work. A test with threads shows a nice improvement running an io_uring based echo server. stock kernel: 0.01% <= 0.1 milliseconds 95.86% <= 0.2 milliseconds 98.27% <= 0.3 milliseconds 99.71% <= 0.4 milliseconds 100.00% <= 0.5 milliseconds 100.00% <= 0.6 milliseconds 100.00% <= 0.7 milliseconds 100.00% <= 0.8 milliseconds 100.00% <= 0.9 milliseconds 100.00% <= 1.0 milliseconds 100.00% <= 1.1 milliseconds 100.00% <= 2 milliseconds 100.00% <= 3 milliseconds 100.00% <= 3 milliseconds 1378930.00 requests per second ~1600% CPU 1.38M requests/second, and all 16 CPUs are maxed out. patched kernel: 0.01% <= 0.1 milliseconds 98.24% <= 0.2 milliseconds 99.47% <= 0.3 milliseconds 99.99% <= 0.4 milliseconds 100.00% <= 0.5 milliseconds 100.00% <= 0.6 milliseconds 100.00% <= 0.7 milliseconds 100.00% <= 0.8 milliseconds 100.00% <= 0.9 milliseconds 100.00% <= 1.2 milliseconds 1666111.38 requests per second ~1450% CPU 1.67M requests/second, and we're no longer just hammering on the sighand lock. The original reporter states: "For 5.7.15 my benchmark achieves 1.6M qps and system cpu is at ~80%. for 5.7.16 or later it achieves only 1M qps and the system cpu is is at ~100%" with the only difference there being that TWA_SIGNAL is used unconditionally in 5.7.16, since it's required to be able to handle the inability to run task_work if the application is waiting in the kernel already on an event that needs task_work run to be satisfied. Also see commit 0ba9c9edcd15. Reported-by: Roman Gershman <romger@xxxxxxxxxx> Signed-off-by: Jens Axboe <axboe@xxxxxxxxx> Signed-off-by: Thomas Gleixner <tglx@xxxxxxxxxxxxx> Reviewed-by: Oleg Nesterov <oleg@xxxxxxxxxx> Link: https://lore.kernel.org/r/20201026203230.386348-5-axboe@xxxxxxxxx Signed-off-by: Greg Kroah-Hartman <gregkh@xxxxxxxxxxxxxxxxxxx> --- kernel/task_work.c | 41 +++++++++++++++++++++++++++++------------ 1 file changed, 29 insertions(+), 12 deletions(-) --- a/kernel/task_work.c +++ b/kernel/task_work.c @@ -5,6 +5,34 @@ static struct callback_head work_exited; /* all we need is ->next == NULL */ +/* + * TWA_SIGNAL signaling - use TIF_NOTIFY_SIGNAL, if available, as it's faster + * than TIF_SIGPENDING as there's no dependency on ->sighand. The latter is + * shared for threads, and can cause contention on sighand->lock. Even for + * the non-threaded case TIF_NOTIFY_SIGNAL is more efficient, as no locking + * or IRQ disabling is involved for notification (or running) purposes. + */ +static void task_work_notify_signal(struct task_struct *task) +{ +#if defined(TIF_NOTIFY_SIGNAL) + set_notify_signal(task); +#else + unsigned long flags; + + /* + * Only grab the sighand lock if we don't already have some + * task_work pending. This pairs with the smp_store_mb() + * in get_signal(), see comment there. + */ + if (!(READ_ONCE(task->jobctl) & JOBCTL_TASK_WORK) && + lock_task_sighand(task, &flags)) { + task->jobctl |= JOBCTL_TASK_WORK; + signal_wake_up(task, 0); + unlock_task_sighand(task, &flags); + } +#endif +} + /** * task_work_add - ask the @task to execute @work->func() * @task: the task which should run the callback @@ -33,7 +61,6 @@ int task_work_add(struct task_struct *ta enum task_work_notify_mode notify) { struct callback_head *head; - unsigned long flags; do { head = READ_ONCE(task->task_works); @@ -49,17 +76,7 @@ int task_work_add(struct task_struct *ta set_notify_resume(task); break; case TWA_SIGNAL: - /* - * Only grab the sighand lock if we don't already have some - * task_work pending. This pairs with the smp_store_mb() - * in get_signal(), see comment there. - */ - if (!(READ_ONCE(task->jobctl) & JOBCTL_TASK_WORK) && - lock_task_sighand(task, &flags)) { - task->jobctl |= JOBCTL_TASK_WORK; - signal_wake_up(task, 0); - unlock_task_sighand(task, &flags); - } + task_work_notify_signal(task); break; default: WARN_ON_ONCE(1); Patches currently in stable-queue which might be from axboe@xxxxxxxxx are queue-5.10/x86-process-setup-io_threads-more-like-normal-user-space-threads.patch queue-5.10/powerpc-add-support-for-tif_notify_signal.patch queue-5.10/eventfd-provide-a-eventfd_signal_mask-helper.patch queue-5.10/fs-provide-locked-helper-variant-of-close_fd_get_file.patch queue-5.10/relay-fix-type-mismatch-when-allocating-memory-in-re.patch queue-5.10/eventfd-change-int-to-__u64-in-eventfd_signal-ifndef.patch queue-5.10/io_uring-pass-in-epoll_uring_wake-for-eventfd-signaling-and-wakeups.patch queue-5.10/blk-mq-fix-possible-memleak-when-register-hctx-faile.patch queue-5.10/fix-handling-of-nd-depth-on-lookup_cached-failures-in-try_to_unlazy.patch queue-5.10/net-provide-__sys_shutdown_sock-that-takes-a-socket.patch queue-5.10/task_work-unconditionally-run-task_work-from-get_signal.patch queue-5.10/openrisc-add-support-for-tif_notify_signal.patch queue-5.10/signal-add-task_sigpending-helper.patch queue-5.10/net-remove-cmsg-restriction-from-io_uring-based-send-recvmsg-calls.patch queue-5.10/alpha-add-support-for-tif_notify_signal.patch queue-5.10/nios32-add-support-for-tif_notify_signal.patch queue-5.10/ia64-don-t-call-handle_signal-unless-there-s-actually-a-signal-queued.patch queue-5.10/task_work-remove-legacy-twa_signal-path.patch queue-5.10/revert-proc-don-t-allow-async-path-resolution-of-proc-self-components.patch queue-5.10/m68k-add-support-for-tif_notify_signal.patch queue-5.10/s390-add-support-for-tif_notify_signal.patch queue-5.10/um-add-support-for-tif_notify_signal.patch queue-5.10/tools-headers-uapi-sync-openat2.h-with-the-kernel-sources.patch queue-5.10/kernel-provide-create_io_thread-helper.patch queue-5.10/iov_iter-add-helper-to-save-iov_iter-state.patch queue-5.10/arc-unbork-5.11-bootup-fix-snafu-in-_tif_notify_signal-handling.patch queue-5.10/arch-ensure-parisc-powerpc-handle-pf_io_worker-in-copy_thread.patch queue-5.10/csky-add-support-for-tif_notify_signal.patch queue-5.10/arm-add-support-for-tif_notify_signal.patch queue-5.10/kernel-stop-masking-signals-in-create_io_thread.patch queue-5.10/fs-expose-lookup_cached-through-openat2-resolve_cached.patch queue-5.10/task_work-add-helper-for-more-targeted-task_work-canceling.patch queue-5.10/nds32-add-support-for-tif_notify_signal.patch queue-5.10/signal-kill-jobctl_task_work.patch queue-5.10/hexagon-add-support-for-tif_notify_signal.patch queue-5.10/sh-add-support-for-tif_notify_signal.patch queue-5.10/riscv-add-support-for-tif_notify_signal.patch queue-5.10/h8300-add-support-for-tif_notify_signal.patch queue-5.10/io_uring-import-5.15-stable-io_uring.patch queue-5.10/sparc-add-support-for-tif_notify_signal.patch queue-5.10/blktrace-fix-output-non-blktrace-event-when-blk_clas.patch queue-5.10/eventpoll-add-epoll_uring_wake-poll-wakeup-flag.patch queue-5.10/parisc-add-support-for-tif_notify_signal.patch queue-5.10/entry-add-support-for-tif_notify_signal.patch queue-5.10/x86-wire-up-tif_notify_signal.patch queue-5.10/task_work-use-tif_notify_signal-if-available.patch queue-5.10/drbd-fix-an-invalid-memory-access-caused-by-incorrec.patch queue-5.10/kernel-don-t-call-do_exit-for-pf_io_worker-threads.patch queue-5.10/kernel-allow-fork-with-tif_notify_signal-pending.patch queue-5.10/pata_ipx4xx_cf-fix-unsigned-comparison-with-less-tha.patch queue-5.10/mips-add-support-for-tif_notify_signal.patch queue-5.10/xtensa-add-support-for-tif_notify_signal.patch queue-5.10/c6x-add-support-for-tif_notify_signal.patch queue-5.10/microblaze-add-support-for-tif_notify_signal.patch queue-5.10/net-add-accept-helper-not-installing-fd.patch queue-5.10/ia64-add-support-for-tif_notify_signal.patch queue-5.10/arm64-add-support-for-tif_notify_signal.patch queue-5.10/arc-add-support-for-tif_notify_signal.patch queue-5.10/revert-proc-don-t-allow-async-path-resolution-of-proc-thread-self-components.patch queue-5.10/fs-make-do_renameat2-take-struct-filename.patch queue-5.10/kernel-remove-checking-for-tif_notify_signal.patch queue-5.10/arch-setup-pf_io_worker-threads-like-pf_kthread.patch queue-5.10/nvme-pci-fix-mempool-alloc-size.patch queue-5.10/fs-add-support-for-lookup_cached.patch