ktask is a generic framework for parallelizing cpu-intensive work in the kernel. The intended use is for big machines that can use their cpu power to speed up large tasks that can't otherwise be multithreaded in userland. The API is generic enough to add concurrency to many different kinds of tasks--for example, zeroing a range of pages or evicting a list of inodes--and aims to save its clients the trouble of splitting up the work, choosing the number of threads to use, starting these threads, and load balancing the work between them. Why do we need ktask when the kernel has other APIs for managing concurrency? After all, kthread_workers and workqueues already provide ways to start threads, and the kernel can handle large tasks with a single thread by periodically yielding the cpu with cond_resched or doing the work in fixed size batches. Of the existing concurrency facilities, kthread_worker isn't suited for providing parallelism because each comes with only a single thread. Workqueues are a better fit for this, and in fact ktask is built on an unbound workqueue, but workqueues aren't designed for splitting up a large task. ktask instead uses unbound workqueue threads to run "chunks" of a task. More background is available in the documentation commit (first commit of the series). This patchset is based on 4.13-rc6 and contains three ktask users so far, with more to come: - clearing gigantic pages - fallocate for HugeTLB pages - deferred struct page initialization at boot time The core ktask code is based on work by Pavel Tatashin, Steve Sistare, and Jonathan Adams. v1 -> v2: - Added deferred struct page initialization use case. - Explained the source of the performance improvement from parallelizing clear_gigantic_page (comment from Dave Hansen). - Fixed Documentation and build warnings from CONFIG_KTASK=n kernels. link to v1: https://lkml.org/lkml/2017/7/14/666 Daniel Jordan (7): ktask: add documentation ktask: multithread cpu-intensive kernel work ktask: add /proc/sys/debug/ktask_max_threads mm: enlarge type of offset argument in mem_map_offset and mem_map_next mm: parallelize clear_gigantic_page hugetlbfs: parallelize hugetlbfs_fallocate with ktask mm: parallelize deferred struct page initialization within each node Documentation/core-api/index.rst | 1 + Documentation/core-api/ktask.rst | 104 ++++++++++ fs/hugetlbfs/inode.c | 117 +++++++++--- include/linux/ktask.h | 235 +++++++++++++++++++++++ include/linux/ktask_internal.h | 19 ++ include/linux/mm.h | 6 + init/Kconfig | 7 + init/main.c | 2 + kernel/Makefile | 2 +- kernel/ktask.c | 396 +++++++++++++++++++++++++++++++++++++++ kernel/sysctl.c | 10 + mm/internal.h | 7 +- mm/memory.c | 35 +++- mm/page_alloc.c | 174 ++++++++++------- 14 files changed, 1014 insertions(+), 101 deletions(-) create mode 100644 Documentation/core-api/ktask.rst create mode 100644 include/linux/ktask.h create mode 100644 include/linux/ktask_internal.h create mode 100644 kernel/ktask.c -- 2.12.2 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>
