Hi Ming, I was running some performance test on latest 4.17-rc and figure out performance drop (approximate 15% drop) due to below patch set. https://marc.info/?l=linux-block&m=150802309522847&w=2 I observed drop on latest 4.16.6 stable and 4.17-rc kernel as well. Taking bisect approach, figure out that Issue is not observed using last stable kernel 4.14.38. I pick 4.14.38 stable kernel as base line and applied above patch to confirm the behavior. lscpu output - Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 72 On-line CPU(s) list: 0-71 Thread(s) per core: 2 Core(s) per socket: 18 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz Stepping: 4 CPU MHz: 1457.182 CPU max MHz: 2701.0000 CPU min MHz: 1200.0000 BogoMIPS: 5400.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 25344K NUMA node0 CPU(s): 0-17,36-53 NUMA node1 CPU(s): 18-35,54-71 I am having 16 SSDs - "SDLL1DLR400GCCA1". Created two R0 VD (each VD consist of 8 SSDs) using MegaRaid Ventura series adapter. fio script - numactl -N 1 fio 2vd.fio --bs=4k --iodepth=128 -rw=randread --group_report --ioscheduler=none --numjobs=4 | v4.14.38-stable | patched v4.14.38-stable | mq-none | mq-none --------------------------------------------------------------------- randread "iops" | 1597k | 1377k Below is perf tool report without patch set. ( Looks like lock contention is causing this drop, so provided relevant snippet) - 3.19% 2.89% fio [kernel.vmlinux] [k] _raw_spin_lock - 2.43% io_submit - 2.30% entry_SYSCALL_64 - do_syscall_64 - 2.18% do_io_submit - 1.59% blk_finish_plug - 1.59% blk_flush_plug_list - 1.59% blk_mq_flush_plug_list - 1.00% __blk_mq_delay_run_hw_queue - 0.99% blk_mq_sched_dispatch_requests - 0.63% blk_mq_dispatch_rq_list 0.60% scsi_queue_rq - 0.57% blk_mq_sched_insert_requests - 0.56% blk_mq_insert_requests 0.51% _raw_spin_lock Below is perf tool report after applying patch set. - 4.10% 3.51% fio [kernel.vmlinux] [k] _raw_spin_lock - 3.09% io_submit - 2.97% entry_SYSCALL_64 - do_syscall_64 - 2.85% do_io_submit - 2.35% blk_finish_plug - 2.35% blk_flush_plug_list - 2.35% blk_mq_flush_plug_list - 1.83% __blk_mq_delay_run_hw_queue - 1.83% __blk_mq_run_hw_queue - 1.83% blk_mq_sched_dispatch_requests - 1.82% blk_mq_do_dispatch_ctx - 1.14% blk_mq_dequeue_from_ctx - 1.11% dispatch_rq_from_ctx 1.03% _raw_spin_lock 0.50% blk_mq_sched_insert_requests Let me know if you want more data or is this something a known implication of patch-set ? Thanks, Kashyap
