On 04/06/2018 10:41 AM, Ming Lei wrote: > On Thu, Apr 05, 2018 at 07:39:56PM +0200, Christian Borntraeger wrote: >> >> >> On 04/05/2018 06:11 PM, Ming Lei wrote: >>>> >>>> Could you please apply the following patch and provide the dmesg boot log? >>> >>> And please post out the 'lscpu' log together from the test machine too. >> >> attached. >> >> As I said before this seems to go way with CONFIG_NR_CPUS=64 or smaller. >> We have 282 nr_cpu_ids here (max 141CPUs on that z13 with SMT2) but only 8 Cores >> == 16 threads. > > OK, thanks! > > The most weird thing is that hctx->next_cpu is computed as 512 since > nr_cpu_id is 282, and hctx->next_cpu should have pointed to one of > possible CPU. > > Looks like it is a s390 specific issue, since I can setup one queue > which has same mapping with yours: > > - nr_cpu_id is 282 > - CPU 0~15 is online > - 64 queues null_blk > - still run all hw queues in .complete handler > > But can't reproduce this issue at all. > > So please test the following patch, which may tell us why hctx->next_cpu > is computed wrong: I see things like [ 8.196907] wrong next_cpu 512, blk_mq_map_swqueue, first_and [ 8.196910] wrong next_cpu 512, blk_mq_map_swqueue, first_and [ 8.196912] wrong next_cpu 512, blk_mq_map_swqueue, first_and [ 8.196913] wrong next_cpu 512, blk_mq_map_swqueue, first_and [ 8.196914] wrong next_cpu 512, blk_mq_map_swqueue, first_and [ 8.196915] wrong next_cpu 512, blk_mq_map_swqueue, first_and [ 8.196916] wrong next_cpu 512, blk_mq_map_swqueue, first_and [ 8.196916] wrong next_cpu 512, blk_mq_map_swqueue, first_and [ 8.196917] wrong next_cpu 512, blk_mq_map_swqueue, first_and [ 8.196918] wrong next_cpu 512, blk_mq_map_swqueue, first_and which is exactly what happens if the find and and operation fails (returns size of bitmap). FWIW, I added a dump stack for the case when we run unbound before I tested your patch: Apr 06 10:47:41 s38lp39 kernel: CPU: 15 PID: 86 Comm: ksoftirqd/15 Not tainted 4.16.0-07249-g864f9fc031e4-dirty #2 Apr 06 10:47:41 s38lp39 kernel: Hardware name: IBM 2964 NC9 704 (LPAR) Apr 06 10:47:41 s38lp39 kernel: Call Trace: Apr 06 10:47:41 s38lp39 kernel: ([<0000000000113946>] show_stack+0x56/0x80) Apr 06 10:47:41 s38lp39 kernel: [<00000000009d8132>] dump_stack+0x82/0xb0 Apr 06 10:47:41 s38lp39 kernel: [<00000000006a05de>] blk_mq_hctx_next_cpu+0x12e/0x138 Apr 06 10:47:41 s38lp39 kernel: [<00000000006a084c>] __blk_mq_delay_run_hw_queue+0x94/0xd8 Apr 06 10:47:41 s38lp39 kernel: [<00000000006a097a>] blk_mq_run_hw_queue+0x82/0x180 Apr 06 10:47:41 s38lp39 kernel: [<00000000006a0ae0>] blk_mq_run_hw_queues+0x68/0x88 Apr 06 10:47:41 s38lp39 kernel: [<000000000069fc4e>] __blk_mq_complete_request+0x11e/0x1d8 Apr 06 10:47:41 s38lp39 kernel: [<000000000069fd94>] blk_mq_complete_request+0x8c/0xc8 Apr 06 10:47:41 s38lp39 kernel: [<0000000000824c50>] dasd_block_tasklet+0x158/0x490 Apr 06 10:47:41 s38lp39 kernel: [<000000000014a952>] tasklet_action_common.isra.5+0x7a/0x100 Apr 06 10:47:41 s38lp39 kernel: [<00000000009f8248>] __do_softirq+0x98/0x368 Apr 06 10:47:41 s38lp39 kernel: [<000000000014a322>] run_ksoftirqd+0x4a/0x68 Apr 06 10:47:41 s38lp39 kernel: [<000000000016dc20>] smpboot_thread_fn+0x108/0x1b0 Apr 06 10:47:41 s38lp39 kernel: [<0000000000168e70>] kthread+0x148/0x160 Apr 06 10:47:41 s38lp39 kernel: [<00000000009f727a>] kernel_thread_starter+0x6/0xc Apr 06 10:47:41 s38lp39 kernel: [<00000000009f7274>] kernel_thread_starter+0x0/0xc
