On 04/06/2018 10:51 AM, Christian Borntraeger wrote:
>
>
> On 04/06/2018 10:41 AM, Ming Lei wrote:
>> On Thu, Apr 05, 2018 at 07:39:56PM +0200, Christian Borntraeger wrote:
>>>
>>>
>>> On 04/05/2018 06:11 PM, Ming Lei wrote:
>>>>>
>>>>> Could you please apply the following patch and provide the dmesg boot log?
>>>>
>>>> And please post out the 'lscpu' log together from the test machine too.
>>>
>>> attached.
>>>
>>> As I said before this seems to go way with CONFIG_NR_CPUS=64 or smaller.
>>> We have 282 nr_cpu_ids here (max 141CPUs on that z13 with SMT2) but only 8 Cores
>>> == 16 threads.
>>
>> OK, thanks!
>>
>> The most weird thing is that hctx->next_cpu is computed as 512 since
>> nr_cpu_id is 282, and hctx->next_cpu should have pointed to one of
>> possible CPU.
>>
>> Looks like it is a s390 specific issue, since I can setup one queue
>> which has same mapping with yours:
>>
>> - nr_cpu_id is 282
>> - CPU 0~15 is online
>> - 64 queues null_blk
>> - still run all hw queues in .complete handler
>>
>> But can't reproduce this issue at all.
>>
>> So please test the following patch, which may tell us why hctx->next_cpu
>> is computed wrong:
>
> I see things like
>
> [ 8.196907] wrong next_cpu 512, blk_mq_map_swqueue, first_and
> [ 8.196910] wrong next_cpu 512, blk_mq_map_swqueue, first_and
> [ 8.196912] wrong next_cpu 512, blk_mq_map_swqueue, first_and
> [ 8.196913] wrong next_cpu 512, blk_mq_map_swqueue, first_and
> [ 8.196914] wrong next_cpu 512, blk_mq_map_swqueue, first_and
> [ 8.196915] wrong next_cpu 512, blk_mq_map_swqueue, first_and
> [ 8.196916] wrong next_cpu 512, blk_mq_map_swqueue, first_and
> [ 8.196916] wrong next_cpu 512, blk_mq_map_swqueue, first_and
> [ 8.196917] wrong next_cpu 512, blk_mq_map_swqueue, first_and
> [ 8.196918] wrong next_cpu 512, blk_mq_map_swqueue, first_and
There are more
# dmesg | grep "wrong next" | cut -d "]" -f 2- | uniq -c
10 wrong next_cpu 512, blk_mq_map_swqueue, first_and
72 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and
1 wrong next_cpu 512, blk_mq_hctx_next_cpu, first_and
1 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and
1 wrong next_cpu 512, blk_mq_hctx_next_cpu, first_and
1 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and
1 wrong next_cpu 512, blk_mq_hctx_next_cpu, first_and
1 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and
1 wrong next_cpu 512, blk_mq_hctx_next_cpu, first_and
1 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and
1 wrong next_cpu 512, blk_mq_hctx_next_cpu, first_and
1 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and
1 wrong next_cpu 512, blk_mq_hctx_next_cpu, first_and
7 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and
1 wrong next_cpu 512, blk_mq_hctx_next_cpu, first_and
1 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and
1 wrong next_cpu 512, blk_mq_hctx_next_cpu, first_and
10 wrong next_cpu 512, blk_mq_hctx_next_cpu, next_and
>
> which is exactly what happens if the find and and operation fails (returns size of bitmap).
>
> FWIW, I added a dump stack for the case when we run unbound before I tested your patch:
>
> Apr 06 10:47:41 s38lp39 kernel: CPU: 15 PID: 86 Comm: ksoftirqd/15 Not tainted 4.16.0-07249-g864f9fc031e4-dirty #2
> Apr 06 10:47:41 s38lp39 kernel: Hardware name: IBM 2964 NC9 704 (LPAR)
> Apr 06 10:47:41 s38lp39 kernel: Call Trace:
> Apr 06 10:47:41 s38lp39 kernel: ([<0000000000113946>] show_stack+0x56/0x80)
> Apr 06 10:47:41 s38lp39 kernel: [<00000000009d8132>] dump_stack+0x82/0xb0
> Apr 06 10:47:41 s38lp39 kernel: [<00000000006a05de>] blk_mq_hctx_next_cpu+0x12e/0x138
> Apr 06 10:47:41 s38lp39 kernel: [<00000000006a084c>] __blk_mq_delay_run_hw_queue+0x94/0xd8
> Apr 06 10:47:41 s38lp39 kernel: [<00000000006a097a>] blk_mq_run_hw_queue+0x82/0x180
> Apr 06 10:47:41 s38lp39 kernel: [<00000000006a0ae0>] blk_mq_run_hw_queues+0x68/0x88
> Apr 06 10:47:41 s38lp39 kernel: [<000000000069fc4e>] __blk_mq_complete_request+0x11e/0x1d8
> Apr 06 10:47:41 s38lp39 kernel: [<000000000069fd94>] blk_mq_complete_request+0x8c/0xc8
> Apr 06 10:47:41 s38lp39 kernel: [<0000000000824c50>] dasd_block_tasklet+0x158/0x490
> Apr 06 10:47:41 s38lp39 kernel: [<000000000014a952>] tasklet_action_common.isra.5+0x7a/0x100
> Apr 06 10:47:41 s38lp39 kernel: [<00000000009f8248>] __do_softirq+0x98/0x368
> Apr 06 10:47:41 s38lp39 kernel: [<000000000014a322>] run_ksoftirqd+0x4a/0x68
> Apr 06 10:47:41 s38lp39 kernel: [<000000000016dc20>] smpboot_thread_fn+0x108/0x1b0
> Apr 06 10:47:41 s38lp39 kernel: [<0000000000168e70>] kthread+0x148/0x160
> Apr 06 10:47:41 s38lp39 kernel: [<00000000009f727a>] kernel_thread_starter+0x6/0xc
> Apr 06 10:47:41 s38lp39 kernel: [<00000000009f7274>] kernel_thread_starter+0x0/0xc
>
