On Wed, 2019-03-27 at 09:38 -0400, Farhan Ali wrote:
> Hi Alex,
>
> I am seeing a warning in the vfio driver with the 5.0 linux kernel on an
> s390 system:
>
> [ 4050.264464] ------------[ cut here ]------------
> [ 4050.264508] do not call blocking ops when !TASK_RUNNING; state=1 set
> at [<00000000b33c00e2>] prepare_to_wait_event+0x14a/0x188
> [ 4050.264529] WARNING: CPU: 12 PID: 35924 at kernel/sched/core.c:6112
> __might_sleep+0x76/0x90
> ....
>
> [ 4050.264636] CPU: 12 PID: 35924 Comm: sh Tainted: G W
> 5.0.0-00016-g4ef69cfc84b1-dirty #12
> [ 4050.264646] Hardware name: IBM 2964 NE1 749 (LPAR)
> [ 4050.264654] Krnl PSW : 0704d00180000000 000000000017bbae
> (__might_sleep+0x76/0x90)
> [ 4050.264668] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1
> PM:0 RI:0 EA:3
> [ 4050.264678] Krnl GPRS: 00000000361d853f 0000000080000000
> 0000000000000072 00000000001d05e2
> [ 4050.264691] 0000000000000007 0000000000bb3c20
> 000003ff800d7abe 000003e013823b58
> [ 4050.264700] 0000000000000002 000003ff800d7abe
> 0000000000000000 000000000000038c
> [ 4050.264708] 0000000000dd96e4 0000000800000000
> 000000000017bbaa 000003e013823910
> [ 4050.264724] Krnl Code: 000000000017bb9e: c0200062dc5d larl
> %r2,dd7458
> 000000000017bba4: c0e5fffe5012 brasl
> %r14,145bc8
> #000000000017bbaa: a7f40001 brc
> 15,17bbac
> >000000000017bbae: b904004a lgr
> %r4,%r10
> 000000000017bbb2: b904003b lgr
> %r3,%r11
> 000000000017bbb6: b904002c lgr
> %r2,%r12
> 000000000017bbba: ebaff0a00004 lmg
> %r10,%r15,160(%r15)
> 000000000017bbc0: c0f4fffffe8c brcl
> 15,17b8d8
> [ 4050.264756] Call Trace:
> [ 4050.264765] ([<000000000017bbaa>] __might_sleep+0x72/0x90)
> [ 4050.264774] [<0000000000b97edc>] __mutex_lock+0x44/0x8c0
> [ 4050.264782] [<0000000000b9878a>] mutex_lock_nested+0x32/0x40
> [ 4050.264793] [<000003ff800d7abe>] vfio_group_get_device+0x36/0xa8 [vfio]
> [ 4050.264803] [<000003ff800d87c0>] vfio_del_group_dev+0x238/0x378 [vfio]
> [ 4050.264813] [<000003ff8015f67c>] mdev_remove+0x3c/0x68 [mdev]
> [ 4050.264825] [<00000000008e01b0>]
> device_release_driver_internal+0x168/0x268
> [ 4050.264834] [<00000000008de692>] bus_remove_device+0x162/0x190
> [ 4050.264843] [<00000000008daf42>] device_del+0x1e2/0x368
> [ 4050.264851] [<00000000008db12c>] device_unregister+0x64/0x88
> [ 4050.264862] [<000003ff8015ed84>] mdev_device_remove+0xec/0x130 [mdev]
> [ 4050.264872] [<000003ff8015f074>] remove_store+0x6c/0xa8 [mdev]
> [ 4050.264881] [<000000000046f494>] kernfs_fop_write+0x14c/0x1f8
> [ 4050.264890] [<00000000003c1530>] __vfs_write+0x38/0x1a8
> [ 4050.264899] [<00000000003c187c>] vfs_write+0xb4/0x198
> [ 4050.264908] [<00000000003c1af2>] ksys_write+0x5a/0xb0
> [ 4050.264916] [<0000000000b9e270>] system_call+0xdc/0x2d8
> [ 4050.264925] 4 locks held by sh/35924:
> [ 4050.264933] #0: 000000001ef90325 (sb_writers#4){.+.+}, at:
> vfs_write+0x9e/0x198
> [ 4050.264948] #1: 000000005c1ab0b3 (&of->mutex){+.+.}, at:
> kernfs_fop_write+0x1cc/0x1f8
> [ 4050.264963] #2: 0000000034831ab8 (kn->count#297){++++}, at:
> kernfs_remove_self+0x12e/0x150
> [ 4050.264979] #3: 00000000e152484f (&dev->mutex){....}, at:
> device_release_driver_internal+0x5c/0x268
> [ 4050.264993] Last Breaking-Event-Address:
> [ 4050.265002] [<000000000017bbaa>] __might_sleep+0x72/0x90
> [ 4050.265010] irq event stamp: 7039
> [ 4050.265020] hardirqs last enabled at (7047): [<00000000001cee7a>]
> console_unlock+0x6d2/0x740
> [ 4050.265029] hardirqs last disabled at (7054): [<00000000001ce87e>]
> console_unlock+0xd6/0x740
> [ 4050.265040] softirqs last enabled at (6416): [<0000000000b8fe26>]
> __udelay+0xb6/0x100
> [ 4050.265049] softirqs last disabled at (6415): [<0000000000b8fe06>]
> __udelay+0x96/0x100
> [ 4050.265057] ---[ end trace d04a07d39d99a9f9 ]---
>
>
> The problem seems to be the function vfio_dev_present which is the
> condition to wait_event_interruptible_timeout.
>
> wait_event_interruptible() sets the state of the current task to
> TASK_INTERRUPTIBLE, then checks the condition. And in vfio_dev_present
> we call vfio_group_get_device which then tries to acquire the mutex
> group->device_lock.
>
>
> I am not too familiar with vfio core driver code yet, and I am not sure
> if the solution to this problem will be something like described in
> https://lwn.net/Articles/628628/. If that is the solution we would want
> I could take a crack at writing a patch.
>
Note that I have seen that a lot while developing my driver too, I assumed that
this is a false positive since I haven't time to study this code more carefully.
Best regards,
Maxim Levitsky
