On Thu, 2021-09-09 at 16:38 -0300, Jason Gunthorpe wrote:
> This addresses Cornelia's remark on the earlier patch that ccw has a
> confusing lifecycle. While it doesn't seem like the original attempt
> was
> functionally wrong, the result can be made better with a lot of
> further
> work.
I thought I'd take a stab at seeing how this works with the hardware
before looking at the code much. git couldn't apply patches 1, 6, or 9
to 5.15-rc1, but I was able to hand-fit them into place. Shutting down
the guest and de-configuring a device ends up bringing my whole system
down. I haven't looked at this any further; hopefully something jumps
to mind for you:
[ 64.585347] vfio_ccw 0.0.08fe: MDEV: Unregistering
[ 64.585357] illegal operation: 0001 ilc:1 [#1] SMP
[ 64.585362] Modules linked in: vhost_vsock
vmw_vsock_virtio_transport_common vsock vhost
[ 64.585364] vfio_ccw_mdev b50bbd4b-eab8-4f8c-9f0c-3cf636f936b9:
Relaying device request to user (#0)
[ 64.585364] vhost_iotlb lcs ctcm fsm kvm xt_MASQUERADE xt_conntrack
ipt_REJECT nf_reject_ipv4 xt_tcpudp iptable_mangle iptable_nat nf_nat
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter bridge stp
llc dm_multipath dm_mod s390_trng eadm_sch zcrypt_cex4 qeth_l2 vfio_ccw
mdev vfio_iommu_type1 vfio configfs zram zsmalloc ip_tables x_tables
mlx5_core ghash_s390 prng aes_s390 des_s390 libdes sha3_512_s390
sha3_256_s390 sha512_s390 sha256_s390 sha1_s390 sha_common pkey zcrypt
rng_core autofs4
[ 64.585392] CPU: 14 PID: 4487 Comm: qemu-system-s39 Kdump: loaded
Not tainted 5.15.0-rc1 #1
[ 64.585395] Hardware name: IBM 3906 M05 780 (LPAR)
[ 64.585396] Krnl PSW : 0704c00180000000 0000000000000002 (0x2)
[ 64.585404] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0
PM:0 RI:0 EA:3
[ 64.585407] Krnl GPRS: 0000000000000001 0000000000000000
00000000005f4800 0000000000000004
[ 64.585410] 0000000000000000 0000000000000002
0000000000000000 000002aa3e65085e
[ 64.585412] 000000008de09100 0000000000003b6f
000003ff8017fa08 00000000005f4800
[ 64.585413] 0000000081450000 000003ff7c032310
000003ff80179db0 000003800bf53da0
[ 64.585418] Krnl Code:#0000000000000000:
0000 illegal
>0000000000000002: 0000 illegal
0000000000000004: 0000 illegal
0000000000000006: 0000 illegal
0000000000000008: 0000 illegal
000000000000000a: 0000 illegal
000000000000000c: 0000 illegal
000000000000000e: 0000 illegal
[ 64.585462] Call Trace:
[ 64.585464] [<0000000000000002>] 0x2
[ 64.585467] ([<000003ff80179d74>] vfio_ccw_mdev_ioctl+0x84/0x318
[vfio_ccw])
[ 64.585476] [<00000000bb7adda6>] __s390x_sys_ioctl+0xbe/0x100
[ 64.585481] [<00000000bbfbf5e4>] __do_syscall+0x1bc/0x1e8
[ 64.585488] [<00000000bbfcc8d8>] system_call+0x78/0xa0
Eric
>
> Reorganize the driver so that the mdev owns the private memory and
> controls the lifecycle, not the css_driver. The memory associated
> with the
> css_driver lifecycle is only the mdev_parent/mdev_type registration.
>
> Along the way we change when the sch is quiescent or not to be linked
> to
> the open/close_device lifetime of the vfio_device, which is sort of
> what
> it was tring to do already, just not completely.
>
> The troublesome racey lifecycle of the css_driver callbacks is made
> clear
> with simple vfio_device refcounting so a callback is only delivered
> into a
> registered vfio_device and has obvious correctness.
>
> Move the only per-css_driver state, the "available instance" counter,
> into
> the core code and share that logic with many of the other drivers.
> The
> value is kept in the mdev_type memory.
>
> v2:
> - Clean up the lifecycle in ccw with 7 new patches
> - Rebase
> v1:
> https://lore.kernel.org/all/7-v2-7667f42c9bad+935-vfio3_jgg@xxxxxxxxxx
>
> Jason Gunthorpe (9):
> vfio/ccw: Use functions for alloc/free of the vfio_ccw_private
> vfio/ccw: Pass vfio_ccw_private not mdev_device to various
> functions
> vfio/ccw: Convert to use vfio_register_group_dev()
> vfio/ccw: Make the FSM complete and synchronize it to the mdev
> vfio/mdev: Consolidate all the device_api sysfs into the core code
> vfio/mdev: Add mdev available instance checking to the core
> vfio/ccw: Remove private->mdev
> vfio: Export vfio_device_try_get()
> vfio/ccw: Move the lifecycle of the struct vfio_ccw_private to the
> mdev
>
> drivers/gpu/drm/i915/gvt/kvmgt.c | 9 +-
> drivers/s390/cio/vfio_ccw_drv.c | 282 +++++++++++-------------
> --
> drivers/s390/cio/vfio_ccw_fsm.c | 152 ++++++++++----
> drivers/s390/cio/vfio_ccw_ops.c | 240 ++++++++++------------
> drivers/s390/cio/vfio_ccw_private.h | 42 +++-
> drivers/s390/crypto/vfio_ap_ops.c | 41 +---
> drivers/s390/crypto/vfio_ap_private.h | 2 -
> drivers/vfio/mdev/mdev_core.c | 13 +-
> drivers/vfio/mdev/mdev_private.h | 2 +
> drivers/vfio/mdev/mdev_sysfs.c | 64 +++++-
> drivers/vfio/vfio.c | 3 +-
> include/linux/mdev.h | 13 +-
> include/linux/vfio.h | 1 +
> samples/vfio-mdev/mbochs.c | 9 +-
> samples/vfio-mdev/mdpy.c | 31 +--
> samples/vfio-mdev/mtty.c | 10 +-
> 16 files changed, 470 insertions(+), 444 deletions(-)
>
