On 4/7/2021 4:45 PM, Michael S. Tsirkin wrote:
On Wed, Apr 07, 2021 at 12:09:23PM +0000, Max Gurtovoy wrote:According to the spec after writing 0 to device_status, the driver MUST wait for a read of device_status to return 0 before reinitializing the device. In case we have a device that won't return 0, the reset operation will loop forever and cause the host/vm to stuck. Set timeout for 3 minutes before giving up on the device. Signed-off-by: Max Gurtovoy <mgurtovoy@xxxxxxxxxx> --- drivers/virtio/virtio_pci_modern.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c index cc3412a96a17..dcee616e8d21 100644 --- a/drivers/virtio/virtio_pci_modern.c +++ b/drivers/virtio/virtio_pci_modern.c @@ -162,6 +162,7 @@ static int vp_reset(struct virtio_device *vdev) { struct virtio_pci_device *vp_dev = to_vp_device(vdev); struct virtio_pci_modern_device *mdev = &vp_dev->mdev; + unsigned long timeout = jiffies + msecs_to_jiffies(180000);/* 0 status means a reset. */vp_modern_set_status(mdev, 0); @@ -169,9 +170,16 @@ static int vp_reset(struct virtio_device *vdev) * device_status to return 0 before reinitializing the device. * This will flush out the status write, and flush in device writes, * including MSI-X interrupts, if any. + * Set a timeout before giving up on the device. */ - while (vp_modern_get_status(mdev)) + while (vp_modern_get_status(mdev)) { + if (time_after(jiffies, timeout)) { + dev_err(&vdev->dev, "virtio: device not ready. " + "Aborting. Try again later\n"); + return -EAGAIN; + } msleep(1); + } /* Flush pending VQ/configuration callbacks. */ vp_synchronize_vectors(vdev); return 0;Problem is everyone just ignores the return code from reset. Timing out like that has a chance to cause a lot of trouble if the device remains active - we need to make reset robust.
But in commit 1/3 I added a code that doesn't ignore the reset return code.
What exactly is going on with the device that get status never returns 0? E.g. maybe it's in a state where it's returning all 1's because it's wedged permanently - using that would be better...
In HW devices you might have situations that the controller is in bad state (maybe bad FW) but still can be seen under the PCI bus.
As long as the device is not returning 0, this is legal. But in today's code, it will cause the kernel to be in endless while loop because of one bad device (that might recover later).
If we have 10 devices, and the first will stuck, all the others will wait forever to be probed.
By Virtio spec, setting FAILED is allowed in case "..driver didn’t like the device for some reason, or even a fatal error during device operation."
For example, in the NVMe spec there is TO (timeout) register that "is the worst case time that host software shall wait for CSTS.RDY to transition from: ..." and the driver wait for this time until it understands that the device is not ready to operate.
I tried to add similar logic to virtio.
-- 2.25.4
