On Thu, Mar 10, 2022 at 03:17:03PM +0800, Xuan Zhuo wrote:
> On Thu, 10 Mar 2022 02:00:39 -0500, "Michael S. Tsirkin" <mst@xxxxxxxxxx> wrote:
> > On Tue, Mar 08, 2022 at 08:35:01PM +0800, Xuan Zhuo wrote:
> > > virtio ring supports reset.
> > >
> > > Queue reset is divided into several stages.
> > >
> > > 1. notify device queue reset
> > > 2. vring release
> > > 3. attach new vring
> > > 4. notify device queue re-enable
> > >
> > > After the first step is completed, the vring reset operation can be
> > > performed. If the newly set vring num does not change, then just reset
> > > the vq related value.
> > >
> > > Otherwise, the vring will be released and the vring will be reallocated.
> > > And the vring will be attached to the vq. If this process fails, the
> > > function will exit, and the state of the vq will be the vring release
> > > state. You can call this function again to reallocate the vring.
> > >
> > > In addition, vring_align, may_reduce_num are necessary for reallocating
> > > vring, so they are retained when creating vq.
> > >
> > > Signed-off-by: Xuan Zhuo <xuanzhuo@xxxxxxxxxxxxxxxxx>
> > > ---
> > > drivers/virtio/virtio_ring.c | 69 ++++++++++++++++++++++++++++++++++++
> > > 1 file changed, 69 insertions(+)
> > >
> > > diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
> > > index e0422c04c903..148fb1fd3d5a 100644
> > > --- a/drivers/virtio/virtio_ring.c
> > > +++ b/drivers/virtio/virtio_ring.c
> > > @@ -158,6 +158,12 @@ struct vring_virtqueue {
> > > /* DMA address and size information */
> > > dma_addr_t queue_dma_addr;
> > > size_t queue_size_in_bytes;
> > > +
> > > + /* The parameters for creating vrings are reserved for
> > > + * creating new vrings when enabling reset queue.
> > > + */
> > > + u32 vring_align;
> > > + bool may_reduce_num;
> > > } split;
> > >
> > > /* Available for packed ring */
> > > @@ -217,6 +223,12 @@ struct vring_virtqueue {
> > > #endif
> > > };
> > >
> > > +static void vring_free(struct virtqueue *vq);
> > > +static void __vring_virtqueue_init_split(struct vring_virtqueue *vq,
> > > + struct virtio_device *vdev);
> > > +static int __vring_virtqueue_attach_split(struct vring_virtqueue *vq,
> > > + struct virtio_device *vdev,
> > > + struct vring vring);
> > >
> > > /*
> > > * Helpers.
> > > @@ -1012,6 +1024,8 @@ static struct virtqueue *vring_create_virtqueue_split(
> > > return NULL;
> > > }
> > >
> > > + to_vvq(vq)->split.vring_align = vring_align;
> > > + to_vvq(vq)->split.may_reduce_num = may_reduce_num;
> > > to_vvq(vq)->split.queue_dma_addr = vring.dma_addr;
> > > to_vvq(vq)->split.queue_size_in_bytes = vring.queue_size_in_bytes;
> > > to_vvq(vq)->we_own_ring = true;
> > > @@ -1019,6 +1033,59 @@ static struct virtqueue *vring_create_virtqueue_split(
> > > return vq;
> > > }
> > >
> > > +static int virtqueue_reset_vring_split(struct virtqueue *_vq, u32 num)
> > > +{
> > > + struct vring_virtqueue *vq = to_vvq(_vq);
> > > + struct virtio_device *vdev = _vq->vdev;
> > > + struct vring_split vring;
> > > + int err;
> > > +
> > > + if (num > _vq->num_max)
> > > + return -E2BIG;
> > > +
> > > + switch (vq->vq.reset) {
> > > + case VIRTIO_VQ_RESET_STEP_NONE:
> > > + return -ENOENT;
> > > +
> > > + case VIRTIO_VQ_RESET_STEP_VRING_ATTACH:
> > > + case VIRTIO_VQ_RESET_STEP_DEVICE:
> > > + if (vq->split.vring.num == num || !num)
> > > + break;
> > > +
> > > + vring_free(_vq);
> > > +
> > > + fallthrough;
> > > +
> > > + case VIRTIO_VQ_RESET_STEP_VRING_RELEASE:
> > > + if (!num)
> > > + num = vq->split.vring.num;
> > > +
> > > + err = vring_create_vring_split(&vring, vdev,
> > > + vq->split.vring_align,
> > > + vq->weak_barriers,
> > > + vq->split.may_reduce_num, num);
> > > + if (err)
> > > + return -ENOMEM;
> > > +
> > > + err = __vring_virtqueue_attach_split(vq, vdev, vring.vring);
> > > + if (err) {
> > > + vring_free_queue(vdev, vring.queue_size_in_bytes,
> > > + vring.queue,
> > > + vring.dma_addr);
> > > + return -ENOMEM;
> > > + }
> > > +
> > > + vq->split.queue_dma_addr = vring.dma_addr;
> > > + vq->split.queue_size_in_bytes = vring.queue_size_in_bytes;
> > > + }
> > > +
> > > + __vring_virtqueue_init_split(vq, vdev);
> > > + vq->we_own_ring = true;
> > > + vq->vq.reset = VIRTIO_VQ_RESET_STEP_VRING_ATTACH;
> > > +
> > > + return 0;
> > > +}
> > > +
> >
> > I kind of dislike this state machine.
> >
> > Hacks like special-casing num = 0 to mean "reset" are especially
> > confusing.
>
> I'm removing it. I'll say in the function description that this function is
> currently only called when vq has been reset. I'm no longer checking it based on
> state.
>
> >
> > And as Jason points out, when we want a resize then yes this currently
> > implies reset but that is an implementation detail.
> >
> > There should be a way to just make these cases separate functions
> > and then use them to compose consistent external APIs.
>
> Yes, virtqueue_resize_split() is fine for ethtool -G.
>
> But in the case of AF_XDP, just execute reset to free the buffer. The name
> virtqueue_reset_vring_split() I think can cover both cases. Or we use two apis
> to handle both scenarios?
>
> Or can anyone think of a better name. ^_^
>
> Thanks.
I'd say resize should be called resize and reset should be called reset.
The big issue is a sane API for resize. Ideally it would resubmit
buffers which did not get used. Question is what to do
about buffers which don't fit (if ring has been downsized)?
Maybe a callback that will handle them?
And then what? Queue them up and readd later? Drop?
If we drop we should drop from the head not the tail ...
> >
> > If we additionally want to track state for debugging then bool flags
> > seem more appropriate for this, though from experience that is
> > not always worth the extra code.
> >
> >
> >
> > > /*
> > > * Packed ring specific functions - *_packed().
> > > @@ -2317,6 +2384,8 @@ static int __vring_virtqueue_attach_split(struct vring_virtqueue *vq,
> > > static void __vring_virtqueue_init_split(struct vring_virtqueue *vq,
> > > struct virtio_device *vdev)
> > > {
> > > + vq->vq.reset = VIRTIO_VQ_RESET_STEP_NONE;
> > > +
> > > vq->packed_ring = false;
> > > vq->we_own_ring = false;
> > > vq->broken = false;
> > > --
> > > 2.31.0
> >
