Wed, Jun 26, 2024 at 01:51:00PM CEST, jiri@xxxxxxxxxxx wrote:
>Wed, Jun 26, 2024 at 11:58:08AM CEST, mst@xxxxxxxxxx wrote:
>>On Wed, Jun 26, 2024 at 10:43:11AM +0200, Jiri Pirko wrote:
>>> Wed, Jun 26, 2024 at 10:08:14AM CEST, mst@xxxxxxxxxx wrote:
>>> >On Wed, Jun 26, 2024 at 09:52:58AM +0200, Jiri Pirko wrote:
>>> >> Thu, Jun 20, 2024 at 12:31:34PM CEST, hengqi@xxxxxxxxxxxxxxxxx wrote:
>>> >> >On Thu, 20 Jun 2024 06:11:40 -0400, "Michael S. Tsirkin" <mst@xxxxxxxxxx> wrote:
>>> >> >> On Thu, Jun 20, 2024 at 06:10:51AM -0400, Michael S. Tsirkin wrote:
>>> >> >> > On Thu, Jun 20, 2024 at 05:53:15PM +0800, Heng Qi wrote:
>>> >> >> > > On Thu, 20 Jun 2024 16:26:05 +0800, Jason Wang <jasowang@xxxxxxxxxx> wrote:
>>> >> >> > > > On Thu, Jun 20, 2024 at 4:21 PM Jason Wang <jasowang@xxxxxxxxxx> wrote:
>>> >> >> > > > >
>>> >> >> > > > > On Thu, Jun 20, 2024 at 3:35 PM Heng Qi <hengqi@xxxxxxxxxxxxxxxxx> wrote:
>>> >> >> > > > > >
>>> >> >> > > > > > On Wed, 19 Jun 2024 17:19:12 -0400, "Michael S. Tsirkin" <mst@xxxxxxxxxx> wrote:
>>> >> >> > > > > > > On Thu, Jun 20, 2024 at 12:19:05AM +0800, Heng Qi wrote:
>>> >> >> > > > > > > > @@ -5312,7 +5315,7 @@ static int virtnet_find_vqs(struct virtnet_info *vi)
>>> >> >> > > > > > > >
>>> >> >> > > > > > > > /* Parameters for control virtqueue, if any */
>>> >> >> > > > > > > > if (vi->has_cvq) {
>>> >> >> > > > > > > > - callbacks[total_vqs - 1] = NULL;
>>> >> >> > > > > > > > + callbacks[total_vqs - 1] = virtnet_cvq_done;
>>> >> >> > > > > > > > names[total_vqs - 1] = "control";
>>> >> >> > > > > > > > }
>>> >> >> > > > > > > >
>>> >> >> > > > > > >
>>> >> >> > > > > > > If the # of MSIX vectors is exactly for data path VQs,
>>> >> >> > > > > > > this will cause irq sharing between VQs which will degrade
>>> >> >> > > > > > > performance significantly.
>>> >> >> > > > > > >
>>> >> >> > > > >
>>> >> >> > > > > Why do we need to care about buggy management? I think libvirt has
>>> >> >> > > > > been teached to use 2N+2 since the introduction of the multiqueue[1].
>>> >> >> > > >
>>> >> >> > > > And Qemu can calculate it correctly automatically since:
>>> >> >> > > >
>>> >> >> > > > commit 51a81a2118df0c70988f00d61647da9e298483a4
>>> >> >> > > > Author: Jason Wang <jasowang@xxxxxxxxxx>
>>> >> >> > > > Date: Mon Mar 8 12:49:19 2021 +0800
>>> >> >> > > >
>>> >> >> > > > virtio-net: calculating proper msix vectors on init
>>> >> >> > > >
>>> >> >> > > > Currently, the default msix vectors for virtio-net-pci is 3 which is
>>> >> >> > > > obvious not suitable for multiqueue guest, so we depends on the user
>>> >> >> > > > or management tools to pass a correct vectors parameter. In fact, we
>>> >> >> > > > can simplifying this by calculating the number of vectors on realize.
>>> >> >> > > >
>>> >> >> > > > Consider we have N queues, the number of vectors needed is 2*N + 2
>>> >> >> > > > (#queue pairs + plus one config interrupt and control vq). We didn't
>>> >> >> > > > check whether or not host support control vq because it was added
>>> >> >> > > > unconditionally by qemu to avoid breaking legacy guests such as Minix.
>>> >> >> > > >
>>> >> >> > > > Reviewed-by: Philippe Mathieu-Daudé <philmd@xxxxxxxxxx
>>> >> >> > > > Reviewed-by: Stefano Garzarella <sgarzare@xxxxxxxxxx>
>>> >> >> > > > Reviewed-by: Stefan Hajnoczi <stefanha@xxxxxxxxxx>
>>> >> >> > > > Signed-off-by: Jason Wang <jasowang@xxxxxxxxxx>
>>> >> >> > >
>>> >> >> > > Yes, devices designed according to the spec need to reserve an interrupt
>>> >> >> > > vector for ctrlq. So, Michael, do we want to be compatible with buggy devices?
>>> >> >> > >
>>> >> >> > > Thanks.
>>> >> >> >
>>> >> >> > These aren't buggy, the spec allows this. So don't fail, but
>>> >> >> > I'm fine with using polling if not enough vectors.
>>> >> >>
>>> >> >> sharing with config interrupt is easier code-wise though, FWIW -
>>> >> >> we don't need to maintain two code-paths.
>>> >> >
>>> >> >Yes, it works well - config change irq is used less before - and will not fail.
>>> >>
>>> >> Please note I'm working on such fallback for admin queue. I would Like
>>> >> to send the patchset by the end of this week. You can then use it easily
>>> >> for cvq.
>>> >>
>>> >> Something like:
>>> >> /* the config->find_vqs() implementation */
>>> >> int vp_find_vqs(struct virtio_device *vdev, unsigned int nvqs,
>>> >> struct virtqueue *vqs[], vq_callback_t *callbacks[],
>>> >> const char * const names[], const bool *ctx,
>>> >> struct irq_affinity *desc)
>>> >> {
>>> >> int err;
>>> >>
>>> >> /* Try MSI-X with one vector per queue. */
>>> >> err = vp_find_vqs_msix(vdev, nvqs, vqs, callbacks, names,
>>> >> VP_VQ_VECTOR_POLICY_EACH, ctx, desc);
>>> >> if (!err)
>>> >> return 0;
>>> >> /* Fallback: MSI-X with one shared vector for config and
>>> >> * slow path queues, one vector per queue for the rest. */
>>> >> err = vp_find_vqs_msix(vdev, nvqs, vqs, callbacks, names,
>>> >> VP_VQ_VECTOR_POLICY_SHARED_SLOW, ctx, desc);
>>> >> if (!err)
>>> >> return 0;
>>> >> /* Fallback: MSI-X with one vector for config, one shared for queues. */
>>> >> err = vp_find_vqs_msix(vdev, nvqs, vqs, callbacks, names,
>>> >> VP_VQ_VECTOR_POLICY_SHARED, ctx, desc);
>>> >> if (!err)
>>> >> return 0;
>>> >> /* Is there an interrupt? If not give up. */
>>> >> if (!(to_vp_device(vdev)->pci_dev->irq))
>>> >> return err;
>>> >> /* Finally fall back to regular interrupts. */
>>> >> return vp_find_vqs_intx(vdev, nvqs, vqs, callbacks, names, ctx);
>>> >> }
>>> >>
>>> >>
>>> >
>>> >
>>> >Well for cvq, we'll need to adjust the API so core
>>> >knows cvq interrupts are be shared with config not
>>> >datapath.
>>>
>>> Agreed. I was thinking about introducing some info struct and pass array
>>> of it instead of callbacks[] and names[]. Then the struct can contain
>>> flag indication. Something like:
>>>
>>> struct vq_info {
>>> vq_callback_t *callback;
>>> const char *name;
>>> bool slow_path;
>>> };
>>>
>>
>>Yes. Add ctx too? There were attempts at it already btw.
>
>Yep, ctx too. I can take a stab at it if noone else is interested.
Since this work is in v4, and I hope it will get merged soon, I plan to
send v2 of admin queue parallelization patchset after that. Here it is:
https://github.com/jpirko/linux_mlxsw/tree/wip_virtio_parallel_aq2
Heng, note the last patch:
virtio_pci: allow to indicate virtqueue being slow path
That is not part of my set, it is ment to be merged in your control
queue patchset. Then you can just indicate cvq to be slow like this:
/* Parameters for control virtqueue, if any */
if (vi->has_cvq) {
vqs_info[total_vqs - 1].callback = virtnet_cvq_done;
vqs_info[total_vqs - 1].name = "control";
vqs_info[total_vqs - 1].slow_path = true;
}
I just wanted to let you know this is in process so you may prepare.
Will keep you informed.
Thanks.
