Wed, Jun 26, 2024 at 11:58:08AM CEST, mst@xxxxxxxxxx wrote:
>On Wed, Jun 26, 2024 at 10:43:11AM +0200, Jiri Pirko wrote:
>> Wed, Jun 26, 2024 at 10:08:14AM CEST, mst@xxxxxxxxxx wrote:
>> >On Wed, Jun 26, 2024 at 09:52:58AM +0200, Jiri Pirko wrote:
>> >> Thu, Jun 20, 2024 at 12:31:34PM CEST, hengqi@xxxxxxxxxxxxxxxxx wrote:
>> >> >On Thu, 20 Jun 2024 06:11:40 -0400, "Michael S. Tsirkin" <mst@xxxxxxxxxx> wrote:
>> >> >> On Thu, Jun 20, 2024 at 06:10:51AM -0400, Michael S. Tsirkin wrote:
>> >> >> > On Thu, Jun 20, 2024 at 05:53:15PM +0800, Heng Qi wrote:
>> >> >> > > On Thu, 20 Jun 2024 16:26:05 +0800, Jason Wang <jasowang@xxxxxxxxxx> wrote:
>> >> >> > > > On Thu, Jun 20, 2024 at 4:21 PM Jason Wang <jasowang@xxxxxxxxxx> wrote:
>> >> >> > > > >
>> >> >> > > > > On Thu, Jun 20, 2024 at 3:35 PM Heng Qi <hengqi@xxxxxxxxxxxxxxxxx> wrote:
>> >> >> > > > > >
>> >> >> > > > > > On Wed, 19 Jun 2024 17:19:12 -0400, "Michael S. Tsirkin" <mst@xxxxxxxxxx> wrote:
>> >> >> > > > > > > On Thu, Jun 20, 2024 at 12:19:05AM +0800, Heng Qi wrote:
>> >> >> > > > > > > > @@ -5312,7 +5315,7 @@ static int virtnet_find_vqs(struct virtnet_info *vi)
>> >> >> > > > > > > >
>> >> >> > > > > > > > /* Parameters for control virtqueue, if any */
>> >> >> > > > > > > > if (vi->has_cvq) {
>> >> >> > > > > > > > - callbacks[total_vqs - 1] = NULL;
>> >> >> > > > > > > > + callbacks[total_vqs - 1] = virtnet_cvq_done;
>> >> >> > > > > > > > names[total_vqs - 1] = "control";
>> >> >> > > > > > > > }
>> >> >> > > > > > > >
>> >> >> > > > > > >
>> >> >> > > > > > > If the # of MSIX vectors is exactly for data path VQs,
>> >> >> > > > > > > this will cause irq sharing between VQs which will degrade
>> >> >> > > > > > > performance significantly.
>> >> >> > > > > > >
>> >> >> > > > >
>> >> >> > > > > Why do we need to care about buggy management? I think libvirt has
>> >> >> > > > > been teached to use 2N+2 since the introduction of the multiqueue[1].
>> >> >> > > >
>> >> >> > > > And Qemu can calculate it correctly automatically since:
>> >> >> > > >
>> >> >> > > > commit 51a81a2118df0c70988f00d61647da9e298483a4
>> >> >> > > > Author: Jason Wang <jasowang@xxxxxxxxxx>
>> >> >> > > > Date: Mon Mar 8 12:49:19 2021 +0800
>> >> >> > > >
>> >> >> > > > virtio-net: calculating proper msix vectors on init
>> >> >> > > >
>> >> >> > > > Currently, the default msix vectors for virtio-net-pci is 3 which is
>> >> >> > > > obvious not suitable for multiqueue guest, so we depends on the user
>> >> >> > > > or management tools to pass a correct vectors parameter. In fact, we
>> >> >> > > > can simplifying this by calculating the number of vectors on realize.
>> >> >> > > >
>> >> >> > > > Consider we have N queues, the number of vectors needed is 2*N + 2
>> >> >> > > > (#queue pairs + plus one config interrupt and control vq). We didn't
>> >> >> > > > check whether or not host support control vq because it was added
>> >> >> > > > unconditionally by qemu to avoid breaking legacy guests such as Minix.
>> >> >> > > >
>> >> >> > > > Reviewed-by: Philippe Mathieu-Daudé <philmd@xxxxxxxxxx
>> >> >> > > > Reviewed-by: Stefano Garzarella <sgarzare@xxxxxxxxxx>
>> >> >> > > > Reviewed-by: Stefan Hajnoczi <stefanha@xxxxxxxxxx>
>> >> >> > > > Signed-off-by: Jason Wang <jasowang@xxxxxxxxxx>
>> >> >> > >
>> >> >> > > Yes, devices designed according to the spec need to reserve an interrupt
>> >> >> > > vector for ctrlq. So, Michael, do we want to be compatible with buggy devices?
>> >> >> > >
>> >> >> > > Thanks.
>> >> >> >
>> >> >> > These aren't buggy, the spec allows this. So don't fail, but
>> >> >> > I'm fine with using polling if not enough vectors.
>> >> >>
>> >> >> sharing with config interrupt is easier code-wise though, FWIW -
>> >> >> we don't need to maintain two code-paths.
>> >> >
>> >> >Yes, it works well - config change irq is used less before - and will not fail.
>> >>
>> >> Please note I'm working on such fallback for admin queue. I would Like
>> >> to send the patchset by the end of this week. You can then use it easily
>> >> for cvq.
>> >>
>> >> Something like:
>> >> /* the config->find_vqs() implementation */
>> >> int vp_find_vqs(struct virtio_device *vdev, unsigned int nvqs,
>> >> struct virtqueue *vqs[], vq_callback_t *callbacks[],
>> >> const char * const names[], const bool *ctx,
>> >> struct irq_affinity *desc)
>> >> {
>> >> int err;
>> >>
>> >> /* Try MSI-X with one vector per queue. */
>> >> err = vp_find_vqs_msix(vdev, nvqs, vqs, callbacks, names,
>> >> VP_VQ_VECTOR_POLICY_EACH, ctx, desc);
>> >> if (!err)
>> >> return 0;
>> >> /* Fallback: MSI-X with one shared vector for config and
>> >> * slow path queues, one vector per queue for the rest. */
>> >> err = vp_find_vqs_msix(vdev, nvqs, vqs, callbacks, names,
>> >> VP_VQ_VECTOR_POLICY_SHARED_SLOW, ctx, desc);
>> >> if (!err)
>> >> return 0;
>> >> /* Fallback: MSI-X with one vector for config, one shared for queues. */
>> >> err = vp_find_vqs_msix(vdev, nvqs, vqs, callbacks, names,
>> >> VP_VQ_VECTOR_POLICY_SHARED, ctx, desc);
>> >> if (!err)
>> >> return 0;
>> >> /* Is there an interrupt? If not give up. */
>> >> if (!(to_vp_device(vdev)->pci_dev->irq))
>> >> return err;
>> >> /* Finally fall back to regular interrupts. */
>> >> return vp_find_vqs_intx(vdev, nvqs, vqs, callbacks, names, ctx);
>> >> }
>> >>
>> >>
>> >
>> >
>> >Well for cvq, we'll need to adjust the API so core
>> >knows cvq interrupts are be shared with config not
>> >datapath.
>>
>> Agreed. I was thinking about introducing some info struct and pass array
>> of it instead of callbacks[] and names[]. Then the struct can contain
>> flag indication. Something like:
>>
>> struct vq_info {
>> vq_callback_t *callback;
>> const char *name;
>> bool slow_path;
>> };
>>
>
>Yes. Add ctx too? There were attempts at it already btw.
Yep, ctx too. I can take a stab at it if noone else is interested.
>
>
>> >
>> >
>> >
>> >>
>> >> >
>> >> >Thanks.
>> >> >
>> >> >>
>> >> >> > > >
>> >> >> > > > Thanks
>> >> >> > > >
>> >> >> > > > >
>> >> >> > > > > > > So no, you can not just do it unconditionally.
>> >> >> > > > > > >
>> >> >> > > > > > > The correct fix probably requires virtio core/API extensions.
>> >> >> > > > > >
>> >> >> > > > > > If the introduction of cvq irq causes interrupts to become shared, then
>> >> >> > > > > > ctrlq need to fall back to polling mode and keep the status quo.
>> >> >> > > > >
>> >> >> > > > > Having to path sounds a burden.
>> >> >> > > > >
>> >> >> > > > > >
>> >> >> > > > > > Thanks.
>> >> >> > > > > >
>> >> >> > > > >
>> >> >> > > > >
>> >> >> > > > > Thanks
>> >> >> > > > >
>> >> >> > > > > [1] https://www.linux-kvm.org/page/Multiqueue
>> >> >> > > > >
>> >> >> > > > > > >
>> >> >> > > > > > > --
>> >> >> > > > > > > MST
>> >> >> > > > > > >
>> >> >> > > > > >
>> >> >> > > >
>> >> >>
>> >> >
>> >
>
