> From: Jason Wang <jasowang@xxxxxxxxxx> > Sent: Wednesday, June 1, 2022 11:54 PM > > On Thu, Jun 2, 2022 at 10:59 AM Parav Pandit <parav@xxxxxxxxxx> wrote: > > > > > > > From: Jason Wang <jasowang@xxxxxxxxxx> > > > Sent: Wednesday, June 1, 2022 10:00 PM > > > > > > On Thu, Jun 2, 2022 at 2:58 AM Parav Pandit <parav@xxxxxxxxxx> wrote: > > > > > > > > > > > > > From: Jason Wang <jasowang@xxxxxxxxxx> > > > > > Sent: Tuesday, May 31, 2022 10:42 PM > > > > > > > > > > Well, the ability to query the virtqueue state was proposed as > > > > > another feature (Eugenio, please correct me). This should be > > > > > sufficient for making virtio-net to be live migrated. > > > > > > > > > The device is stopped, it won't answer to this special vq config done > here. > > > > > > This depends on the definition of the stop. Any query to the device > > > state should be allowed otherwise it's meaningless for us. > > > > > > > Programming all of these using cfg registers doesn't scale for > > > > on-chip > > > memory and for the speed. > > > > > > Well, they are orthogonal and what I want to say is, we should first > > > define the semantics of stop and state of the virtqueue. > > > > > > Such a facility could be accessed by either transport specific > > > method or admin virtqueue, it totally depends on the hardware > architecture of the vendor. > > > > > I find it hard to believe that a vendor can implement a CVQ but not AQ and > chose to expose tens of hundreds of registers. > > But maybe, it fits some specific hw. > > You can have a look at the ifcvf dpdk driver as an example. > Ifcvf is an example of using registers. It is not an answer why AQ is hard for it. :) virtio spec has definition of queue now and implementing yet another queue shouldn't be a problem. So far no one seem to have problem with the additional queue. So I take it as AQ is ok. > But another thing that is unrelated to hardware architecture is the nesting > support. Having admin virtqueue in a nesting environment looks like an > overkill. Presenting a register in L1 and map it to L0's admin should be good > enough. So may be a optimized interface can be added that fits nested env. At this point in time real users that we heard are interested in non-nested use cases. Let's enable them first. > > > > > I like to learn the advantages of such method other than simplicity. > > > > We can clearly that we are shifting away from such PCI registers with SIOV, > IMS and other scalable solutions. > > virtio drifting in reverse direction by introducing more registers as > transport. > > I expect it to an optional transport like AQ. > > Actually, I had a proposal of using admin virtqueue as a transport, it's > designed to be SIOV/IMS capable. And it's not hard to extend it with the > state/stop support etc. > > > > > > > > > > > Next would be to program hundreds of statistics of the 64 VQs > > > > through a > > > giant PCI config space register in some busy polling scheme. > > > > > > We don't need giant config space, and this method has been > > > implemented by some vDPA vendors. > > > > > There are tens of 64-bit counters per VQs. These needs to programmed on > destination side. > > Programming these via registers requires exposing them on the registers. > > In one of the proposals, I see them being queried via CVQ from the device. > > I didn't see a proposal like this. And I don't think querying general virtio state > like idx with a device specific CVQ is a good design. > My example was not for the idx. But for VQ statistics that is queried via CVQ. > > > > Programming them via cfg registers requires large cfg space or synchronous > programming until receiving ACK from it. > > This means one entry at a time... > > > > Programming them via CVQ needs replicate and align cmd values etc on all > device types. All duplicate and hard to maintain. > > > > > > > > > > > > I can clearly see how all these are inefficient for faster LM. > > > > We need an efficient AQ to proceed with at minimum. > > > > > > I'm fine with admin virtqueue, but the stop and state are orthogonal to > that. > > > And using admin virtqueue for stop/state will be more natural if we > > > use admin virtqueue as a transport. > > Ok. > > We should have defined it bit earlier that all vendors can use. :( > > I agree. I remember few months back, you acked in the weekly meeting that TC has approved the AQ direction. And we are still in this circle of debating the AQ.
