在 2022/11/14 15:04, Leon Romanovsky 写道:
On Sun, Nov 13, 2022 at 09:47:12PM +0800, Longpeng (Mike, Cloud Infrastructure Service Product Dept.) wrote:
Hi leon,
在 2022/11/12 0:39, Leon Romanovsky 写道:
On Fri, Nov 11, 2022 at 10:27:18PM +0800, Longpeng(Mike) wrote:
From: Longpeng <longpeng2@xxxxxxxxxx>
We can enable SRIOV and add VFs by /sys/bus/pci/devices/..../sriov_numvfs, but
this operation needs to spend lots of time if there has a large amount of VFs.
For example, if the machine has 10 PFs and 250 VFs per-PF, enable all the VFs
concurrently would cost about 200-250ms. However most of them are not need to be
used at the moment, so we can enable SRIOV first but add VFs on demand.
It is unclear what took 200-250ms, is it physical VF creation or bind of
the driver to these VFs?
It is neither. In our test, we already created physical VFs before, so we
skipped the 100ms waiting when writing PCI_SRIOV_CTRL. And our driver only
probes PF, it just returns an error if the function is VF.
It means that you didn't try sriov_drivers_autoprobe. Once it is set to
true, It won't even try to probe VFs.
The hotspot is the sriov_add_vfs (but no driver probe in fact) which is a
long procedure. Each step costs only a little, but the total cost is not
acceptable in some time-sensitive cases.
This is also cryptic to me. In standard SR-IOV deployment, all VFs are
created and configured while operator booted the machine with sriov_drivers_autoprobe
set to false. Once this machine is ready, VFs are assigned to relevant VMs/users
through orchestration SW (IMHO, it is supported by all orchestration SW).
And only last part (assigning to users) is time-sensitive operation.
The VF creation and configuration are also time-sensitive in some cases,
for example, the hypervisor live update case (such as [1]):
save VMs -> kexec -> restore VMs
After the new kernel starts, the VFs must be added into the system, and
then assign the original VFs to the QEMU. This means we must enable all
2K+ VFs at once and increase the downtime.
If we can enable the VFs that are used by existing VMs then restore the
VMs and enable other unused VFs at last, the downtime would be
significantly reduced.
[1]
https://static.sched.com/hosted_files/kvmforum2022/65/kvmforum2022-Preserving%20IOMMU%20states%20during%20kexec%20reboot-v4.pdf
What’s more, the sriov_add_vfs adds the VFs of a PF one by one. So we can
mostly support 10 concurrent calls if there has 10 PFs.
I wondered, are you using real HW? or QEMU SR-IOV? What is your server
that supports such large number of VFs?
Physical device. Some devices in the market support the large number of
VFs, especially in the hardware offloading area, e.g DPU/IPU. I think
the SR-IOV software should keep pace with times too.
BTW, Your change will probably break all SR-IOV devices in the market as
they rely on PCI subsystem to have VFs ready and configured.
I see, but maybe this change could be a choice for some users.
Thanks
.
