At 05/11/2011 03:55 PM, Daniel P. Berrange Write: > On Wed, May 11, 2011 at 10:30:43AM +0800, Wen Congyang wrote: >> At 05/10/2011 06:00 PM, Daniel P. Berrange Write: >>> On Tue, May 10, 2011 at 02:00:17PM +0800, Wen Congyang wrote: >>>> We want to use more than 200+ device. Libvirt does not use multi >>>> function PCI device and PCI-to-PCI bridge. So we can not use more >>>> than 200+ device if it's a PCI device or it's controller is a PCI >>>> device. >>> >>> IMHO using multi function support is a dead end, because it >>> makes hotplug completely unusable and is not even possible >>> for most PCI devices. The way we want to raise the device >>> limit is >>> >>> a. Supporting multiple PCI domains (multiple PCI root complexes) >> >> IIUC, qemu do not support multiple PCI domains now. > > There are people who are working on making that supported in KVM > real soon. Good news. > >>> b. Adding PCI bridges >> >> I do not know whether qemu supports it. I find hw/pci_bridge.c in qemu's source, >> so qemu may support it. I will confirm it. > > I'm told this is already supported, but I've not tested it myself. I'm still reading the qemu's code. Do you know the driver's name(For example: the driver for scsi controller in qemu is lsi.) > >>> c. A virtio-scsi controller to allow > 1 virtio disk per PCI device >> >> Hmm, do you mean this: >> -device virtio-blk-pci,ports=4,drive0=hda,drive1=hdb,drive2=hdc,... > > No it is actually going to be a new device type, and I'm expecting > it will work very like the LSI scsi does in terms of device setup. > eg create a controller device, and then create devices for each drive. > > -device virtio-scsi-pci,id=vscsi1 > -drive file=/some/disk,id=vscsi1.1 > -device virtio-blk-pci,bus=vscsi1,drive=vscsi1.1 > -drive file=/some/disk,id=vscsi1.2 > -device virtio-blk-pci,bus=vscsi1,drive=vscsi1.2 > -drive file=/some/disk,id=vscsi1.2 > -device virtio-blk-pci,bus=vscsi1,drive=vscsi1.2 > >>> any of those will dramatically increase the number of devices >>> we can use without the horrible hotplug problems that multifunction >>> introduces. >>> >>>> This patch adds the support of multi function PCI device. It >>>> does not support to hot plug/unplug multi function PCI device. >>>> >>>> TODO: >>>> 1. support to hot plug multi function PCI device >>>> We only can hot plug one device at one time. I think we should >>>> introduce another command 'virsh attach-devices XXX' to support >>>> hot plug more than one device at one time? >>> >>> Since you can't do practical hotplug of multifunction devices at the >>> QEMU layer at all, there's nothing useful we can do at libvirt either. >> >> We can hotplug multifunction PCI devices like this: >> 1. Before hot pluging, >> # lspci >> 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) >> 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] >> 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] >> 00:01.2 USB Controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) >> 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) >> 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 >> 00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20) >> 00:04.0 RAM memory: Red Hat, Inc Virtio memory balloon >> 00:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a >> >> 2. hot plug multi function PCI devices: >> # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi2,bus=pci.0,addr=0x06.0x07' >> # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi3,bus=pci.0,addr=0x06.0x06' >> # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi4,bus=pci.0,addr=0x06.0x05' >> # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi5,bus=pci.0,addr=0x06.0x04' >> # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi6,bus=pci.0,addr=0x06.0x03' >> # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi7,bus=pci.0,addr=0x06.0x02' >> # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi8,bus=pci.0,addr=0x06.0x01' >> # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi9,multifunction=on,bus=pci.0,addr=0x06.0x00' > > Hmm, that's kinda wierd & i'm suprised it works, particularly for LSI > since I thought guest drivers would need support for multifunction too. I only test LSI. If we can hotplug the device by single PCI device, I think it can work fine for multi function. > >> >> 3. After hot plug multi function PCI devices: >> # lspci >> 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) >> 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] >> 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] >> 00:01.2 USB Controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) >> 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) >> 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 >> 00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20) >> 00:04.0 RAM memory: Red Hat, Inc Virtio memory balloon >> 00:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a >> 00:06.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a >> 00:06.1 SCSI storage controller: LSI Logic / Symbios Logic 53c895a >> 00:06.2 SCSI storage controller: LSI Logic / Symbios Logic 53c895a >> 00:06.3 SCSI storage controller: LSI Logic / Symbios Logic 53c895a >> 00:06.4 SCSI storage controller: LSI Logic / Symbios Logic 53c895a >> 00:06.5 SCSI storage controller: LSI Logic / Symbios Logic 53c895a >> 00:06.6 SCSI storage controller: LSI Logic / Symbios Logic 53c895a >> 00:06.7 SCSI storage controller: LSI Logic / Symbios Logic 53c895a >> >> >>> >>>> >>>> 2. support to hot unplug multi function PCI device >>>> hot unpluging multi function PCI device meas that all the other >>>> devices on the same slot will be hot unpluged. So we should do >>>> some cleanup and remove the other devices too. If the other >>>> device on the same slot does not support hot unpluged, or it is a >>>> a controller and some other devices still use this controller, >>>> I think we should refuse to hot unplug this mutlti function PCI >>>> device. >>> >>> IMHO these kind of restrictions will make life really unpleasant >>> for applications and are a reason we should *not* support the >>> multifunction code. Instead we should focus on one of the other >>> 3 options I mention above. >> >> Yes, there's too many restrictions about hot plug/unplug multifunction >> PCI devices. > > > Regards, > Daniel -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list