Hi All ,
I have tried the same using dell R710 server and RHEL6.1 after enabled the VT-D option enabled . and checked it was not working with my enviromment ..
any body tested below on Dell server ...If i have missed anything ..
PCI passthrough
This chapter covers using PCI passthrough with KVM.
Certain hardware platforms allow virtualized guests to directly access various hardware devices and
components. This process in virtualization is known as passthrough. Passthrough is known as device
assignment in some of the KVM documentation and the KVM code.
The KVM hypervisor supports attaching PCI devices on the host system to virtualized guests.
PCI passthrough allows guests to have exclusive access to PCI devices for a range of tasks. PCI
passthrough allows PCI devices to appear and behave as if they were physically attached to the guest
operating system. PCI passthrough can improve the I/O performance of devices attached to virtualized
guests.
Almost all PCI and PCI Express devices that support passthrough, except for graphics cards, can be
directly attached to virtualized guests with PCI passthrough.
PCI passthrough is only available on hardware platforms supporting either Intel VT-d or AMD IOMMU.
These Intel VT-d or AMD IOMMU extensions must be enabled in BIOS for PCI passthrough to
function.
Red Hat Enterprise Linux 6.0 and newer supports hot plugging PCI passthrough devices into
virtualized guests.
Out of the 32 available PCI devices for a guest 4 are not removable. This means there are only 28
PCI slots available for additional devices per guest. Every para-virtualized network or block device
uses one slot. Each guest can use up to 28 additional devices made up of any combination of paravirtualized
network, para-virtualized disk devices, or other PCI devices using VT-d.
Procedure 12.1. Preparing an Intel system for PCI passthrough
1. Enable the Intel VT-d extensions
The Intel VT-d extensions provides hardware support for directly assigning a physical devices to
guest.
The VT-d extensions are required for PCI passthrough with Red Hat Enterprise Linux. The
extensions must be enabled in the BIOS. Some system manufacturers disable these extensions
by default.
These extensions are often called various terms in BIOS which differ from manufacturer to
manufacturer. Consult your system manufacturer's documentation.
2. Activate Intel VT-d in the kernel
Activate Intel VT-d in the kernel by appending the intel_iommu=on parameter to the kernel line
of the kernel line in the /boot/grub/grub.conf file.
The example below is a modified grub.conf file with Intel VT-d activated.
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux Server (2.6.32-36.x86-645)
root (hd0,0)
kernel /vmlinuz-2.6.32-36.x86-64 ro root=/dev/VolGroup00/LogVol00 rhgb
quiet intel_iommu=on
initrd /initrd-2.6.32-36.x86-64.img
Chapter 12. PCI passthrough
108
3. Ready to use
Reboot the system to enable the changes. Your system is now PCI passthrough capable.
Procedure 12.2. Preparing an AMD system for PCI passthrough
• Enable AMD IOMMU extensions
The AMD IOMMU extensions are required for PCI passthrough with Red Hat Enterprise Linux.
The extensions must be enabled in the BIOS. Some system manufacturers disable these
extensions by default.
AMD systems only require that the IOMMU is enabled in the BIOS. The system is ready for PCI
passthrough once the IOMMU is enabled.
12.1. Adding a PCI device with virsh
These steps cover adding a PCI device to a virtualized guest on a KVM hypervisor using hardwareassisted
PCI passthrough.
This example uses a USB controller device with the PCI identifier code, pci_8086_3a6c, and a fully
virtualized guest named win2k3.
1. Identify the device
Identify the PCI device designated for passthrough to the guest. The virsh nodedev-list
command lists all devices attached to the system. The --tree option is useful for identifying
devices attached to the PCI device (for example, disk controllers and USB controllers).
# virsh nodedev-list --tree
For a list of only PCI devices, run the following command:
# virsh nodedev-list | grep pci
In the output from this command, each PCI device is identified by a string, as shown in the
following example output:
pci_0000_00_00_0
pci_0000_00_02_0
pci_0000_00_02_1
pci_0000_00_03_0
pci_0000_00_03_2
pci_0000_00_03_3
pci_0000_00_19_0
pci_0000_00_1a_0
pci_0000_00_1a_1
pci_0000_00_1a_2
pci_0000_00_1a_7
pci_0000_00_1b_0
pci_0000_00_1c_0
Tip: determining the PCI device
Comparing lspci output to lspci -n (which turns off name resolution) output can assist in
deriving which device has which device identifier code.
Adding a PCI device with virsh
109
Record the PCI device number; the number is needed in other steps.
2. Information on the domain, bus and function are available from output of the virsh nodedevdumpxml
command:
# virsh nodedev-dumpxml pci_8086_3a6c
<device>
<name>pci_8086_3a6c</name>
<parent>computer</parent>
<capability type='pci'>
<domain>0</domain>
<bus>0</bus>
<slot>26</slot>
<function>7</function>
<id='0x3a6c'>82801JD/DO (ICH10 Family) USB2 EHCI Controller #2</product>
<vendor id='0x8086'>Intel Corporation</vendor>
</capability>
</device>
3. Detach the device from the system. Attached devices cannot be used and may cause various
errors if connected to a guest without detaching first.
# virsh nodedev-dettach pci_8086_3a6c
Device pci_8086_3a6c dettached
4. Convert slot and function values to hexadecimal values (from decimal) to get the PCI bus
addresses. Append "0x" to the beginning of the output to tell the computer that the value is a
hexadecimal number.
For example, if bus = 0, slot = 26 and function = 7 run the following:
$ printf %x 0
0
$ printf %x 26
1a
$ printf %x 7
7
The values to use:
bus='0x00'
slot='0x1a'
function='0x7'
5. Run virsh edit (or virsh attach device) and added a device entry in the <devices> section to
attach the PCI device to the guest.
# virsh edit win2k3
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x00' slot='0x1a' function='0x7'/>
</source>
</hostdev>
6. Once the guest system is configured to use the PCI address, the host system must be configured
to stop using the device. The ehci driver is loaded by default for the USB PCI controller.
Chapter 12. PCI passthrough
110
$ readlink /sys/bus/pci/devices/0000\:00\:1a.7/driver
../../../bus/pci/drivers/ehci_hcd
7. Detach the device:
$ virsh nodedev-dettach pci_8086_3a6c
8. Verify it is now under the control of pci_stub:
$ readlink /sys/bus/pci/devices/0000\:00\:1d.7/driver
../../../bus/pci/drivers/pci-stub
9. Set a sebool to allow the management of the PCI device from the guest:
$ setsebool -P virt_manage_sysfs 1
10. Start the guest system :
# virsh start win2k3
The PCI device should now be successfully attached to the guest and accessible to the guest
This chapter covers using PCI passthrough with KVM.
Certain hardware platforms allow virtualized guests to directly access various hardware devices and
components. This process in virtualization is known as passthrough. Passthrough is known as device
assignment in some of the KVM documentation and the KVM code.
The KVM hypervisor supports attaching PCI devices on the host system to virtualized guests.
PCI passthrough allows guests to have exclusive access to PCI devices for a range of tasks. PCI
passthrough allows PCI devices to appear and behave as if they were physically attached to the guest
operating system. PCI passthrough can improve the I/O performance of devices attached to virtualized
guests.
Almost all PCI and PCI Express devices that support passthrough, except for graphics cards, can be
directly attached to virtualized guests with PCI passthrough.
PCI passthrough is only available on hardware platforms supporting either Intel VT-d or AMD IOMMU.
These Intel VT-d or AMD IOMMU extensions must be enabled in BIOS for PCI passthrough to
function.
Red Hat Enterprise Linux 6.0 and newer supports hot plugging PCI passthrough devices into
virtualized guests.
Out of the 32 available PCI devices for a guest 4 are not removable. This means there are only 28
PCI slots available for additional devices per guest. Every para-virtualized network or block device
uses one slot. Each guest can use up to 28 additional devices made up of any combination of paravirtualized
network, para-virtualized disk devices, or other PCI devices using VT-d.
Procedure 12.1. Preparing an Intel system for PCI passthrough
1. Enable the Intel VT-d extensions
The Intel VT-d extensions provides hardware support for directly assigning a physical devices to
guest.
The VT-d extensions are required for PCI passthrough with Red Hat Enterprise Linux. The
extensions must be enabled in the BIOS. Some system manufacturers disable these extensions
by default.
These extensions are often called various terms in BIOS which differ from manufacturer to
manufacturer. Consult your system manufacturer's documentation.
2. Activate Intel VT-d in the kernel
Activate Intel VT-d in the kernel by appending the intel_iommu=on parameter to the kernel line
of the kernel line in the /boot/grub/grub.conf file.
The example below is a modified grub.conf file with Intel VT-d activated.
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux Server (2.6.32-36.x86-645)
root (hd0,0)
kernel /vmlinuz-2.6.32-36.x86-64 ro root=/dev/VolGroup00/LogVol00 rhgb
quiet intel_iommu=on
initrd /initrd-2.6.32-36.x86-64.img
Chapter 12. PCI passthrough
108
3. Ready to use
Reboot the system to enable the changes. Your system is now PCI passthrough capable.
Procedure 12.2. Preparing an AMD system for PCI passthrough
• Enable AMD IOMMU extensions
The AMD IOMMU extensions are required for PCI passthrough with Red Hat Enterprise Linux.
The extensions must be enabled in the BIOS. Some system manufacturers disable these
extensions by default.
AMD systems only require that the IOMMU is enabled in the BIOS. The system is ready for PCI
passthrough once the IOMMU is enabled.
12.1. Adding a PCI device with virsh
These steps cover adding a PCI device to a virtualized guest on a KVM hypervisor using hardwareassisted
PCI passthrough.
This example uses a USB controller device with the PCI identifier code, pci_8086_3a6c, and a fully
virtualized guest named win2k3.
1. Identify the device
Identify the PCI device designated for passthrough to the guest. The virsh nodedev-list
command lists all devices attached to the system. The --tree option is useful for identifying
devices attached to the PCI device (for example, disk controllers and USB controllers).
# virsh nodedev-list --tree
For a list of only PCI devices, run the following command:
# virsh nodedev-list | grep pci
In the output from this command, each PCI device is identified by a string, as shown in the
following example output:
pci_0000_00_00_0
pci_0000_00_02_0
pci_0000_00_02_1
pci_0000_00_03_0
pci_0000_00_03_2
pci_0000_00_03_3
pci_0000_00_19_0
pci_0000_00_1a_0
pci_0000_00_1a_1
pci_0000_00_1a_2
pci_0000_00_1a_7
pci_0000_00_1b_0
pci_0000_00_1c_0
Tip: determining the PCI device
Comparing lspci output to lspci -n (which turns off name resolution) output can assist in
deriving which device has which device identifier code.
Adding a PCI device with virsh
109
Record the PCI device number; the number is needed in other steps.
2. Information on the domain, bus and function are available from output of the virsh nodedevdumpxml
command:
# virsh nodedev-dumpxml pci_8086_3a6c
<device>
<name>pci_8086_3a6c</name>
<parent>computer</parent>
<capability type='pci'>
<domain>0</domain>
<bus>0</bus>
<slot>26</slot>
<function>7</function>
<id='0x3a6c'>82801JD/DO (ICH10 Family) USB2 EHCI Controller #2</product>
<vendor id='0x8086'>Intel Corporation</vendor>
</capability>
</device>
3. Detach the device from the system. Attached devices cannot be used and may cause various
errors if connected to a guest without detaching first.
# virsh nodedev-dettach pci_8086_3a6c
Device pci_8086_3a6c dettached
4. Convert slot and function values to hexadecimal values (from decimal) to get the PCI bus
addresses. Append "0x" to the beginning of the output to tell the computer that the value is a
hexadecimal number.
For example, if bus = 0, slot = 26 and function = 7 run the following:
$ printf %x 0
0
$ printf %x 26
1a
$ printf %x 7
7
The values to use:
bus='0x00'
slot='0x1a'
function='0x7'
5. Run virsh edit (or virsh attach device) and added a device entry in the <devices> section to
attach the PCI device to the guest.
# virsh edit win2k3
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x00' slot='0x1a' function='0x7'/>
</source>
</hostdev>
6. Once the guest system is configured to use the PCI address, the host system must be configured
to stop using the device. The ehci driver is loaded by default for the USB PCI controller.
Chapter 12. PCI passthrough
110
$ readlink /sys/bus/pci/devices/0000\:00\:1a.7/driver
../../../bus/pci/drivers/ehci_hcd
7. Detach the device:
$ virsh nodedev-dettach pci_8086_3a6c
8. Verify it is now under the control of pci_stub:
$ readlink /sys/bus/pci/devices/0000\:00\:1d.7/driver
../../../bus/pci/drivers/pci-stub
9. Set a sebool to allow the management of the PCI device from the guest:
$ setsebool -P virt_manage_sysfs 1
10. Start the guest system :
# virsh start win2k3
The PCI device should now be successfully attached to the guest and accessible to the guest
On Mon, Mar 5, 2012 at 9:47 PM, Eric Blake <eblake@xxxxxxxxxx> wrote:
[top-posting on technical lists tends to be frowned on]
If the kernel presents PCI express devices to user space in the same way
On 03/05/2012 09:06 AM, Daniel Gonzalez wrote:
> Thanks Eric,
>
> Does this also apply for PCI express?
as it presents traditional PCI devices, then the answer is yes - PCI
device passthrough in both cases is a matter of using kernel passthrough
to let the qemu user space process control that device, so that the
guest OS in turn can manage the device as it would in a bare metal
situation.
_______________________________________________
libvirt-users mailing list
libvirt-users@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvirt-users
--
With Best Regards,
Nitin Kumar
Sr. System Engineer-Unix (CSC India Pvt. Ltd, Noida )
+91-9999706444
