在qemu-kvm虚拟化环境中，为提高虚拟机性能，需要将主机（PCI）设备直通给虚拟机（vm），即：设备透传技术（也叫设备直通技术），该技术需要主机支持Intel(VT-d) 或 AMD (IOMMU) 硬件虚拟化加速技术。

前提要求

linux kernel>3.6
qemu 版本>1.4
虚拟机系统以uefi模式启动
宿主机服务器开启IOMMU
宿主机加载vfio和vfio-pci驱动
宿主机CPU支持intel-vt/vd

实验环境

  Centos7.2-AIO-3.6.0.0.1
  qemu-2.2

准备工作

服务器宿主机

修改系统内核启动参数，重启系统，查看是否开启IOMMU

dmesg | grep -e DMAR -e IOMMU
[ 0.000000] DMAR: IOMMU enabled

查看是否开启intel-vt-x/vt-d

cat /proc/cpuinfo | grep vmx

执行如下脚本检查是否支持中断重定向

#!/bin/sh
if [ $(dmesg | grep ecap | wc -l) -eq 0 ]; then
  echo "No interrupt remapping support found"
  exit 1
fi
for i in $(dmesg | grep ecap | awk '{print $NF}'); do
  if [ $(( (0x$i & 0xf) >> 3 )) -ne 1 ]; then
    echo "Interrupt remapping not supported"
    exit 1
  fi
done

如果硬件不支持interrupt remapping，需要执行

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf

加载vfio驱动

modprobe vfio
modprobe vfio-pci

安装OVMF引导虚拟机（OVMF支持uefi启动）

wget http://www.kraxel.org/repos/firmware.repo
yum install edk2.git-ovmf-x64.noarch

透传

设备在主机中解绑

[root@localhost ~]# lspci -nn|more
00:00.0 Host bridge [0600]: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge [8086:7190] (rev 01)
00:01.0 PCI bridge [0604]: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge [8086:7191] (rev 01)
00:07.0 ISA bridge [0601]: Intel Corporation 82371AB/EB/MB PIIX4 ISA [8086:7110] (rev 08)
00:07.1 IDE interface [0101]: Intel Corporation 82371AB/EB/MB PIIX4 IDE [8086:7111] (rev 01)
00:07.3 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI [8086:7113] (rev 08)
00:07.7 System peripheral [0880]: VMware Virtual Machine Communication Interface [15ad:0740] (rev 10)
00:0f.0 VGA compatible controller [0300]: VMware SVGA II Adapter [15ad:0405]
00:10.0 SCSI storage controller [0100]: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI [1000:0030] (rev 01)
00:11.0 PCI bridge [0604]: VMware PCI bridge [15ad:0790] (rev 02)
00:15.0 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
00:15.1 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
00:15.2 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)

找到设备，解绑

echo 0000:00:07.1 > /sys/bus/pci/devices/0000\:00\:07.1/driver/unbind

生成vfio设备

echo 8086 7111 > /sys/bus/pci/drivers/vfio-pci/new_id

此时：/dev/vfio下面会有个以阿拉伯数字命名的文件，对应vfio设备组

启动虚拟机

加参数：-device vfio-pci,host=00:05:00.0  #透传的设备
加参数：-drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd #虚拟机bios

/usr/libexec/qemu-kvm -M pc-i440fx-rhel7.0.0 -enable-kvm -m 2048 -smp 2 -drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd -drive id=disk0,if=none,format=qcow2,file=test.qcow2 -device virtio-blk-pci,drive=disk0,bootindex=0 -drive id=cd0,if=none,format=raw,readonly,file=CentOS-7-x86_64-Minimal-1503-01.iso -device ide-cd,bus=ide.1,drive=cd0,bootindex=1 -global PIIX4_PM.disable_s3=0 -global isa-debugcon.iobase=0x402 -debugcon file:fedora.ovmf.log -monitor stdio -device piix3-usb-uhci -device usb-tablet -netdev id=net0,type=user -device virtio-net-pci,netdev=net0,romfile= -device qxl-vga -spice port=3000,disable-ticketing

/usr/libexec/qemu-kvm -M pc-i440fx-rhel7.0.0 -enable-kvm -m 2048 -smp 2 -drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd -drive id=disk0,if=none,format=qcow2,file=test.qcow2 -device virtio-blk-pci,drive=disk0,bootindex=0 -global PIIX4_PM.disable_s3=0 -global isa-debugcon.iobase=0x402 -debugcon file:fedora.ovmf.log -monitor stdio -device piix3-usb-uhci -device usb-tablet -netdev id=net0,type=user -device virtio-net-pci,netdev=net0,romfile= -device qxl-vga -spice port=3000,disable-ticketing -device vfio-pci,host=00:05:00.0

客户端访问

 spicy -h ip -p 3000

在虚拟机内部执行

lspci -nn
lsblk

查看是否透传成功

参考资料

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
https://www.kernel.org/doc/Documentation/vfio.txt
http://pve.proxmox.com/wiki/Pci_passthrough