Testing VFIO_PLATFORM with the PL330 DMA Controller
A user space VFIO_PLATFORM test and an example of device passthrough in QEMU
VFIO Test Cases Introduction
Modern hardware components include Direct Memory Access (DMA) capabilities, which allow them to quickly deliver data to the running operating system. DMA accesses are problematic when mapping a device either to a user space application or to a guest virtual machine. Not only the hardware will be configured using virtual (or intermediate physical) memory addresses, other security concerns also apply since the guest operating system cannot be trusted with direct access to the host’s or other VMs memory space.
An IOMMU is a hardware block, similar to a MMU of CPU which intercepts any memory access from programmable or not hardware accelerators, and translates the memory addresses, from virtual addresses to physical addresses. Hardware components behind a properly configured IOMMU will operate under the illusion that they are directly accessing the physical memory of the machine, while they have actually been mapped to the memory space of a virtual machine. Additionally, the IOMMU will prevent the associated devices from accessing memory beyond the mappings that have been assigned to them in the page tables.
The VFIO framework is featured in the Linux kernel in order to fulfill a precise requirement, which is the possibility to realize user space device drivers, exploiting the capabilities of a DMA capable device behind an IOMMU. This further applies to Virtual Machines, driven by QEMU/KVM, as the capability to assign hardware devices directly to a guest OS, while preserving the isolation properties of virtualization and the security of the system.
Virtual Open Systems, is actively developing the VFIO_PLATFORM driver, which allows to use VFIO on ARM based systems. With this driver, devices on an ARM based SoC can be bound to VFIO, and be used via the VFIO API. This can be used with KVM on ARM virtual machines for device assignment to dramatically improve I/O performance. The goal of this document is to guide the reader, step by step, not only on building the VFIO_PLATFORM driver on ARM, but also through two different test cases that Virtual Open Systems has implemented to validate VFIO on ARM:
The first will run through all the steps necessary to compile and configure an user space test case and VFIO driver example, based on the DMA-330 DMA Controller (also known as PL330 DMA controller). This device is an AMBA peripheral developed by ARM; it is easily obtainable using FastModels, building the corresponding model available among all the examples included in the portfolio. As a DMAC, its purpose is to copy data memory-to-memory, memory-to-device or device-to-memory. However, in order to validate the VFIO implementation on ARM, the VFIO test case we are going to use will instruct the device to only make memory-to-memory data transfers.
The second use-case shows how we can configure QEMU to directly expose a real device to a virtual machine. The QEMU upstream version does not yet allow binding of platform devices to virtual machines; this feature is reserved for PCI devices (PCI passthrough) only. However, the community is now moving towards integrating the VFIO API for platform devices into QEMU, in order to guarantee the passthrough feature for this type of devices as well. Virtual Open Systems is contributing towards VFIO and we present the latest state of VFIO integration in QEMU, at the time of writing of this guide. This use-case illustrates the steps required to bind the DMA-330 controller to a virtual machine: in this scenario QEMU will mostly do all of the job for us, requiring only some basic information for locating the device to be bound. The guest system will not be aware of the underlying mechanism and will probe the device seamlessly with an untouched kernel driver.
This guide has the following mandatory requirements:
FastModels, since we will deal with the PL330 DMA controller and the ARM SMMU-400, a version 7.0 or newer is required. The guide has been tested on FastModels version 8.2 and 8.3 (the latest versions available at the time of writing). The DMA controller model is included inside the FastModels Portfolio which is provided in every FastModels release. As an inherited requirement, a proper license to run FastModels is also needed.
A computer with a Linux distribution such as Ubuntu 13.10 or similar.
Setting up an ARM Linux Host with VFIO_PLATFORM
In this section we are going to prepare a Linux distribution which will be used through the rest of the guide. We will set up all the required elements:
install the required packages for cross compiling
compile the model which includes the PL330 DMA Controller and an ARM SMMU-400
set up a NFS resource to accommodate the user space file system
compile the host Linux kernel
configure the final details to boot the model
These first steps are in common with the two alternative uses of VFIO that we feature in this guide. After this chapter, we will look into the necessary steps to build the two test cases.
Install the cross compiler
In order to cross compile for the ARM architecture, we will need some basic tools. Considering that these commands refer to an Ubuntu distribution, they may need to be adapted in order to suit other configurations. First of all we start by installing the GCC compiler, git and other useful tools:
$ sudo apt-get update && sudo apt-get install build-essential git libncurses5-dev
In order to install the cross compiler and the necessary dependencies:
$ sudo apt-get install gcc-arm-linux-gnueabi coreutils-armel-cross libc6-armel-cross \
libc6-dev-armel-cross linux-libc-dev-armel-cross binutils-arm-linux-gnueabi \
Compilation of the Model from the FastModels Portfolio
Once FastModels has been installed on the target machine in the installation folder, the FastModelsPortfolio folder will contain the models of interest located in the directory FastModelsPortfolio_8.3/examples/FVP_VE/Build_Cortex-A15x4-A7x4-MMU400-DMA330.
In order to compile the model, we may launch System Canvas (the executable is FastModelsTools_8.3/bin/sgcanvas) and open and compile the project FastModelsPortfolio_8.3/examples/FVP_VE/Build_Cortex-A15x4-A7x4-MMU400-DMA330/FVP_VE_Cortex-A15x4-A7x4-MMU400-DMA330.sgproj as an ISIM system.
As soon as the compilation has finished, we can find the binary isim_system in a sub-directory inside the project folder. The name of the sub-directory depends on the profile and compiler used at the time of compilation. We need now to take note of the location of the binary representing the model, as it will be used later.
Since we will need a file system to boot together with the Linux kernel, it is highly recommended to configure the NFS service in order to export some directory that will contain the user space files. To install the NFS server one needs to simply run:
$ sudo apt-get install nfs-kernel-server nfs-common
A new directory is needed for the host files:
$ sudo mkdir /srv/vfio_host
To add it to the exported NFS resources, the following line can be added to the /etc/exports file:
Please note that the previous line should be adapted to match the system's configuration.
Creating the File System
There are several ways to obtain a bootable file system; a relatively simple one is to use qemu-debootstrap to create an Ubuntu Quantal chroot:
$ cd /srv/
$ sudo qemu-debootstrap --arch=armel quantal ./vfio_host
When the installation of the file system has finished, it needs to be configured. Enabling the repositories for ATP enables the possibility to install several useful packages, which can be done by adding the following to the /etc/apt/sources.list file inside the Quantal chroot folder:
###### Ubuntu Main Repos
deb http://ports.ubuntu.com/ quantal main restricted universe
###### Ubuntu Update Repos
deb http://ports.ununtu.com/ quantal-security main restricted universe
The system will also need to be modified to use the right serial console for the FastModels:
$ sudo cp /srv/vfio_host/etc/init/tty1.conf /srv/vfio_host/etc/init/ttyAMA0.conf
Occurrences of tty1 will need to be changed with ttyAMA0 in the new file, so that the exec directive will look like this:
exec /sbin/getty -8 38400 ttyAMA0
Compilation of the Host Linux kernel with VFIO_PLATFORM
The latest kernel including the VFIO_PLATFORM patches can be cloned from the Virtual Open Systems repository, by following these few steps:
$ git clone git://github.com/virtualopensystems/linux-kvm-arm.git
$ cd linux-kvm-arm
$ git checkout vfio-platform-v6-guide
$ CROSS_COMPILE=arm-linux-gnueabi- ARCH=arm make menuconfig
We need to enable at least the following configuration options for the host kernel:
Virtualization ---> Kernel-based Virtual Machine (KVM) support
Device Drivers ---> IOMMU Hardware Support ---> ARM Ltd. System MMU (SMMU) Support
Device Drivers ---> VFIO Non-Privileged userspace driver framework ---> VFIO support for platform devices
Starting from scratch, you can obtain a working configuration file with the following steps:
$ cp arch/arm/configs/vexpress_defconfig .config
$ scripts/config -e CONFIG_ARM_LPAE
$ scripts/config -e CONFIG_PROC_DEVICETREE
$ scripts/config -e CONFIG_EXT3_FS
$ scripts/config -e CONFIG_VIRTUALIZATION
$ scripts/config -e CONFIG_KVM
$ scripts/config -e CONFIG_KVM_ARM_TIMER
$ scripts/config -e CONFIG_ARM_SMMU
$ scripts/config -e CONFIG_VFIO
$ scripts/config -e CONFIG_VFIO_PLATFORM
$ scripts/config -e CONFIG_KVM_ARM_VGIC
$ scripts/config --set-val CONFIG_KVM_ARM_MAX_VCPUS 4
$ scripts/config --set-val CONFIG_ARM_DMA_IOMMU_ALIGNMENT 8
$ scripts/config -e CONFIG_EXT3_DEFAULTS_TO_ORDERED
$ scripts/config -e CONFIG_EXT3_FS_XATTR
$ scripts/config -e CONFIG_EXT3_FS_POSIX_ACL
$ scripts/config -e CONFIG_EXT3_FS_SECURITY
$ scripts/config -e CONFIG_VIRTIO
$ scripts/config -e CONFIG_VIRTIO_BLK
$ scripts/config -e CONFIG_VIRTIO_RING
$ scripts/config -e CONFIG_VIRTIO_MMIO
$ scripts/config -e CONFIG_DMA_ENGINE
$ scripts/config -e CONFIG_DMADEVICES
$ scripts/config -e CONFIG_PL330_DMA
$ scripts/config -e CONFIG_VOSYS_DMATEST
Configuring the kernel for cross compilation does not differ much than usual, however it is important to set the ARCH and CROSS_COMPILE variables. And finally, we can compile our host kernel:
$ CROSS_COMPILE=arm-linux-gnueabi- ARCH=arm make zImage
The kernel requires also a device tree which describes all the hardware components of the target model. The cloned kernel directory includes an appropriate device tree for the model that includes the SMMU, which can be build with:
$ scripts/dtc/dtc -I dts -O dtb -o /path/to/tree.dtb arch/arm/boot/dts/rtsm_ve-cortex_a15x4_a7x4.dts
Booting the Model
We are almost ready to run the model, but some parameters are required to make the DMA controller working. We can create the following script to launch the model:
-a coretile.cluster0.*=boot-wrapper/linux-system-semi.axf \
-a coretile.cluster1.*=boot-wrapper/linux-system-semi.axf \
-C motherboard.smsc_91c111.enabled=1 \
-C motherboard.hostbridge.interfaceName="tap0" \
-C coretile.cluster0.cpu0.semihosting-enable=1 \
-C coretile.dmac0_int_base=64 \
-C coretile.dmac1_int_base=96 \
-C coretile.dmac2_int_base=128 \
-C coretile.dmac3_int_base=168 \
-C coretile.mmu400_dma330.dmac0.p_irq_nsecure="true" \
-C coretile.mmu400_dma330.dmac1.p_irq_nsecure="true" \
-C coretile.mmu400_dma330.dmac2.p_irq_nsecure="true" \
-C coretile.mmu400_dma330.dmac3.p_irq_nsecure="true" \
--kernel <PATH_TO_ZIMAGE> \
--dtb tree.dtb -- earlyprintk console=ttyAMA0 mem=2048M \
root=/dev/nfs nfsroot=<YOUR_IP>:/srv/vfio_host/ rw ip=dhcp"
taking care to substitute <PATH_TO_MODEL>, <PATH_TO_ZIMAGE> and <YOUR_IP> accordingly. The configuration above refers to a tap0 interface which allows the model to be connected to the same network of the workstation running it. To add such an interface a bash script like the following can be used:
sudo brctl addbr br0
sudo tunctl -u <USERNAME>
sudo ifconfig eth0 0.0.0.0 promisc up
sudo ifconfig tap0 0.0.0.0 promisc up
sudo brctl addif br0 eth0
sudo brctl addif br0 tap0
sudo brctl stp br0 off
sudo ifconfig br0 $bridge_ip netmask 255.255.255.0 broadcast 192.168.1.255
sudo route add default gw $gateway
Care should be taken to adapt it to the system configuration and network. Finally we can launch the model:
$ sh boot_model.sh
We can confirm that all is set up correctly if all four DMA controllers of the model have been correctly probed:
$ ls /sys/devices/ | grep dma
and the DMA test completes successfully:
# echo 1 > /sys/kernel/debug/vosys_dmatest/start
# dmesg |grep "vosys dmatest"
Finally we may also verify that the virtio_platform module is actually loaded:
$ ls /sys/module/ | grep vfio_platform
Test Case 1: A VFIO_PLATFORM Userspace Driver
As already mentioned in the introduction, the first VFIO test case should validate the possibility to have a complete user space driver for a device behind the IOMMU. Virtual Open Systems has implemented a reference test case based on the PL330 DMA controller in the FastModels. To clone the code from the Virtual Open Systems repository:
# apt-get install build-essential libglib2.0-dev git
$ git clone https://github.com/virtualopensystems/vfio-host-test.git
$ cd vfio-vhost-test
This is a suite intended to test VFIO for platform devices. The suite relays on a bash script to check the devices available that could be potentially used as target for the VFIO tests. The test will perform some general, yet vital checks and then execute some specific code of the device being tested. At the moment, only the DMA330 has a specific test code available that will instruct the device to perform a simple DMA transfer, from Userspace.
Since we are dealing with the DMA330, we can instruct the test suite to perform directly the device specific code:
# ./vfio_test_device.sh -s
When asked, press the index of the device to test ("0" for 2c0a0000.dma); if everything goes like it should, the output will confirm the successful outcome of the test.
If you want to have more visibility on what the test actually does, we suggest to execute the following commands instead of using the automated script. First of all, we need to compile the DMA330 test:
$ cd src_test/pl330/ && make
Before running the test, we need to configure VFIO to bind with the target device. Since the model provides four different DMA controllers, we need to choose one of them as target. Here we have picked the first one (2c0a0000.dma).
# echo vfio-platform > /sys/bus/platform/devices/2c0a0000.dma/driver_override
# echo 2c0a0000.dma > /sys/bus/platform/drivers_probe
Now that the device is bound, we need to find the IOMMU group it is associated with. To do that, we may simply run:
$ ls -l /sys/devices/2c0a0000.dma/iommu_groups
and look at the last number of the output (in our case "0"). Depending on this value, a new file will be created in /dev/vfio/ which will be the first parameter of the next command. Finally, we can run the test:
# ./test_pl330_vfio_driver /dev/vfio/0 2c0a0000.dma
If you wish to test another DMA controller, you can change the above commands accordingly. Please note that, in order to test a controller other than the first available, an additional modification in the device tree might be required so that the kernel can see the device as attached to the platform bus. To do that, the value "arm,primecell" has to be removed from the compatible string of the corresponding dma device.
Test Case 2: Device Passthrough on a QEMU/KVM Guest
In this section we will deal with the second, and more interesting, use case of VFIO: an example of device assignment for a QEMU/KVM virtual machine. This time the scenario is a little more complicated; it is formed by an host (we will use the same machine we were using previously) and by a QEMU guest, that we will build soon.
The guest will use the kernel DMA test module to set up a DMA transfer which in turn will use the (unmodified) PL330 guest kernel driver. The execution of the driver code in the guest will not involve any trap to the host for emulation, but will instead instruct directly the real hardware device.
The first thing to do is building QEMU from our sources; we will then copy the binary inside the host file system. We also create a new folder to put all the files necessary to run the guest (we assume that we will be later logged as root in the host, thus we use /home/root as working directory).
Login or register to access full information