Ryzen GPU Passthrough Guide – Linux Gaming VFIO
I built this system for my review of the Gigabyte M27Q, a monitor with a KVM built in making it perfect for the enthusiast linux gamer. You can check out that video in the cards above, but this one is a getting started guide on how to get this set up. This is advice both from a beginner, and as a first step to get it running, but I can’t hold a candle to Wendell and his wealth of knowledge so if you haven’t already, go subscribe to his channel Level1Techs, and his Linux channel too.
So lets start with parts. I’m using a Ryzen 3900X here giving us plenty of cores to share with our Windows VM, and I’ve got 32GB of RAM too, again plenty to share. The motherboard you choose is incredibly important, because that’s what determines if you will be able to pass your secondary GPU through to the VM on its own, or if it’s going to be more difficult. I’ve got a Gigabyte X570 Master here, which has excellent IOMMU grouping. Check out r/VFIO or the Level1Techs forum for more info on what boards are best for this – or advice as they are great communities.
You will also need 2 GPUs, one for your host OS, in our case that’ll be Ubunutu, and one for your VM. In my case I’ve got a 2080ti and a 2060, but you can use what you like. In theory AMD cards are potentially better as driver support is more solid, but NVIDIA cards are generally more plentiful including on the used market and they do still work (unlike if you were building a hackintosh) so it’s up to you.
Either way, we will need to create a bootable USB stick with ubuntu on it – I’m using 20.04LTS here, and to make the bootable USB I’m using a tool called Rufus. It’s free and simple to use, just pick the ISO file, select your USB stick, and click start. Let it do it’s thing, then we can plug it into our VFIO system.
When you boot the PC up, you’ll want to spam the DELETE key to get into the BIOS as there are some settings we need to change first. You’ll want to enable XMP if you haven’t already, then find the setting for IOMMU – there might actually be 2, set them to enabled. You’ll also want to find “SVM” on AMD, or “VT-x” on Intel, and enable it. That lets us run virtualisation on the CPU well. Once that’s done, boot to the USB stick (often F12 to see boot options, or set it to first priority in the BIOS itself), then install Ubuntu.
A key word of warning, if you are using NVIDIA GPUs, whatever you do, do not press the “automatically log in” or the “download third party drivers” checkboxes. I learned the hard way that bricked the OS for me, I think due to the version of the third part NV driver it installs and a bug in 20.04.
Once you’ve got it installed, we can get to work on the passthrough. I like to check the IOMMU groupings first before we get too far, so the first thing to do is enable it in your grub config file. Run: “sudo nano /etc/default/grub” and navigate to the line “GRUB_CMDLINE_LINUX_DEFAULT” and inside the quotes add “amd_iommu=on iommu=pt”. Use CTRL+O to write out the file, press enter to save, then CTRL+X to exit. Then run “sudo update-grub” and reboot the system once that’s finished. Once you are back in, you can run “dmesg |grep AMD-Vi”, which should read something like this.
Then you can create this bash script to check your groupings. In terminal, type “nano iommugroups.sh” then copy and paste, or type out if you really want, this script. CTRL+O, enter and CTRL+X to save and exit, then use “sudo chmod 755 iommugroups.sh” then “./iommugroups.sh”. That should give you an output like this, which will tell you what group your components are in.
What we are looking for is our secondary GPU to be in a group on its own, which ours is. It’s technically in a group with the audio and USB controller that is built into the card, but that’s fine as we will pass those through too. To save time, write down the PCI IDs of those parts, those will look like this “10de:1e07”. I’m only passing through the GPU as it has a USB C port that does both display and USB, but if you want to pass a USB hub too, find one in it’s own group, or install a PCIe one that will be in either the same group as the GPU, or on it’s own.
So, we can now start installing all the tools we’ll need. Run this, “sudo apt install qemu-kvm qemu-utils libvirt-daemon-system libvirt-clients bridge-utils virt-manager ovmf”. It’s also a good idea to “sudo apt update” and “sudo apt upgrade” while we are at it too.
We can now go back to our grub file, “sudo nano /etc/default/grub” and add to the same line we modified earlier, “kvm.ignore_msrs=1” as we are using a Windows version newer than 1803 and we don’t want it blue screening, and “vfio-pci.ids=” and the ids of the devices we want to passthrough. Separate them with a comma. Write out the file, exit, then “sudo update-grub” again, and reboot. Remember you will now only have access to the host GPU, the one we didn’t passthrough, from now on. You will want the secondary GPU connected to a display though.
You can verify the passthrough worked by running “lspci -nnv”, finding your GPU and there should be a line that reads “Kernel Driver in use: vfio-pci”. If not, check you got the PCI ID right, and that it’s plugged in to a display.
Now we are ready to make our VM. You can do it through terminal, but using “virt-manager” is a lot easier. Make sure you have a Windows ISO ready, you can download the media creation tool on a Windows machine and create the ISO there, and download the virtio drivers from the link in the description as you will need both. Select the Windows ISO file as the ISO image, pick how many cores you want to give it, and how much RAM, in my case I’m giving it half of each. You can use a whole drive if you’d like, although I’m just using a virtual image instead. You can also create an ethernet bridge, although NAT passthrough works fine for getting it up and running.
Once it is created, it may start. Stop it, and go to the configuration page and add a storage device. Select the Virtio ISO. You can then press the add hardware button again and add a PCIe device, selecting all the devices we’ve passed through, in my case the card, it’s USB controller, and the audio controller.
There are some other optimisations you might need to do, but if you are using an NVIDIA card the one you will need to do is fixing the Code 43 error – that’s a driver level thing that NVIDIA put in place because you didn’t pay them extra for a Quadro. Terrible, I know, but easily defeated (for now).. Go to the XML section, and press “edit XML”, then under , add “” and under add “”.
Then you can start the system. You should see Windows boot up through your second GPU, you’ll need to plug a second mouse and keyboard into the passed through USB ports, or use a KVM like the one in the M27Q. Either way, that’s it up and running. Install the NVIDIA drivers and you are ready to go. There are some Ryzen optimisations you might want to do that’ll be linked in the description, as well as Mathias Hueber’s excellent written guide for all of this which includes plenty of extra details. It is written for a slightly older version of QEMU and Virt-Manager, but still very relevant.
