Hardware & PCI Passthrough

Vapor allows you to assign physical hardware devices directly to Virtual Machines using PCIe Passthrough (VFIO).

Prerequisites

To use PCI Passthrough, your host hardware must support IOMMU (Intel VT-d or AMD-Vi), and it must be enabled in the host's BIOS and Kernel parameters.

Assigning a PCI Device

1. Ensure the VM is powered off.
2. Navigate to the VM's Hardware tab.
3. Click Add PCI Device.
4. Vapor will list all available PCI devices on the host (e.g., GPUs, Network Cards, NVMe controllers).
5. Select the device and save.

WARNING

Assigning a PCI device to a VM removes that device from the host OS. Do not assign the network card you are using to access the Vapor web console, or you will lose connectivity!

Hotplugging

Vapor supports hotplugging for certain virtual hardware components without powering down the VM:

• Disks: You can attach new virtual disks to a running VM.
• Network Interfaces: You can add or remove virtual network interfaces.

(Note: CPU and Memory hotplugging require specific Guest OS support and may not be available for all OS variants).

---

NVIDIA vGPU (Mediated Devices)

PCI Passthrough gives one whole GPU to one VM. NVIDIA vGPU slices a single enterprise GPU into several virtual GPUs so multiple VMs can share it at once — for example, eight VMs each receiving a 2 GB slice of a 16 GB Tesla T4.

Each slice is a mediated device (mdev): an instance created on the host from a profile that fixes its framebuffer size, display heads and maximum resolution.

Passthrough or vGPU?

	PCI Passthrough	vGPU
VMs per GPU	One	Many
GPU model	Any	NVIDIA vGPU-capable only
Extra licensing	None	NVIDIA vGPU software required
Host driver	None (device is hidden from the host)	NVIDIA vGPU Manager
Guest driver	Standard NVIDIA driver	NVIDIA vGPU (GRID) guest driver

IMPORTANT

A GPU cannot do both at once. While a card is assigned to a VM as a passthrough device it is bound to vfio-pci and cannot serve vGPU, and the reverse is also true.

A. Requirements

• A vGPU-capable NVIDIA GPU — Tesla T4, V100, A100, H100, L40, RTX 6000 Ada and similar datacenter cards. Consumer GeForce cards do not support vGPU.
• An NVIDIA vGPU software licence and entitlement, which gives you access to the NVIDIA Licensing Portal.
• IOMMU enabled on the host (Intel VT-d or AMD-Vi), as for PCI passthrough.
• Host kernel headers and build tools, because the driver is compiled for your running kernel.

TIP

On GPUs based on Ampere and newer architectures, SR-IOV virtual functions must be enabled on the host before vGPU profiles appear (/usr/lib/nvidia/sriov-manage -e <pci_address>), and instances are created against the virtual function rather than the physical card. Older cards such as the Tesla T4 do not need this step.

B. Obtaining the driver

Vapor cannot ship or download NVIDIA's drivers for you. They are licensed software and NVIDIA's portal requires your own account — so you download them under your entitlement and upload them to Vapor.

1. Sign in to the NVIDIA Licensing Portal with your enterprise account.
2. Download the NVIDIA vGPU software package for Linux KVM. It is a single archive containing both host and guest drivers.
3. Extract it. You need two files:
   • Host driver: NVIDIA-Linux-x86_64-<version>-vgpu-kvm.run — installed on the Vapor host.
   • Guest driver: from the Guest_Drivers folder — installed inside each VM.

WARNING

The host and guest drivers must come from the same release branch. A host driver of 550.144.02 pairs with a guest driver of 550.144.03. Mixing branches — for example a 570.x guest driver against a 550.x host — is not supported and will fail. Because NVIDIA's archives contain several versions with near-identical filenames, check the version before you upload.

C. Uploading the driver to the host

1. Navigate to Third-Party Libraries.
2. Find the NVIDIA vGPU Manager tile.
3. Choose the *-vgpu-kvm.run host driver and click Upload.

Uploads are resumable, so a dropped connection continues where it left off rather than restarting the transfer.

D. Checking host readiness

The NVIDIA vGPU Manager tile lists every prerequisite with its current state. Each is checked independently, so you can see exactly which one is standing in the way:

Check	What it means
NVIDIA GPU present	A vGPU-capable card was found on the PCI bus.
IOMMU enabled	The host booted with intel_iommu=on or amd_iommu=on.
GPU free for the vGPU driver	The card is not bound to vfio-pci. If it is, the check names the VM passing it through.
Kernel headers for the running kernel	Headers matching the booted kernel are installed.
Build toolchain	gcc, make and dkms are available.
Secure Boot	Secure Boot is not blocking unsigned modules.
nouveau not holding the GPU	The open-source nouveau driver is not loaded.
vGPU Manager package uploaded	A host driver package is stored on the host.
vGPU driver active	The driver is loaded and GPUs are advertising profiles.

Resolve anything reported as missing before installing. Two are worth calling out:

• Kernel headers. Install the package matching your running kernel — linux-headers-$(uname -r) — not the generic linux-headers-amd64 metapackage, which may track a newer kernel than the one you have booted. The driver builds against the running kernel and will fail if they differ.
• GPU still in use. If the card is passed through to a VM, stop that VM and release the card:

  virsh nodedev-reattach pci_0000_81_00_0

E. Installing the host driver

1. In the NVIDIA vGPU Manager tile, find Guided install.
2. Click Install driver, then confirm.

The install builds a kernel module, loads it, starts the nvidia-vgpud and nvidia-vgpu-mgr services, and confirms that vGPU profiles have appeared. Progress is shown as it runs and typically completes in one to two minutes. If it fails, expand Show log — the build output explains why.

CAUTION

Installing replaces the running NVIDIA kernel modules. Vapor refuses to start while any vGPU instance is active, because unloading the driver would take the GPU away from a running VM. Stop those VMs and remove their vGPU instances first.

When it succeeds you will see a message such as "vGPU Manager installed — 1 GPU(s) offering 14 profiles", and the vGPU driver active check turns green. A reboot is usually not required; if one is needed, Vapor says so explicitly.

F. Adding a vGPU to a VM

vGPUs are configured in the VM wizard, for both new and existing machines.

1. Create a new VM, or edit an existing one that is powered off.
2. Go to Step 4 — Hardware, then Advanced Configuration.
3. Under NVIDIA vGPU (Mediated Device), click + Add vGPU (Mediated Device).
4. Choose a profile. The picker is searchable by name, profile ID, VRAM size or GPU, and shows how many slices of each remain. Profiles with no capacity left cannot be selected.
5. Leave Existing mdev Device UUID empty — Vapor creates the instance for you and destroys it when the VM is deleted. Fill it in only to attach an instance you created yourself.
6. Guest PCI Address is optional; leave it blank unless you need the device pinned to a specific slot inside the guest.
7. Save and finish the wizard.

IMPORTANT

vGPUs can only be added or removed while the VM is powered off. NVIDIA vGPU supports neither hot-plug nor hot-unplug, so these controls are disabled on a running VM. Changes made through the API against a running VM are written to its saved configuration and take effect at the next restart.

Profile naming

Profiles follow the pattern GRID <card>-<size><series>, for example GRID T4-2Q:

• Q — Virtual Workstation. Professional graphics and CUDA. The usual choice for compute.
• C — Compute Server. Compute-only, no display.
• B — Virtual PC. Office and browser workloads, no CUDA.
• A — Virtual Application. Application streaming.

The number is the framebuffer size in gigabytes, so T4-2Q is a 2 GB workstation profile and a 16 GB T4 can host eight of them.

G. Installing the guest driver

The VM sees the vGPU as a PCI device as soon as it boots, but it stays inert until the guest driver is installed inside the VM. This is an ordinary in-guest software installation, performed by the VM's own administrator — Vapor is not involved.

Before starting, confirm the guest can see the device:

lspci -nn | grep -i nvidia
# 05:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8]

If nothing is listed, the vGPU is not attached — return to section F.

TIP

Install the driver once in a golden image or template, then clone VMs from it. The driver survives cloning, so this whole section becomes a one-time job per image rather than per VM.

G.1 Getting the installer into the VM

Use whichever suits your environment:

• Over the network, if the VM has connectivity and SSH:

  scp NVIDIA-Linux-x86_64-<version>-grid.run user@<vm-ip>:~/

• From the host over HTTP, useful for VMs on a libvirt NAT network with no inbound SSH. On the host, serve the directory holding the guest driver:

  cd /path/to/Guest_Drivers && python3 -m http.server 8000 --bind 192.168.122.1

  Then inside the VM, where 192.168.122.1 is the libvirt gateway:

  curl -O http://192.168.122.1:8000/NVIDIA-Linux-x86_64-<version>-grid.run

  Stop the server when finished.
• As a virtual CD-ROM, if the VM has no networking: build an ISO containing the driver, upload it under ISO Images, and attach it to the VM.

G.2 Debian and Ubuntu

# Prerequisites. Use the headers matching the RUNNING kernel — the generic
# linux-headers-amd64 metapackage may track a newer kernel than the one booted,
# and the driver builds against the running one.
sudo apt-get update
sudo apt-get install -y dkms build-essential "linux-headers-$(uname -r)"

# Install the driver
sudo dpkg -i nvidia-linux-grid-<branch>_<version>_amd64.deb

sudo reboot

WARNING

On Debian, NVIDIA's .deb declares a dependency on a package called linux-headers, which does not exist in Debian's archive — the real packages are linux-headers-amd64 and linux-headers-<version>, and neither provides that name. This is an Ubuntu-ism in NVIDIA's packaging and will stop dpkg mid-install.

Resolve it cleanly with an equivs shim rather than forcing dpkg, which leaves the package database inconsistent:

sudo apt-get install -y equivs
cat > linux-headers-shim <<EOF
Section: misc
Priority: optional
Standards-Version: 3.9.2
Package: linux-headers
Version: $(uname -r)
Provides: linux-headers
Depends: linux-headers-$(uname -r)
Description: Satisfies the NVIDIA guest driver's bare linux-headers dependency
EOF
equivs-build linux-headers-shim
sudo dpkg -i linux-headers_*.deb

# Now the driver installs normally
sudo dpkg -i nvidia-linux-grid-<branch>_<version>_amd64.deb
sudo reboot

Alternatively, skip the packaging problem entirely by using the .run installer described in G.4.

G.3 RHEL, Rocky and AlmaLinux

sudo dnf install -y dkms gcc make "kernel-devel-$(uname -r)" "kernel-headers-$(uname -r)"
sudo rpm -ivh nvidia-linux-grid-<branch>-<version>.x86_64.rpm
sudo reboot

G.4 Any Linux distribution (.run installer)

The .run installer works anywhere and avoids distribution packaging quirks, but it does not update itself through your package manager.

sudo sh NVIDIA-Linux-x86_64-<version>-grid.run --dkms
sudo reboot

--dkms registers the module with DKMS so it rebuilds automatically when the guest kernel is updated. Without it, the driver stops loading after the next kernel upgrade.

If the installer reports a conflict with nouveau, let it write the blacklist, then reboot and run it again.

G.5 Windows

1. Copy the guest driver .exe into the VM.
2. Run it and accept the defaults.
3. Reboot.
4. Confirm the adapter appears in Device Manager under Display adapters.

G.6 Verifying

Inside the guest:

nvidia-smi

It should report the profile name and framebuffer size you selected, for example:

| GPU  Name                 Persistence-M | Bus-Id          Disp.A |
|   0  GRID T4-2Q                     On  |   00000000:05:00.0 Off |
|                                         |       1MiB /   2048MiB |

To confirm the guest is genuinely using a vGPU rather than a passed-through card:

nvidia-smi -q | grep -A2 "GPU Virtualization Mode"
#     Virtualization Mode : VGPU

On the host, the same session should be visible from the other side, which also confirms the host and guest drivers are talking to each other:

nvidia-smi vgpu
#   0  Tesla T4    | 00000000:81:00.0 |   0%
#      3251634273  GRID T4-2Q | 0c6a...  my-vm |   0%

G.7 Licensing the guest

Without a licence server the driver still loads and nvidia-smi works, reporting Unlicensed. Configure your Delegated License Service (DLS) or Cloud License Service (CLS) address in /etc/nvidia/gridd.conf inside the guest (or via the NVIDIA control panel on Windows) to license it fully.

H. Managing vGPU devices on the host

Virtualization → vGPU Devices shows every mediated device on the host, independently of any VM.

• Capacity per profile, with exhausted profiles dimmed.
• Assigned To — the VM holding each instance, or an Unassigned badge.
• Create a new instance ahead of time.
• Remove an instance to return its capacity to the GPU.

TIP

Removing a vGPU from a VM does not always destroy the underlying instance. Instances Vapor created for a VM are destroyed with it, but ones you created yourself are only released. An unassigned instance still consumes GPU capacity, so check this page if free slices seem to be missing.

I. Troubleshooting

No vGPU profiles appear in the wizard. Check the NVIDIA vGPU Manager tile. The most common cause is that the vGPU Manager host driver is not installed — the standard NVIDIA datacenter driver and the open-source nouveau driver do not provide vGPU, and only the vgpu-kvm package creates the profile tree.

"No capacity left for profile … available_instances is 0". On GPUs without SR-IOV (including the Tesla T4) a card is locked to a single framebuffer size while any instance exists. If you have 2 GB instances running, only other 2 GB profiles can be created. To switch sizes, remove every existing instance on that GPU first — including any unassigned ones, which are visible under vGPU Devices.

The VM will not start after a host reboot. Vapor creates instances persistently so they are recreated automatically at boot. An instance created outside Vapor — with a direct sysfs write, for example — is transient and disappears on reboot, leaving the VM referencing a device that no longer exists. Check with:

mdevctl list -d

Instances marked auto are recreated at boot.

nvidia-smi is missing inside the VM. The guest driver is not installed. See section G.

The guest driver installs but nvidia-smi reports an error. The host and guest driver branches probably do not match. Compare nvidia-smi on the host with the guest driver version — they must be from the same branch.