Back

02_Hardware_Based_Virtualization

Loading views...
← PreviousNext →

Hardware-Based Virtualization

Deep Technical Analysis

Table of Contents

  1. CPU Virtualization
  2. Memory Virtualization
  3. I/O Virtualization
  4. GPU Virtualization
  5. Security Features

CPU Virtualization (Processor-Level)

Hardware Virtualization Extensions

CPU virtualization extensions allow the processor to directly support virtualization, eliminating the need for expensive binary translation.

Intel VT-x (Virtualization Technology)

VMX (Virtual Machine Extensions)

Introduces two modes of operation:

┌─────────────────────────────────────┐
│         VMX Root Mode               │
│   Ring 0: Hypervisor runs here      │
│   Ring 3: Host user processes       │
├─────────────────────────────────────┤
│       VMX Non-Root Mode             │
│   Ring 0: Guest OS kernel           │
│           (thinks it's in Ring 0)   │
│   Ring 3: Guest user applications   │
└─────────────────────────────────────┘

Key Components

VMCS (Virtual Machine Control Structure)

  • Data structure storing guest and host state
  • Controls VM execution and exit conditions
  • Separate VMCS for each virtual CPU
  • Contains:

- Guest state area (registers, segment selectors)

- Host state area (loaded on VM exit)

- VM execution control fields

- VM exit control and information fields

VM Entry/Exit Operations

  • VM Entry (VMLAUNCH/VMRESUME): Transition from host to guest
  • VM Exit: Guest → Host transition on privileged operations
  • Exit reasons include: I/O instructions, interrupts, exceptions, CPUID, etc.

Critical Instructions:

VMXON   - Enable VMX operation
VMXOFF  - Disable VMX operation
VMLAUNCH - Launch VM (first time)
VMRESUME - Resume VM execution
VMREAD/VMWRITE - Access VMCS fields
VMPTRLD  - Load VMCS pointer
VMPTRST  - Store VMCS pointer

AMD-V (AMD Virtualization)

SVM (Secure Virtual Machine)

  • Similar functionality to Intel VT-x with different implementation
  • Uses VMCB (Virtual Machine Control Block) instead of VMCS

Key Instructions:

VMRUN   - Start guest execution
VMSAVE  - Save guest state
VMLOAD  - Load guest state
STGI    - Set Global Interrupt Flag
CLGI    - Clear Global Interrupt Flag
SKINIT  - Secure kernel initialization

ARM Virtualization Extensions

ARM processors support virtualization through:

  • EL2 (Exception Level 2) - Hypervisor mode
  • Stage-2 Translation - Additional memory translation layer
  • Trapped operations automatically route to hypervisor
  • Hardware support added in ARMv7 and extended in ARMv8

Privilege Ring Architecture

Traditional x86 Rings (without virtualization):

Ring 0: Kernel (most privileged)
Ring 1: Device drivers (rarely used)
Ring 2: Device drivers (rarely used)
Ring 3: User applications (least privileged)

With Hardware Virtualization:

VMX Root Mode:
  Ring 0: Hypervisor (full control)
  Ring 3: Host user processes

VMX Non-Root Mode (Each VM):
  Ring 0: Guest OS kernel
  Ring 3: Guest applications

Sensitive Instructions Handling

The Problem:

  • x86 architecture has 17 sensitive but non-privileged instructions
  • These instructions don't naturally trap when executed in user mode
  • Examples: SGDT, SIDT, SLDT (read descriptor tables)
  • Without hardware support, these must be detected and emulated (slow)

Hardware Solution:

  • VT-x/AMD-V trap ALL sensitive instructions automatically
  • Automatic VM exit to hypervisor occurs
  • Hypervisor emulates the behavior
  • Results returned to guest transparently
  • No binary scanning or translation needed

Memory Virtualization

Memory Translation Layers

Without Virtualization:

Virtual Address → Physical Address
    (Single translation via page tables)

With Virtualization (Two-Level Translation):

Guest Virtual Address (GVA)
        ↓ Guest Page Tables
Guest Physical Address (GPA)
        ↓ EPT/NPT (Hypervisor)
Host Physical Address (HPA)

Extended Page Tables (EPT) - Intel

Architecture:

EPT provides hardware-managed second level of page translation:

EPT PML4 (512 entries)
        ↓
EPT PDPT (512 entries)
        ↓
EPT PD (512 entries)
        ↓
EPT PT (512 entries)
        ↓
Physical Page

How It Works:

  1. Guest OS manages its own page tables (GVA → GPA) freely
  2. Hypervisor maintains EPT structures (GPA → HPA)
  3. Hardware walks both tables during address translation
  4. No VM exits required for guest page table modifications

Benefits:

  • Guest OS operates normally without modification
  • Massive performance improvement over shadow page tables
  • Reduced hypervisor intervention
  • Supports large/huge pages for efficiency

EPT Violations:

Hardware triggers EPT violation when:

  • Access to unmapped GPA
  • Permission violations (read/write/execute)
  • Accessed/dirty bit tracking
  • Hypervisor handles violation and updates EPT

Nested Page Tables (NPT) - AMD

AMD's equivalent to Intel EPT:

  • Also called RVI (Rapid Virtualization Indexing)
  • Same two-level translation mechanism
  • gCR3: Guest page table base
  • nCR3: Nested page table base

Shadow Page Tables (Legacy)

Historical Approach Before EPT/NPT:

How it Works:
1. Hypervisor maintains shadow page tables
2. Direct mapping: GVA → HPA (bypassing GPA)
3. Guest page table writes trapped and emulated
4. Shadow tables kept synchronized with guest tables

Problems:

  • Very high VM exit overhead on page table updates
  • Complex maintenance logic
  • Memory overhead for shadow structures
  • Largely obsolete with EPT/NPT availability

Memory Management Features

Memory Ballooning

Mechanism:

1. Hypervisor wants memory from VM
         ↓
2. Balloon driver in guest inflates
         ↓
3. Guest OS releases pages to balloon
         ↓
4. Hypervisor reclaims physical pages
         ↓
(Reverse process deflates balloon)
  • No specific hardware required
  • Cooperative between guest and hypervisor
  • Guest OS can manage reclaimed memory
  • Common in VMware, KVM, Hyper-V

Transparent Huge Pages (THP)

  • Uses 2MB or 1GB pages instead of 4KB
  • Reduces TLB misses significantly
  • Better memory access performance
  • Hardware TLB supports huge page entries

Kernel Same-page Merging (KSM)

  • Introduced in Linux kernel 2.6.32 (2009)
  • Scans for identical memory pages across VMs
  • Merges duplicate pages using copy-on-write
  • Significant memory savings with similar VMs

NUMA Awareness

Modern multi-socket servers have Non-Uniform Memory Access:

NUMA Topology:
┌────────────────┐    ┌────────────────┐
│   Socket 0     │    │   Socket 1     │
│  ┌──────────┐  │    │  ┌──────────┐  │
│  │   CPU    │  │    │  │   CPU    │  │
│  └────┬─────┘  │    │  └────┬─────┘  │
│       │        │    │       │        │
│  ┌────▼─────┐  │    │  ┌────▼─────┐  │
│  │ Memory   │  │    │  │ Memory   │  │
│  │ Node 0   │  │    │  │ Node 1   │  │
│  └──────────┘  │    │  └──────────┘  │
└────────────────┘    └────────────────┘
        ↕ XBAR/QPI (slower) ↕

Virtualization Considerations:

  • Pin vCPUs to physical NUMA nodes
  • Allocate guest memory from same NUMA node
  • Reduce cross-NUMA memory traffic
  • Modern hypervisors are NUMA-aware

I/O Virtualization

Traditional I/O Virtualization (Emulation)

Full Device Emulation:

Guest Driver → Virtual Device → Hypervisor
                                    ↓
                              Physical Device
  • Software emulates real hardware (e1000, IDE, etc.)
  • High overhead due to many VM exits
  • Compatible with unmodified guests
  • Slow but fully functional

Para-virtualized I/O (VirtIO)

Architecture:

Guest VM:
  VirtIO Frontend Driver
    (virtio-net, virtio-blk)
         ↓
    VirtQueues (shared memory rings)
         ↓
Host/QEMU:
  VirtIO Backend
         ↓
  Physical Device

Components:

  • VirtQueues: Ring buffers in shared memory
  • Kick mechanism: Notify other side of data
  • Available ring: Buffers guest makes available
  • Used ring: Buffers host has processed

Benefits:

  • Guest knows it's virtualized (cooperative)
  • Batch multiple operations
  • Fewer VM exits
  • Much better performance than emulation

Intel VT-d (Virtualization Technology for Directed I/O)

Purpose:

  • Direct assignment of physical devices to VMs
  • DMA remapping for security
  • Interrupt remapping

DMA Remapping (IOMMU)

Device DMA Request
        ↓
IOMMU Translation
        ↓
Guest Physical Address
        ↓
EPT Translation
        ↓
Host Physical Address

Benefits:

  • Protects host memory from malicious DMA
  • Each device assigned to specific address space
  • Hardware-enforced isolation
  • Enables safe device passthrough

Interrupt Remapping

  • Maps device interrupts to specific VMs
  • Prevents interrupt injection attacks
  • Supports MSI/MSI-X interrupts

Posted Interrupts

  • Direct interrupt delivery to guest vCPU
  • No VM exit required for routine interrupts
  • Hardware posts interrupt directly to virtual APIC

SR-IOV (Single Root I/O Virtualization)

Concept:

Single physical device appears as multiple virtual devices with hardware-level isolation.

Physical Function (PF) - Managed by hypervisor
        ↓
├─ Virtual Function 1 (VF) → VM 1
├─ Virtual Function 2 (VF) → VM 2
├─ Virtual Function 3 (VF) → VM 3
└─ Virtual Function N (VF) → VM N

Components:

Physical Function (PF):

  • Full PCIe function with SR-IOV capability
  • Managed by hypervisor
  • Creates and manages VFs
  • Handles configuration

Virtual Functions (VFs):

  • Lightweight PCIe functions
  • Directly assigned to VMs
  • Independent queues and resources
  • Near-native performance (95%+ of bare metal)

Example: SR-IOV NIC:

Intel X710 10GbE NIC
├─ PF (Hypervisor manages)
└─ Up to 128 VFs
    ├─ VF0 → VM1 (direct DMA access)
    ├─ VF1 → VM2 (direct DMA access)
    └─ VF2 → VM3 (direct DMA access)

Benefits:

  • Near-native performance
  • Low latency
  • Direct DMA access
  • Hardware isolation

Limitations:

  • Live migration complexity (must re-attach VF)
  • Device-specific driver support
  • Limited VF count per device

GPU Virtualization

GPU Pass-through

  • Entire GPU assigned to single VM
  • Full performance, no sharing
  • Requires VT-d/AMD-Vi
  • One GPU per VM limitation

vGPU (NVIDIA GRID/vGPU)

Physical GPU
        ↓
NVIDIA vGPU Manager (in hypervisor)
        ↓
├─ vGPU1 → VM1 (1/4 GPU slice)
├─ vGPU2 → VM2 (1/4 GPU slice)
├─ vGPU3 → VM3 (1/4 GPU slice)
└─ vGPU4 → VM4 (1/4 GPU slice)

Hardware Requirements:

  • NVIDIA Tesla/Quadro GPUs
  • Can be time-sliced or hardware-partitioned
  • MIG (Multi-Instance GPU) on A100 provides true hardware isolation

Intel GVT-g

  • GPU virtualization for Intel integrated GPUs
  • Mediated pass-through
  • Para-virtualization with hardware assist

Security Features

AMD SEV (Secure Encrypted Virtualization)

Levels of Protection:

  • SEV: Encrypts VM memory with per-VM key
  • SEV-ES: Encrypted State (registers protected)
  • SEV-SNP: Secure Nested Paging (integrity protection)
Each VM gets unique AES encryption key
            ↓
Memory encrypted/decrypted by hardware
            ↓
Even hypervisor cannot read VM memory

Intel TDX (Trust Domain Extensions)

  • Intel's confidential computing technology
  • Hardware-isolated VMs (Trust Domains)
  • Memory encryption and integrity protection
  • Attestation support

Intel TXT (Trusted Execution Technology)

  • Secure boot for hypervisors
  • Measured launch environment
  • Protects against firmware rootkits

Hardware Virtualization Evolution Timeline

2005: Intel VT-x, AMD-V introduced
2006: Intel VT-d (IOMMU for I/O virtualization)
2008: Intel EPT, AMD NPT (memory virtualization)
2010: SR-IOV widespread adoption
2015: NVIDIA vGPU technology
2017: AMD SEV (encrypted virtualization)
2020: Intel TDX announced
2021: DPU/SmartNIC mainstream adoption
2022: AMD SEV-SNP, Intel TDX available

Modern Server Hardware Requirements

For production virtualization environments:

Feature Purpose Technology
VT-x/AMD-V CPU virtualization Required
EPT/NPT Memory virtualization Required
VT-d/AMD-Vi Device assignment Highly recommended
SR-IOV NICs High-performance networking Recommended
Hardware AES Storage/memory encryption Recommended
NUMA Memory locality Multi-socket servers
SEV/TDX Confidential computing Security-critical workloads

This hardware support enables modern virtualization to achieve 95%+ of bare-metal performance for most workloads.

Previous: 01IntroductiontoVirtualization.md

Next: 03HypervisorArchitectures.md

← PreviousNext →