Back

NVMe

Loading views...
← PreviousNext →

NVMe

Category: Storage

Overview

NVMe (Non-Volatile Memory Express) is a storage protocol for flash over PCIe, replacing SATA AHCI. It supports up to 65,535 queues of 65,535 commands each for massive parallelism, achieving ~100µs latency and >7 GB/s throughput on PCIe 4.0 x4.

1. Theory & Fundamentals

  • Interface: PCIe (x1/x2/x4); also NVMe-oF over TCP/RDMA
  • Latency: ~100 µs consumer; ~10 µs Optane/datacenter
  • Throughput: 7 GB/s read on PCIe 4.0 x4
  • Queues: 1 Admin + up to 65,535 I/O queues
  • Commands/queue: Up to 65,535 (vs AHCI single queue of 32)
  • Form factors: M.2 2280, U.2, PCIe AIC, EDSFF

2. Frame / Packet Structure

NVMe Submission Queue Entry (64 bytes):
  CDW0: Opcode(8b)|FUSE|PSDT|CID(16b)
  NSID: Namespace ID
  PRP1/PRP2: Data pointers (DMA)
  CDW10–CDW15: Command-specific

Completion Queue Entry (16 bytes):
  DW0: Result
  DW2: SQ Head | SQ ID
  DW3: CID | Phase | Status

Key opcodes:
  Admin: 0x06=Identify, 0x09=Get Features
  I/O: 0x01=Write, 0x02=Read, 0x00=Flush

3. Protocol Mechanics

  • Doorbell: Host writes tail pointer to signal new SQ entries
  • Completion: Controller writes CQE; asserts MSI-X interrupt
  • Phase bit: Alternates 0/1 to distinguish old/new completions
  • PRP: Physical Region Pages for scatter-gather DMA
  • Namespace: Logical storage partition (NSID)
  • Admin queue: Device config, firmware update, NS management

4. Hardware Implementation

  • M.2 slot: 2280 most common (PCIe x4)
  • SSDs: Samsung 990 Pro, WD Black SN850, SK Hynix P41
  • SoC: i.MX8M, RK3588, Raspberry Pi CM4 with PCIe
  • Industrial: M.2 SSD modules for embedded data logging
  • Host controller: CPU PCIe root complex + Linux NVMe driver

5. Register-Level / Configuration

// NVMe BAR0 register definitions
#define NVME_REG_CAP  0x00  // Capabilities
#define NVME_REG_CC   0x14  // Controller Config
#define NVME_REG_CSTS 0x1C  // Controller Status
#define NVME_REG_AQA  0x24  // Admin Queue Attrs
#define NVME_REG_ASQ  0x28  // Admin SQ Base Addr
#define NVME_REG_ACQ  0x30  // Admin CQ Base Addr

// Enable controller
*(uint32_t*)(bar0+NVME_REG_CC) = 0x00460001;
// Wait CSTS.RDY=1
while(!(*(uint32_t*)(bar0+NVME_REG_CSTS)&1));

6. Driver / Software Development

void NVMe_Read(uint32_t nsid, uint64_t slba,
               uint16_t nlb, void *buf) {
    NVMe_SQE cmd = {0};
    cmd.cdw0 = 0x02;        // Read opcode
    cmd.nsid = nsid;
    cmd.prp1 = virt_to_phys(buf);
    cmd.cdw10 = slba & 0xFFFFFFFF;
    cmd.cdw11 = slba >> 32;
    cmd.cdw12 = nlb - 1;    // 0-based
    submit_sq_entry(&cmd);
    ring_doorbell(io_sq_tail++);
    wait_cq_completion();
}

7. Debugging & Testing

  • nvme-cli: nvme id-ctrl /dev/nvme0, nvme smart-log, nvme format
  • fio: --ioengine=libaio --iodepth=32 --rw=randread --bs=4k
  • PCIe analyzer for link-level debug
  • Common issues: PCIe link not training; DMA coherency; IOMMU issues; command timeout

8. Real-World Applications

  1. High-performance embedded systems (edge AI, video)
  2. Industrial PCs and HMIs
  3. Network appliances
  4. Medical imaging storage
  5. Autonomous vehicle data recording

9. Advanced Topics & Edge Cases

  • NVMe-oF: NVMe over TCP or RDMA for disaggregated storage
  • ZNS (Zoned Namespace): Append-only zones for write ordering
  • Computational Storage: Execute code on NVMe device
  • Multi-path I/O: Two paths to same SSD for availability
  • PCIe 5.0: 14 GB/s per x4 for next-gen NVMe

10. Standards & Variants

Version Notes
NVMe 1.4 Current base
NVMe 2.0 Unified (base+ZNS+NVMe-oF)
NVMe-oF Over TCP/RDMA
NVMe ZNS Zoned namespace

💡 Practical Examples

Ex 1: nvme id-ctrl /dev/nvme0 — model, serial, firmware, queue depths.

Ex 2: nvme smart-log /dev/nvme0 — temperature, wear indicator, error count.

Ex 3: fio benchmark: fio --name=rr --ioengine=libaio --iodepth=32 --rw=randread --bs=4k --size=10G --filename=/dev/nvme0n1

🧪 Practice Questions

Beginner: 1) NVMe interface? 2) Queue count? 3) What is a namespace? 4) Replaces what? 5) M.2 form factor?

Intermediate: 1) SQ/CQ doorbell mechanism. 2) Phase bit purpose. 3) Bare-metal Identify command. 4) PRP vs SGL. 5) NVMe vs SATA latency.

Advanced: 1) Bare-metal NVMe driver on ARM. 2) NVMe-oF TCP target. 3) ZNS log device. 4) Command timeout debug. 5) Firmware update via admin queue.

Checklist

  • [ ] Understand NVMe queue model
  • [ ] Read BAR0 capability registers
  • [ ] Submit Admin Identify command
  • [ ] Submit I/O Read and Write
  • [ ] Handle completions correctly
  • [ ] Use nvme-cli for management
  • [ ] Benchmark with fio
  • [ ] Debug with PCIe analyzer
  • [ ] Implement multi-queue I/O
  • [ ] Understand ZNS and NVMe-oF
← PreviousNext →