NVMe over Fabrics (NVMe-oF): Implementing Low-Latency Network Storage

For decades, network storage in datacenters relied on legacy protocols like iSCSI or Fibre Channel. While these systems allowed organizations to pool storage arrays across multiple servers, they were designed in the era of spinning hard disks. They utilize legacy SCSI commands, introducing significant protocol latencies and processing bottlenecks when paired with modern solid-state drives.

With the development of NVMe (Non-Volatile Memory Express), storage read/write limits shifted from drive mechanics to bus speeds. To extend these flash speeds across network boundaries, the storage industry developed NVMe over Fabrics (NVMe-oF). By transporting native NVMe commands over high-speed networks, NVMe-oF delivers network storage with latencies matching local PCIe-mounted drives.

This guide explores the architecture of NVMe-oF, comparing network transport layers, configuring Remote Direct Memory Access (RDMA), and analyzing latency metrics.

---

Legacy SCSI vs. Native NVMe Protocols

To understand why legacy storage arrays degrade solid-state drive performance, we must compare the command queues of legacy SCSI systems with the NVMe architecture:

1. Legacy SCSI Queue Limits

The SCSI command set, designed for hard disks with rotating platters and read heads, supports:

• A Single Command Queue: All read/write requests wait in a single line.
• Queue Depth of 64 Commands: If a server sends more than 64 commands simultaneously, the storage controller drops connection speeds to prevent queue crashes.
• High Lock Contention: Multi-core CPUs must synchronize access to the single queue, wasting cycles on lock states.

2. Native NVMe Queue Parallelism

NVMe was designed specifically for non-volatile flash memory, supporting:

• 64,000 Parallel Command Queues: Each CPU core can have its own dedicated write pathway.
• 64,000 Commands Per Queue: Enabling massive parallel processing.
• Lock-Free Access: Bypassing OS kernel locks to write directly to flash channels.

NVMe-oF maps this parallel command set directly across network fabrics, allowing remote servers to read and write to storage targets with minimal protocol conversion overhead.

---

Fabric Transport Layers: RDMA, TCP, and Fibre Channel

NVMe-oF can be implemented across diverse network transport fabrics, each offering different cost-to-performance profiles:

                  NVMe over Fabrics (NVMe-oF) Transport Layers
                                      |
         +----------------------------+----------------------------+
         |                            |                            |
    NVMe-oF / RDMA              NVMe-oF / TCP               NVMe-oF / FC
   (RoCE or iWARP)            (Standard Ethernet)         (Fibre Channel SAN)
   * Sub-10μs Latency         * Standard Hardware         * Legacy Integration
   * Zero-Copy Transfers      * High CPU Overhead         * High Cost Hardware

1. NVMe-oF over RDMA

Remote Direct Memory Access (RDMA) allows two servers to read and write directly to each other’s memory buffers without involving either operating system’s kernel or CPU, achieving zero-copy data transfers.

• RoCE (RDMA over Converged Ethernet): Transports RDMA packets over standard Ethernet networks. It requires a lossless network configuration (Priority Flow Control, PFC), demanding specialized enterprise switches.
• iWARP: Transports RDMA packets over standard TCP/IP streams. iWARP is more robust and runs on standard switches, but it introduces higher latency compared to RoCE.

2. NVMe-oF over TCP

NVMe-oF over TCP transports NVMe commands inside standard TCP packets over standard Ethernet hardware. This eliminates the need for specialized RDMA network interface cards (RNICs) or switches, making it easy to deploy on existing infrastructure. However, it lacks zero-copy capabilities, placing a higher processing load on the host CPU and increasing network latency.

---

Network Storage Protocols Comparison

The following table compares the performance profiles of the primary network storage transport options:

Storage Metric	Legacy iSCSI (Ethernet)	NVMe-oF / TCP (Ethernet)	NVMe-oF / RDMA (RoCEv2)
Command Protocol	SCSI command set	Native NVMe commands	Native NVMe commands
Transport Layer	Standard TCP/IP	Standard TCP/IP	RDMA (Hardware bypass)
Zero-Copy Support	No (Kernel copy)	No (Kernel copy)	Yes (Direct memory writes)
Protocol Latency	High (50μs – 150μs)	Medium (15μs – 30μs)	Low (sub-10μs)
CPU Overhead	Medium	High (TCP stack processing)	Low (Bypasses host CPU)

---

Key Takeaways

• SCSI Bottleneck: Legacy storage protocols limit modern flash speeds due to single queue structures and high CPU lock overheads.
• Protocol Mapping: NVMe-oF extends local PCIe speeds across network fabrics, utilizing parallel queue layouts to handle thousands of concurrent requests.
• RDMA Efficiency: Transporting NVMe-oF over RDMA (RoCE) bypasses guest operating system kernels, delivering sub-10 microsecond latencies.

---

FAQ

Here are answers to the most frequently asked questions about this topic:

What is the latency impact of NVMe-oF over TCP compared to RDMA?

NVMe-oF over TCP typically adds 10 to 20 microseconds of latency compared to RDMA transport. While this makes it slightly slower, it remains significantly faster than legacy iSCSI protocols and is much cheaper to implement because it runs on standard network switches.

Can NVMe-oF run on standard consumer networks?

While NVMe-oF over TCP can technically run on standard gigabit networks, you will not experience its speed benefits. To realize the latency advantages of NVMe-oF, you need network links of at least 10 Gbps (preferably 25 Gbps or higher) to avoid bandwidth bottlenecks.

---

Related Inquiries

• Learn more about Self-Hosting Independent Database Vaults: Cloud vs. Co-location.
• Learn more about loss functions.
• Learn more about vector databases.