NVMe Storage Performance: io_uring and Queue Depth Profiling

Modern NVMe solid-state drives possess immense parallel capabilities: enterprise NVMe controllers support up to 64,000 independent hardware queues with each queue capable of holding up to 64,000 commands simultaneously. However, legacy Linux asynchronous I/O architectures (such as POSIX aio and libaio) were built for single-queue spinning disks. libaio requires multiple kernel syscalls per I/O submission, requires unbuffered O_DIRECT access, and frequently falls back to blocking behavior during file metadata updates.

io_uring, created by kernel engineer Jens Axboe, replaces legacy I/O subsystems with two lockless ring buffers shared directly between user space and kernel space: the Submission Queue (SQ) and Completion Queue (CQ). By eliminating context switches and system call overhead, io_uring unlocks the true hardware potential of high-speed NVMe storage.

Expand

Benchmarking io_uring against libaio with FIO

To measure the real-world throughput difference between legacy libaio and modern io_uring on high-speed NVMe storage, run identical 4KB random read tests:

# Benchmark 1: Legacy libaio engine
fio --name=libaio-test   --filename=/dev/nvme0n1   --direct=1   --rw=randread   --bs=4k   --ioengine=libaio   --iodepth=64   --numjobs=4   --runtime=30   --time_based   --group_reporting

# Benchmark 2: Modern io_uring engine
fio --name=io_uring-test   --filename=/dev/nvme0n1   --direct=1   --rw=randread   --bs=4k   --ioengine=io_uring   --iodepth=64   --numjobs=4   --runtime=30   --time_based   --group_reporting

Across identical hardware, io_uring regularly delivers 25% to 40% higher IOPS with significantly reduced CPU utilization due to the avoidance of enter/exit syscall interrupts.

Tuning Block Layer Queue Depths

For Linux to submit sufficient commands to keep the NVMe controller queues full, tune the block layer request queue settings in sysfs:

# Inspect current request queue length
cat /sys/block/nvme0n1/queue/nr_requests

# Increase queue depth to 1024 for high-throughput database workloads
echo 1024 | sudo tee /sys/block/nvme0n1/queue/nr_requests

# Verify I/O scheduler is set to 'none' on fast NVMe drives
# (Hardware multi-queue controllers do not benefit from CPU-bound elevator schedulers)
cat /sys/block/nvme0n1/queue/scheduler
# Expected: [none] mq-deadline

Setting the scheduler to none delegates all queuing and sorting directly to the NVMe controller's internal hardware queues.

Modern Filesystem Architecture Video

Review how operating systems interface with high-speed persistence hardware:

Activating io_uring Kernel Polling (IORING_SETUP_SQPOLL)

For ultra-low-latency applications (such as high-frequency trading or real-time event brokers), io_uring can be initialized with IORING_SETUP_SQPOLL. In this mode, the kernel creates a dedicated kernel thread that continually polls the Submission Ring buffer in memory. The user application writes I/O entries into the ring buffer without executing any system call whatsoever, achieving sub-10 microsecond read latencies.

Unlock maximum NVMe SSD IOPS on dedicated Intel Xeon Platinum nodes at netbayhosts.in.