Network Interface Queue Tuning: RSS, RPS, and XPS Guide

On high-traffic cloud instances handling hundreds of thousands of network connections, network interface cards (NICs) become the primary source of CPU contention. When a network packet arrives, the NIC raises a hardware interrupt (IRQ) to alert the processor. In unoptimized configurations, all incoming packet interrupts are routed to CPU Core 0, leaving one CPU core at 100% utilization in softirq while remaining cores sit completely idle.

Scaling packet processing across multi-core processors requires coordinating three key mechanisms: Receive Side Scaling (RSS, hardware multi-queue hashing), Receive Packet Steering (RPS, software packet distribution for virtual NICs), and Transmit Packet Steering (XPS, pinning outbound transmit queues to the CPU core generating the data).

Expand

Checking and Configuring Hardware Multi-Queue with ethtool

Use ethtool to verify how many hardware channels (queues) your interface supports:

# Inspect current and maximum supported queues
sudo ethtool -l eth0

# Sample output:
# Channel parameters for eth0:
# Pre-set maximums:
# RX:      0
# TX:      0
# Other:   0
# Combined: 8
# Current hardware settings:
# Combined: 4

If Combined is less than the number of vCPU cores on your system, increase the active queue count:

# Expand queue channels to 8 to match available vCPUs
sudo ethtool -L eth0 combined 8

Configuring Receive Packet Steering (RPS) for Virtual NICs

When operating on cloud virtual network devices (such as virtio-net) with a single hardware queue, configure software Receive Packet Steering (RPS) to distribute packet processing across all CPU cores:

# Calculate CPU core bitmask for an 8-core server:
# Cores 0-7 = binary 11111111 = hex 'ff'
echo "ff" | sudo tee /sys/class/net/eth0/queues/rx-0/rps_cpus

# Increase flow limit to buffer traffic bursts
echo 4096 | sudo tee /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
echo 32768 | sudo tee /proc/sys/net/core/rps_sock_flow_entries

Now the kernel hashes incoming TCP connections and distributes software interrupt processing evenly across all 8 cores.

Transmit Packet Steering (XPS) for Cache Locality

While RPS handles incoming packets, Transmit Packet Steering (XPS) ensures that outbound packets sent by a thread running on CPU Core 2 use Transmit Queue 2. This prevents CPU cores from contending over transmit locks and eliminates cross-core CPU cache invalidations:

# Map TX queues 1:1 to their corresponding CPU core bitmasks
# Queue 0 -> Core 0 (0x1), Queue 1 -> Core 1 (0x2), Queue 2 -> Core 2 (0x4), Queue 3 -> Core 3 (0x8)
echo 1 | sudo tee /sys/class/net/eth0/queues/tx-0/xps_cpus
echo 2 | sudo tee /sys/class/net/eth0/queues/tx-1/xps_cpus
echo 4 | sudo tee /sys/class/net/eth0/queues/tx-2/xps_cpus
echo 8 | sudo tee /sys/class/net/eth0/queues/tx-3/xps_cpus

Packet Processing Video

Learn how packets traverse network queues and socket buffers inside the operating system:

Verifying Balanced Interrupt Distribution

Monitor live hardware interrupt distribution using mpstat and /proc/interrupts:

# Display per-core softirq utilization (%soft)
mpstat -P ALL 1 5

# Inspect active network interrupt counters across all cores
watch -n 1 "cat /proc/interrupts | grep eth0"

With RSS and RPS tuned, %soft is distributed evenly across all CPU cores, eliminating Core 0 saturation.

Achieve low jitter and maximum packet throughput on Netbay DDoS-protected cloud servers at netbayhosts.in.