NUMA Architecture Tuning: CPU Pinning and Memory Affinity

Modern multi-socket cloud servers feature Non-Uniform Memory Access (NUMA) architecture. In a dual-socket system powered by enterprise Intel Xeon Platinum processors, each physical CPU socket controls its own integrated memory controllers and local DDR4/DDR5 memory channels. While any CPU core can access memory attached to the opposing socket via Ultra Path Interconnect (UPI) links, remote memory access suffers from approximately 1.8x higher latency and consumes valuable inter-socket bandwidth.

When operating high-performance databases, cache engines (such as Redis or Memcached), and virtual machine hypervisors, careless thread placement across NUMA boundaries creates unpredictable latency jitter. Aligning thread execution with local memory allocation through NUMA pinning eliminates remote cross-talk penalties.

Expand

Inspecting System NUMA Topology

Use numactl to inspect the physical socket layout, CPU core assignments, and distance matrices on your host:

# Inspect NUMA hardware distribution
numactl --hardware

# Sample output:
# available: 2 nodes (0-1)
# node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23
# node 0 size: 64182 MB
# node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31
# node 1 size: 64512 MB
# node distances:
# node   0   1
#   0:  10  21
#   1:  21  10

The distance value 10 represents local memory access, while 21 indicates an access penalty traversing the interconnect.

Pinning Applications with numactl and taskset

To guarantee that an in-memory database process runs strictly on Node 0 with 100% local memory allocations:

# Launch PostgreSQL or Redis bound strictly to NUMA Node 0
numactl --cpunodebind=0 --membind=0 redis-server /etc/redis/redis.conf

# Verify active NUMA memory distribution for the running process
numastat -p $(pgrep redis-server)

Inspect numastat to ensure numa_miss and numa_foreign counters remain at 0, confirming that all memory allocations occurred on local memory channels.

Declarative NUMA Pinning in systemd

In production environments, configure NUMA affinity directly within the systemd service unit:

# /etc/systemd/system/high-perf-app.service
[Unit]
Description=NUMA-Aware Low Latency Engine
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/worker
CPUAffinity=0-7 16-23
NUMAPolicy=bind
NUMAMask=0

# Ensure memory cannot spill across nodes
LimitMEMLOCK=infinity

CPU Caching and Hierarchy Video

Explore computer memory hierarchies, CPU caches, and bus interconnects in this deep-dive guide:

Managing Automatic NUMA Balancing

The Linux kernel includes an automatic NUMA balancing thread (numabalancing) that periodically scans process page tables, unmaps pages, and generates minor page faults to migrate pages closer to the CPU executing the code. While beneficial for general workloads, this scanning introduces periodic 1-2ms latency spikes. For dedicated databases pinned with numactl, disable automatic balancing via sysctl:

# Disable kernel automatic NUMA balancing to eliminate scanning overhead
echo 0 | sudo tee /proc/sys/kernel/numa_balancing
echo "kernel.numa_balancing = 0" | sudo tee /etc/sysctl.d/99-numa.conf
sudo sysctl -p /etc/sysctl.d/99-numa.conf

Run compute-heavy applications with optimal CPU affinity on Netbay cloud servers at netbayhosts.in.