Container Storage Deep Dive: Tuning OverlayFS and NVMe Mounts on Linux

OverlayFS is the default storage driver for Docker, Podman, and containerd on modern Linux systems. It merges multiple directory trees into a single coherent filesystem view using an upper read-write directory and a stack of lower read-only directories. When a process inside a container modifies a file that originated from an image layer, the Linux kernel performs a Copy-on-Write (CoW) operation, copying the entire file from the lower layer to the upper layer before applying the change.

While this mechanism provides rapid container instantiation and memory deduplication across shared base layers, unoptimized I/O in the upper layer can degrade disk performance, exhaust inodes, and saturate disk queues during write-heavy operations.

Under the hood, OverlayFS mounts four distinct paths for each active container:

• **lowerdir:** A colon-separated list of read-only image layers containing the base OS and application files.
• **upperdir:** The writable layer specific to the container where all newly created, modified, or deleted files reside.
• **workdir:** An internal working directory required by the kernel for atomic filesystem operations (such as renaming or creating whiteout files).
• **merged:** The final unified mount point presented to the container as its root filesystem (/).

When an application deletes a file that exists in lowerdir, OverlayFS cannot alter the immutable image layer. Instead, it creates a special character device called a "whiteout" file (with device number 0/0) in upperdir that masks the lower file from directory listings.

Expand

Kernel Copy-on-Write Mechanics and Metacopy Optimization

When an application modifies only the metadata of a file (such as permissions with chmod or timestamps with touch), standard OverlayFS copies the entire file content up to UpperDir. For large files (such as a 2GB database seed or cache archive), this triggers severe write amplification and disk latency spikes. Enabling the kernel metacopy feature instructs OverlayFS to copy only metadata, creating a lightweight redirect inode in upperdir without duplicating the underlying data blocks:

# Check if your kernel supports OverlayFS metacopy
cat /sys/module/overlay/parameters/metacopy

# Enable metacopy at kernel boot or module load time
echo 'options overlay metacopy=on' | sudo tee /etc/modprobe.d/overlay.conf
sudo update-initramfs -u

Configuring Docker Storage Options and Kernel Dirty Ratios

To ensure maximum I/O throughput on high-speed SSD drives, configure /etc/docker/daemon.json and tune the Linux kernel virtual memory subsystem to handle rapid bursts of container disk writes without stalling application threads:

{
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ],
  "data-root": "/var/lib/docker"
}

Apply kernel virtual memory tuning in /etc/sysctl.d/99-storage.conf:

# Smooth out disk write flushes for high-throughput SSDs
cat << 'EOF' | sudo tee /etc/sysctl.d/99-storage.conf
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
vm.dirty_expire_centisecs = 3000
vm.dirty_writeback_centisecs = 500
EOF

sudo sysctl --system

The parameter vm.dirty_background_ratio = 5 instructs the Linux pdflush/flush kernel threads to begin asynchronously committing dirty memory pages to disk as soon as dirty memory hits 5 percent of system RAM. Setting vm.dirty_ratio = 10 forces synchronous write blocking only when dirty memory exceeds 10 percent, preventing sudden multi-gigabyte disk flushes that freeze container process threads.

Inspecting Layer Sprawl and Inode Health

Containers that write millions of small temporary files to the writable layer can rapidly consume filesystem inodes even when available disk gigabytes appear healthy. Always inspect inode utilization:

# Check inode utilization across all mounted filesystems
df -i /var/lib/docker

# Inspect OverlayFS layer mount tree for active containers
docker inspect --format '{{json .GraphDriver.Data}}' $(docker ps -q) | jq .

# Identify containers with bloated writable UpperDirs
docker ps -s --format 'table {{.Names}}	{{.Size}}'

For database engines (PostgreSQL, MySQL, Redis) and logging agents, bypass OverlayFS entirely by using dedicated bind mounts or Docker named volumes. Named volumes write directly to the underlying host filesystem, circumventing the Copy-on-Write penalty and achieving raw SSD performance.

Deploying high-throughput workloads on Netbay takes under 60 seconds with Intel Xeon Platinum compute and high-speed SSD storage in Lucknow — netbayhosts.in.