Preventing Disk Exhaustion with Docker Logging Drivers and Rotation Policies

One of the most frequent causes of unplanned VPS downtime is disk exhaustion caused by unbounded container log files. By default, the standard Docker engine captures container stdout and stderr using the json-file logging driver. If an application enters a rapid crash loop or emits high-volume debug logs, the log file inside /var/lib/docker/containers/container-id/ grows indefinitely until the root filesystem hits 100 percent capacity.

Once a server runs out of disk space, essential operating system operations fail: temporary files cannot be created, systemd journals freeze, databases suffer write transaction errors, and administrators are frequently locked out of remote SSH sessions.

Understanding how Docker captures container output clarifies why this happens. When a containerized application writes to stdout or stderr, the Linux kernel forwards those writes through a pseudo-terminal (pty) or pipe directly to the dockerd or containerd process. The logging driver receives this raw stream, formats each line with a timestamp and stream identifier, and writes it to disk. In a default configuration without explicit file size or rotation limits, Docker never truncates or deletes historical log entries.

Expand

Global Engine Hardening via daemon.json

The most robust solution is to establish global log rotation boundaries in /etc/docker/daemon.json. Modern Docker releases include the high-performance local logging driver, which provides automatic file rotation, disk compression, and dual-file retention without external cron jobs:

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "25m",
    "max-file": "3",
    "compress": "true"
  }
}

The local logging driver stores logs in an internal binary format optimized for rapid reads with docker logs while keeping disk consumption capped. In the configuration above, a container will never consume more than 75MB of disk space for logging across its lifetime.

Alternatively, if your centralized log monitoring stack (such as Promtail, Fluentbit, or Datadog) requires standard JSON logs, retain json-file while enforcing hard size limits and enabling non-blocking delivery:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "20m",
    "max-file": "5"
  }
}

After editing the daemon configuration, restart the Docker service to apply the settings to all newly created containers:

sudo systemctl restart docker

Note that modifying daemon.json applies only to newly created containers. Existing running containers will continue using their initial logging configuration until they are removed and recreated with docker compose down followed by docker compose up -d.

Service-Specific Log Controls in Docker Compose

When running individual applications through Docker Compose, you can define granular logging policies for services with distinct output volumes:

services:
  web-app:
    image: myorg/web-service:v2
    restart: unless-stopped
    ports:
      - "8080:8080"
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
        mode: "non-blocking"
        max-buffer-size: "4m"

The mode: "non-blocking" option is particularly valuable in production environments. In blocking mode (the default), if the host disk subsystem encounters temporary saturation or high I/O wait times, the Docker engine's write call blocks. This causes the kernel pipe buffer to fill up, which in turn halts the application process writing to stdout. In non-blocking mode, if the 4MB memory buffer fills up during a disk write freeze, Docker drops excess log lines rather than hanging application worker threads.

Proactive Disk Monitoring and Emergency Truncation

If a container has already written a massive log file and filled the disk, do not delete the file with rm, as the open file descriptor held by the Docker daemon will prevent the kernel from reclaiming the allocated disk blocks. Instead, truncate the file to zero bytes:

# Locate the largest container log files on the host
sudo find /var/lib/docker/containers/ -name "*-json.log" -exec du -h {} + | sort -rh | head -n 10

# Safely truncate an active runaway log file without restarting the container
sudo truncate -s 0 /var/lib/docker/containers/abc12345/abc12345-json.log

Adopting strict container logging drivers guarantees that your host filesystem remains protected against runaway log growth regardless of application error states.

You can spin up an Ubuntu 24.04 instance on Netbay in under 60 seconds with Intel Xeon Platinum compute and high-speed SSD storage in Lucknow — netbayhosts.in.