Chaos Engineering with eBPF: Injecting Granular Network Faults into Production

Reliability engineering requires validating how distributed microservices behave when the underlying network degrades. In traditional environments, testing network failures relied on crude mechanisms: pulling virtual network cables, stopping entire Docker containers, or injecting indiscriminate latency using iptables that affected every process on the host.

In 2026, chaos engineering on Linux hosts has evolved with extended Berkeley Packet Filter (eBPF). By attaching eBPF programs directly to Traffic Control (TC) kernel hooks, engineers can inject granular network faults—such as dropping 15% of egress packets specifically destined for a PostgreSQL replica on port 5432—with zero impact on other services and zero daemon restarts.

Kernel-Level Packet Interception with eBPF

Attaching eBPF to the Linux network stack provides precision control:

• Ingress/Egress TC Hooks: eBPF programs inspect socket buffers (sk_buff) directly in the kernel before TCP retransmission queues.
• Process & Port Targeting: Faults are applied strictly matching destination IP addresses, destination ports, or process cgroups.
• Safety-First Circuit Breakers: eBPF map values can be updated atomically from user space or set to expire automatically via kernel timers, preventing runaway outages.

eBPF Kernel Code Snippet

Below is the C implementation of an eBPF TC filter that inspects packets and injects packet drops based on a pseudo-random probability generator:

#include <linux/bpf.h>
#include <linux/pkt_cls.h>
#include <linux/ip.h>
#include <linux/tcp.h>
#include <bpf/bpf_helpers.h>

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __type(key, __u32);
    __type(value, __u32);
    __uint(max_entries, 1);
} drop_rate_map SEC(".maps");

SEC("classifier")
int inject_fault(struct __sk_buff *skb) {
    void *data = (void *)(long)skb->data;
    void *data_end = (void *)(long)skb->data_end;

    struct iphdr *ip = data + sizeof(struct ethhdr);
    if ((void *)(ip + 1) > data_end)
        return TC_ACT_OK;

    if (ip->protocol == IPPROTO_TCP) {
        struct tcphdr *tcp = (void *)ip + (ip->ihl * 4);
        if ((void *)(tcp + 1) > data_end)
            return TC_ACT_OK;

        // Target PostgreSQL default port 5432
        if (tcp->dest == __constant_htons(5432)) {
            __u32 rand = bpf_get_prandom_u32() % 100;
            if (rand < 20) { // 20% packet drop
                bpf_printk("eBPF Chaos: Dropped packet to port 5432
");
                return TC_ACT_SHOT;
            }
        }
    }
    return TC_ACT_OK;
}

char _license[] SEC("license") = "GPL";

User Space Activation Script

Attaching this eBPF bytecode to the primary network interface requires just two commands:

# Compile eBPF program with clang
clang -O2 -target bpf -c chaos_filter.c -o chaos_filter.o

# Attach filter to egress Traffic Control qdisc
tc qdisc add dev eth0 clsact
tc filter add dev eth0 egress bpf da obj chaos_filter.o sec classifier

To instantly remove the chaos test and restore flawless connectivity, simply delete the filter:

tc filter del dev eth0 egress

Expand

Video Guide on DevOps Reliability & Fault Tolerance

Explore foundational DevOps reliability practices and site reliability engineering principles:

Testing systems under realistic, kernel-level network degradation ensures that automated retries, circuit breakers, and connection pools work when true outages strike.

Conduct realistic reliability and chaos drills on Netbay Linux VPS instances with full root kernel capabilities at netbayhosts.in.