Architecting an OpenTelemetry Collector Pipeline for Metrics, Logs, and Traces

Observability in modern distributed systems has historically suffered from vendor fragmentation: developers instrumented distinct client libraries for Prometheus metrics, fluent-bit for log streaming, and Jaeger or Zipkin for distributed tracing. This fragmented approach bloated application dependencies, wasted CPU cycles formatting data multiple times, and made correlating logs with trace spans notoriously difficult.

The OpenTelemetry (OTel) Collector provides a unified, vendor-neutral proxy pipeline. Applications emit telemetry in OpenTelemetry Protocol (OTLP) format over gRPC or HTTP to a single local collector daemon. The collector buffers, filters sensitive attributes, and exports telemetry to Prometheus, Grafana Loki, and Grafana Tempo simultaneously.

The Architecture of the OpenTelemetry Collector

The OpenTelemetry Collector operates on a modular pipeline architecture consisting of four fundamental components:

1. **Receivers**: The ingress point. Receivers accept telemetry data in various formats (OTLP, Zipkin, Jaeger, Prometheus) and translate it into internal OTel memory representations.
2. **Processors**: Applied in sequential order. Processors modify, filter, batch, and scrub data. The memory_limiter processor ensures the process never causes kernel OOM panics, while the batch processor groups records to minimize network syscalls.
3. **Exporters**: The egress point. Exporters translate internal OTel data into the formats required by downstream storage engines (e.g. OTLP gRPC for Tempo, remote write for Prometheus, or OTLP HTTP for Loki).
4. **Connectors**: Bridges that connect the egress of one pipeline to the ingress of another (such as generating span metrics from raw traces).

Installing OpenTelemetry Collector Contrib on Ubuntu 24.04

Always deploy the otelcol-contrib distribution, which includes production processors, transform tools, and multi-backend exporters:

# Download and install the official OTel Contrib debian package
sudo apt-get update
sudo apt-get install -y curl

OTEL_VERSION="0.108.0"
curl -fsSL -O "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTEL_VERSION}/otelcol-contrib_${OTEL_VERSION}_linux_amd64.deb"
sudo dpkg -i "otelcol-contrib_${OTEL_VERSION}_linux_amd64.deb"
rm "otelcol-contrib_${OTEL_VERSION}_linux_amd64.deb"

# Verify binary installation and enabled components
otelcol-contrib components

The collector installs as a systemd service managed under the otelcol-contrib user, reading its configuration from /etc/otelcol-contrib/config.yaml.

Configuring Receivers, Processors, and Exporters

Edit /etc/otelcol-contrib/config.yaml to define pipelines for traces, metrics, and logs:

# /etc/otelcol-contrib/config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # Crucial: memory limiter protects host from out-of-memory crashes
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20

  # Batch processor combines items to reduce network calls
  batch:
    send_batch_size: 1024
    timeout: 5s

  # Transform processor redacts sensitive authorization and cookie tokens
  transform:
    log_statements:
      - context: log
        statements:
          - replace_pattern(attributes["http.request.headers.authorization"], "Bearer .*", "Bearer [REDACTED]")
          - replace_pattern(attributes["http.request.headers.cookie"], "session=.*", "session=[REDACTED]")

exporters:
  # Export traces to Grafana Tempo
  otlp/tempo:
    endpoint: 10.0.0.8:4317
    tls:
      insecure: true

  # Export logs to Grafana Loki via OTLP HTTP
  otlphttp/loki:
    endpoint: http://10.0.0.8:3100/otlp

  # Prometheus metrics scrape endpoint
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo]

    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]

    logs:
      receivers: [otlp]
      processors: [memory_limiter, transform, batch]
      exporters: [otlphttp/loki]

  telemetry:
    metrics:
      address: 0.0.0.0:8888

Notice the memory_limiter processor: it continuously monitors system memory, dropping incoming payloads if RAM consumption nears 75% to prevent OS-level OOM kills.

Expand

Starting and Verifying the Collector Service

Restart the service and inspect the ports:

sudo systemctl restart otelcol-contrib
sudo systemctl status otelcol-contrib
sudo ss -lntp | grep -E "4317|4318|8889"

Send a test OTLP trace via curl to verify ingestion:

curl -X POST http://localhost:4318/v1/traces   -H "Content-Type: application/json"   -d '{
    "resourceSpans": [{
      "resource": { "attributes": [{ "key": "service.name", "value": { "stringValue": "checkout-service" } }] },
      "scopeSpans": [{
        "spans": [{
          "traceId": "5b8aa5a2d2c872e8321cf37308d69de2",
          "spanId": "051581bf3cb46c5e",
          "name": "process-payment",
          "kind": 1,
          "startTimeUnixNano": "1725350400000000000",
          "endTimeUnixNano": "1725350400500000000"
        }]
      }]
    }]
  }'

The OpenTelemetry Collector provides complete architectural independence, preventing vendor lock-in while streamlining telemetry pipelines.

Deploy your observability gateways on Netbay Linux VPS in under 60 seconds with High-Speed SSD in Lucknow DC01 — netbayhosts.in.