API & Automation·7 min read·2026-10-24

Scaling Event-Driven Background Workers with BullMQ and Valkey

Offload compute-heavy background tasks from the HTTP event loop using BullMQ and Valkey on Linux, managing concurrency, retries, and job telemetry.

Netbay Cloud Team

Netbay Engineering

On this page

Architectural Fundamentals of BullMQ
Installing and Configuring BullMQ Workers
Supervising Workers with Systemd
Dispatching Jobs and Monitoring Metrics

Node.js and asynchronous runtimes excel at high-concurrency IO-bound operations. However, when an incoming HTTP request triggers intensive computational work—such as generating PDF invoices, transposing video assets, compiling zip archives, or sending batch transactional emails—running those tasks inside the request loop freezes the main event loop, causing latency spikes and dropped connections.

BullMQ is an enterprise-grade message queue library built on Redis and Valkey primitives (Streams and Sorted Sets). Running BullMQ workers supervised by systemd on a Linux VPS allows you to process thousands of background jobs per second with granular concurrency limits, priority scheduling, parent-child job workflows, and automatic stalled-job recovery.

Architectural Fundamentals of BullMQ

BullMQ avoids complex message broker daemons by leveraging Valkey's atomic operations:

**Waiting / Delayed Jobs**: Stored in Valkey Sorted Sets (ZSET) scored by execution timestamp.
**Active Jobs**: Managed via atomic Lua scripts and stream groups.
**Stalled Job Detection**: Workers regularly update a heartbeat lock. If a worker process abruptly dies (OOM kill or power failure), another worker detects the expired lock and automatically re-queues the task.
**Parent-Child Flows**: Enable complex orchestration (e.g. download five data chunks in parallel, then compile them in a parent aggregation step).

Installing and Configuring BullMQ Workers

Install BullMQ and ioredis in your TypeScript project:

bash

npm install bullmq ioredis
npm install -D typescript @types/node

Define the queue producer and worker processor:

typescript

// queue-worker.ts
import { Queue, Worker, Job } from "bullmq";
import Redis from "ioredis";

const connection = new Redis("redis://10.0.0.4:6379", {
  maxRetriesPerRequest: null, // Required by BullMQ
});

// 1. Queue Definition (Producer)
export const exportQueue = new Queue("file-exports", { connection });

// 2. Worker Processor (Consumer)
const worker = new Worker(
  "file-exports",
  async (job: Job) => {
    console.log("[Job " + job.id + "] Processing export for user " + job.data.userId);

    // Report progress to dashboard
    await job.updateProgress(25);

    // Simulate compute-heavy file creation
    await new Promise((resolve) => setTimeout(resolve, 2000));
    await job.updateProgress(75);

    // Return final result payload
    return { downloadUrl: "https://storage.example.com/exports/" + job.id + ".zip", sizeBytes: 1048576 };
  },
  {
    connection,
    concurrency: 8, // Run 8 tasks concurrently on multi-core CPU
    limiter: {
      max: 100, // Maximum 100 jobs processed
      duration: 1000, // per 1000ms (1s)
    },
  }
);

worker.on("completed", (job) => {
  console.log("[Job " + job.id + "] Completed successfully! Result:", job.returnvalue);
});

worker.on("failed", (job, err) => {
  console.error("[Job " + (job?.id ?? "") + "] Failed with error:", err.message);
});

// Graceful shutdown handling
process.on("SIGTERM", async () => {
  console.log("Gracefully closing worker...");
  await worker.close();
  process.exit(0);
});

BullMQ Background Workers — BullMQ Asynchronous Job Processing

Supervising Workers with Systemd

Supervise background workers using systemd to ensure automatic restart upon failure and clean SIGTERM handling during system updates:

ini

# /etc/systemd/system/bullmq-worker.service
[Unit]
Description=BullMQ Production Background Worker
After=network.target valkey-server.service
Wants=valkey-server.service

[Service]
Type=simple
User=www-data
WorkingDirectory=/srv/app
Environment=NODE_ENV=production
ExecStart=/usr/bin/node dist/queue-worker.js
Restart=always
RestartSec=3s
KillSignal=SIGTERM
TimeoutStopSec=60

# Resource limits and sandboxing
LimitNOFILE=65535
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

The TimeoutStopSec=60 setting gives active background jobs up to 60 seconds to finish processing before systemd forcefully issues SIGKILL.

bash

sudo systemctl daemon-reload
sudo systemctl enable --now bullmq-worker
sudo systemctl status bullmq-worker

Message Queues, Background Workers, and Event-Driven Architecture

Watch on YouTubeYouTube

Dispatching Jobs and Monitoring Metrics

Add jobs from your API endpoints and inspect real-time queue states:

typescript

// api-route.ts
app.post("/api/export", async (req, res) => {
  const job = await exportQueue.add(
    "generate-pdf",
    { userId: req.user.id, filters: req.body },
    {
      attempts: 5,
      backoff: { type: "exponential", delay: 2000 },
      removeOnComplete: 1000,
      removeOnFail: 5000,
    }
  );

  return res.status(202).json({ jobId: job.id, status: "queued" });
});

BullMQ ensures compute tasks scale horizontally without degrading user-facing HTTP performance.

Deploy high-concurrency BullMQ worker pools on Netbay Linux VPS in under 60 seconds with High-Speed SSD in Lucknow DC01 — netbayhosts.in.

Keep reading

API & Automation

Deploy Linux in under 60 seconds

These guides are written against Ubuntu, Debian, and RHEL-family images — the same ones on NetBay.

Deploy an instance