Stream LLM Tokens Through nginx Without Buffering

Chat UIs feel fast when the first token arrives in a few hundred milliseconds and the rest drip in. They feel broken when the browser waits for the full completion, then paints a wall of text. The usual cause on a VPS is not the model. It is nginx buffering the upstream response until the app closes the body. A production streaming path is Server-Sent Events or a chunked JSON stream, proxy_buffering off, gzip off for that location, and timeouts long enough for a slow cheap-model call.

This is architecture, not a model install. The app on 127.0.0.1 still talks to OpenAI or another remote API, or to a local 7B bound on loopback. nginx on :443 is the only public listener. Lucknow DC01 gives you Intel Xeon Platinum, High-Speed SSD, and L3/L4 DDoS filtering on the public NIC. None of that helps if the reverse proxy holds 32 KB of tokens in a buffer.

Why nginx swallows the stream

Default proxy_buffering is on. nginx reads from the upstream and writes to disk or memory buffers before sending to the client. That is correct for a REST JSON blob. It is wrong for text/event-stream. gzip on the same location waits for a full buffer to compress. proxy_read_timeout at 60 seconds kills a long completion. HTTP/1.1 keepalive to the app is fine; just do not let the proxy sit on the bytes.

SSE is the least surprising browser API. The app writes data: plus a JSON payload plus two newlines, repeatedly, then data: [DONE]. The client is EventSource or fetch with a ReadableStream. Do not use WebSockets unless you already have them. You do not need them to stream tokens.

The app must flush. In Node, res.write for each event and disable Nagle if you are on a tiny payload. In Python with gunicorn, that means no fully buffered WSGI worker holding the iterator; use a worker that supports streaming or run uvicorn. If you cannot flush, nginx has nothing to forward.

nginx location for SSE

Terminate TLS at nginx. Proxy to the app on loopback. Turn buffering off only on the stream path so the rest of the site can still cache.

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

server {
    listen 443 ssl http2;
    server_name ai.example.in;
    ssl_certificate     /etc/letsencrypt/live/ai.example.in/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ai.example.in/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Request-Id $request_id;
        proxy_read_timeout 60s;
    }

    location /v1/stream {
        proxy_pass http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header Connection '';
        proxy_set_header X-Request-Id $request_id;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_buffering off;
        proxy_cache off;
        gzip off;
        proxy_read_timeout 300s;
        proxy_send_timeout 300s;
        chunked_transfer_encoding on;
        add_header X-Accel-Buffering no;
        add_header Cache-Control no-cache;
    }
}

X-Accel-Buffering no is a second belt if some other layer re-enables buffering. proxy_read_timeout 300s covers a long completion. Do not set it to a day; a stuck vendor call should die. $request_id lets you join nginx access logs with app logs without inventing a header scheme.

App: write events and never buffer the vendor

The handler opens the vendor stream, then forwards tokens. If you collect the full vendor body and then loop, you have not streamed anything. Parse SSE or data-url encoded chunks from the vendor as they arrive.

const http = require('http');

function writeEvent(res, obj) {
  res.write('data: ' + JSON.stringify(obj) + '\n\n');
}

http.createServer(async function (req, res) {
  if (req.url !== '/v1/stream' || req.method !== 'POST') {
    res.statusCode = 404;
    res.end('not found');
    return;
  }
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');
  res.flushHeaders();

  const vendor = await fetch('https://api.openai.com/v1/chat/completions', {
    method: 'POST',
    headers: {
      Authorization: 'Bearer ' + process.env.OPENAI_API_KEY,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model: process.env.CHEAP_MODEL || 'gpt-4o-mini',
      stream: true,
      messages: [{ role: 'user', content: 'Say hello in one sentence.' }],
    }),
  });
  if (!vendor.ok || !vendor.body) {
    writeEvent(res, { error: 'upstream' });
    res.end();
    return;
  }
  const reader = vendor.body.getReader();
  const decoder = new TextDecoder();
  let buf = '';
  for (;;) {
    const chunk = await reader.read();
    if (chunk.done) break;
    buf += decoder.decode(chunk.value, { stream: true });
    const parts = buf.split('\n');
    buf = parts.pop();
    for (let i = 0; i < parts.length; i++) {
      const line = parts[i];
      if (line.indexOf('data: ') !== 0) continue;
      const data = line.slice(6);
      if (data === '[DONE]') continue;
      writeEvent(res, { raw: data });
    }
  }
  writeEvent(res, { done: true });
  res.end();
}).listen(3000, '127.0.0.1');

That sample forwards vendor data as-is so you can inspect it. In production, parse delta.content and send only that field. Cap the number of concurrent streams per IP. A stream holds a vendor connection and a worker; 200 idle EventSource clients will ruin a small plan even with L3/L4 filtering in front.

Timeouts, clients, and what to measure

Browsers drop EventSource if a proxy sits silent. Send a comment line every 15 seconds if the model is slow: a line that starts with a colon is an SSE comment. That keeps NAT and nginx from assuming the socket is dead.

Measure time to first byte from nginx, time to first token in the app, and total tokens. If TTFB is seconds and the app log shows the first token in 200 ms, buffering is still on. curl -N is the right test; without -N curl buffers too.

The takeaway: streaming is a three-party contract. The vendor must stream, the app must flush, nginx must not buffer. Fail any one and the UI waits for the whole answer.

You can spin up an Ubuntu 24.04 instance on Netbay in under 60 seconds and follow along — netbayhosts.in.