Observability-Driven Architecture for VPS Apps

Most architecture reviews argue from diagrams. Production argues from latency, error rate, and how long a job has been sitting. Observability-driven architecture means those signals are allowed to change the shape of the system: when a process is split onto another VPS, when a route is shed, when a dependency is isolated, and when you stop adding replicas that cannot help. Dashboards that nobody uses to make a decision are decoration. This post is the decision loop, not a tour of exporters.

Start From SLOs, Not From Hosts

Pick a few user-visible promises. Checkout p95 under 400ms. Invoice jobs under 5 minutes old. Error rate under 0.5 percent on /api. Those numbers are the architecture's budget. CPU on Intel Xeon Platinum is an explanation, not a promise. If checkout p95 is fine and CPU is 80 percent, you do not split the node. If checkout p95 is 2 seconds and CPU is 20 percent, you have a lock, a slow query, or a partner call on the request path. The metric that is allowed to spend money is the SLO, not the comfort of a flat CPU graph.

Instrument the edges you might redesign. Every outbound HTTP call gets a name, a duration, and a status. Every queue pop gets age-at-start. Every database query on the checkout path gets a statement id you can find in pg_stat_statements. Without names, you cannot decide whether to cache, to queue, or to put the partner behind a worker.

var http = require('http');
var orig = http.request;
http.request = function (opts, cb) {
  var start = Date.now();
  var req = orig.call(http, opts, function (res) {
    var host = (opts && opts.hostname) || 'unknown';
    console.log('out http ' + host + ' ' + res.statusCode + ' ' + (Date.now() - start) + 'ms');
    if (cb) cb(res);
  });
  return req;
};

That is not a tracing product. It is a log line you can grep during an incident. Add a trace id per request and copy it onto jobs you enqueue so the worker log can be joined. Architecture questions such as "does the GST call belong in the request" are answered by those durations, not by opinions.

Signals That Force a Split

Split a role onto another VPS when a signal says the shared node is the coupling:

• Queue age rises while API p95 is still healthy: workers need their own CPU, not more gunicorn workers.
• Memory of Redis eviction starts while the API is idle: Redis needs its own node with noeviction and its own High-Speed SSD.
• Postgres checkpoints stall the API: move Postgres, do not add another app replica that will hit the same disk.
• One tenant's traffic moves everyone else's p95: that tenant needs its own node, which is isolation as an SLO fix.

The anti-pattern is splitting because the diagram looked crowded. Extra nodes add network, secret, and failure surface. Pay-as-you-go INR billing makes the split cheap to try and cheap to undo if the SLO does not move.

# Cheap SLO export: p95 of /checkout from access log latency field
awk '$7 == "/checkout" {print $NF}' /var/log/nginx/access.log   | sort -n   | awk ' { a[NR]=$1 } END { if (NR) print a[int(NR*0.95)] }'

# Queue age from Redis (seconds since oldest job was enqueued, if you store ts)
redis-cli LLEN jobs:invoice

Alert on the SLO and on the leading indicator, not on both plus CPU plus disk plus random 5xx on a healthcheck you hit from the same machine. An architecture that pages on everything never gets a clean signal to change.

Shed Load as an Architecture Choice

When the SLO is already burning, adding capacity that will not be ready for 60 seconds is not the first move. Return 503 with Retry-After on non-critical routes. Disable a flag that fans out to a dying partner. Stop accepting new jobs when the queue age exceeds a budget. Those are design features you build when you are calm. They are also observability features: a shed counter tells you the system protected itself instead of melting every worker.

Logs, metrics, and traces need a home that is not the production database. A second small VPS for Prometheus and Loki is a reasonable split once the API node is spending iowait on its own logs. Until then, journald plus a few textfile gauges is enough to drive the first year of decisions. Do not wait for a perfect stack to start naming spans.

Takeaway

Let a short list of SLOs spend money and reshape the fleet. Name the work on the request path, watch queue age, and split roles when the signal says they contend. Host graphs stay in the explanation column.

Stand up API, worker, and a tiny metrics VPS on Netbay in Lucknow (DC01) and let a week of p95 data tell you whether the split was worth the invoice — netbayhosts.in.