A One-Weekend Observability Stack That Stays Small

The pieces in this series are useful only if they still fit next to the application on Monday. A weekend stack is a budget: processes, ports, disk, and a list of things you will not install. Prometheus, node_exporter, Alertmanager, Grafana, journald, logrotate, and one external uptime check are enough. Loki is optional. Kubernetes exporters, tracing meshes, and a second Grafana for fun are how the observers eat the app.

This is the assembly guide. You already know the individual tools. The work is to put them on one Intel Xeon Platinum VPS with High-Speed SSD, bind every UI to localhost, cap RAM, and write down the four pages you will actually answer.

The process list and the ports

Saturday morning: node_exporter on 127.0.0.1:9100, Prometheus on 127.0.0.1:9090, Alertmanager on 127.0.0.1:9093, Grafana on 127.0.0.1:3000. Optional: blackbox on 127.0.0.1:9115, Loki on 127.0.0.1:3100, Promtail on 127.0.0.1:9080. None of those ports on the public NIC. SSH tunnel or a reverse proxy with auth is how you look. L3/L4 DDoS filtering in front of a public Grafana is still a public Grafana; do not do that.

systemd units with Restart=on-failure and MemoryMax. Prometheus 512M, Grafana 256M, Alertmanager 128M, Loki 256M if present, exporters tiny. On a 4 GB plan the app still owns most of RAM. On a 2 GB plan skip Loki and maybe skip Grafana; Prometheus plus Alertmanager plus an SSH habit can wait a month.

[Service]
MemoryMax=512M
MemorySwapMax=0
Restart=on-failure
RestartSec=5

Apply the same drop-in pattern to each observer unit. MemorySwapMax=0 stops a metrics process from thrashing the app into swap. If Prometheus hits the cap, you have too many series, not too little RAM. Drop collectors before you raise MemoryMax.

Disk budget you can say out loud

Prometheus TSDB 4 GB and 15 days. journald 500 MB and 14 days. Loki 2 GB and 7 days if it exists. logrotate 50 MB times 7 for nginx. Leave at least 20 percent of the root filesystem empty for the application, updates, and a dump. Record those numbers in a file next to prometheus.yml. DiskAlmostFull at 10 percent free is the backstop, not the plan.

Saturday afternoon: take the CLI baseline from the first post. Quiet-hour CPU, MemAvailable, disk await, listen ports. That page is the dashboard spec. Build three Grafana rows that match it. Do not import Node Exporter Full.

Sunday morning: four Prometheus rules, Alertmanager with page versus ticket, one inhibit on InstanceDown. Stop node_exporter for three minutes and confirm a single page. Rotate a dummy log with logrotate --force. Vacuum the journal once so you have done it.

Sunday afternoon: either add blackbox against the public URL, or stand up the external check on another host. If you only have time for one, take the external check. On-box blackbox is a luxury. Off-box GET is the hole in the design.

What you skip on purpose

No tracing until a request path is actually mysterious. No remote write. No Grafana onboarding slideshow of 12 data sources. No cAdvisor if you do not run containers. No exporters for things you do not run. No 90-day Loki. No public :9090. No default Grafana password. No alerting on CPU at 70 percent. No alerting on public NIC drops. No second copy of logs in files and Loki and syslog.

The skip list is the stack. Every extra process is RAM the app does not get and a unit you will not patch. Write the skip list in the same repo as the configs so future-you does not "just add" a toy.

# weekend-stack.yml notes, not a compose file
# bind: 127.0.0.1 only
# retain: prom 15d / journal 14d / loki 7d optional
# alerts: InstanceDown, DiskAlmostFull, MemoryLow, ExternalProbeFailed
# skip: tracing, remote_write, public UIs, CPU pages, NIC drop pages

If you want compose instead of systemd, that is fine as long as the ports, memory limits, and skip list stay. Compose is not an excuse to add six more images because the sample did. Pin versions. Do not use latest on an observer you need during an incident.

Monday: operate, do not decorate

Look at Grafana once a day for a week and compare it to the baseline notes. Silence the backup window. Add one panel only if you asked a question the screen could not answer. Delete panels you never look at. When you rebuild the VPS, the Git repo of units and yml files is the stack; the TSDB is not.

The stack stays small because you revisit the budget after a month. du the data dirs. systemctl status the units. RSS from ps. If Prometheus is 200 MB and the app is healthy, leave it. If Loki is 1.2 GB because you ingested access logs, stop ingesting access logs. The observers work for the application, on a Lucknow VPS that should still have room to install a security update.

Takeaway

One weekend is enough to leave a VPS with baselines, Prometheus, four alerts, capped logs, and an external check. The stack stays small when the skip list is real. Spin up Ubuntu 24.04 on Netbay in Lucknow and keep the observers on localhost — netbayhosts.in.