An observability starter kit for a small team

Start small on purpose

Observability tools can measure almost anything, and a new team often tries to. The result is a dashboard nobody looks at and alerts everybody mutes.

This guide is the opposite: the smallest set of signals, checks and alerts that will tell a small team when its users are having a bad time, and give it enough context to fix it. Add more only when an incident shows you something was missing.

The four golden signals

For every service that handles requests, watch:

• Latency. How long requests take. Look at a high percentile (p95 or p99), not the average, because the average hides the users who are suffering.
• Traffic. How many requests per second. A sudden drop is often a worse sign than a spike.
• Errors. The rate of failed requests, usually 5xx responses. Track it as a percentage of traffic, not an absolute count.
• Saturation. How full the service is: CPU, memory, connection pools, queue depth. It tells you what is about to break.

For background workers, adapt them: job duration, jobs processed, job failures, and queue depth or age of the oldest job.

Health and readiness checks

A check is only useful if it answers a real question.

• Liveness: is the process stuck? Keep it trivial. If it fails, the platform restarts the container. Do not check dependencies here, or a database blip will restart every replica at once.
• Readiness: can this instance take traffic now? Check what the service needs to do its job, usually its database connection. If it fails, traffic stops being routed to that instance until it recovers.

Rollouts are gated on readiness. A version that never becomes ready never receives traffic.

Structured logs

Write logs as JSON lines to standard output, with a consistent set of fields:

{"ts":"2026-10-13T09:41:07Z","level":"error","service":"api","env":"production","request_id":"9f2c…","route":"POST /orders","status":500,"duration_ms":812,"msg":"payment provider timeout"}

• Always include timestamp, level, service, environment and a request or trace ID.
• Log events, not narration. One line per meaningful event, with the data needed to understand it.
• Never log secrets or personal data. Redact tokens, passwords and card numbers at the logger, not by hoping.
• Use levels honestly. error means someone may need to act. If nobody would, it is warn or info.

Propagate the request ID across services, so one user action can be followed from the frontend to the worker.

Your first three alerts

Alert on symptoms users feel, not on every cause you can imagine. Three alerts are enough to start:

1. Error rate too high. The percentage of 5xx responses on a public service stays above a threshold for several minutes.
2. Latency too high. The p95 latency of a public service stays above a threshold for several minutes.
3. Something is about to run out. Database storage, connection pool or disk is above a high-water mark.

Set the thresholds from a week of normal data, not from a guess. Require the condition to persist for a few minutes so a single slow request does not page anyone.

Every alert must answer three questions in its message: what is wrong, where, and where to start looking. Link to the runbook.

Runbooks

A runbook is a short page per alert:

• What this alert means in one sentence.
• First checks: the latest deployment, the dashboard for the service, the logs filtered to errors.
• Common causes you have seen before, with the fix for each.
• How to mitigate: roll back, scale, disable a flag, fail over.
• Who to escalate to.

Write them after incidents, not before. The first version can be three lines; it gets better every time it is used.

Start from what changed

Most incidents are caused by a change: a deployment, a configuration update, a migration, a dependency that changed under you. The fastest first question is “what changed just before this started?”

Make that question easy to answer: keep deployments, config changes and migrations on the same timeline as your metrics, so a spike can be lined up against the release that preceded it.

How Lookout uses this

The Lookout is the agent in Kapten’s crew responsible for observability and incidents. It works best with the setup above:

• It starts every investigation from the deployment and config timeline for the affected environment.
• It reads the golden signals of the affected service and its neighbours on the architecture canvas.
• It queries structured logs by service, environment, request ID and time window, instead of searching free text.
• It proposes a likely cause with the evidence attached (the log lines, the metric change, the release) and a mitigation for the Helmsman to plan.
• It drafts the incident timeline and follow-ups for your runbook.

Investigation is read-only. Mitigations such as a rollback are proposed as plans for you to approve. See the incident triage workflow for the full sequence.

Log availability, retention and monitoring coverage depend on the early-access setup agreed during onboarding.

Checklist

• Each service exposes liveness and readiness checks that mean something.
• Logs are structured JSON with service, environment and request ID.
• Golden signals are visible per service and per environment.
• Three alerts: error rate, latency, saturation of a critical resource.
• Each alert links to a runbook.
• Deployments and config changes appear alongside metrics.