Monitoring on business outcomes: green dashboards, empty checkout

Every platform team knows the moment: CPU at 30 percent, no errors in the logs, all dashboards green, and customers have not been able to pay for two hours. Classic monitoring watches the machines. What the business needs to know is whether people can do what they came for: book, pay, sign up. This is how I set up monitoring that watches outcomes, catches problems from the outside before customers report them, and stays quiet enough that people still read the alerts.

Dominik Heinrich

November 8, 202610 min read

1. Watch symptoms, not machines

Google's SRE book frames monitoring around two questions: what is broken, and why. The "what" is a symptom a user feels: a page that doesn't load, a payment that fails, a booking that never arrives. The "why" is a cause: a full disk, a slow query, a certificate that expired. Symptoms deserve alerts because they always matter; causes belong on dashboards, where you look when a symptom fires.

The Prometheus project gives the same advice: alert on symptoms, keep alerting simple, and avoid pages where there is nothing to do. A CPU alert at 3 a.m. for a machine that is busy but serving every customer is the opposite, and it trains people to ignore the pager.

2. Name the journeys and give them targets

Start with three to five journeys that carry the business, and describe each from the customer's side: search and book a stay, pay, receive the confirmation, sign up for a course. For each, pick one indicator that says whether it worked, such as the share of started bookings that complete or the share of payments that succeed, and agree on an objective with the business owner.

An objective of 99.5 percent successful bookings per day also defines an error budget: the 0.5 percent you can lose before the business is hurt. The SRE book uses that budget to balance reliability work against new features. In practice it gives every alert a clear threshold, and every release discussion a number.

Outcome monitoring: check from outside, compare with the source of truth, alert on journeys, one quiet issue per outage.

3. Check from the outside, like a customer

Metrics from inside the platform can't see a broken DNS record, an expired certificate or a CDN that serves an old page. Synthetic checks can: small scripts that run from outside and do what a customer does. On one platform I run, they check the vital pages every 10 minutes, walk through the booking funnel every hour and make a real test payment every day. When the vendor of the hostel management system changed its API without notice, the probes caught it and stays were bookable again the same day.

This website has the same thing in miniature: a scheduled workflow requests the main pages, every article, the AI guide, every PDF guide, the sitemap and the booking link once an hour, and a failure lands in my inbox.

yaml

# GitHub Actions: a synthetic check that fails (and emails the owner) when a vital page breaks
on:
  schedule:
    - cron: "*/10 * * * *"
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - run: |
          for path in / /book /checkout; do
            code=$(curl -s -o /dev/null -L --max-time 20 -w '%{http_code}' "https://www.example.com$path")
            [ "$code" = "200" ] || { echo "FAIL $path -> $code"; exit 1; }
          done

Tip from practice

Give synthetic checks their own test customer and mark their bookings and payments, so they never show up in revenue reports or confuse the team.

4. Watchdogs against the source of truth

Some failures only show up when two systems disagree. A customer paid, but no reservation exists. The site had visitors all morning, but not a single booking. A webhook arrived, but the order never changed state. Each of these is a short query against the database or the payment provider, run every few minutes, that should always return nothing.

These watchdogs are the most valuable checks I know, because they catch exactly the failures that technical metrics miss: everything was fast and returned 200, and the outcome was still wrong. Export the result as a metric, so it alerts like everything else.

sql

-- Paid, but no reservation after 15 minutes: should always be empty
SELECT p.id, p.amount, p.created_at
FROM payments p
LEFT JOIN reservations r ON r.payment_id = p.id
WHERE p.status = 'succeeded'
  AND r.id IS NULL
  AND p.created_at < now() - interval '15 minutes';

5. Alert on the objective, not on every blip

Turn the journey indicators into recording rules in Prometheus, then alert when they drop below the objective for long enough to matter. A minimum traffic condition prevents alerts at night when two failed bookings out of three look like a disaster. Every alert carries a summary in plain words and a link to the runbook that says what to check first.

On the platform mentioned above, Prometheus, Grafana, Loki and Alertmanager run 143 alert rules on Kubernetes. That sounds like a lot, but most of them feed dashboards and tickets; only the ones tied to customer journeys page a person.

yaml

groups:
  - name: booking-slo
    rules:
      - record: booking:success_ratio:rate1h
        expr: |
          sum(rate(bookings_completed_total[1h]))
            / sum(rate(bookings_started_total[1h]))
      - alert: BookingSuccessBelowObjective
        expr: |
          booking:success_ratio:rate1h < 0.995
            and sum(rate(bookings_started_total[1h])) * 3600 > 20   # enough traffic to judge
        for: 15m
        labels:
          severity: page
        annotations:
          summary: "Fewer than 99.5% of started bookings complete"
          runbook_url: "https://runbooks.example.com/booking-success"

6. Keep it quiet enough to be read

An alerting system dies of noise long before it dies of missing coverage. What works for me: one GitHub issue per outage, opened automatically when a check fails and closed automatically when it recovers, deduplicated so that a flapping check updates the same issue instead of opening twenty, and hourly caps so a storm can't flood the team chat. The issue becomes the record of the incident, with timestamps, for free.

Review the alerts once a month. Every alert that fired without anyone needing to act is a candidate to delete, move to a dashboard or tune. The goal is that every page means something.

7. Make every release prove itself

The cheapest incident is the one a release gate stops. Two gates do most of the work: database migrations run as a dry run against a copy before a change can merge, and a smoke test runs right after every deploy and turns the deploy red if a vital route breaks. Both reuse the same checks as monitoring, so a broken booking page fails the release before a customer ever sees it.

The checklist below covers these basics, from backups that have actually been restored to business checks that alert a person. Every gap it shows is a place where your dashboards can stay green while the business is not.

Takeaway
Watch what customers do, from outside and against the source of truth, alert only on the journeys that make money, and keep it quiet enough that every page means something.

Real projects behind this

Monitoring on business outcomes, backups proven nightly

Try it yourself

Log retention cost

Want this in your company? Let's talk mail@dominikheinrich.ch

Sources and official guides