Two years ago, I was on-call for a microservices platform that handled payment processing. At 3:17 AM, PagerDuty lit up: latency p95 for checkout had jumped from 200ms to 12 seconds. My monitoring stack—Grafana dashboards on top of Prometheus metrics—showed every service was green. CPU, memory, request rate: all normal. The only signal was a single red line on the checkout latency chart. I spent the next 90 minutes SSHing into boxes, tailing logs, and restarting services at random. Eventually the problem went away. I never found the root cause.
That night is the textbook definition of the monitoring vs. observability divide. Monitoring told me something was wrong. Observability would have told me why.
Monitoring: The Known Knowns
Monitoring is the practice of collecting and alerting on predefined metrics and logs. You decide ahead of time that CPU > 90% is bad, or 5xx rate > 1% is bad, and you build dashboards and alerts around those signals. This works great for known failure modes—when a server dies, when a database fills its disk, when a certificate expires.
But monitoring is fundamentally reactive and constrained. Every dashboard you build encodes a hypothesis about what might go wrong. The universe of possible failures in a distributed system is infinite; you can't graph everything.
Monitoring encodes your past failures. Observability encodes your system's actual behavior.
Observability: The Unknown Unknowns
Observability is a property of a system—the degree to which you can infer its internal state from its external outputs. In practice, it means that when something goes wrong, you can ask arbitrary questions about what happened, without having pre-built a dashboard for that specific scenario.
This requires three things: high-cardinality data (every request carries dozens of dimensions like user_id, region, feature_flag, shard), structured events (logs, traces, metrics all with consistent context), and a query engine that can slice and dice across millions of events in seconds.
// Bad: unstructured log with no context
console.log('Payment timeout for order ' + orderId);
// Good: structured event with high cardinality dimensions
tracer.startSpan('payment.timeout', {
attributes: {
'order.id': orderId,
'user.id': userId,
'payment.gateway': 'stripe',
'payment.amount_cents': amountCents,
'payment.currency': 'USD',
'region': 'us-east-1',
'feature.payment_v2': true
}
}).end();The Cardinality Trap
A common mistake is thinking that dumping more logs into your existing monitoring tool makes it observability. It doesn't. Most monitoring tools (Prometheus, Datadog, Grafana) work well with low cardinality—like 10 label values per metric. But when you add user_id (millions of values) or request_id (billions), the storage and query performance collapses.
Observability tools (Honeycomb, Lightstep, Grafana Tempo with traceql) are built for high cardinality. They use columnar storage and aggressive indexing so you can group by any field without pre-aggregation.
The 3 AM Payment Fail that Monitoring Missed
- 03:17PagerDuty alert: checkout latency p95 > 10s
- 03:18On-call checks dashboards: all service metrics green. No anomaly.
- 03:25On-call begins SSHing into payment service instances, grepping logs for 'timeout'
- 03:45No pattern found. Restarts payment service. Latency drops.
- 04:30Incident resolved. Root cause unknown.
- Weeks laterPostmortem reveals a new feature flag 'payment_v2' was rolled to 5% of users. It caused a slow SQL query on a specific user segment.
Lesson
With observability, the on-call engineer could have queried: 'show me all requests with latency > 5s, grouped by feature_flag and user_id.' The flag would have been visible immediately. No restart, no SSH, no 90-minute MTTR.
How to Actually Adopt Observability
- 1Instrument with OpenTelemetry: emit traces with spans, structured logs, and metrics from every service. Use context propagation to connect them.
- 2Choose a backend that handles high-cardinality: Honeycomb, Grafana Tempo + Loki, or SigNoz. Avoid shoving traces into traditional log stores.
- 3Build a culture of debugging with ad-hoc queries: when an alert fires, don't just look at dashboards—open the query interface and ask 'what dimensions correlate with this anomaly?'
- 4Start with one critical user journey: trace a checkout flow end-to-end. Add dimensions like user tier, region, payment provider, A/B test variant. Then when latency spikes, you can slice by any of those.
Reduction in MTTR reported by teams that adopt high-cardinality observability practices (source: Honeycomb customer surveys)
You Don't Have to Throw Away Your Dashboards
I still use Grafana for SLO burn rate alerts and capacity planning. Monitoring is not dead—it's just not sufficient. Observability is the debugger you never knew you needed. The next time PagerDuty wakes you up, don't SSH into a box. Open a query and ask the system what happened.
Frequently asked questions
What is the main difference between monitoring and observability?
Monitoring is the practice of collecting predefined metrics and logs to detect known failure modes. Observability is a system property that lets you ask arbitrary questions about its internal state without writing new code. Monitoring answers 'what broke?' Observability answers 'why is it broken?'
Do I need observability if I already have monitoring?
Yes, if your system is distributed or dynamic. Monitoring covers the 'known unknowns'—the dashboards you built. Observability covers the 'unknown unknowns'—the weird latency spike you never predicted. In practice, you need both.
What are the key components of an observable system?
Structured, high-cardinality events (logs, traces, metrics) with correlation IDs, distributed tracing with context propagation, and a query engine that can aggregate and filter without pre-defined dimensions. Tools like OpenTelemetry + Honeycomb or Grafana Tempo are common choices.
Can I make my system observable without buying new tools?
To some extent. You can start by adding structured logging (JSON with IDs), ensuring correlation IDs pass across services, and storing logs in a queryable store. But true observability requires a purpose-built backend that handles high-cardinality and fast ad-hoc queries—most traditional log aggregators can't.