The Three Pillars of Observability
The observability framework most commonly referenced in platform engineering organizes telemetry into three categories: metrics, logs, and traces. Each pillar serves a distinct investigative purpose, and effective observability systems provide all three with sufficient correlation capability to move between them during incident investigation.
Metrics provide quantitative time-series data about system behavior — request rates, error rates, and latency percentiles are the most common examples. Logs provide detailed event-level records that capture the context of specific actions. Traces provide a causal chain of events across service boundaries, enabling identification of where in a distributed transaction latency is being introduced or failures are occurring.
Metrics Collection and Storage
Metrics collection involves instrumenting application and infrastructure components to emit numeric measurements at regular intervals or on specific events. The choice of metrics storage system affects the query capabilities available for analysis and alerting.
Push vs. Pull Models
In a push model, instrumented services send metrics to a collection endpoint. In a pull model, a collection service periodically scrapes metrics from instrumented service endpoints. Pull models are generally easier to reason about for service discovery and don't require services to know where to send metrics. Push models may be more appropriate for short-lived processes that may not be running when the scrape interval occurs.
High-Cardinality Metrics
High-cardinality metric dimensions — where a label can take a large number of distinct values, such as user IDs or request IDs — create significant storage and query performance challenges in time-series databases designed for low-cardinality labels. Distributed tracing systems are better suited than metrics systems for per-request observability. Metrics should be designed with labels that take a bounded set of values representing meaningful operational categories rather than per-entity identifiers.
Structured Logging
Structured logging emits log records in a machine-parseable format — typically JSON — rather than free-form text strings. Structured logs can be indexed and queried efficiently on any field, enabling log-based analysis and alerting on specific event attributes without regex pattern matching against free-form text.
Log Correlation
Effective log correlation requires consistent propagation of request identifiers across service boundaries. A trace ID included in the log records for every request, propagated from the originating service through all downstream calls, enables reconstruction of the complete activity log for a given request across all participating services.
Log Levels and Retention
Log volume management requires a clear policy for log levels (debug, info, warn, error) and for which level is emitted in each deployment environment. Verbose debug logging in production generates storage and query costs that must be justified by the observability value. A tiered retention policy — high-fidelity logs retained for seven to thirty days, aggregated event summaries retained for longer periods — provides investigation capability for recent incidents while managing long-term storage costs.
Distributed Tracing
Distributed tracing captures the causal chain of calls across service boundaries that constitute a single logical operation. Each step in the chain is recorded as a span; spans are linked by a common trace ID, allowing the complete tree of service calls to be reconstructed and visualized.
OpenTelemetry
OpenTelemetry is a vendor-neutral instrumentation framework providing standardized APIs, SDKs, and data export formats for metrics, logs, and traces. Using OpenTelemetry for instrumentation decouples application code from specific observability backend choices, allowing the telemetry backend to be changed without reinstrumentation. It has reached stable status for traces and metrics across major language SDKs and has significant adoption across cloud-native tooling ecosystems.
Sampling Strategies
Tracing all requests in a high-throughput system generates impractical data volumes. Sampling strategies reduce trace volume while preserving observability value. Head-based sampling makes the sampling decision at the trace origin, with the decision propagated to all services in the trace. Tail-based sampling defers the decision until the trace is complete, enabling selection based on trace characteristics such as presence of errors or high latency — at the cost of buffering complete traces before the sampling decision is made.
Alerting Design
Effective alerting surfaces actionable conditions that require human attention without generating alert fatigue from conditions that are non-actionable or transient. Alert design principles include: alerting on symptoms rather than causes where possible, ensuring every alert has a documented response action, and setting thresholds that reflect the conditions under which service quality is actually affected rather than static percentages of resource utilization.
Alert Routing
Alert routing directs notifications to the team responsible for responding to the specific condition. Routing configurations should be maintained alongside service ownership documentation and updated when service ownership changes. Multi-stage routing — initial alert to primary on-call, escalation to secondary on-call if unacknowledged within a defined interval — provides backup coverage for alerts that are not addressed within the expected timeframe.
SLOs and SLAs
Service Level Objectives (SLOs) define the target reliability level for a service over a measurement window, expressed as a percentage of successful operations. Error budgets — the permissible quantity of non-successful operations within the measurement window before the SLO is breached — provide a framework for balancing reliability investment against feature development velocity.
Service Level Agreements (SLAs) are contractual commitments to external parties based on service levels. SLOs should be set more conservatively than SLAs, providing a buffer between the internal target and the external commitment that gives time to address reliability degradation before it affects the contractual commitment. The gap between SLO and SLA should reflect the confidence interval around the organization's ability to detect and remediate SLO-threatening conditions before the measurement window closes.