Installing an OpenTelemetry SDK does not automatically create useful observability. Unbounded attributes can increase cost, broken context propagation fragments traces, and excessive sampling can hide the failures engineers most need to understand.
Production telemetry requires deliberate instrumentation and governance. This guide covers semantic conventions, sampling, sensitive data, collector design, and the operational practices that keep signals useful as systems grow.
Instrument meaningful boundaries
Begin with inbound requests, database calls, queues, external APIs, and critical business operations. Use standard semantic conventions and stable service names. Add custom spans only when they answer an operational question; instrumentation volume is not the same as diagnostic value.
Preserve context across systems
Propagate trace context through HTTP, messaging, and scheduled work, including retries and asynchronous consumers. Validate propagation in integration tests. Correlate logs with trace and span identifiers so engineers can move from an alert to relevant events without searching by timestamps.
Govern data and sampling
Define approved attributes, cardinality limits, redaction rules, and retention. Never attach unrestricted payloads or customer identifiers casually. Combine baseline sampling with policies that retain errors, slow operations, and important transactions while controlling routine traffic.
Operate the pipeline
Run collectors with buffering, resource limits, health monitoring, and failure handling. Track dropped spans and exporter latency. Start with one user journey, verify that a support engineer can diagnose a realistic failure, and expand instrumentation from unanswered questions rather than blanket auto-instrumentation.