Fundamentals & Vendor Neutrality
How does OpenTelemetry achieve vendor neutrality, and what are its benefits?
What is OpenTelemetry and why was it created?
How does OpenTelemetry facilitate distributed tracing, and why is this significant for monitoring modern applications?
What is OpenTelemetry and what is it not (e.g., generation + collection + export, not storage/analysis/visualization)?
How is OpenTelemetry built (API & SDK, auto-instrumentation, shared spec across languages)?
Why is OpenTelemetry considered a 'big deal' (universal standard, CNCF, consistent format, data model, APIs, SDK, Collector, auto-instrumentation)?
What is OpenTelemetry instrumentation?
What is the OpenTelemetry Demo application, and how is it used to learn and evaluate the framework?
How does OpenTelemetry differ from older standards like OpenTracing or OpenCensus?
Why would you choose OpenTelemetry over vendor-specific agents?
Explain the difference between pull-based and push-based monitoring architectures, and where OpenTelemetry fits in.
How does an observability solution support OpenTelemetry data, given that OTel defines instrumentation and semantic conventions but not storage/visualization?
Project Structure & Compatibility
How is the OpenTelemetry project structured, and what role do the SIGs and the specification play across languages?
Why did OTel emerge as the front-runner for semantic conventions compared to other projects like Elastic Common Schema or Splunk's CIM?
Explain the significance of 'vendor consolidation around OTLP' in the observability ecosystem.
How does OpenTelemetry provide backward-compatibility shims or bridges for OpenTracing and OpenCensus instrumentation?
Architecture & Core Components
What are the core components of OpenTelemetry (API, SDK, Collector, Exporters, Context Propagation)?
In OpenTelemetry's architecture, which component is responsible for collecting telemetry data from different sources and preparing it for export?
How do the OTel SDK, Collector, and Exporters work together?
What are the key components of the OpenTelemetry API (Tracers, Meters, Loggers)?
In what ways do the OpenTelemetry SDK and Collector differ in their operation and purpose within the OpenTelemetry ecosystem?
Explain the differences between the OpenTelemetry API and SDK in terms of purpose, audience, stability, implementation, dependency, and resource usage.
How does OpenTelemetry's API and SDK separation contribute to its flexibility and adoption?
What are the downsides of wrapping the OpenTelemetry API/SDK?
Why is the OpenTelemetry API designed to be a no-op by default, and why is this important for library authors?
Signals & Data Model
Explain how the three pillars of observability—logs, metrics, and traces—are interconnected via trace ID and span ID in OpenTelemetry.
Explain the four main signals in OpenTelemetry (Traces, Metrics, Logs, and Profiles). What is each signal used for, and how do they inter-correlate within the OpenTelemetry data model?
What does signal stability or maturity (stable vs. experimental) mean in OpenTelemetry, and why does it matter when adopting a signal?
What is the emerging Profiles signal in OpenTelemetry, and what problem does continuous profiling solve alongside traces, metrics, and logs?
What is the cross-cutting data model in OpenTelemetry, and how does a shared model across signals enable correlation?
Tracing Concepts
Explain the tracing concepts: Trace, Span, Trace ID, Span ID, and parent-child relationships.
What is the span status in OpenTelemetry, and what do the Unset, Ok, and Error values signify?
Describe the structure of an OpenTelemetry Span. What are its key components, such as SpanContext, SpanKind, attributes, events, and links, and what role does each play in distributed tracing?
How does distributed tracing track a single request across multiple microservices, and what is context propagation in OpenTelemetry?
What is an OpenTelemetry Context and what's special about how it follows the thread of execution?
How are traces represented in OTLP, specifically through spans and their key fields like trace_id and span_id?
What is the difference between span attributes and span events in OpenTelemetry, and how do you use each during root-cause analysis?
What are span events and span links, and when would you use each?
Why is it important to configure proper span status and error recording for actionable traces?
Explain the TracerProvider → Tracer → Span hierarchy in OpenTelemetry. What is the role of each level and how do you obtain a tracer?
What are the different SpanKind values (INTERNAL, SERVER, CLIENT, PRODUCER, CONSUMER) and why does SpanKind matter?
What is the 'active' or current span in OpenTelemetry, and what is the difference between implicit and explicit parenting?
How do you record an exception on a span in OpenTelemetry, and how does that differ from setting the span status to Error?
Instrumentation Approaches
How do you instrument an application for observability using OpenTelemetry?
Explain the role of OpenTelemetry SDKs and libraries in instrumenting an application for observability.
What is auto-instrumentation in OpenTelemetry, and how does it eliminate the need for manual instrumentation?
What is programmatic instrumentation in OpenTelemetry?
What are OpenTelemetry instrumentation libraries, and how does the registry help you find the right one for a framework or library?
What is the difference between manual (code-based) and automatic (zero-code) instrumentation in OpenTelemetry? Provide examples of scenarios where each would be preferred.
How do you decide which spans to add beyond auto-instrumentation in OpenTelemetry?
How do you configure the OpenTelemetry API, for example to set a global tracer provider and sampler?
How does OpenTelemetry provide automatic instrumentation in multiple languages and what are the approaches (e.g., agent-based)?
Describe the three primary types of instrumentation supported by OpenTelemetry: manual, automatic, and using the Kubernetes Operator.
What is the difference between library (native) instrumentation and separate instrumentation libraries in OpenTelemetry?
What are OpenTelemetry's best practices for where to instrument, such as instrumenting at service boundaries?
Explain the role of eBPF in OpenTelemetry auto-instrumentation.
Opentelemetry Collector
What is the OpenTelemetry Collector and what role does it play?
What are the advantages of using an OpenTelemetry Collector in a production environment compared to directly exporting telemetry from applications?
Describe the architecture of the OpenTelemetry Collector. Explain the role of Receivers, Processors, and Exporters, and how they are wired into pipelines.
Which deployment mode of the OpenTelemetry Collector is most suitable for processing telemetry data from multiple applications on the same host (agent vs. gateway)?
How do you configure the OpenTelemetry Collector pipeline (receivers, processors, exporters)?
How does the OpenTelemetry Collector improve the scalability and performance of telemetry data handling?
What are Collector extensions like health_check, pprof, and zpages used for?
What does the memory_limiter processor do in the OpenTelemetry Collector, and why is it important in production?
What do the batch, attributes, and resource processors do in the OpenTelemetry Collector?
Discuss the common deployment topologies for the OpenTelemetry Collector (e.g., agent/DaemonSet vs. gateway/Deployment) and the trade-offs of each.
How do you implement tail-based sampling with the OpenTelemetry Collector?
What are Connectors in the OpenTelemetry Collector, and how do they bridge one pipeline to another?
What is the difference between the Core and Contrib distributions of the OpenTelemetry Collector, and what is the OpenTelemetry Collector Builder (ocb)?
What is OTTL (OpenTelemetry Transformation Language), and how is it used in the transform and filter processors?
Logs & Correlation
What are OpenTelemetry logs and how would you describe their structure (timestamp, severity, attributes, trace/span identifiers)?
How do you correlate logs and traces using Trace ID and Span ID injected into logs, and why is this critical during an incident?
How do OpenTelemetry logs differ from the traditional logging model, and how do they enable correlation with traces and metrics?
How do application logs enter the OpenTelemetry pipeline through SDKs and log bridges?
What is a log appender in OpenTelemetry, and how does it route existing logging framework output into the OTel pipeline?
How do exemplars link metrics to traces in OpenTelemetry?
What is the OpenTelemetry Logs Bridge API, and why does OpenTelemetry deliberately not provide a logging facade for application developers?
Semantic Conventions & Resources
What are OpenTelemetry resource attributes and how do you use them for service identification?
Why is service.name a required resource attribute in OpenTelemetry, and what happens if it is not set?
What are OpenTelemetry Semantic Conventions, and why are they crucial for making telemetry data portable and queryable across different backends?
Explain resource semantic conventions, especially service.name, and how they describe the entity producing telemetry.
Why should you adopt OpenTelemetry semantic conventions for all custom instrumentation?
How does OpenTelemetry take a pragmatic approach to logging semantic conventions, and what is the goal?
What are Resource detectors in OpenTelemetry, and how do they automatically populate resource attributes?
What is the schema_url in OpenTelemetry semantic conventions, and how does schema versioning help telemetry evolve over time?
Context Propagation
What is context propagation in OpenTelemetry?
What is the benefit of context propagation?
How does OpenTelemetry context propagation follow the flow of execution across network calls (e.g., W3C Trace Context headers)?
What is baggage in OpenTelemetry and how does it differ from span attributes?
What is the difference between inject and extract in the OpenTelemetry propagation API?
What propagator formats does OpenTelemetry support (W3C Trace Context, W3C Baggage, B3, Jaeger), and how do you configure which are used?
In the context of W3C Trace Context, what is the correct behavior when receiving an invalid traceparent header?
Why is it important to implement context propagation correctly across async boundaries and message queues?
Why can context propagation be tricky to implement, and how does OpenTelemetry address it?
What is the tracestate header in W3C Trace Context, and how does it differ from traceparent?
Otlp & Protocols
Where is OTLP typically used?
What are the default OTLP ports 4317 and 4318 used for, and which transport does each correspond to?
What is the OpenTelemetry Protocol (OTLP) and how does it compare to other protocols like Jaeger and Zipkin?
Why is OTLP considered the future of observability (vendor-neutral, decouples instrumentation from backend, standardizes data movement)?
How does the OpenTelemetry Prometheus exporter work, and how does OTel interoperate with Prometheus pull-based scraping and remote write?
What is OTLP, and why is it considered the lingua franca for OpenTelemetry data? Discuss its transport options (gRPC vs. HTTP/protobuf vs. HTTP/JSON) and their use cases.
What is partial success in OTLP, and how does the protocol handle retries and backoff on export failures?
Metrics
Explain the different types of metric instruments available in OpenTelemetry (Counter, UpDownCounter, Histogram, Gauge) and when you would use each.
What is the effect of setting aggregation to drop in an OpenTelemetry metric view?
How do you create custom metrics using the OpenTelemetry Metrics API (counters, histograms, gauges)?
How does the structure of metric semantic conventions allow for querying related metrics?
What are asynchronous (observable) instruments in the OpenTelemetry metrics API, and how do their callbacks work compared to synchronous instruments?
What is a MetricReader in OpenTelemetry, and what is the difference between periodic and manual (pull) readers?
What is the difference between a Gauge and an UpDownCounter in OpenTelemetry, and when would you choose one over the other?
What is the difference between delta and cumulative temporality in OpenTelemetry metrics, and why is this distinction important for backend analysis?
What aggregation types does the OpenTelemetry metrics SDK support (sum, last-value, explicit-bucket histogram, exponential histogram), and when is each used?
What are Views in the OpenTelemetry metrics SDK, and how can they be used to rename metrics, drop attributes, or change aggregation?
Exporters & Span Processing
What are OpenTelemetry exporters and what are some common ones?
How do you export OpenTelemetry data to backends like Datadog, Prometheus, or Jaeger?
How do you implement distributed tracing across microservices with OpenTelemetry?
How do you filter logs or apply sampling in OTel?
What is the difference between the SimpleSpanProcessor and the BatchSpanProcessor, and when would you use each?
Sdk Configuration & Lifecycle
What are the key OTEL_* environment variables used to configure the OpenTelemetry SDK, and what do OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, and OTEL_TRACES_SAMPLER control?
Why is it important to call shutdown or force-flush on OpenTelemetry providers, and what happens if you don't?
What is declarative/file-based configuration in OpenTelemetry, and why is it being introduced alongside environment variables?
Sampling
Explain the terminology 'sampled' vs 'not sampled' in OpenTelemetry.
How do samplers like AlwaysOn, AlwaysOff, TraceIdRatioBased, and ParentBased work within the OpenTelemetry SDK?
Why is sampling important in OpenTelemetry, especially for high-volume systems?
Explain the concepts of head-based and tail-based sampling in OpenTelemetry. When would you use one over the other, and what are the implications for cost and data completeness?
What is the primary purpose of using the tail-based sampling strategy in OpenTelemetry?
Why is it important to generate span metrics carefully and consider sampling decisions at the SDK level?
How does the ParentBased sampler combine with other samplers to form OpenTelemetry's composite default sampling behavior?
What is consistent probability sampling and the sampling threshold in OpenTelemetry, and what problem does it solve?
Kubernetes & Fleet Management
How do you implement OpenTelemetry in a Kubernetes environment with the OpenTelemetry Operator?
How does the OpenTelemetry Operator for Kubernetes facilitate auto-instrumentation (e.g., injecting agents via initContainers)?
What is OpAMP (OpenTelemetry Agent Management Protocol), and how does it relate to managing a fleet of OpenTelemetry Collectors and agents?
How does the OpenTelemetry Collector CRD managed by the Operator differ from the auto-instrumentation injection it provides?
Production Operations & Security
How should sensitive data be handled when using OpenTelemetry to ensure security and privacy?
How do you handle high-cardinality attributes and control telemetry costs in OpenTelemetry?
How do you troubleshoot missing spans, broken traces, and context propagation failures in OpenTelemetry?
What challenges have you faced while implementing OpenTelemetry in production?