Opentelemetry Collector
Describe the architecture of the OpenTelemetry Collector. Explain the role of Receivers, Processors, and Exporters, and how they are wired into pipelines.
Discuss the common deployment topologies for the OpenTelemetry Collector (e.g., agent/DaemonSet vs. gateway/Deployment) and the trade-offs of each.
How do you configure the OpenTelemetry Collector pipeline (receivers, processors, exporters)?
How do you implement tail-based sampling with the OpenTelemetry Collector?
How does the OpenTelemetry Collector improve the scalability and performance of telemetry data handling?
What are Collector extensions like health_check, pprof, and zpages used for?
What are Connectors in the OpenTelemetry Collector, and how do they bridge one pipeline to another?
What are the advantages of using an OpenTelemetry Collector in a production environment compared to directly exporting telemetry from applications?
What do the batch, attributes, and resource processors do in the OpenTelemetry Collector?
What does the memory_limiter processor do in the OpenTelemetry Collector, and why is it important in production?
What is OTTL (OpenTelemetry Transformation Language), and how is it used in the transform and filter processors?
What is the difference between the Core and Contrib distributions of the OpenTelemetry Collector, and what is the OpenTelemetry Collector Builder (ocb)?
What is the OpenTelemetry Collector and what role does it play?
Which deployment mode of the OpenTelemetry Collector is most suitable for processing telemetry data from multiple applications on the same host (agent vs. gateway)?
Tracing Concepts
Describe the structure of an OpenTelemetry Span. What are its key components, such as SpanContext, SpanKind, attributes, events, and links, and what role does each play in distributed tracing?
Explain the TracerProvider → Tracer → Span hierarchy in OpenTelemetry. What is the role of each level and how do you obtain a tracer?
Explain the tracing concepts: Trace, Span, Trace ID, Span ID, and parent-child relationships.
How are traces represented in OTLP, specifically through spans and their key fields like trace_id and span_id?
How do you record an exception on a span in OpenTelemetry, and how does that differ from setting the span status to Error?
How does distributed tracing track a single request across multiple microservices, and what is context propagation in OpenTelemetry?
What are span events and span links, and when would you use each?
What are the different SpanKind values (INTERNAL, SERVER, CLIENT, PRODUCER, CONSUMER) and why does SpanKind matter?
What is an OpenTelemetry Context and what's special about how it follows the thread of execution?
What is the 'active' or current span in OpenTelemetry, and what is the difference between implicit and explicit parenting?
What is the difference between span attributes and span events in OpenTelemetry, and how do you use each during root-cause analysis?
What is the span status in OpenTelemetry, and what do the Unset, Ok, and Error values signify?
Why is it important to configure proper span status and error recording for actionable traces?
Instrumentation Approaches
Describe the three primary types of instrumentation supported by OpenTelemetry: manual, automatic, and using the Kubernetes Operator.
Explain the role of eBPF in OpenTelemetry auto-instrumentation.
Explain the role of OpenTelemetry SDKs and libraries in instrumenting an application for observability.
How do you configure the OpenTelemetry API, for example to set a global tracer provider and sampler?
How do you decide which spans to add beyond auto-instrumentation in OpenTelemetry?
How do you instrument an application for observability using OpenTelemetry?
How does OpenTelemetry provide automatic instrumentation in multiple languages and what are the approaches (e.g., agent-based)?
What are OpenTelemetry instrumentation libraries, and how does the registry help you find the right one for a framework or library?
What are OpenTelemetry's best practices for where to instrument, such as instrumenting at service boundaries?
What is auto-instrumentation in OpenTelemetry, and how does it eliminate the need for manual instrumentation?
What is programmatic instrumentation in OpenTelemetry?
What is the difference between library (native) instrumentation and separate instrumentation libraries in OpenTelemetry?
What is the difference between manual (code-based) and automatic (zero-code) instrumentation in OpenTelemetry? Provide examples of scenarios where each would be preferred.
Signals & Data Model
Explain how the three pillars of observability—logs, metrics, and traces—are interconnected via trace ID and span ID in OpenTelemetry.
Explain the four main signals in OpenTelemetry (Traces, Metrics, Logs, and Profiles). What is each signal used for, and how do they inter-correlate within the OpenTelemetry data model?
What does signal stability or maturity (stable vs. experimental) mean in OpenTelemetry, and why does it matter when adopting a signal?
What is the cross-cutting data model in OpenTelemetry, and how does a shared model across signals enable correlation?
What is the emerging Profiles signal in OpenTelemetry, and what problem does continuous profiling solve alongside traces, metrics, and logs?
Semantic Conventions & Resources
Explain resource semantic conventions, especially service.name, and how they describe the entity producing telemetry.
How does OpenTelemetry take a pragmatic approach to logging semantic conventions, and what is the goal?
What are OpenTelemetry resource attributes and how do you use them for service identification?
What are OpenTelemetry Semantic Conventions, and why are they crucial for making telemetry data portable and queryable across different backends?
What are Resource detectors in OpenTelemetry, and how do they automatically populate resource attributes?
What is the schema_url in OpenTelemetry semantic conventions, and how does schema versioning help telemetry evolve over time?
Why is service.name a required resource attribute in OpenTelemetry, and what happens if it is not set?
Why should you adopt OpenTelemetry semantic conventions for all custom instrumentation?
Sampling
Explain the concepts of head-based and tail-based sampling in OpenTelemetry. When would you use one over the other, and what are the implications for cost and data completeness?
Explain the terminology 'sampled' vs 'not sampled' in OpenTelemetry.
How do samplers like AlwaysOn, AlwaysOff, TraceIdRatioBased, and ParentBased work within the OpenTelemetry SDK?
How does the ParentBased sampler combine with other samplers to form OpenTelemetry's composite default sampling behavior?
What is consistent probability sampling and the sampling threshold in OpenTelemetry, and what problem does it solve?
What is the primary purpose of using the tail-based sampling strategy in OpenTelemetry?
Why is it important to generate span metrics carefully and consider sampling decisions at the SDK level?
Why is sampling important in OpenTelemetry, especially for high-volume systems?
Fundamentals & Vendor Neutrality
Explain the difference between pull-based and push-based monitoring architectures, and where OpenTelemetry fits in.
How does an observability solution support OpenTelemetry data, given that OTel defines instrumentation and semantic conventions but not storage/visualization?
How does OpenTelemetry achieve vendor neutrality, and what are its benefits?
How does OpenTelemetry differ from older standards like OpenTracing or OpenCensus?
How does OpenTelemetry facilitate distributed tracing, and why is this significant for monitoring modern applications?
How is OpenTelemetry built (API & SDK, auto-instrumentation, shared spec across languages)?
What is OpenTelemetry and what is it not (e.g., generation + collection + export, not storage/analysis/visualization)?
What is OpenTelemetry and why was it created?
What is OpenTelemetry instrumentation?
What is the OpenTelemetry Demo application, and how is it used to learn and evaluate the framework?
Why is OpenTelemetry considered a 'big deal' (universal standard, CNCF, consistent format, data model, APIs, SDK, Collector, auto-instrumentation)?
Why would you choose OpenTelemetry over vendor-specific agents?
Architecture & Core Components
Explain the differences between the OpenTelemetry API and SDK in terms of purpose, audience, stability, implementation, dependency, and resource usage.
How do the OTel SDK, Collector, and Exporters work together?
How does OpenTelemetry's API and SDK separation contribute to its flexibility and adoption?
In OpenTelemetry's architecture, which component is responsible for collecting telemetry data from different sources and preparing it for export?
In what ways do the OpenTelemetry SDK and Collector differ in their operation and purpose within the OpenTelemetry ecosystem?
What are the core components of OpenTelemetry (API, SDK, Collector, Exporters, Context Propagation)?
What are the downsides of wrapping the OpenTelemetry API/SDK?
What are the key components of the OpenTelemetry API (Tracers, Meters, Loggers)?
Why is the OpenTelemetry API designed to be a no-op by default, and why is this important for library authors?
Metrics
Explain the different types of metric instruments available in OpenTelemetry (Counter, UpDownCounter, Histogram, Gauge) and when you would use each.
How do you create custom metrics using the OpenTelemetry Metrics API (counters, histograms, gauges)?
How does the structure of metric semantic conventions allow for querying related metrics?
What aggregation types does the OpenTelemetry metrics SDK support (sum, last-value, explicit-bucket histogram, exponential histogram), and when is each used?
What are asynchronous (observable) instruments in the OpenTelemetry metrics API, and how do their callbacks work compared to synchronous instruments?
What are Views in the OpenTelemetry metrics SDK, and how can they be used to rename metrics, drop attributes, or change aggregation?
What is a MetricReader in OpenTelemetry, and what is the difference between periodic and manual (pull) readers?
What is the difference between a Gauge and an UpDownCounter in OpenTelemetry, and when would you choose one over the other?
What is the difference between delta and cumulative temporality in OpenTelemetry metrics, and why is this distinction important for backend analysis?
What is the effect of setting aggregation to drop in an OpenTelemetry metric view?
Project Structure & Compatibility
Explain the significance of 'vendor consolidation around OTLP' in the observability ecosystem.
How does OpenTelemetry provide backward-compatibility shims or bridges for OpenTracing and OpenCensus instrumentation?
How is the OpenTelemetry project structured, and what role do the SIGs and the specification play across languages?
Why did OTel emerge as the front-runner for semantic conventions compared to other projects like Elastic Common Schema or Splunk's CIM?
Logs & Correlation
How do application logs enter the OpenTelemetry pipeline through SDKs and log bridges?
How do exemplars link metrics to traces in OpenTelemetry?
How do OpenTelemetry logs differ from the traditional logging model, and how do they enable correlation with traces and metrics?
How do you correlate logs and traces using Trace ID and Span ID injected into logs, and why is this critical during an incident?
What are OpenTelemetry logs and how would you describe their structure (timestamp, severity, attributes, trace/span identifiers)?
What is a log appender in OpenTelemetry, and how does it route existing logging framework output into the OTel pipeline?
What is the OpenTelemetry Logs Bridge API, and why does OpenTelemetry deliberately not provide a logging facade for application developers?
Exporters & Span Processing
How do you export OpenTelemetry data to backends like Datadog, Prometheus, or Jaeger?
How do you filter logs or apply sampling in OTel?
How do you implement distributed tracing across microservices with OpenTelemetry?
What are OpenTelemetry exporters and what are some common ones?
What is the difference between the SimpleSpanProcessor and the BatchSpanProcessor, and when would you use each?
Production Operations & Security
How do you handle high-cardinality attributes and control telemetry costs in OpenTelemetry?
How do you troubleshoot missing spans, broken traces, and context propagation failures in OpenTelemetry?
How should sensitive data be handled when using OpenTelemetry to ensure security and privacy?
What challenges have you faced while implementing OpenTelemetry in production?
Kubernetes & Fleet Management
How do you implement OpenTelemetry in a Kubernetes environment with the OpenTelemetry Operator?
How does the OpenTelemetry Collector CRD managed by the Operator differ from the auto-instrumentation injection it provides?
How does the OpenTelemetry Operator for Kubernetes facilitate auto-instrumentation (e.g., injecting agents via initContainers)?
What is OpAMP (OpenTelemetry Agent Management Protocol), and how does it relate to managing a fleet of OpenTelemetry Collectors and agents?
Context Propagation
How does OpenTelemetry context propagation follow the flow of execution across network calls (e.g., W3C Trace Context headers)?
In the context of W3C Trace Context, what is the correct behavior when receiving an invalid traceparent header?
What is baggage in OpenTelemetry and how does it differ from span attributes?
What is context propagation in OpenTelemetry?
What is the benefit of context propagation?
What is the difference between inject and extract in the OpenTelemetry propagation API?
What is the tracestate header in W3C Trace Context, and how does it differ from traceparent?
What propagator formats does OpenTelemetry support (W3C Trace Context, W3C Baggage, B3, Jaeger), and how do you configure which are used?
Why can context propagation be tricky to implement, and how does OpenTelemetry address it?
Why is it important to implement context propagation correctly across async boundaries and message queues?
Otlp & Protocols
How does the OpenTelemetry Prometheus exporter work, and how does OTel interoperate with Prometheus pull-based scraping and remote write?
What are the default OTLP ports 4317 and 4318 used for, and which transport does each correspond to?
What is OTLP, and why is it considered the lingua franca for OpenTelemetry data? Discuss its transport options (gRPC vs. HTTP/protobuf vs. HTTP/JSON) and their use cases.
What is partial success in OTLP, and how does the protocol handle retries and backoff on export failures?
What is the OpenTelemetry Protocol (OTLP) and how does it compare to other protocols like Jaeger and Zipkin?
Where is OTLP typically used?
Why is OTLP considered the future of observability (vendor-neutral, decouples instrumentation from backend, standardizes data movement)?
Sdk Configuration & Lifecycle
What are the key OTEL_* environment variables used to configure the OpenTelemetry SDK, and what do OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, and OTEL_TRACES_SAMPLER control?
What is declarative/file-based configuration in OpenTelemetry, and why is it being introduced alongside environment variables?
Why is it important to call shutdown or force-flush on OpenTelemetry providers, and what happens if you don't?