As a system scales, the cost of observability can exceed the cost of the infrastructure itself: what strategies can you use to reduce telemetry volume without losing visibility?
Senior
How do data retention policies affect observability, and how do you balance retention duration against storage cost?
Mid
How do you approach making a system 'observable' for other engineers? What are the trade-offs between providing deep technical insights and keeping the telemetry cost-effective?
Senior
What are the cost and performance trade-offs of 'Full-fidelity' tracing versus 'Sampling'?
Senior
What are the trade-offs of log sampling? How do you decide which logs are 'interesting' enough to keep and which to drop to save on storage costs?
Senior
What is the difference between head-based and tail-based sampling? What are the pros and cons of each when trying to capture intermittent errors?
Senior
What is the impact of data downsampling or rollups on long-term observability? What information is lost when you move from 1-minute to 1-hour granularity?
Senior
Monitoring Methodologies
Compare the RED method (Rate, Errors, Duration) and the USE method (Utilization, Saturation, Errors). Which one is more appropriate for monitoring a database versus a public-facing API?
Senior
Identify the 'Four Golden Signals' of monitoring. If you could only pick two to alert on for a user-facing service, which would they be and why?
Mid
What is Apdex, and how does it attempt to summarize user satisfaction with response times into a single score?
Mid
Why is it dangerous to use averages to measure request latency, and why are p95 or p99 percentiles preferred for understanding user experience?
Mid
Slis Slos & Error Budgets
Define SLIs, SLOs, and SLAs. Who is the primary audience for each, and how do they relate to one another?
Junior
Explain the concept of an SLO burn rate. Why is it more useful than a simple threshold alert for long-term reliability?
Senior
If you were tasked with defining SLIs for a new asynchronous message-processing service, which metrics would you prioritize and why?
Senior
What is an 'Error Budget'? If a team has exhausted its error budget for the month, what practical actions should they take regarding their deployment pipeline?
Mid
What is the difference between a good SLI and a bad SLI, and what makes a metric a meaningful indicator of user experience?
Senior
Three Pillars & Signal Types
Explain the 'Three Pillars of Observability' (Metrics, Logs, Traces). In what specific scenarios would you prioritize one over the others for root-cause analysis?
Mid
Explain the relationship between a Trace and a Span. What kind of metadata should be included in a span to make it useful for root-cause analysis?
Mid
What is structured logging, and why is it superior to plain-text logs for modern observability pipelines?
Junior
What is time-series data, and what characteristics distinguish a time-series database from a general-purpose database?
Mid
Instrumentation & Collection
Explain the concept of a 'telemetry collector' or 'agent.' Why would you use a collector as a buffer/proxy instead of sending data directly from the application to the backend?
Mid
What are the conceptual differences between a 'Push' model (e.g., StatsD) and a 'Pull' model (e.g., Prometheus) for telemetry collection? What are the scaling implications of each?
Senior
What are the performance trade-offs of adding heavy instrumentation to a high-throughput production service?
Senior
What are the pros and cons of manual instrumentation versus auto-instrumentation?
Mid
What is OpenTelemetry (OTel), and why is the industry moving toward it as a vendor-neutral standard?
Junior
What is the difference between OpenTracing and OpenTelemetry, and how did the standards converge?
Senior
What is the role of the OpenTelemetry Collector? Why would a developer use the OTel SDK instead of a vendor-specific agent?
Mid
Why is buffering and back-pressure important in a telemetry pipeline? What happens to your observability if the monitoring system itself becomes the bottleneck?
Senior
Logging Pipelines
Explain the cost and performance tradeoffs between log ingestion and log indexing. When would you choose to 'live tail' logs instead of indexing them?
Senior
Explain the ELK/EFK stack conceptually. What role does each component play in a centralized logging pipeline?
Mid
What are log levels (DEBUG, INFO, WARN, ERROR, etc.), and how do you decide what to log at each level in production?
Junior
What are log-based metrics, and when would you derive a metric from logs instead of instrumenting a metric directly?
Mid
Alerting & On Call
How do alert deduplication, grouping, and routing work, and why are they important for an on-call system?
Mid
How do you design an alerting system to minimize 'alert fatigue' and 'noise', and what criteria make an alert 'actionable'?
Senior
What is a runbook, and what role does it play in responding to alerts?
Junior
What is an escalation policy, and how should paging and escalation be structured for an on-call rotation?
Mid
What is the difference between monitoring and alerting, and why shouldn't every monitored metric have an alert?
Junior
What is the difference between static threshold alerting and anomaly-detection-based alerting, and when is each appropriate?
Senior
What is the difference between symptom-based alerting and cause-based alerting, and why is symptom-based alerting generally preferred for on-call rotations?
Mid
Distributed Tracing
How do you correlate a specific log line to a specific distributed trace? Why is this correlation critical during an incident?
Mid
How does distributed tracing track a single request across multiple microservices? Explain the concept of 'context propagation.'
Mid
What is a correlation ID, and how does it help trace a request through logs across multiple services?
Mid
What is a flame graph or waterfall view in distributed tracing, and how do you read one to find a bottleneck?
Mid
What is the difference between span attributes and span events, and how do you use each during root-cause analysis?
Mid
Foundations & Concepts
How do you debug an intermittent issue in production that you cannot reproduce, and which observability signals help most?
Senior
Observability is a term borrowed from control theory. In a software context, what does it mean to 'infer the internal state of a system from its external outputs'?
Mid
What does it mean for a system to have 'Unknown Unknowns,' and how does observability help address them?
Mid
What is the difference between 'White-box' and 'Black-box' monitoring, and when would you prefer one over the other?
Mid
What is the difference between an event, a metric, and a log, and when should telemetry be captured as one versus another?
Mid
What is the fundamental difference between monitoring and observability? When does a system transition from being 'monitored' to being 'observable'?
Junior
Why is it important to distinguish correlation from causation when investigating an incident using telemetry?
Senior
Metrics & Cardinality
What are labels or dimensions on a metric, and how do they enable slicing and dicing of telemetry?
Junior
What does 'saturation' mean as a golden signal, and how do you measure it for different types of resources?
Mid
What is 'cardinality' in the context of metrics, and why is a 'cardinality explosion' dangerous for a monitoring system? How would you mitigate it?
Senior
What is the difference between a Counter, a Gauge, and a Histogram? Why would you never use a Gauge to track the total number of requests handled?
Junior
What is the difference between a Histogram and a Summary metric type, and why do histograms aggregate better across multiple instances?
Senior
What is the difference between the 'rate' and 'increase' of a counter, and why can't you alert on a raw counter value directly?
Mid
Reliability Metrics & Monitoring Practices
What are MTTR (Mean Time to Resolve) and MTTD (Mean Time to Detect), and which observability signals most directly impact each?
Mid
What are the unique challenges of observing an LLM-based application compared to a traditional CRUD app?
Senior
What does it mean to practice 'Observability-Driven Development', and how does it change the way a developer writes code?
Senior
What is a health check or heartbeat, and how does it differ from a full metrics-based assessment of service health?
Junior
What is Application Performance Monitoring (APM), and what does an APM tool give you beyond raw metrics and logs?
Mid
What is MTBF (Mean Time Between Failures), and how does it relate to MTTR and MTTD when reasoning about reliability?
Mid
What is the concept of 'Observability 2.0' or 'Unified Telemetry,' and how does it differ from the traditional siloed approach of having separate tools for logs and metrics?
Senior
What is the difference between a Liveness probe and a Readiness probe? What happens if you misconfigure a readiness probe to check a downstream database that is currently down?
Mid
What is the difference between Real User Monitoring (RUM) and Synthetic Monitoring? When would synthetic tests fail to catch an issue that RUM would identify?
Mid
What is uptime or availability monitoring, and how is availability typically expressed and measured?
Junior
What makes a good dashboard, and what are common anti-patterns when visualizing metrics?