Sampling & Telemetry Cost
As a system scales, the cost of observability can exceed the cost of the infrastructure itself: what strategies can you use to reduce telemetry volume without losing visibility?
How do data retention policies affect observability, and how do you balance retention duration against storage cost?
How do you approach making a system 'observable' for other engineers? What are the trade-offs between providing deep technical insights and keeping the telemetry cost-effective?
What are the cost and performance trade-offs of 'Full-fidelity' tracing versus 'Sampling'?
What are the trade-offs of log sampling? How do you decide which logs are 'interesting' enough to keep and which to drop to save on storage costs?
What is the difference between head-based and tail-based sampling? What are the pros and cons of each when trying to capture intermittent errors?
What is the impact of data downsampling or rollups on long-term observability? What information is lost when you move from 1-minute to 1-hour granularity?
Monitoring Methodologies
Compare the RED method (Rate, Errors, Duration) and the USE method (Utilization, Saturation, Errors). Which one is more appropriate for monitoring a database versus a public-facing API?
Identify the 'Four Golden Signals' of monitoring. If you could only pick two to alert on for a user-facing service, which would they be and why?
What is Apdex, and how does it attempt to summarize user satisfaction with response times into a single score?
Why is it dangerous to use averages to measure request latency, and why are p95 or p99 percentiles preferred for understanding user experience?
Slis Slos & Error Budgets
Define SLIs, SLOs, and SLAs. Who is the primary audience for each, and how do they relate to one another?
Explain the concept of an SLO burn rate. Why is it more useful than a simple threshold alert for long-term reliability?
If you were tasked with defining SLIs for a new asynchronous message-processing service, which metrics would you prioritize and why?
What is an 'Error Budget'? If a team has exhausted its error budget for the month, what practical actions should they take regarding their deployment pipeline?
What is the difference between a good SLI and a bad SLI, and what makes a metric a meaningful indicator of user experience?
Three Pillars & Signal Types
Explain the 'Three Pillars of Observability' (Metrics, Logs, Traces). In what specific scenarios would you prioritize one over the others for root-cause analysis?
Explain the relationship between a Trace and a Span. What kind of metadata should be included in a span to make it useful for root-cause analysis?
What is structured logging, and why is it superior to plain-text logs for modern observability pipelines?
What is time-series data, and what characteristics distinguish a time-series database from a general-purpose database?
Instrumentation & Collection
Explain the concept of a 'telemetry collector' or 'agent.' Why would you use a collector as a buffer/proxy instead of sending data directly from the application to the backend?
What are the conceptual differences between a 'Push' model (e.g., StatsD) and a 'Pull' model (e.g., Prometheus) for telemetry collection? What are the scaling implications of each?
What are the performance trade-offs of adding heavy instrumentation to a high-throughput production service?
What are the pros and cons of manual instrumentation versus auto-instrumentation?
What is OpenTelemetry (OTel), and why is the industry moving toward it as a vendor-neutral standard?
What is the difference between OpenTracing and OpenTelemetry, and how did the standards converge?
What is the role of the OpenTelemetry Collector? Why would a developer use the OTel SDK instead of a vendor-specific agent?
Why is buffering and back-pressure important in a telemetry pipeline? What happens to your observability if the monitoring system itself becomes the bottleneck?
Logging Pipelines
Explain the cost and performance tradeoffs between log ingestion and log indexing. When would you choose to 'live tail' logs instead of indexing them?
Explain the ELK/EFK stack conceptually. What role does each component play in a centralized logging pipeline?
What are log levels (DEBUG, INFO, WARN, ERROR, etc.), and how do you decide what to log at each level in production?
What are log-based metrics, and when would you derive a metric from logs instead of instrumenting a metric directly?
Alerting & On Call
How do alert deduplication, grouping, and routing work, and why are they important for an on-call system?
How do you design an alerting system to minimize 'alert fatigue' and 'noise', and what criteria make an alert 'actionable'?
What is a runbook, and what role does it play in responding to alerts?
What is an escalation policy, and how should paging and escalation be structured for an on-call rotation?
What is the difference between monitoring and alerting, and why shouldn't every monitored metric have an alert?
What is the difference between static threshold alerting and anomaly-detection-based alerting, and when is each appropriate?
What is the difference between symptom-based alerting and cause-based alerting, and why is symptom-based alerting generally preferred for on-call rotations?
Distributed Tracing
How do you correlate a specific log line to a specific distributed trace? Why is this correlation critical during an incident?
How does distributed tracing track a single request across multiple microservices? Explain the concept of 'context propagation.'
What is a correlation ID, and how does it help trace a request through logs across multiple services?
What is a flame graph or waterfall view in distributed tracing, and how do you read one to find a bottleneck?
What is the difference between span attributes and span events, and how do you use each during root-cause analysis?
Foundations & Concepts
How do you debug an intermittent issue in production that you cannot reproduce, and which observability signals help most?
Observability is a term borrowed from control theory. In a software context, what does it mean to 'infer the internal state of a system from its external outputs'?
What does it mean for a system to have 'Unknown Unknowns,' and how does observability help address them?
What is the difference between 'White-box' and 'Black-box' monitoring, and when would you prefer one over the other?
What is the difference between an event, a metric, and a log, and when should telemetry be captured as one versus another?
What is the fundamental difference between monitoring and observability? When does a system transition from being 'monitored' to being 'observable'?
Why is it important to distinguish correlation from causation when investigating an incident using telemetry?
Metrics & Cardinality
What are labels or dimensions on a metric, and how do they enable slicing and dicing of telemetry?
What does 'saturation' mean as a golden signal, and how do you measure it for different types of resources?
What is 'cardinality' in the context of metrics, and why is a 'cardinality explosion' dangerous for a monitoring system? How would you mitigate it?
What is the difference between a Counter, a Gauge, and a Histogram? Why would you never use a Gauge to track the total number of requests handled?
What is the difference between a Histogram and a Summary metric type, and why do histograms aggregate better across multiple instances?
What is the difference between the 'rate' and 'increase' of a counter, and why can't you alert on a raw counter value directly?
Reliability Metrics & Monitoring Practices
What are MTTR (Mean Time to Resolve) and MTTD (Mean Time to Detect), and which observability signals most directly impact each?
What are the unique challenges of observing an LLM-based application compared to a traditional CRUD app?
What does it mean to practice 'Observability-Driven Development', and how does it change the way a developer writes code?
What is a health check or heartbeat, and how does it differ from a full metrics-based assessment of service health?
What is Application Performance Monitoring (APM), and what does an APM tool give you beyond raw metrics and logs?
What is MTBF (Mean Time Between Failures), and how does it relate to MTTR and MTTD when reasoning about reliability?
What is the concept of 'Observability 2.0' or 'Unified Telemetry,' and how does it differ from the traditional siloed approach of having separate tools for logs and metrics?
What is the difference between a Liveness probe and a Readiness probe? What happens if you misconfigure a readiness probe to check a downstream database that is currently down?
What is the difference between Real User Monitoring (RUM) and Synthetic Monitoring? When would synthetic tests fail to catch an issue that RUM would identify?
What is uptime or availability monitoring, and how is availability typically expressed and measured?
What makes a good dashboard, and what are common anti-patterns when visualizing metrics?