68 Monitoring & Observability Interview Questions and Answers (2026)

Monitoring tells you whether predefined conditions are healthy (you ask known questions with known dashboards and alerts); observability is a property of the system that lets you ask arbitrary new questions about its behavior without shipping new code. A system becomes observable when its telemetry is rich and high-cardinality enough to explain novel failure modes you never anticipated.

• Monitoring: known-unknowns:

  • You decide in advance what to measure and alert on (CPU, error rate, latency thresholds).

  • Answers "is it working?" but struggles with "why is this specific request slow?"

• Observability: unknown-unknowns:

  • A property: can you understand any internal state from outputs without new deploys?

  • Relies on wide, context-rich events you can slice by arbitrary dimensions (user ID, region, build).

• The transition point:

  • When you can freely explore and correlate signals to answer questions you didn't predefine.

  • High cardinality and dimensionality matter more than more dashboards.

• Relationship: monitoring is a subset of what an observable system enables, not a competitor.

Structured logging emits logs as machine-parseable key/value data (typically JSON) instead of free-form text sentences. It's superior because it can be reliably queried, filtered, and aggregated by field without fragile regex parsing, which is essential for modern pipelines that index and correlate at scale.

• What it is: Each log is a set of typed fields (timestamp, level, user_id, latency_ms, trace_id) rather than a prose string.

• Why it beats plain text:

  • Queryable: filter/group on fields directly instead of brittle regex over strings.

  • Consistent schema survives message wording changes that break text parsers.

  • Correlatable: embed trace_id/request_id to link logs to traces and metrics.

  • Enables aggregation and alerting on log-derived numeric fields.

• Caveats:

  • Slightly less human-readable raw, so tooling matters.

  • Keep field names consistent across services or you lose the ability to query uniformly.

OpenTelemetry is an open-source, vendor-neutral standard (a CNCF project) that unifies how you generate, collect, and export telemetry data: traces, metrics, and logs. It gives you one set of APIs, SDKs, and wire protocols so instrumentation is decoupled from any specific backend.

• What it provides:

  • A cross-language API/SDK to instrument code, plus the OTLP wire protocol and the Collector for receiving and exporting data.

  • Covers all three signals (traces, metrics, logs) under one correlated model.

• Why the industry is converging on it:

  • No vendor lock-in: instrument once, then swap backends (Jaeger, Prometheus, Datadog, etc.) by changing exporter config, not code.

  • It's the merger of OpenTracing and OpenCensus, so the community consolidated around a single standard instead of competing ones.

  • Broad ecosystem support and auto-instrumentation for common libraries reduce boilerplate.

• Practical payoff: Consistent semantic conventions (standardized attribute names) make data portable and comparable across services and tools.

SLIs, SLOs, and SLAs form a hierarchy of reliability measurement: an SLI is what you measure, an SLO is the internal target for that measurement, and an SLA is the external contract with consequences. They build on each other, with the SLA typically looser than the SLO to give you a safety margin.

• SLI (Service Level Indicator):

  • A quantitative measure of service behavior, e.g. request success rate, p99 latency, or availability.

  • Audience: engineers/SREs who instrument and monitor it.

• SLO (Service Level Objective):

  • A target for an SLI over a window, e.g. 99.9% of requests succeed over 30 days.

  • Audience: internal teams and product; it drives the error budget and prioritization.

• SLA (Service Level Agreement):

  • A formal contract with customers promising a level of service, with penalties (credits/refunds) if breached.

  • Audience: customers, legal, business stakeholders.

• How they relate:

  • SLI is the metric, SLO is the goal built on it, SLA is the externally promised (usually looser) version of the SLO.

  • The gap between SLO and SLA is your buffer: you aim to violate the SLO before you ever breach the SLA.

A Counter is a monotonically increasing value that only goes up (or resets to zero), a Gauge is a value that can go up and down, and a Histogram samples observations into buckets to describe a distribution. You'd never use a Gauge for total requests because a Gauge can decrease and has no reset semantics, so you lose the guarantees that make rate calculations correct.

• Counter: cumulative, only increases:

  • For counts of events: requests served, errors, bytes sent.

  • You query it with rate(), which understands resets on restart.

• Gauge: point-in-time value that rises and falls: For things you measure now: memory used, queue depth, active connections, temperature.

• Histogram: distribution of observations: Counts observations into buckets plus a sum and count, enabling percentiles (request latency, payload sizes).

• Why not a Gauge for total requests:

  • A Gauge can go down, so tooling can't distinguish a real decrease from a counter reset.

  • Counters carry monotonic + reset semantics that let rate() compute accurate per-second throughput; a Gauge would break that.

Labels (called dimensions or tags) are key-value pairs attached to a metric that break a single measurement into many independent time series. They let you slice and dice: query, filter, group, and aggregate telemetry along whatever axes you attached, without emitting a separate metric name per variation.

• What they are: A metric like http_requests_total with labels method, status, route becomes one series per unique combination.

• How they enable slicing:

  • Filter: only status=500 to see errors.

  • Group/aggregate: sum by (route) to compare endpoints.

  • Drill down: pivot from a global rate to per-region or per-instance without new instrumentation.

• The trade-off: Every label combination is a stored series, so high-cardinality labels drive cost and cardinality explosions: keep values bounded.

A runbook is a documented, step-by-step guide for diagnosing and resolving a specific alert or failure scenario. It turns tribal knowledge into a repeatable procedure so any on-call engineer, not just the expert, can respond quickly and consistently.

• What it contains:

  • What the alert means and its likely impact.

  • Diagnostic steps: which dashboards, queries, and logs to check.

  • Remediation steps: common fixes, rollback/failover procedures, and how to verify recovery.

  • Escalation contacts if the steps don't resolve it.

• Role in alert response:

  • Reduces mean-time-to-resolution and cognitive load during high-stress incidents.

  • Makes response consistent regardless of who is on-call, reducing key-person dependence.

  • Best practice: link the runbook directly in the alert (a runbook_url annotation).

• Keeping it useful:

  • Living document: update after each incident and post-mortem.

  • An alert without a runbook is a signal the alert may not be actionable enough.

Monitoring is collecting and storing signals about a system's state; alerting is the subset of that data important enough to demand human attention. Every metric is worth observing for diagnosis, but only a few represent conditions urgent and actionable enough to interrupt a person.

• Monitoring:

  • Broad, continuous collection of metrics, logs, and traces for visibility and debugging.

  • Its value is in dashboards and post-hoc investigation; more data is generally fine.

• Alerting:

  • A rule on top of monitoring that fires when a condition needs action.

  • Its value is precision: every alert should map to a decision or response.

• Why not alert on everything:

  • Alerting on every metric creates noise and alert fatigue, so real problems get ignored.

  • Most metrics (CPU, cache hit rate) are causes/context, useful for diagnosis but not worth paging on.

  • Alert on user-facing symptoms and SLO breaches; keep the rest as monitored-but-not-alerted.

Log levels are a severity ranking (typically DEBUG < INFO < WARN < ERROR < FATAL) that lets you control verbosity and filter by importance. In production you set a threshold (usually INFO) and choose each level by asking who needs to see this message and how urgently.

• The levels:

  1. DEBUG: detailed developer diagnostics; noisy, usually off in prod.

  2. INFO: normal significant events (startup, request handled, job completed).

  3. WARN: something unexpected but recoverable, worth watching (retry, degraded mode, deprecation).

  4. ERROR: an operation failed and needs attention (unhandled exception, failed request).

  5. FATAL/CRITICAL: the process/service can't continue.

• How to decide in production:

  • Run at INFO by default; enable DEBUG dynamically/temporarily when investigating.

  • Reserve ERROR for things a human may need to act on, so error logs stay meaningful and alertable.

  • Use WARN for early-warning signals that don't yet warrant action.

  • Watch cost: DEBUG in prod is expensive and noisy; too much at ERROR desensitizes responders.

• Pair with structured logging: Attach context (request id, user id) as fields so level-based filtering plus fields make logs queryable.

A health check is a lightweight endpoint or probe that reports whether a service is alive and able to serve, usually as a simple yes/no. It's a coarse liveness signal, whereas metrics-based health assessment judges quality of service (latency, error rate, saturation) over time.

• Health check / heartbeat:

  • A periodic ping or endpoint (/healthz) returning up/down; used by load balancers and orchestrators.

  • Liveness (is the process alive?) vs readiness (can it accept traffic yet?) are distinct checks in systems like Kubernetes.

  • Shallow (process responds) vs deep (checks dependencies like DB) trades speed for accuracy.

• Metrics-based assessment:

  • Continuous, quantitative: is it serving well, not just up? A service can pass a health check while p99 latency or error rate is terrible.

  • Feeds SLOs and trend analysis; health checks only give a binary snapshot.

• Rule of thumb: health checks decide routing/restarts; metrics decide whether users are actually being served.

Availability monitoring tracks the proportion of time a service is up and successfully serving requests. It's usually expressed as a percentage over a window (the 'nines') and measured either by external probes or by the ratio of good to total requests.

• How it's expressed:

  • Percentage of uptime: 99.9% ('three nines') ≈ 43 min downtime/month; 99.99% ≈ 4.3 min/month.

  • Time-based: uptime / (uptime + downtime).

  • Request-based (SRE style): successful requests / total requests, better reflecting user experience.

• How it's measured:

  • External synthetic probes / uptime checks from multiple regions.

  • Server-side success/error ratios derived from real traffic.

• Tie to SLOs: An availability SLO defines a target; the gap to 100% is the error budget you can spend on releases and risk.

White-box monitoring inspects a system's internal state (metrics, logs, traces it exposes), while black-box monitoring probes it from the outside as a user would, without knowledge of internals. You want both: black-box tells you something is broken for users, white-box tells you why.

• White-box: based on internals:

  • Uses instrumented signals the app emits: request rates, queue depth, GC pauses, DB connection pool usage.

  • Great for root-cause analysis and catching problems before they surface (symptom prediction).

• Black-box: based on external behavior:

  • Synthetic probes, health checks, HTTP pings that mimic a real user with no view into internals.

  • Catches user-facing outages including things your internal metrics can't see (DNS, TLS, load balancer).

• When to prefer which:

  • Black-box for alerting on symptoms actually affecting users (aligns with SLOs).

  • White-box for diagnosing cause and for predictive/impending-failure signals.

  • Google SRE guidance: page primarily on black-box/symptom-based alerts, debug with white-box.

In control theory a system is observable if you can determine its complete internal state purely from its outputs. In software, the internal state is what your code is doing right now (which code path, what data, why it's slow), and the external outputs are the telemetry it emits: metrics, logs, traces, and events. Inferring state means reconstructing what happened inside from those signals, without attaching a debugger to production.

• Internal state you want to know: Which branch executed, which dependency was slow, what the request payload was, which node served it.

• External outputs you actually have:

  • Emitted telemetry: spans, structured log fields, metric time series.

  • You never observe state directly, you infer it from these outputs.

• Inference quality depends on instrumentation:

  • If a state has no observable output, it's unobservable, so you literally can't diagnose it.

  • Rich context (IDs, attributes, correlations) makes more internal states recoverable from outputs.

"Unknown unknowns" are failure modes you didn't anticipate and therefore never built a dashboard or alert for: you don't even know the question to ask. Observability addresses them by capturing rich, high-cardinality telemetry that lets you explore and ask new questions after the fact, rather than only checking predefined conditions.

• The knowledge quadrants:

  • Known-knowns: things you monitor and understand (CPU high, restart it).

  • Known-unknowns: things you know can happen but not the value (which node will fail).

  • Unknown-unknowns: novel emergent behavior you never predicted, common in distributed systems.

• Why traditional monitoring fails here: You can't pre-build a dashboard for a problem you've never imagined.

• How observability helps:

  • Wide events with many dimensions let you slice/group arbitrarily during investigation.

  • Exploratory, iterative querying: form a hypothesis, test against data, refine, without redeploying.

An event is a structured record of one discrete thing that happened (with rich context); a log is traditionally a timestamped text line describing an occurrence; a metric is a numeric measurement aggregated over time. Capture as a metric when you need cheap trends and alerting, as an event/log when you need the full context of individual occurrences to explain them.

• Metric: aggregated number over time:

  • Cheap to store and query, great for dashboards, SLOs, and threshold alerts.

  • Loses per-request detail and struggles with high cardinality (per-user labels explode storage).

• Log: a record of a discrete occurrence:

  • Plain-text logs are human-readable; structured logs carry key/value fields.

  • Best for detailed narrative of what happened at a point in time.

• Event: a structured, context-rich occurrence:

  • Think of a wide event per unit of work carrying all dimensions (IDs, timings, outcome).

  • The foundation of observability: you can aggregate events into metrics but not the reverse.

• Choosing:

  • Need a trend or alert cheaply, use a metric.

  • Need to debug specific occurrences with full context, capture rich events (and derive metrics from them).

The three pillars are complementary telemetry signals: metrics tell you that something is wrong, traces tell you where it's wrong, and logs tell you why. Together they let you move from detection to root cause.

• Metrics:

  • Numeric, aggregated measurements over time (request rate, error count, CPU). Cheap to store and ideal for dashboards and alerting.

  • Weakness: aggregated, so they lose per-request detail.

• Logs:

  • Discrete, timestamped event records with rich context (stack traces, parameters, messages).

  • High cardinality and volume, so expensive to index and query at scale.

• Traces:

  • Follow a single request across services, showing latency per hop and causal ordering.

  • Best for distributed systems where a request spans many components.

• Prioritization by scenario:

  • Latency spike in a microservice mesh: start with traces to find the slow span.

  • A crash or specific exception: logs give the stack trace and inputs.

  • Resource exhaustion or capacity trends: metrics show saturation over time.

In practice you correlate all three: an alert fires (metric), you find the offending request path (trace), then read the error detail (log).

A trace represents one end-to-end request through a system; a span is a single named operation within that trace (a DB call, an HTTP hop). A trace is a tree of spans linked by parent-child relationships, sharing a common trace_id.

• The relationship:

  • All spans in a trace share the same trace_id; each span has its own span_id and a parent_span_id that builds the tree.

  • The root span covers the whole request; child spans are nested sub-operations with their own start/end timestamps.

• Useful span metadata:

  • Timing: start time and duration, to spot the slow hop.

  • Identity: operation name, service name, and the IDs above.

  • Attributes/tags: HTTP method and status, DB statement, user_id, region, so you can filter and group.

  • Status: an error/OK flag plus exception events with stack traces.

• Why it aids root cause: Rich attributes let you answer "is it slow for all users or just one tenant, one endpoint, one downstream?" without guessing.

Time-series data is a sequence of measurements indexed by time, where each point is a timestamp plus a value (and usually labels). A time-series database (TSDB) is optimized for the specific access pattern of such data: heavy append-only writes and time-range queries.

• What time-series data looks like:

  • Each record is (timestamp, value) with dimensions/labels like host, region.

  • Writes are almost always newest-timestamp appends; old points rarely change.

• Distinguishing characteristics of a TSDB:

  • Ingestion optimized for high-volume sequential writes.

  • Compression exploits regular intervals and slowly-changing values (delta and delta-of-delta encoding).

  • Time-based partitioning and retention/downsampling (keep raw data days, rollups for months).

  • Query engine built around time windows and aggregation (rate(), avg over 5m).

• vs a general-purpose DB: A relational DB assumes updates, joins, and ACID transactions; a TSDB trades those for far higher write throughput and cheaper storage of ordered metrics.

Averages are dangerous because they hide the tail: a mean latency blends fast and slow requests into one number that no real user necessarily experiences. Percentiles like p95/p99 describe the worst experiences that a meaningful fraction of users actually see.

• The problem with averages:

  • They are skewed by distribution: a few very slow requests can be masked by many fast ones, or a spike can drag the mean without any single user hitting that value.

  • Latency distributions are typically long-tailed, not normal, so the mean is a poor summary.

  • You can't average percentiles across instances, but people wrongly average means and lose signal.

• Why percentiles are better:

  • p95 means 95% of requests were faster than this value: a concrete guarantee.

  • p99 exposes tail latency that affects power users and requests that fan out to many services (fan-out amplifies the tail).

• Practical note: Track a distribution (histogram) and alert on high percentiles; report averages only as a rough cross-check.

The Four Golden Signals (from Google's SRE book) are Latency, Traffic, Errors, and Saturation. For a user-facing service where I could alert on only two, I'd choose Latency and Errors, because they most directly reflect whether users are being harmed right now.

• The four signals:

  • Latency: how long requests take, ideally split into successful vs failed.

  • Traffic: demand on the system (requests/sec).

  • Errors: rate of failed requests.

  • Saturation: how full the system is (the most constrained resource).

• Why Latency and Errors:

  • They are symptom-based and user-visible: a spike in either means users are experiencing slowness or failures immediately.

  • Traffic alone isn't a problem (high traffic is often good) and saturation is a leading cause that usually surfaces as latency/errors anyway.

  • This aligns with alerting on symptoms, not causes: page on what users feel, use traffic/saturation for diagnosis.

Apdex (Application Performance Index) compresses response-time satisfaction into a single 0-to-1 score by bucketing requests against a target threshold T that you define as the acceptable response time.

• The three buckets:

  • Satisfied: response time ≤ T.

  • Tolerating: between T and 4T.

  • Frustrated: > 4T.

• The formula:

  • Apdex = (Satisfied + Tolerating/2) / total requests. Tolerating counts half; Frustrated counts zero.

  • Score ranges 0 (all frustrated) to 1 (all satisfied).

• Strengths and caveats:

  • Turns a latency distribution into one intuitive KPI stakeholders can track.

  • It's only as meaningful as T, and like any single number it hides the tail; pair it with percentiles for detail.

A collector (or agent) is a separate process that receives telemetry from your applications, processes it, and forwards it to one or more backends. Instead of each app shipping data straight to a vendor, you route it through this intermediary, which acts as a buffer, proxy, and central control point.

• Decouples app from backend: Apps just send to a local endpoint; changing backends or adding new ones is a collector config change, no redeploy.

• Buffering and resilience: The collector absorbs bursts, retries, and handles backend outages, so slow exports don't block the application.

• Central processing: Batching, sampling, filtering, redaction of PII, and enriching with metadata (e.g. Kubernetes labels) happen in one place.

• Offloads work from the app: Heavy export logic and network I/O move out of the request path, keeping the SDK lightweight.

• Deployment modes: As an agent (sidecar/per-host) close to the app, or as a gateway (centralized pool) for aggregation, often both in a pipeline.

Auto-instrumentation hooks into known libraries to emit telemetry with little or no code change, giving fast broad coverage; manual instrumentation is code you write to capture domain-specific spans, attributes, and metrics. Most real systems use both: auto for the plumbing, manual for what matters to your business.

• Auto-instrumentation:

  • Pros: quick to enable, consistent coverage of HTTP, DB, and RPC calls, no app code changes.

  • Cons: generic (no business context), can add overhead, may miss custom code, and gives less control over naming/attributes.

• Manual instrumentation:

  • Pros: precise, captures business-relevant spans and attributes (e.g. order_id), full control over what and how you measure.

  • Cons: labor-intensive, easy to be inconsistent across teams, and adds maintenance burden.

• Best practice: Start with auto-instrumentation for baseline coverage, then layer manual spans on critical business logic and hot paths.

The OpenTelemetry Collector is a standalone service that receives, processes, and exports telemetry, acting as a vendor-neutral pipeline between your apps and backends. You'd use the OTel SDK (rather than a proprietary agent) to instrument your code once against an open standard, keeping your application free of vendor lock-in.

• Collector architecture:

  • Built from pipelines of receivers (ingest, e.g. OTLP), processors (batch, filter, sample, enrich), and exporters (send to backends).

  • Centralizes buffering, retries, and data transformation outside the app.

• Why the SDK over a vendor agent:

  • Portability: instrument once with OTel APIs, switch backends via exporter config without touching code.

  • Standardized semantic conventions make data consistent and correlated across services.

  • Avoids re-instrumenting every time you change or evaluate vendors.

• Common pattern: App uses OTel SDK to emit OTLP to a local Collector, which processes and fans out to Prometheus, Jaeger, or a SaaS backend.

An error budget is the amount of unreliability you are allowed to spend over a period, derived directly from your SLO: if your SLO is 99.9% availability, the budget is the remaining 0.1% of failed requests (or downtime) for that window. It reframes reliability as a resource you can trade against feature velocity.

• Definition:

  • Budget = 100% - SLO target, measured over a window (e.g. 30 days).

  • Spent by errors, latency violations, or downtime, whatever your SLI measures.

• Why it exists:

  • Aligns dev and ops: shipping fast is fine as long as budget remains.

  • Turns "is it reliable enough?" into an objective, data-driven decision.

• When the budget is exhausted, the pipeline reacts:

  1. Freeze non-essential feature deploys: only reliability fixes, security patches, and rollbacks go out.

  2. Redirect engineering effort to hardening: fix the top error sources, add tests, improve rollback speed.

  3. Conduct/postmortem the incidents that burned the budget and track remediation.

  4. Resume normal velocity once the rolling window recovers and budget is available again.

• Key nuance: the freeze is a policy agreed in advance, not a punishment: it keeps the SLO honest instead of being ignored.

Distributed tracing follows one request by assigning it a shared trace ID at the entry point and having every service record spans (timed units of work) tagged with that ID and a parent span ID. Context propagation is the mechanism that carries this trace context across service boundaries, typically as HTTP headers or message metadata, so downstream services attach their spans to the same trace.

• The building blocks:

  • Trace ID: one ID shared by every span in the request.

  • Span: a single operation with start/end time, parent reference, and attributes.

  • Parent/child links reconstruct the causal tree of calls.

• Context propagation:

  • On an outgoing call the current span's context is serialized into headers (W3C traceparent, or older b3 headers).

  • The receiving service extracts those headers, continues the same trace, and creates child spans.

  • Works across HTTP, gRPC, and async queues (context travels in message metadata).

• Why it matters: without propagation each service produces isolated spans that can't be stitched into one end-to-end picture. Instrumentation via OpenTelemetry handles inject/extract automatically for common frameworks.

You correlate a log line to a trace by including the trace ID (and often span ID) as a field in every structured log entry, injected from the active trace context at log time. Then in your tooling you can jump from a span to all logs carrying that trace ID, and vice versa. This is critical in an incident because it collapses the gap between "something is slow/broken" (the trace) and "here's exactly why" (the log message and stack trace).

• How the correlation is wired:

  • Log in structured (JSON) form and add trace_id and span_id fields.

  • Pull those IDs from the current context: OpenTelemetry log instrumentation can auto-inject them via MDC / log processors.

  • Index the trace ID in your log store (e.g. Loki, Elasticsearch) so it's queryable.

• Why it's critical during incidents:

  • A trace shows which span is slow or errored; the linked logs show the exception, query, or payload behind it.

  • Removes guesswork of grepping logs by timestamp across many services under time pressure.

  • Enables exemplars: jumping from a spiking latency metric straight to a representative trace and its logs.

A correlation ID is a unique identifier attached to a request at its entry point and passed unchanged to every downstream service and log line involved in handling that request. It lets you filter logs across all services by one ID and see the complete story of a single request, even when work fans out to many systems.

• How it flows:

  • Generated (or accepted from the client) at the edge, e.g. an X-Correlation-ID or X-Request-ID header.

  • Propagated on every outbound call and included in each service's structured logs.

  • Often stored per-request in context/MDC so every log statement picks it up automatically.

• Why it helps:

  • One query (correlation_id = X) reconstructs the full request path through disparate log streams.

  • Ties customer support tickets to backend logs when the ID is surfaced to the client.

• Relation to tracing: A trace ID is essentially a formalized correlation ID: tracing adds span-level timing and parent/child structure, whereas a plain correlation ID just groups logs.

A waterfall (or flame graph) view visualizes a trace as horizontal bars where each bar is a span, positioned by start time and sized by duration, and nested to show parent/child calls. You read it to find the bottleneck by looking for the widest bars and gaps: the longest span on the critical path is where time is actually being spent.

• How to read it:

  • X-axis is time; bar length is span duration. Longer = slower.

  • Vertical nesting shows call hierarchy: a child span sits under its parent.

  • Look for the single dominant wide bar: that's the biggest contributor to total latency.

• Patterns to spot:

  • Sequential staircase: calls happening one after another that could be parallelized.

  • N+1 calls: many small identical spans (e.g. repeated DB queries) that should be batched.

  • Gaps with no child spans: time spent in local compute, GC, or lock waiting, not downstream calls.

• Waterfall vs flame graph:

  • Waterfall is time-ordered per trace, best for latency on the critical path.

  • Flame graphs typically aggregate many samples by stack depth, best for finding where CPU/time concentrates across many executions.

Span attributes are key-value metadata describing the whole span (its context), while span events are timestamped points marking something that happened at a moment within the span. Attributes tell you what the operation was; events tell you when things occurred during it.

• Span attributes: dimensions on the span:

  • Key-value pairs like http.method, db.statement, user.id that hold true for the span's duration.

  • Used to filter and group traces (find all slow spans where db.system=postgres).

• Span events: timestamped moments inside the span:

  • A named log-like point with its own timestamp and optional attributes (e.g. cache.miss, retry.attempt).

  • Exceptions are recorded as a special event with stack trace attributes.

• Using them in root-cause analysis:

  • Start with attributes to isolate the failing population (which endpoint, tenant, host).

  • Then read events chronologically within a suspect span to see the sequence: when a retry fired, when a lock was acquired, where the exception was thrown.

• Rule of thumb: attribute for a property of the operation, event for a discrete occurrence during it.

Both operate on counters over a time window: rate() gives the per-second average increase, while increase() gives the total growth over the window. You can't alert on a raw counter because it's cumulative and ever-growing (and resets on restart), so its absolute value is meaningless as a threshold.

• rate: per-second rate of increase:

  • For throughput and comparisons across intervals (rate(http_requests_total[5m]) = requests/sec).

  • Normalized by time, so it's window-length independent.

• increase: total count over the window: Essentially rate * window; useful for "how many errors in the last hour".

• Why not alert on the raw counter:

  • It monotonically increases forever, so errors_total > 1000 will eventually always fire.

  • It resets to zero on process restart, so absolute values aren't comparable.

  • Both functions handle counter resets correctly, giving you a meaningful "current activity" signal to alert on.

Saturation, one of Google's four golden signals, measures how full a resource is: how close it is to its capacity limit. It's a leading indicator of trouble because performance usually degrades as saturation approaches 100%, often before errors or latency spike.

• What it captures:

  • The most constrained resource, expressed as a fraction of its usable maximum.

  • Often best watched via queue depth, which grows before hard limits are hit.

• How to measure it by resource type:

  • CPU: run-queue length / load average relative to core count (utilization alone hides queuing).

  • Memory: used vs available, plus swap activity and OOM pressure.

  • Disk: I/O queue depth and throughput vs device limits; also space used.

  • Network: bandwidth used vs link capacity, dropped packets.

  • Application: thread pool / connection pool usage, work queue backlog.

• Why it matters: High saturation predicts latency and error spikes, so alerting on it gives lead time to add capacity.

Symptom-based alerting fires on what the user actually experiences (high error rate, slow responses), while cause-based alerting fires on a potential internal reason (a full disk, a down node). Symptom-based is generally preferred for on-call because it catches real user impact regardless of cause and produces far less noise.

• Symptom-based: alert on user-visible impact:

  • Tied to SLOs: error rate, latency, availability.

  • Catches problems you never anticipated, since it watches the outcome, not the mechanism.

• Cause-based: alert on internal conditions:

  • Examples: disk 90% full, high CPU, one replica down.

  • Many causes never affect users (redundancy absorbs them), so they generate false pages.

• Why symptom-based wins for paging:

  • Fewer, more actionable pages: you only wake someone when users are actually hurting.

  • Avoids alert fatigue from redundant causes and every possible failure mode.

• The balance:

  • Page on symptoms; keep cause-based signals as dashboards or low-urgency tickets to speed diagnosis.

  • Exception: page on a cause when it reliably precedes impact with lead time (e.g. disk will fill in 4 hours).

Deduplication collapses repeated firings of the same alert into one, grouping bundles related alerts into a single notification, and routing sends that notification to the right team via the right channel. Together they turn a flood of raw alerts into a few coherent, addressable incidents.

• Deduplication:

  • The same alert firing every evaluation cycle is merged by a fingerprint/identity so on-call gets one notification, not hundreds.

  • Prevents re-paging for an incident already acknowledged.

• Grouping:

  • Alerts sharing labels (same service, cluster, cause) are bundled into one notification: e.g. 50 pods down becomes one "service X degraded".

  • Reduces cognitive load and reflects that they're one underlying problem.

• Routing:

  • Matches alerts (by labels like team or severity) to the correct receiver: page the owning team, ticket the rest.

  • Also picks the channel: PagerDuty/phone for critical, Slack/email for low priority.

• Why it matters for on-call:

  • Without it, one outage generates a storm of duplicate pages: fatigue, missed real signals, slow response.

  • Correct routing means the person who can actually fix it is the one paged.

An escalation policy is the ordered set of rules that decides who gets notified, and what happens if they don't respond, ensuring no alert is silently dropped. It structures on-call so a page always reaches a human capable of acting, escalating up the chain when it isn't acknowledged.

• Core mechanism:

  • Level 1: primary on-call is paged and given an acknowledge window (e.g. 5 min).

  • No ack: escalate to secondary on-call, then to team lead / manager.

  • Escalation guarantees the alert isn't lost if someone is asleep, offline, or overwhelmed.

• Structuring the rotation:

  • Use fair, time-bounded shifts (weekly) with a clear handoff and follow-the-sun for global teams.

  • Keep a backup/secondary always defined so escalation has a target.

  • Route by severity: only critical, actionable alerts should page; lower priority goes to tickets/chat.

• Healthy on-call practices:

  • Track page volume per shift; excessive paging is a signal to fix noise or reliability.

  • Attach runbooks so whoever is paged (including escalations) can act quickly.

  • Post-incident review to refine thresholds and escalation timings.

Log-based metrics are numeric time series derived by parsing, filtering, or counting log lines rather than emitting a metric at the source. You use them when you can't (or didn't) instrument the code directly, but the signal already exists in the logs.

• How they work:

  • A pipeline or backend matches a pattern (e.g. count of lines with status=500) and turns it into a counter or gauge.

  • Can also extract a value from the log (latency field) into a distribution.

• When to derive from logs:

  • You can't change the code: third-party apps, legacy systems, or infra that only emits logs.

  • The signal is rare or ad hoc and not worth a permanent metric.

  • You need to retroactively measure something that already happened (logs are historical; a metric only starts once instrumented).

• When to prefer a direct metric:

  • High-volume, always-on signals: parsing logs is far more expensive than incrementing a counter.

  • You need low cardinality and predictable cost; log-derived metrics can explode cardinality via unbounded label values.

• Trade-off: log-based metrics couple your metric accuracy to log format stability and add parsing latency and cost.

The ELK stack (Elasticsearch, Logstash, Kibana) is a centralized logging pipeline; EFK swaps Logstash for Fluentd/Fluent Bit as the collector. Conceptually it covers four stages: ship, process, store/index, and visualize.

• Collector/shipper: Beats, Logstash, or Fluentd/Fluent Bit:

  • Reads logs from files, containers, or stdout and forwards them.

  • In EFK, Fluentd is preferred in Kubernetes for its lightweight footprint and plugin ecosystem.

• Processing: Logstash (or Fluentd filters): Parses unstructured lines (grok), enriches, redacts, and transforms into structured JSON before indexing.

• Storage/index: Elasticsearch: A distributed search engine that indexes documents so you can query and aggregate logs quickly.

• Visualization: Kibana: UI for searching, dashboards, and alerting over the indexed data.

• Common addition: a buffer like Kafka or Redis between shipper and processor absorbs spikes and decouples producers from Elasticsearch.

RUM measures the experience of actual users from their real browsers/devices, while synthetic monitoring runs scripted probes against your system on a schedule from controlled locations. RUM tells you what really happened; synthetics tell you what should happen even when no one is using the app.

• Real User Monitoring (RUM):

  • Passive: collects real page loads, latencies, errors, and Core Web Vitals across real devices, networks, and geographies.

  • Reflects the true diversity of your traffic but only covers paths users actually take.

• Synthetic monitoring:

  • Active: scripted transactions (login, checkout) run at fixed intervals, giving consistent baselines and pre-launch/low-traffic coverage.

  • Great for uptime SLAs and catching regressions before users hit them.

• Where synthetics miss but RUM catches:

  • Issues specific to real conditions: a certain browser version, mobile network, ad-blocker, or geography your probes don't emulate.

  • Problems on user flows or inputs your scripts never exercise.

  • Client-side failures from real third-party scripts or personalized/logged-in content the synthetic account never sees.

• Best practice: use both. Synthetics for consistent early detection, RUM for real-world impact and long-tail issues.

MTTD is the average time from when an incident begins to when you detect it; MTTR is the average time from detection (or onset) to full resolution. Detection is driven by your alerting signals; resolution is driven by your diagnostic signals.

• MTTD (Mean Time to Detect):

  • Most influenced by metrics and their alerts: high-level SLO/RED/USE metrics with good thresholds catch problems fast.

  • Poor MTTD comes from missing alerts, noisy thresholds, or metrics too coarse to reveal degradation.

• MTTR (Mean Time to Resolve):

  • Most influenced by traces and logs: they let you localize the failing service and read the exact error to fix it.

  • Distributed tracing shortens MTTR by pinpointing the slow/failing span across services.

• How they relate:

  • Total downtime roughly equals MTTD + time to diagnose + time to fix; metrics attack the first term, traces/logs the rest.

  • Correlating all three signals (jump from a metric alert to the trace to the log) is what drives both down.

In Kubernetes, a liveness probe asks "is this container still healthy, or should it be restarted?" while a readiness probe asks "is this container ready to receive traffic right now?" Liveness failure triggers a restart; readiness failure only removes the pod from the Service endpoints.

• Liveness probe:

  • Detects a wedged/deadlocked process; on failure the kubelet kills and restarts the container.

  • Should check only the process itself, never external dependencies.

• Readiness probe:

  • Gates traffic: on failure the pod is pulled from load-balancing but keeps running.

  • Useful during startup, warm-up, or when temporarily overloaded.

• The misconfiguration: readiness checks a shared downstream DB:

  • If the DB goes down, every pod's readiness fails at once, so all pods leave the Service endpoints simultaneously.

  • The Service now has zero endpoints: clients get connection refused instead of a graceful degraded response, turning a partial DB issue into a total outage.

  • It also blocks recovery: pods can't serve cached or health-check traffic, and if the DB check were in liveness instead, pods would enter a crash-restart loop.

• Rule of thumb: probe only what the pod itself controls; handle downstream failures with retries, timeouts, and graceful degradation in application code.

A good dashboard answers a specific question for a specific audience at a glance: it tells you whether the system is healthy and, when it isn't, points toward why. It favors clarity over completeness.

• Purpose-driven: Built around a question or audience (on-call triage, SLO tracking, capacity planning), not a dumping ground of every available metric.

• Structured around signals that matter: Lead with user-facing symptoms (latency, errors, traffic, saturation) then supporting causes; top-left is most important.

• Contextual and readable: Show thresholds/SLO lines, consistent units and time ranges, and use percentiles (p50/p95/p99) not just averages.

• Common anti-patterns:

  • Wall of graphs: too many panels so nothing stands out.

  • Averages that hide outliers; a mean latency masks a bad p99.

  • Dual y-axes and misleading scales that imply false correlation.

  • Vanity metrics with no threshold or action tied to them.

  • Pie charts / gauges for time-series data where trends matter.

MTBF is the average time a system operates correctly between failures: a measure of how often things break. Together with MTTD (how fast you detect) and MTTR (how fast you recover), it lets you reason about both failure frequency and failure impact, which drives availability.

• MTBF (Mean Time Between Failures):

  • Total operational (uptime) divided by number of failures; higher is better (more reliable).

  • Applies to repairable systems; MTTF is the equivalent for non-repairable ones.

• MTTD (Mean Time To Detect): Time from a failure starting to being noticed; good monitoring/alerting shrinks it.

• MTTR (Mean Time To Recovery/Repair): Time from detection to restored service; good runbooks, automation, and rollback shrink it.

• How they combine:

  • Availability ≈ MTBF / (MTBF + MTTR): you improve reliability either by failing less often (raise MTBF) or recovering faster (lower MTTR + MTTD).

  • Modern SRE leans on lowering MTTD/MTTR because complex distributed systems will fail regardless.

APM is monitoring focused on application-level performance and behavior: it instruments code to track request latency, throughput, errors, and traces, correlating them so you can see not just that the app is slow but which code path, query, or downstream call caused it.

• What it measures:

  • Request rate, error rate, and latency percentiles per endpoint/transaction.

  • Distributed traces showing a request's path across services.

  • Code-level detail: slow DB queries, external calls, function spans.

• Beyond raw metrics and logs:

  • Correlation: ties a metric spike to the exact trace and log lines behind it, replacing manual joining.

  • Transaction context: groups telemetry by business operation (POST /checkout) rather than isolated numbers.

  • Automatic dependency mapping and code-level root cause, so you find the bottleneck faster.

• Examples: Datadog APM, New Relic, Dynatrace, Elastic APM, and OpenTelemetry-based backends.

Retention policies set how long telemetry is kept before deletion, directly bounding how far back you can investigate. Longer retention enables trend analysis, audits, and post-mortems on old incidents, but storage cost grows roughly linearly with duration and data volume, so the aim is to keep data only as long as it delivers value.

• How retention shapes observability:

  • Debugging: you can only investigate incidents that fall inside the window; expired data is gone.

  • Baselining: year-over-year comparisons and seasonality analysis need long retention.

  • Compliance: some logs (audit, security) have legally mandated minimum retention.

• Levers to balance cost vs. duration:

  • Tier by signal: keep cheap aggregated metrics for months/years, expensive high-volume logs and traces for days/weeks.

  • Tier by resolution: full-resolution recent data plus downsampled rollups for the long tail.

  • Tier by storage class: move older data to cheaper cold/object storage that is slower but far less expensive per GB.

• How to set the window:

  • Anchor to real use: how far back do on-call and post-mortems actually query? That defines the hot window.

  • Separate policies per data type: raw debug logs may live days while SLO metrics live a year.

  • Account for cardinality and growth: high-cardinality data gets costly fast, so shorten its retention first.

• Rule of thumb: retain full detail for the active operational window, downsample and demote to cheap storage beyond it, and let compliance set hard minimums.

During an incident many signals move together, but two things changing at the same time doesn't mean one caused the other. Confusing correlation with causation leads you to fix a symptom or an innocent bystander while the real cause keeps failing, wasting time and eroding trust in the data.

• Why correlation is seductive:

  • Telemetry surfaces many things spiking at once (latency, errors, CPU) and the brain grabs the first pattern.

  • A common third cause can move two metrics together (a deploy raises both latency and error rate).

• Risks of assuming causation:

  • Fixing a downstream symptom while the upstream trigger persists.

  • Rolling back an innocent change and missing the real regression.

• How to establish causation:

  • Check temporal ordering: the cause must precede the effect (traces and timestamps help).

  • Follow dependency direction with distributed traces to see what actually blocked what.

  • Correlate the change timeline (deploys, config, feature flags) with symptom onset.

For a rare, non-reproducible issue you can't step through it in a debugger, so you rely on telemetry captured when it actually happened. The key is high-cardinality, correlatable data: find the few affected requests, isolate what makes them different, and trace them end to end.

• Approach:

  1. Find the affected requests by slicing events on dimensions (user, region, version, endpoint).

  2. Compare the bad population to the good one to see which attribute correlates with failure.

  3. Pull the distributed trace for a failing request to locate the slow or erroring span.

  4. Read the correlated logs for that trace/request ID for the detailed narrative.

• Most useful signals:

  • Distributed traces: reveal which service/dependency caused the latency or error on that exact request.

  • High-cardinality wide events: let you filter down to the rare intermittent cases.

  • Exemplars and trace IDs in logs/metrics: jump from an aggregate spike to a concrete example.

• Enablers:

  • Consistent trace/request ID propagation across services for correlation.

  • Tail-based sampling so rare errors are actually retained, not sampled away.

RED describes the workload from the request's perspective (Rate, Errors, Duration) and suits request-driven services; USE describes a resource from the resource's perspective (Utilization, Saturation, Errors) and suits infrastructure and hardware. For a public-facing API prefer RED; for a database's underlying resources prefer USE (though a good database story uses both).

• RED (per service/endpoint):

  • Rate: requests per second.

  • Errors: failed requests per second.

  • Duration: latency distribution of requests.

  • Best when you care about user-facing request behavior.

• USE (per resource):

  • Utilization: percent of time the resource was busy.

  • Saturation: queued/backlogged work the resource can't service yet.

  • Errors: error events for that resource.

  • Best for CPU, memory, disk, connection pools.

• Applying them:

  • Public-facing API: RED, because users experience rate, errors, and latency directly.

  • Database: USE, because the failure modes are resource-driven (disk saturation, CPU utilization, connection pool exhaustion). Layer RED on top to watch query rate/errors/latency.

In a push model the application actively sends metrics to a collector (StatsD, the OpenTelemetry OTLP exporter); in a pull model the monitoring server scrapes metrics from an endpoint each target exposes (Prometheus scraping /metrics). The core difference is who initiates the connection, which drives discovery, reliability, and scaling behavior.

• Push model:

  • Clients emit as events happen; good for short-lived jobs and batch tasks that die before a scrape.

  • Works well through NAT/firewalls since clients dial out.

  • Scaling risk: a traffic surge means everyone pushes harder, and the collector can be overwhelmed (thundering herd); needs buffering/aggregation.

  • Harder to know if a silent client is down vs just idle.

• Pull model:

  • Server controls scrape interval and rate, so it can't be overrun by clients.

  • A failed scrape is itself a signal (up == 0), giving free liveness detection.

  • Needs service discovery to know what to scrape, and struggles with ephemeral jobs (mitigated by a push gateway).

• Scaling implications:

  • Pull scales by sharding/federating scrapers across target sets; the server is the bottleneck to plan around.

  • Push scales by adding relay/aggregation tiers (a StatsD aggregator) but must handle backpressure so bursts don't drop or overwhelm.

Buffering and back-pressure keep a telemetry pipeline stable when data is produced faster than it can be exported. Without them, a slow or down backend can push failure back into your application, and if the monitoring system becomes the bottleneck you can degrade or crash the very service you're trying to observe, and go blind exactly when you need visibility most.

• Buffering absorbs bursts:

  • Queues (in-memory or on-disk) hold telemetry during traffic spikes or transient backend outages so data isn't lost immediately.

  • Batching from the buffer also improves export efficiency.

• Back-pressure signals overload:

  • When buffers fill, the pipeline must decide: block, drop, or sample down. Blocking the app is usually the worst choice.

  • Good practice is to shed telemetry (drop/sample) rather than harm request-serving.

• If monitoring becomes the bottleneck:

  • Synchronous exports can add latency or exhaust memory/threads and take the service down with it.

  • You lose observability during an incident, which is when it's most valuable.

• Golden rule: Telemetry should fail independently of the application: use async export, bounded buffers, and a drop policy so the observability system never brings down production.

Instrumentation is never free: it costs CPU, memory, and I/O, and in a high-throughput service those costs compound per request. The goal is enough visibility to debug and measure without materially degrading latency or throughput, which you achieve through sampling, batching, and async export.

• CPU and latency: Creating spans, serializing attributes, and context propagation add per-request work; excessive attributes or high span counts hurt.

• Memory and GC pressure: Buffered spans/metrics and high-cardinality labels consume memory and can trigger more garbage collection.

• Network and export cost: Shipping telemetry uses bandwidth and connections; synchronous export in the request path is especially dangerous.

• Mitigations:

  • Sampling (head or tail) to reduce trace volume while keeping representative data.

  • Async/batched export so telemetry leaves the hot path.

  • Control cardinality: avoid unbounded label values like user IDs on metrics.

  • Offload processing to a Collector rather than the app.

• Bottom line: Measure the overhead (benchmark with/without), and treat observability cost as a tunable dial, not all-or-nothing.