Incident Communication
After an incident, a manager requests a formal explanation for remediation — how would you approach this communication and what information would you provide?
How do you decide on an update cadence for stakeholders during a prolonged incident?
How do you handle internal and external communication during a production incident?
How do you manage the 'fog of war' and avoid publishing speculation in incident communications?
What is a status page and how does external communication during an incident differ from internal communication?
Postmortems & Continuous Improvement
After an incident, how do you debug and harden a system to prevent a similar class of regressions from shipping again?
Explain the importance of a blameless postmortem — what are its key components and objectives?
How do game days and fire drills prepare a team for real incidents?
How do you ensure that learnings from incidents are shared across teams and lead to lasting system improvements?
How do you reconstruct an accurate incident timeline for a postmortem?
What are "action items" from a postmortem, and how do you ensure follow-through?
What does 'blameless and just culture' really mean, and how do you keep a postmortem from turning into finger-pointing?
What is a near-miss, and why is it worth reviewing incidents that didn't cause customer impact?
What is a postmortem and why is it important?
What is the 5 Whys technique, and what are its limitations for root-cause analysis?
What is the difference between a postmortem that generates real improvement and one that is just "theater"?
What steps do you take to prevent recurring incidents?
What would you do after the fire is out?
What's your approach to root cause analysis and post-incident reviews?
Why do many practitioners argue there is no single 'root cause', and how do you think about contributing factors instead?
Debugging Methodology
Are you able to find where things are diverging and trace to the root cause?
Do you follow a repeatable process instead of guessing when debugging?
How can dynamically adjusting log levels or adding targeted instrumentation help narrow down a live issue?
How do correlation or request IDs help you trace a single failing request across services during debugging?
How do you approach debugging a live production system when you cannot attach a debugger?
How do you approach the 'works on my machine' problem and environment-parity issues when a bug only appears in production?
How do you check recent changes and system signals during debugging?
How do you clarify the symptom and impact of an issue?
How do you close loops and summarize conclusions before moving on in debugging?
How do you debug with a plan and iteratively bisect rather than randomly tweaking code?
How do you distinguish correlation from causation when a metric spike lines up with a symptom during an incident?
How do you make rapid, data-informed decisions during an incident?
How do you narrate calmly and articulate what you're checking and why during debugging?
How do you navigate an unfamiliar codebase, forming different hypotheses about the bug?
How do you read error messages and leverage stack traces?
How do you reason hierarchically and test the highest-probability causes first?
How do you reproduce an issue in a production-like environment?
How do you use a known-good baseline to spot what's abnormal during an incident?
What are heisenbugs, and how do you go about diagnosing a bug that disappears when you try to observe it?
What do you do if you can't find the exact cause during debugging?
What is hypothesis-driven debugging, and how do you apply it during an incident?
What is your standard approach to debugging a production issue?
What steps do you take before starting to debug an issue?
When faced with an incident, what is your methodology for narrowing down the scope and localizing the fault?
When is first-principles reasoning better than pattern-matching from past incidents, and vice versa?
Why is 'what changed?' usually the first question in production debugging, and how do you bisect over time or over a change set?
Why is debugging about systems, not symptoms?
Incident Lifecycle & Foundations
Can you explain the difference between an incident, a problem, and a change, and why it matters?
Can you list some incident management best practices?
Describe common production failure modes you've encountered and how you would diagnose and mitigate them.
Explain the concepts of MTTD, MTTA, and MTTR and how they relate to incident response.
Explain the difference between proactive and reactive incident response strategies.
Explain the typical lifecycle of a production incident, from detection to resolution and recovery.
How do MTBF and MTTF differ from MTTR, and how do they inform reliability improvements?
What are common causes of production failures?
What are the operational practices for running services in production and responding to incidents?
What does the 'you build it, you run it' ownership model mean for incident response?
What is a production failure?
What is the difference between incident response and incident management?
What is toil from incident load, and how does high incident frequency signal deeper problems?
What is your incident response process?
When would you implement an incident management system?
Incident Command & Coordination
Describe the different roles involved in a major incident and their responsibilities.
How do you determine what else might be affected if a service is compromised?
How do you escalate to leadership during a major outage?
How would you build trust and alignment with technical teams during an incident?
How would you work with global teams during an incident?
If we had a suspected security breach during an availability incident, how would you coordinate the response?
Walk me through how you run a SEV-1 incident from detection to resolution.
What are common challenges in incident response, such as communication issues?
What does 'span of control' mean in the Incident Command System, and why shouldn't the Incident Commander also be hands-on debugging?
What is the importance of cross-functional collaboration in the preparation phase of incident response?
What is the role of a scribe during an incident, and why is capturing a real-time timeline valuable?
What is your approach when you first join a major incident call?
When a major issue occurs, how do you allocate and manage resources?
You've just declared yourself Incident Commander and assigned Operations Lead and Communications Lead roles during an incident — what are your immediate next steps as IC?
Performance & Resource Diagnostics
Describe your methodology for debugging a slow or unreachable service in production.
How do you approach diagnosing tail latency issues in a production environment?
How do you detect and diagnose a deadlock or lock contention in a running production system?
How would you diagnose connection-pool or thread-pool exhaustion as the cause of an outage?
How would you diagnose long GC pauses in a production JVM service?
Logging was increased for debugging and production went down — how?
The application runs fine for hours and then suddenly slows down — what could be wrong?
What are flame graphs and how does continuous profiling help you find hotspots in production?
What are thread dumps and heap dumps, and when would you use them for production debugging?
What production debugging techniques would you use to diagnose high CPU usage or memory leaks?
You receive an OutOfMemoryError even though the heap appears sufficient — how is that possible?
Your Java service is slow but CPU usage is low — what do you check first?
Severity Priority & Triage
Do you assess user impact before diving into root cause?
How do severity levels act as escalation triggers and communication protocols?
How do severity levels optimize resource allocation during an incident?
How do teams build the judgment to accurately assign severity in the first few minutes of a live incident?
How do you assess the severity and priority of an incident?
How do you decide when an incident is truly over and can be closed, versus prematurely declaring 'all clear'?
How do you determine incident severity and priority when information is incomplete?
How do you prioritize which alerts to respond to first during an incident?
How do you weigh impact against urgency when assessing an incident's severity?
How does a defined severity level help with faster triage and prioritization?
How many severity levels should an organization use, and why?
What are incident severity levels and how do they impact the incident response process?
What are the genuine limitations of severity frameworks under pressure?
What criteria or thresholds do you use to decide when to formally declare an incident versus just handling it quietly?
What is the difference between severity and priority?
Why is a standardized severity framework crucial?
Observability & Fault Localization
During an incident, how would you investigate latency, errors, and restarts to pinpoint the root cause?
Explain the USE method or the RED method for diagnosing system performance.
How do you bound the blast radius during an incident?
How do you differentiate between "our service is slow" and "something our service depends on is slow"?
How do you identify performance bottlenecks in a live production system?
How do you use logs, metrics, and traces to triangulate the source of a problem in a distributed system?
How do you use traces to localize latency in a microservices architecture?
How do you validate downstream dependencies during debugging?
How would you diagnose a production service returning intermittent 500 errors?
What are the four golden signals, and how do you use them to diagnose a live incident?
Mitigation Recovery & Rollback
Explain the difference between a rollback and a roll-forward as mitigation strategies.
Explain the difference between mitigation and resolution in the context of a production incident.
How do feature flags and kill switches serve as mitigation levers during an incident?
How do traffic draining, failover, and shifting traffic away from a bad node work as mitigation techniques?
How do you decide whether to roll back during an incident?
How do you handle failed deployments?
How do you handle high traffic or traffic spikes?
How do you handle incidents affecting cross-region services?
How do you restore critical applications quickly?
How do you stabilize the system if needed during an incident?
How do you triage a production incident, and what immediate production changes would you consider to restore reliability without compromising critical functionality?
How do you validate the fix carefully after an incident?
How does load shedding or emergency rate limiting protect a system that's being overwhelmed?
How would you use a circuit breaker to immediately shed load during a latency SLO breach?
What are some common mitigation and recovery levers you would use during an incident?
What do RTO and RPO mean as disaster-recovery targets, and how do they shape your recovery approach during an incident?
What is graceful degradation, and how do fallbacks help you keep serving users during a partial outage?
What is the "stop the bleeding first" principle in incident response, and why is it important?
What is the criticality of data backups and disaster recovery planning in the recovery phase?
When an SLO for latency is breached, what immediate mitigation strategies might you employ, and why?
When is restarting or rebooting only a bandaid, and what are the risks of reaching for it too quickly?
When would you quarantine a bad host or restore from backup, and what are the tradeoffs?
Why is capturing system state before restarting a hung process important, and what would you capture?
Distributed Failure Modes
How can clock skew cause production incidents, and how would you diagnose it?
How do slow dependencies and timeout propagation cause outages, and how do you contain them?
How do you detect and respond to split-brain in a clustered or replicated system during an incident?
How do you diagnose and mitigate a disk-full, inode, or file-descriptor exhaustion incident?
How do you handle partial failures in a system to prevent a complete outage?
How would you diagnose and handle a poison-pill message or a growing queue backlog?
How would you recognize and respond to a DNS failure or an expired TLS certificate causing an outage?
What are hot shards or noisy-neighbor problems, and how would you diagnose one in production?
What is a cache stampede or dogpile effect, and how would you diagnose and mitigate it?
What is a cascading failure, how do you recognize one in progress, and how do you stop it?
What is a gray failure, and why are partial or 'limping' failures harder to detect and diagnose than hard crashes?
What is a metastable failure, and why can a system stay broken even after the original trigger is gone?
What is a retry storm or thundering herd, and how do you mitigate it during an incident?
Resilience & Deployment Safety
How do canary or blue-green deployments limit the blast radius of a bad release?
What are runbooks and playbooks, and how do they aid in incident preparedness and response?
What is shadow traffic or traffic replay, and when would you use it to debug a production problem safely?
On Call & Alerting
How do primary and secondary on-call responders and escalation policies work together?
How do you balance operational work with project work, especially during on-call?
How do you handle on-call handoffs and shift changes so context isn't lost mid-incident?
How do you manage alert fatigue and noisy alerts from a responder's perspective?
How does linking severity to specific notification rules reduce alert fatigue?
Tell me about on-call: what are the challenges and how do you address them?
What does a healthy on-call rotation look like, and how do you protect responder well-being?
What role do incident-management tools like PagerDuty or Opsgenie play in the response process?