Incident Response

0%
Theory
Quiz

    Debugging Methodology

    • What steps do you take before starting to debug an issue?

      Junior
    • Do you follow a repeatable process instead of guessing when debugging?

      Junior
    • How do you clarify the symptom and impact of an issue?

      Junior
    • How do you read error messages and leverage stack traces?

      Junior
    • What is your standard approach to debugging a production issue?

      Mid
    • How do you check recent changes and system signals during debugging?

      Mid
    • How do you debug with a plan and iteratively bisect rather than randomly tweaking code?

      Mid
    • How do you navigate an unfamiliar codebase, forming different hypotheses about the bug?

      Mid
    • Are you able to find where things are diverging and trace to the root cause?

      Mid
    • How do you narrate calmly and articulate what you're checking and why during debugging?

      Mid
    • How do you close loops and summarize conclusions before moving on in debugging?

      Mid
    • What do you do if you can't find the exact cause during debugging?

      Mid
    • How do you reproduce an issue in a production-like environment?

      Mid
    • Why is 'what changed?' usually the first question in production debugging, and how do you bisect over time or over a change set?

      Mid
    • How do you use a known-good baseline to spot what's abnormal during an incident?

      Mid
    • How do correlation or request IDs help you trace a single failing request across services during debugging?

      Mid
    • How do you approach the 'works on my machine' problem and environment-parity issues when a bug only appears in production?

      Mid
    • How can dynamically adjusting log levels or adding targeted instrumentation help narrow down a live issue?

      Mid
    • What is hypothesis-driven debugging, and how do you apply it during an incident?

      Senior
    • How do you make rapid, data-informed decisions during an incident?

      Senior
    • How do you reason hierarchically and test the highest-probability causes first?

      Senior
    • Why is debugging about systems, not symptoms?

      Senior
    • How do you approach debugging a live production system when you cannot attach a debugger?

      Senior
    • When faced with an incident, what is your methodology for narrowing down the scope and localizing the fault?

      Senior
    • How do you distinguish correlation from causation when a metric spike lines up with a symptom during an incident?

      Senior
    • When is first-principles reasoning better than pattern-matching from past incidents, and vice versa?

      Senior
    • What are heisenbugs, and how do you go about diagnosing a bug that disappears when you try to observe it?

      Senior

    Severity Priority & Triage

    • What is the difference between severity and priority?

      Junior
    • Do you assess user impact before diving into root cause?

      Junior
    • How do you assess the severity and priority of an incident?

      Mid
    • What are incident severity levels and how do they impact the incident response process?

      Mid
    • How do you prioritize which alerts to respond to first during an incident?

      Mid
    • Why is a standardized severity framework crucial?

      Mid
    • How does a defined severity level help with faster triage and prioritization?

      Mid
    • What criteria or thresholds do you use to decide when to formally declare an incident versus just handling it quietly?

      Mid
    • How do you decide when an incident is truly over and can be closed, versus prematurely declaring 'all clear'?

      Mid
    • How do you weigh impact against urgency when assessing an incident's severity?

      Mid
    • How do severity levels optimize resource allocation during an incident?

      Senior
    • How many severity levels should an organization use, and why?

      Senior
    • How do severity levels act as escalation triggers and communication protocols?

      Senior
    • How do you determine incident severity and priority when information is incomplete?

      Senior
    • What are the genuine limitations of severity frameworks under pressure?

      Senior
    • How do teams build the judgment to accurately assign severity in the first few minutes of a live incident?

      Senior

    Incident Lifecycle & Foundations

    • What are common causes of production failures?

      Junior
    • What is a production failure?

      Junior
    • Can you list some incident management best practices?

      Junior
    • Explain the typical lifecycle of a production incident, from detection to resolution and recovery.

      Mid
    • Explain the concepts of MTTD, MTTA, and MTTR and how they relate to incident response.

      Mid
    • What is your incident response process?

      Mid
    • Explain the difference between proactive and reactive incident response strategies.

      Mid
    • Can you explain the difference between an incident, a problem, and a change, and why it matters?

      Mid
    • What are the operational practices for running services in production and responding to incidents?

      Mid
    • What is the difference between incident response and incident management?

      Mid
    • When would you implement an incident management system?

      Mid
    • How do MTBF and MTTF differ from MTTR, and how do they inform reliability improvements?

      Mid
    • What is toil from incident load, and how does high incident frequency signal deeper problems?

      Mid
    • What does the 'you build it, you run it' ownership model mean for incident response?

      Mid
    • Describe common production failure modes you've encountered and how you would diagnose and mitigate them.

      Senior

    Mitigation Recovery & Rollback

    • Explain the difference between mitigation and resolution in the context of a production incident.

      Junior
    • What is the "stop the bleeding first" principle in incident response, and why is it important?

      Mid
    • How do you decide whether to roll back during an incident?

      Mid
    • How do you handle high traffic or traffic spikes?

      Mid
    • How do you handle failed deployments?

      Mid
    • How do you stabilize the system if needed during an incident?

      Mid
    • How do you validate the fix carefully after an incident?

      Mid
    • How do you restore critical applications quickly?

      Mid
    • What is the criticality of data backups and disaster recovery planning in the recovery phase?

      Mid
    • What are some common mitigation and recovery levers you would use during an incident?

      Mid
    • Explain the difference between a rollback and a roll-forward as mitigation strategies.

      Mid
    • How do feature flags and kill switches serve as mitigation levers during an incident?

      Mid
    • When is restarting or rebooting only a bandaid, and what are the risks of reaching for it too quickly?

      Mid
    • What is graceful degradation, and how do fallbacks help you keep serving users during a partial outage?

      Mid
    • When would you quarantine a bad host or restore from backup, and what are the tradeoffs?

      Mid
    • What do RTO and RPO mean as disaster-recovery targets, and how do they shape your recovery approach during an incident?

      Mid
    • Why is capturing system state before restarting a hung process important, and what would you capture?

      Mid
    • How do you triage a production incident, and what immediate production changes would you consider to restore reliability without compromising critical functionality?

      Senior
    • How do you handle incidents affecting cross-region services?

      Senior
    • When an SLO for latency is breached, what immediate mitigation strategies might you employ, and why?

      Senior
    • How would you use a circuit breaker to immediately shed load during a latency SLO breach?

      Senior
    • How do traffic draining, failover, and shifting traffic away from a bad node work as mitigation techniques?

      Senior
    • How does load shedding or emergency rate limiting protect a system that's being overwhelmed?

      Senior

    Postmortems & Continuous Improvement

    • What is a postmortem and why is it important?

      Junior
    • What would you do after the fire is out?

      Junior
    • What steps do you take to prevent recurring incidents?

      Mid
    • What's your approach to root cause analysis and post-incident reviews?

      Mid
    • Explain the importance of a blameless postmortem — what are its key components and objectives?

      Mid
    • What are "action items" from a postmortem, and how do you ensure follow-through?

      Mid
    • What is the 5 Whys technique, and what are its limitations for root-cause analysis?

      Mid
    • How do you reconstruct an accurate incident timeline for a postmortem?

      Mid
    • What is a near-miss, and why is it worth reviewing incidents that didn't cause customer impact?

      Mid
    • How do game days and fire drills prepare a team for real incidents?

      Mid
    • After an incident, how do you debug and harden a system to prevent a similar class of regressions from shipping again?

      Senior
    • What is the difference between a postmortem that generates real improvement and one that is just "theater"?

      Senior
    • How do you ensure that learnings from incidents are shared across teams and lead to lasting system improvements?

      Senior
    • Why do many practitioners argue there is no single 'root cause', and how do you think about contributing factors instead?

      Senior
    • What does 'blameless and just culture' really mean, and how do you keep a postmortem from turning into finger-pointing?

      Senior

    Observability & Fault Localization

    • How do you validate downstream dependencies during debugging?

      Mid
    • How would you diagnose a production service returning intermittent 500 errors?

      Mid
    • During an incident, how would you investigate latency, errors, and restarts to pinpoint the root cause?

      Mid
    • What are the four golden signals, and how do you use them to diagnose a live incident?

      Mid
    • How do you use logs, metrics, and traces to triangulate the source of a problem in a distributed system?

      Senior
    • Explain the USE method or the RED method for diagnosing system performance.

      Senior
    • How do you bound the blast radius during an incident?

      Senior
    • How do you use traces to localize latency in a microservices architecture?

      Senior
    • How do you differentiate between "our service is slow" and "something our service depends on is slow"?

      Senior
    • How do you identify performance bottlenecks in a live production system?

      Senior

    Incident Command & Coordination

    • What are common challenges in incident response, such as communication issues?

      Junior
    • What is the importance of cross-functional collaboration in the preparation phase of incident response?

      Mid
    • What is your approach when you first join a major incident call?

      Mid
    • Describe the different roles involved in a major incident and their responsibilities.

      Mid
    • What is the role of a scribe during an incident, and why is capturing a real-time timeline valuable?

      Mid
    • How do you escalate to leadership during a major outage?

      Senior
    • How would you build trust and alignment with technical teams during an incident?

      Senior
    • When a major issue occurs, how do you allocate and manage resources?

      Senior
    • How would you work with global teams during an incident?

      Senior
    • If we had a suspected security breach during an availability incident, how would you coordinate the response?

      Senior
    • How do you determine what else might be affected if a service is compromised?

      Senior
    • Walk me through how you run a SEV-1 incident from detection to resolution.

      Senior
    • You've just declared yourself Incident Commander and assigned Operations Lead and Communications Lead roles during an incident — what are your immediate next steps as IC?

      Senior
    • What does 'span of control' mean in the Incident Command System, and why shouldn't the Incident Commander also be hands-on debugging?

      Senior

    On Call & Alerting

    • How do primary and secondary on-call responders and escalation policies work together?

      Junior
    • What role do incident-management tools like PagerDuty or Opsgenie play in the response process?

      Junior
    • Tell me about on-call: what are the challenges and how do you address them?

      Mid
    • How do you balance operational work with project work, especially during on-call?

      Mid
    • How do you handle on-call handoffs and shift changes so context isn't lost mid-incident?

      Mid
    • What does a healthy on-call rotation look like, and how do you protect responder well-being?

      Mid
    • How do you manage alert fatigue and noisy alerts from a responder's perspective?

      Senior
    • How does linking severity to specific notification rules reduce alert fatigue?

      Senior

    Incident Communication

    • What is a status page and how does external communication during an incident differ from internal communication?

      Junior
    • How do you handle internal and external communication during a production incident?

      Mid
    • After an incident, a manager requests a formal explanation for remediation — how would you approach this communication and what information would you provide?

      Mid
    • How do you decide on an update cadence for stakeholders during a prolonged incident?

      Mid
    • How do you manage the 'fog of war' and avoid publishing speculation in incident communications?

      Senior

    Performance & Resource Diagnostics

    • The application runs fine for hours and then suddenly slows down — what could be wrong?

      Mid
    • Logging was increased for debugging and production went down — how?

      Mid
    • What are flame graphs and how does continuous profiling help you find hotspots in production?

      Mid
    • How would you diagnose connection-pool or thread-pool exhaustion as the cause of an outage?

      Mid
    • Describe your methodology for debugging a slow or unreachable service in production.

      Senior
    • What are thread dumps and heap dumps, and when would you use them for production debugging?

      Senior
    • Your Java service is slow but CPU usage is low — what do you check first?

      Senior
    • You receive an OutOfMemoryError even though the heap appears sufficient — how is that possible?

      Senior
    • What production debugging techniques would you use to diagnose high CPU usage or memory leaks?

      Senior
    • How do you approach diagnosing tail latency issues in a production environment?

      Senior
    • How would you diagnose long GC pauses in a production JVM service?

      Senior
    • How do you detect and diagnose a deadlock or lock contention in a running production system?

      Senior

    Distributed Failure Modes

    • How do you diagnose and mitigate a disk-full, inode, or file-descriptor exhaustion incident?

      Mid
    • How would you diagnose and handle a poison-pill message or a growing queue backlog?

      Mid
    • How would you recognize and respond to a DNS failure or an expired TLS certificate causing an outage?

      Mid
    • How do you handle partial failures in a system to prevent a complete outage?

      Senior
    • What is a cascading failure, how do you recognize one in progress, and how do you stop it?

      Senior
    • What is a retry storm or thundering herd, and how do you mitigate it during an incident?

      Senior
    • What is a cache stampede or dogpile effect, and how would you diagnose and mitigate it?

      Senior
    • What is a metastable failure, and why can a system stay broken even after the original trigger is gone?

      Senior
    • What is a gray failure, and why are partial or 'limping' failures harder to detect and diagnose than hard crashes?

      Senior
    • How do slow dependencies and timeout propagation cause outages, and how do you contain them?

      Senior
    • What are hot shards or noisy-neighbor problems, and how would you diagnose one in production?

      Senior
    • How can clock skew cause production incidents, and how would you diagnose it?

      Senior
    • How do you detect and respond to split-brain in a clustered or replicated system during an incident?

      Senior

    Resilience & Deployment Safety

    • What are runbooks and playbooks, and how do they aid in incident preparedness and response?

      Junior
    • How do canary or blue-green deployments limit the blast radius of a bad release?

      Mid
    • What is shadow traffic or traffic replay, and when would you use it to debug a production problem safely?

      Senior