Distributed Transactions & Messaging
Compare 2PC vs. Sagas. When would you choose one over the other in terms of atomicity vs. availability?
Explain the difference between 'At-least-once', 'At-most-once', and 'Exactly-once' delivery guarantees.
Explain the Two-Phase Commit (2PC) protocol. What is its biggest blocking weakness if the coordinator fails?
What is distributed deadlock, and how can it be detected or prevented across multiple nodes?
What is the difference between 'At-least-once' and 'Exactly-once' delivery? Why is 'Exactly-once' often described as an abstraction rather than a literal network guarantee?
What is the Saga pattern, and how do 'Choreography' and 'Orchestration' compare in the context of long-running distributed transactions?
What is the Saga pattern, and how does it maintain data consistency without using a distributed lock?
What is the TCC (Try-Confirm-Cancel) pattern, and how does it differ from a Saga?
What is the Three-Phase Commit (3PC) protocol, and how does it attempt to solve the blocking problem of 2PC?
What is the Transactional Outbox pattern, and how does it solve the problem of atomically updating a database and sending a message to another system?
Why is Idempotency critical in distributed systems, and how would you design an idempotent API?
Consistency Models
Compare Strong Consistency vs. Causal Consistency. When would you prefer the latter?
Explain 'Read-Your-Writes' consistency. How would you implement it in a system with asynchronous replication?
Explain the trade-off between Latency and Consistency in a system that is not currently experiencing a network partition.
What are monotonic reads and monotonic writes consistency, and what anomalies do they prevent?
What is the difference between Linearizability and Eventual Consistency? When is Eventual Consistency 'not enough'?
What is the difference between Linearizability and Sequential Consistency? Which one is harder to achieve and why?
What is the difference between Linearizability and Serializability, and can a system be both?
What is the difference between Strong Consistency and Eventual Consistency? When would you choose one over the other?
Scalability Resilience & Data Processing
Explain 'Backpressure'. Why is it critical for a downstream service to signal its inability to keep up with requests?
Explain the 'Thundering Herd' problem. How do jitter and exponential backoff help mitigate it?
Explain the MapReduce execution model. Why is 'Data Locality' a core principle of its design?
How does load balancing work as a distributed-systems building block, and what are common strategies for distributing requests across nodes?
What are 'Hedged Requests' and how do they help reduce tail latency in a distributed environment?
What is a distributed file system like GFS or HDFS at a conceptual level, and how does it achieve fault tolerance for large data?
What is the difference between 'Batch Processing' and 'Stream Processing' at a conceptual level?
Why do retries in a distributed system need backoff and jitter, and how can naive retries make an outage worse?
Time & Logical Clocks
Explain how a Lamport Clock works. If two events have the same Lamport timestamp, can we determine which one happened first? Why or why not?
Explain the 'Happens-Before' relationship. How do we determine if two events are concurrent?
Explain the concept of 'Clock Skew' and its impact on distributed transaction ordering.
How do Vector Clocks allow us to detect 'causal' relationships between events? In what scenario would a Vector Clock show that two writes are 'concurrent' or 'conflicting'?
What are Hybrid Logical Clocks (HLC), and why are they used in modern databases like CockroachDB or Spanner?
What is a Lamport Clock, and how does it allow us to determine the partial ordering of events without a synchronized physical clock?
What is a Lamport Clock, and how does it differ from a Vector Clock? What specific problem do Vector Clocks solve that Lamport Clocks cannot?
What is Clock Skew and Clock Drift, and how do these physical realities impact distributed algorithms like Spanner's TrueTime?
What is Last-Write-Wins (LWW), and why is it dangerous in systems where physical clocks are not perfectly synchronized?
What is TrueTime in Google Spanner, and how does bounding clock uncertainty enable externally consistent transactions?
Why can't we rely on physical system clocks (NTP) for ordering events in a distributed system? What is 'clock skew' and how does it affect distributed logs?
Replication & Partitioning
Explain how Consistent Hashing works. How does it minimize data movement when a node is added or removed compared to a simple hash(key) mod N approach?
Explain the difference between Leader-based, Multi-leader, and Leaderless replication.
How does a request get routed to the correct partition or shard, and what are the trade-offs between different routing approaches?
What are sloppy quorums and hinted handoff, and how do they improve availability in Dynamo-style systems?
What are the trade-offs between synchronous and asynchronous replication? If you choose 'Semi-synchronous' replication, what failure scenario are you trying to protect against?
What are Virtual Nodes in consistent hashing, and what problem do they solve regarding data distribution?
What is a Hotspot or Hot Key in a sharded system, and how can you detect and mitigate it at the architectural level?
What is anti-entropy, and how do read-repair and Merkle trees help replicas converge in a leaderless system?
What is chain replication, and what are its trade-offs compared to leader-based replication?
What is data rebalancing when partitions are added or removed, and what strategies avoid moving too much data at once?
What is redundancy, and how does replication contribute to availability versus consistency?
What is the difference between Range Partitioning and Hash Partitioning, and what are the tradeoffs regarding range queries vs. load balancing?
Cap & Pacelc Theorems
Explain the CAP theorem. In a real-world network partition, why is it technically impossible to choose both Consistency and Availability? Can you give an example of a 'CP' vs. an 'AP' system?
Explain the PACELC theorem. How does it extend the CAP theorem, and what does it tell us about a system when there is no network partition?
Explain the relationship R + W > N. If I want to increase read throughput in a distributed database, how should I tune these variables, and what is the trade-off regarding consistency?
What is a 'Quorum' in a distributed system, and how do the values of N, R, and W affect consistency and availability?
Why is 'Partition Tolerance' generally considered non-negotiable in distributed systems?
Foundations & Motivation
Explain the difference between 'Horizontal Scaling' and 'Vertical Scaling' from a state-management perspective.
Explain the Two Generals Problem. Why can two nodes never be 100% certain they have reached a common agreement over an unreliable network?
What are the Fallacies of Distributed Computing? Name at least three and explain how ignoring them leads to system failure.
What is an RPC, and what challenges arise from treating a remote call like a local function call?
What is partial failure, and why does it make distributed systems fundamentally harder to reason about than single-machine programs?
What is the difference between synchronous and asynchronous communication models in distributed systems, and what are the trade-offs?
What is the difference between the request/response and publish/subscribe coordination paradigms?
Why do stateless services scale more easily than stateful ones, and how do we handle state when scaling horizontally?
Why do we build distributed systems in the first place: what problems do scalability, fault tolerance, low latency, and geographic distribution solve?
Coordination & Cluster Management
Explain the Gossip (Epidemic) Protocol. How does it allow a cluster of thousands of nodes to maintain a membership list without a central registry?
How do coordination services like ZooKeeper, etcd, and Chubby provide primitives such as locks and configuration, and why not build these yourself?
What is failover, and how does a system detect failure and promote a standby without causing inconsistency?
What is service discovery, and why is it a coordination problem in dynamic distributed systems?
What is the 'Split Brain' problem, and how do quorums and fencing tokens prevent it?
What is the coordinator/worker (master/worker) pattern, and what are the risks of having a single coordinator?
What is the SWIM protocol for membership and failure detection, and how does it improve on naive gossip?
Why is a simple 'distributed lock' often dangerous? Explain how 'Leases' and 'Fencing Tokens' solve the problem of a process holding a lock but being paused.
Consensus & Coordination Protocols
Explain the high-level roles in the Raft consensus algorithm (Leader, Follower, Candidate). How does the system ensure that two leaders are never elected for the same term?
How do heartbeats and election timeouts work together in a consensus protocol? What happens if the election timeout is set too low or too high?
How does PBFT achieve consensus in the presence of Byzantine nodes, and how many faulty nodes can it tolerate?
In the context of distributed algorithms, what is the difference between a 'safety' property and a 'liveness' property? Why is it often impossible to guarantee both in an asynchronous system (FLP impossibility)?
What are the different message ordering guarantees, FIFO, causal, and total order, and when does each matter?
What is 'Leader Election', and what happens to a distributed system when the leader becomes partitioned from the followers?
What is state machine replication, and why is a replicated log central to building consistent distributed systems?
What is the Chandy-Lamport algorithm for capturing a consistent global snapshot of a distributed system?
What is the difference between basic Paxos and Multi-Paxos, and why is Multi-Paxos more practical for replicated state machines?
What is the difference between Paxos and Raft, and why is Raft often preferred for practical implementations despite Paxos being the theoretical gold standard?
What is the FLP Impossibility Result, and why is it impossible to guarantee consensus in a purely asynchronous system if even one node can fail?
What is the role of a 'Term' or 'Epoch' in consensus protocols like Raft or Paxos?
What is the ZAB protocol used by ZooKeeper, and how does it differ from Paxos and Raft?
What is total order broadcast (atomic broadcast), and why is it considered equivalent to consensus?
Eventual Consistency & Conflict Resolution
What are CRDTs (Conflict-free Replicated Data Types), and how do they allow for concurrent updates without a central coordinator?
What are the practical risks of Eventual Consistency? How do concepts like 'Read-your-writes' or 'Monotonic Reads' help mitigate the poor user experience of eventual consistency?
What are the trade-offs of Eventual Consistency, and how does a system eventually converge if multiple nodes receive conflicting updates?
What is 'Replication Lag,' and what specific anomalies can a user see because of it? How would you design a system to prevent a user from seeing their own update disappear after a refresh?
Failure Models & Detection
What are the omission and timing failure models, and how do they sit between crash and Byzantine failures?
What is a 'Lease' in distributed systems, and how does it improve upon a standard heartbeat for failure detection?
What is a cascading failure in a distributed system, and what techniques like load shedding help contain it?
What is a Phi-Accrual Failure Detector, and how does it differ from a simple heartbeat timeout?
What is graceful degradation, and how does a distributed system stay partially functional when components fail?
What is the Byzantine Generals Problem, and how does it differ from a simple 'fail-stop' or 'crash' failure? When would a software engineer actually need to care about Byzantine Fault Tolerance (BFT)?