Foundations & Motivation
What are the Fallacies of Distributed Computing? Name at least three and explain how ignoring them leads to system failure.
Explain the difference between 'Horizontal Scaling' and 'Vertical Scaling' from a state-management perspective.
What is an RPC, and what challenges arise from treating a remote call like a local function call?
What is the difference between the request/response and publish/subscribe coordination paradigms?
Why do we build distributed systems in the first place: what problems do scalability, fault tolerance, low latency, and geographic distribution solve?
What is partial failure, and why does it make distributed systems fundamentally harder to reason about than single-machine programs?
Explain the Two Generals Problem. Why can two nodes never be 100% certain they have reached a common agreement over an unreliable network?
What is the difference between synchronous and asynchronous communication models in distributed systems, and what are the trade-offs?
Why do stateless services scale more easily than stateful ones, and how do we handle state when scaling horizontally?
Consistency Models
What is the difference between Strong Consistency and Eventual Consistency? When would you choose one over the other?
Explain 'Read-Your-Writes' consistency. How would you implement it in a system with asynchronous replication?
Explain the trade-off between Latency and Consistency in a system that is not currently experiencing a network partition.
What is the difference between Linearizability and Eventual Consistency? When is Eventual Consistency 'not enough'?
What are monotonic reads and monotonic writes consistency, and what anomalies do they prevent?
Compare Strong Consistency vs. Causal Consistency. When would you prefer the latter?
What is the difference between Linearizability and Serializability, and can a system be both?
What is the difference between Linearizability and Sequential Consistency? Which one is harder to achieve and why?
Cap & Pacelc Theorems
Explain the CAP theorem. In a real-world network partition, why is it technically impossible to choose both Consistency and Availability? Can you give an example of a 'CP' vs. an 'AP' system?
What is a 'Quorum' in a distributed system, and how do the values of N, R, and W affect consistency and availability?
Explain the relationship R + W > N. If I want to increase read throughput in a distributed database, how should I tune these variables, and what is the trade-off regarding consistency?
Why is 'Partition Tolerance' generally considered non-negotiable in distributed systems?
Explain the PACELC theorem. How does it extend the CAP theorem, and what does it tell us about a system when there is no network partition?
Failure Models & Detection
What is the Byzantine Generals Problem, and how does it differ from a simple 'fail-stop' or 'crash' failure? When would a software engineer actually need to care about Byzantine Fault Tolerance (BFT)?
What is graceful degradation, and how does a distributed system stay partially functional when components fail?
What is a Phi-Accrual Failure Detector, and how does it differ from a simple heartbeat timeout?
What is a 'Lease' in distributed systems, and how does it improve upon a standard heartbeat for failure detection?
What is a cascading failure in a distributed system, and what techniques like load shedding help contain it?
What are the omission and timing failure models, and how do they sit between crash and Byzantine failures?
Replication & Partitioning
What is redundancy, and how does replication contribute to availability versus consistency?
Explain how Consistent Hashing works. How does it minimize data movement when a node is added or removed compared to a simple hash(key) mod N approach?
What are Virtual Nodes in consistent hashing, and what problem do they solve regarding data distribution?
What are the trade-offs between synchronous and asynchronous replication? If you choose 'Semi-synchronous' replication, what failure scenario are you trying to protect against?
Explain the difference between Leader-based, Multi-leader, and Leaderless replication.
What is the difference between Range Partitioning and Hash Partitioning, and what are the tradeoffs regarding range queries vs. load balancing?
What is data rebalancing when partitions are added or removed, and what strategies avoid moving too much data at once?
How does a request get routed to the correct partition or shard, and what are the trade-offs between different routing approaches?
What is a Hotspot or Hot Key in a sharded system, and how can you detect and mitigate it at the architectural level?
What is chain replication, and what are its trade-offs compared to leader-based replication?
What is anti-entropy, and how do read-repair and Merkle trees help replicas converge in a leaderless system?
What are sloppy quorums and hinted handoff, and how do they improve availability in Dynamo-style systems?
Time & Logical Clocks
What is a Lamport Clock, and how does it differ from a Vector Clock? What specific problem do Vector Clocks solve that Lamport Clocks cannot?
Explain the 'Happens-Before' relationship. How do we determine if two events are concurrent?
Why can't we rely on physical system clocks (NTP) for ordering events in a distributed system? What is 'clock skew' and how does it affect distributed logs?
What is a Lamport Clock, and how does it allow us to determine the partial ordering of events without a synchronized physical clock?
What is Last-Write-Wins (LWW), and why is it dangerous in systems where physical clocks are not perfectly synchronized?
Explain the concept of 'Clock Skew' and its impact on distributed transaction ordering.
Explain how a Lamport Clock works. If two events have the same Lamport timestamp, can we determine which one happened first? Why or why not?
How do Vector Clocks allow us to detect 'causal' relationships between events? In what scenario would a Vector Clock show that two writes are 'concurrent' or 'conflicting'?
What are Hybrid Logical Clocks (HLC), and why are they used in modern databases like CockroachDB or Spanner?
What is Clock Skew and Clock Drift, and how do these physical realities impact distributed algorithms like Spanner's TrueTime?
What is TrueTime in Google Spanner, and how does bounding clock uncertainty enable externally consistent transactions?
Eventual Consistency & Conflict Resolution
What is 'Replication Lag,' and what specific anomalies can a user see because of it? How would you design a system to prevent a user from seeing their own update disappear after a refresh?
What are the practical risks of Eventual Consistency? How do concepts like 'Read-your-writes' or 'Monotonic Reads' help mitigate the poor user experience of eventual consistency?
What are the trade-offs of Eventual Consistency, and how does a system eventually converge if multiple nodes receive conflicting updates?
What are CRDTs (Conflict-free Replicated Data Types), and how do they allow for concurrent updates without a central coordinator?
Consensus & Coordination Protocols
Explain the high-level roles in the Raft consensus algorithm (Leader, Follower, Candidate). How does the system ensure that two leaders are never elected for the same term?
What is 'Leader Election', and what happens to a distributed system when the leader becomes partitioned from the followers?
How do heartbeats and election timeouts work together in a consensus protocol? What happens if the election timeout is set too low or too high?
What are the different message ordering guarantees, FIFO, causal, and total order, and when does each matter?
What is the FLP Impossibility Result, and why is it impossible to guarantee consensus in a purely asynchronous system if even one node can fail?
What is the difference between Paxos and Raft, and why is Raft often preferred for practical implementations despite Paxos being the theoretical gold standard?
What is the role of a 'Term' or 'Epoch' in consensus protocols like Raft or Paxos?
In the context of distributed algorithms, what is the difference between a 'safety' property and a 'liveness' property? Why is it often impossible to guarantee both in an asynchronous system (FLP impossibility)?
What is the difference between basic Paxos and Multi-Paxos, and why is Multi-Paxos more practical for replicated state machines?
What is the ZAB protocol used by ZooKeeper, and how does it differ from Paxos and Raft?
What is state machine replication, and why is a replicated log central to building consistent distributed systems?
What is total order broadcast (atomic broadcast), and why is it considered equivalent to consensus?
What is the Chandy-Lamport algorithm for capturing a consistent global snapshot of a distributed system?
How does PBFT achieve consensus in the presence of Byzantine nodes, and how many faulty nodes can it tolerate?
Distributed Transactions & Messaging
Why is Idempotency critical in distributed systems, and how would you design an idempotent API?
Explain the Two-Phase Commit (2PC) protocol. What is its biggest blocking weakness if the coordinator fails?
What is the Saga pattern, and how does it maintain data consistency without using a distributed lock?
Explain the difference between 'At-least-once', 'At-most-once', and 'Exactly-once' delivery guarantees.
What is the Transactional Outbox pattern, and how does it solve the problem of atomically updating a database and sending a message to another system?
Compare 2PC vs. Sagas. When would you choose one over the other in terms of atomicity vs. availability?
What is the difference between 'At-least-once' and 'Exactly-once' delivery? Why is 'Exactly-once' often described as an abstraction rather than a literal network guarantee?
What is the Saga pattern, and how do 'Choreography' and 'Orchestration' compare in the context of long-running distributed transactions?
What is the Three-Phase Commit (3PC) protocol, and how does it attempt to solve the blocking problem of 2PC?
What is the TCC (Try-Confirm-Cancel) pattern, and how does it differ from a Saga?
What is distributed deadlock, and how can it be detected or prevented across multiple nodes?
Coordination & Cluster Management
Explain the Gossip (Epidemic) Protocol. How does it allow a cluster of thousands of nodes to maintain a membership list without a central registry?
What is the 'Split Brain' problem, and how do quorums and fencing tokens prevent it?
What is failover, and how does a system detect failure and promote a standby without causing inconsistency?
What is service discovery, and why is it a coordination problem in dynamic distributed systems?
What is the coordinator/worker (master/worker) pattern, and what are the risks of having a single coordinator?
Why is a simple 'distributed lock' often dangerous? Explain how 'Leases' and 'Fencing Tokens' solve the problem of a process holding a lock but being paused.
How do coordination services like ZooKeeper, etcd, and Chubby provide primitives such as locks and configuration, and why not build these yourself?
What is the SWIM protocol for membership and failure detection, and how does it improve on naive gossip?
Scalability Resilience & Data Processing
What is the difference between 'Batch Processing' and 'Stream Processing' at a conceptual level?
How does load balancing work as a distributed-systems building block, and what are common strategies for distributing requests across nodes?
Explain the 'Thundering Herd' problem. How do jitter and exponential backoff help mitigate it?
Explain the MapReduce execution model. Why is 'Data Locality' a core principle of its design?
Explain 'Backpressure'. Why is it critical for a downstream service to signal its inability to keep up with requests?
What is a distributed file system like GFS or HDFS at a conceptual level, and how does it achieve fault tolerance for large data?
Why do retries in a distributed system need backoff and jitter, and how can naive retries make an outage worse?
What are 'Hedged Requests' and how do they help reduce tail latency in a distributed environment?