Fundamentals & Core Concepts
When would you choose a Message Queue over a direct API call (REST/gRPC)?
What is the difference between synchronous and asynchronous communication, and where does a message queue fit in?
What is a Message Queue, and what problem does it solve?
Explain the concept of "decoupling" in the context of message-driven systems.
What is the role of a "Message Broker" in a distributed system?
How does a message queue help with 'Temporal Decoupling' in a microservices architecture?
Why would you use a Message Queue instead of just writing tasks to a database table and polling it?
What are the trade-offs of using a Message Queue?
Messaging Models & Patterns
Explain the difference between a Point-to-Point (Queue) model and a Publish/Subscribe (Topic) model.
Explain the 'competing consumers' pattern and what problem it solves.
What is the difference between a Pull-based and a Push-based messaging model? What are the trade-offs for consumer scaling?
When should you use a Message Queue versus an Event Stream (like Kafka)?
When would you choose a 'Smart Broker / Dumb Consumer' architecture over a 'Dumb Broker / Smart Consumer' architecture?
In the context of microservices, what is the difference between orchestration and choreography when using an event-driven architecture?
Delivery Semantics & Acknowledgements
In a message queue system, what is the purpose of an 'Acknowledgement' (ACK)?
Explain the concept of "Idempotency." Why is it critical for message consumers?
Explain the three message delivery semantics (At-most-once, At-least-once, Exactly-once) and the trade-offs of each.
What happens if a consumer crashes after it has processed a message but before it sends the acknowledgment (ACK) back to the broker?
What does the producer 'acks' setting (acks=0, acks=1, acks=all) control, and how does it trade off durability against latency?
How do you achieve message deduplication, and when is it needed despite at-least-once delivery?
Explain the challenges of achieving 'exactly-once' delivery semantics in a distributed system. Is it ever truly possible?
Broker Architectures & Comparisons
Explain the difference between a Direct exchange, a Fanout exchange, and a Topic exchange in RabbitMQ.
When would you choose a managed service like AWS SQS over a self-hosted Kafka cluster?
Compare RabbitMQ and Kafka: in what scenarios is one fundamentally better than the other?
What is the fundamental architectural difference between a 'Distributed Commit Log' (like Kafka) and a 'Traditional Message Queue' (like RabbitMQ)?
Explain the concept of 'Message Retention.' Why does Kafka keep messages after they are consumed, while RabbitMQ typically deletes them?
What role does a coordination service like ZooKeeper or KRaft play in a Kafka cluster?
Failure Handling & Retries
What is a Dead Letter Queue (DLQ), and when should a message be moved there?
What is a 'poison pill' message, and how do you prevent it from stalling your entire ingestion pipeline?
When would you choose to route a message to a Dead Letter Queue versus simply retrying the operation?
What are the trade-offs of using exponential backoff versus a fixed retry interval when a consumer fails?
What is a "Poison Pill" message, and how does it impact a consumer group?
What is a Dead Letter Queue (DLQ), and what is the danger of automatically retrying messages from a DLQ without manual intervention?
Ordering & Partitioning
How does the number of partitions in a Kafka topic affect both parallelism and message ordering?
Why does "Partitioning" or "Sharding" exist in message queues, and how does it affect scalability?
Why is message ordering typically only guaranteed within a single partition or shard, rather than across the entire queue?
How does a partition (or routing) key determine which partition a message lands in, and why does that matter for ordering?
What is the difference between a FIFO queue and a standard queue (e.g. in SQS), and what are the trade-offs?
How do you ensure message ordering in a distributed environment where multiple consumers are active?
Under what circumstances does a message queue guarantee ordering, and when might that guarantee be broken in a distributed environment?
How do you handle 'Out-of-Order' messages if your business logic requires strict sequential processing?
Observability & Operations
What is 'consumer lag,' and what are the common architectural reasons it occurs?
Explain the concept of 'Visibility Timeout' in Amazon SQS. What happens if the timeout is too short for the processing time?
What is 'Consumer Lag,' and why is it a critical metric to monitor?
What are the primary trade-offs (latency, complexity, consistency) when moving from a synchronous REST call to an asynchronous message queue?
How do you implement distributed tracing across a message queue?
Durability Replication & Availability
What does it mean for a message to be 'durable,' and how does that impact system performance?
How does a distributed queue maintain High Availability if one of the broker nodes fails?
Explain the trade-offs between 'Memory-first' storage (RabbitMQ) vs. 'Disk-first' storage (Kafka) regarding performance and durability.
How do you handle "Reliable Delivery" if the message broker itself crashes?
What is a replication factor and an in-sync replica (ISR) set, and how do they ensure durability in a distributed broker?
How does leader/follower replication work for a partition, and what happens during a leader election?
Offsets Retention & Replay
What is message TTL or expiration, and what use cases require it?
What is an 'offset' in Kafka, and how does offset management affect message consumption and replay?
How do you replay or reprocess messages, and how does this differ between a traditional queue and a log-based system?
What is 'Log Compaction' in Kafka, and in what specific use cases would you enable it?
What is the difference between automatic and manual offset commits, and what are the risks of each?
Scaling Throughput & Flow Control
What is backpressure and how does a message queue help manage it?
How does Kafka's 'Consumer Group' concept allow for horizontal scaling of message processing?
How do you handle QoS or priority between different types of messages in the same system?
Explain the trade-off between 'Latency' and 'Throughput' when configuring message batching on the producer side.
What is consumer prefetch (or prefetch count), and how does it affect throughput and fairness between consumers?
How do you handle 'Long-Running Tasks' in a message queue to prevent them from blocking other messages or causing timeouts?
Explain the 'Thundering Herd' problem in the context of message consumers. How do you prevent it?
What is a 'Consumer Group Rebalance,' and why is it often a performance bottleneck in high-throughput systems?
If a producer goes offline for 48 hours and then sends a massive 'catch-up' batch, how should the consumer/queue architecture handle the sudden load?
In a serverless environment (e.g., AWS Lambda), how does the scaling behavior of a message queue differ from a traditional worker-based model?
Data Consistency Patterns
How do you maintain data consistency across multiple microservices using the Saga pattern without a global transaction coordinator?
How do you handle schema evolution or versioning in a message-driven architecture?
How do you handle "Eventual Consistency" when using message queues for inter-service communication?
Explain the 'Transactional Outbox Pattern' and why it's used to solve the dual-write problem.
What is the 'Dual Write' problem, and how does using a message queue potentially complicate or solve data consistency?