182 Kafka Interview Questions and Answers (2026)

Kafka is popular because it decouples producers from consumers via a durable, replayable, horizontally scalable log that handles millions of events per second with low latency, something traditional message queues and point-to-point integrations struggle to match.

• High throughput and low latency: Sequential disk writes, zero-copy transfer, and batching let it move millions of messages/sec.

• Durability and replayability: Messages persist on disk with configurable retention; consumers can re-read by resetting offsets, unlike queues that delete on consume.

• Horizontal scalability: Topics split into partitions spread across brokers, so you scale by adding brokers and partitions.

• Decoupling via pub/sub: Many independent consumer groups read the same stream without producers knowing about them.

• Fault tolerance: Partition replication survives broker failures with no data loss when configured with acks and in-sync replicas.

• vs traditional techniques: Classic brokers (RabbitMQ, JMS) delete messages after delivery and scale less; direct DB/ETL integrations are batch-oriented and tightly coupled.

Apache Kafka is a distributed, append-only commit log used as a streaming platform: producers write events to topics, brokers store them durably, and consumers read them at their own pace.

• Core idea: An event is an immutable record (key, value, timestamp) appended to a partitioned, replicated log.

• Primary use cases:

  1. Messaging / event backbone between services.

  2. Log and metrics aggregation from many sources.

  3. Stream processing with Kafka Streams or Flink (aggregations, joins, enrichment).

  4. Data integration / pipelines via Kafka Connect (DBs, object stores, search).

  5. Event sourcing and CDC as a source of truth for state changes.

• Why it fits these: Retention plus replay makes it both a transport and a temporary storage layer for streams.

REST is synchronous request/response between two known parties; Kafka is asynchronous event streaming that decouples producers from an unknown number of consumers and buffers data, making it better for high-volume, multi-consumer, and resilient pipelines.

• Decoupling: Producer doesn't need to know or wait for consumers; new consumers can be added without touching the producer.

• Buffering and backpressure: If a consumer is slow or down, events wait in the log; a REST call would fail or time out.

• Replay and durability: Consumers reprocess history by rewinding offsets; a REST response is gone once returned.

• Fan-out: One event feeds many consumer groups; REST needs one call per target.

• When REST still wins: Immediate synchronous responses, simple CRUD, and low-volume request/response interactions.

Kafka's defining features are its partitioned distributed log, replication for fault tolerance, consumer groups for scalable parallel reads, configurable retention, and strong ordering guarantees within a partition.

• Partitioned log: Each topic is split into partitions, the unit of parallelism and ordering; records within a partition keep insertion order.

• Replication: Each partition has a leader and follower replicas; in-sync replicas take over on failure, giving durability.

• Consumer groups: Partitions are divided among group members so consumption scales; different groups read the same data independently.

• Retention: Messages are kept by time or size, or compacted by key to keep the latest value per key.

• Offset-based reads: Consumers track their own offset, enabling replay and independent progress.

Yes: Kafka is a core platform for real-time processing, both as a low-latency transport and, with its stream-processing libraries, as an engine that transforms events as they arrive.

• Low-latency delivery: Events are available to consumers within milliseconds of being produced.

• Stream processing: Kafka Streams and ksqlDB do filtering, joins, and windowed aggregations continuously; external engines like Flink integrate too.

• Stateful operations: Local state stores backed by compacted topics enable running counts, sessions, and enrichment.

• Typical real-time uses: Fraud detection, monitoring/alerting, recommendations, and live dashboards.

• Caveat: Kafka is near-real-time (event streaming), not hard real-time with guaranteed sub-millisecond deadlines.

Kafka was created at LinkedIn around 2010 (open-sourced in 2011) to replace a tangle of point-to-point data pipelines with a single, unified, high-throughput log for moving activity and operational data across systems.

• The problem:

  • LinkedIn had many systems (databases, search, analytics, monitoring) each needing the same data, wired together by brittle custom pipelines (the N-to-N integration explosion).

  • Existing message queues couldn't handle the volume of user activity events and log/metrics data at low latency.

• The insight:

  • Model everything as an append-only, replayable commit log that producers write once and any number of consumers read independently.

  • Decouple producers from consumers so new systems can subscribe without touching sources.

• Design goals: High throughput via sequential disk I/O and zero-copy, horizontal scale via partitions, durability via replication.

• Original use cases: activity tracking (page views, clicks) and operational metrics/log aggregation feeding analytics and monitoring.

A topic is a named, logical stream/category of records that producers write to and consumers read from. Physically it is not a single log but a set of partitions, each an ordered, immutable, append-only sequence of records replicated across brokers.

• Partitions: A topic is divided into one or more partitions spread across brokers for scale and parallelism.

• Records and offsets: Each record has a key, value, timestamp, and headers, and is assigned a monotonically increasing offset within its partition.

• Append-only and immutable: Records are only appended; they aren't modified, and consumers track position by offset.

• Retention: Data persists per a retention policy (time/size) or is compacted, independent of whether it's been consumed.

• Replication: Each partition has a leader and follower replicas (set by `replication.factor`) for fault tolerance.

Partitioning scales a topic by splitting it into independent logs that can live on different brokers and be processed in parallel, so both storage and throughput grow horizontally as you add partitions and machines.

• Distributes load across brokers: Partitions (and their leaders) are spread across the cluster, so no single broker is a bottleneck for a topic's reads and writes.

• Enables parallel consumption: Each partition is consumed by one consumer in a group, so adding partitions lets you add consumers and process more in parallel.

• Parallel production: Producers write to many partitions concurrently, and each partition benefits from fast sequential disk writes.

• Independent scaling of storage: A topic's data need not fit on one machine; partitions divide it across nodes.

• Limit: consumer parallelism is capped by partition count, so partitions must be provisioned ahead of scaling needs.

A topic is a named, logical stream of records (like a category or feed), while a partition is the physical, ordered log that a topic is split into for scalability and parallelism.

• Topic: the logical unit:

  • Producers write to it and consumers read from it by name; it is a pure abstraction with no physical storage of its own.

  • Configured with a replication factor and a partition count.

• Partition: the physical unit:

  • An append-only, ordered commit log stored on disk; each record gets a monotonically increasing offset.

  • A topic is the set of one or more partitions; the partitions hold the actual data.

• Why the split matters:

  • Partitions enable parallelism (each can be consumed by a different consumer in a group) and horizontal scaling across brokers.

  • Ordering is guaranteed only within a partition, not across the topic.

A partition key is a value attached to a message that Kafka hashes to decide which partition the record lands in, ensuring all messages with the same key go to the same partition (and thus keep their relative order).

• Purpose: deterministic routing: By default the partition is hash(key) % numPartitions, so identical keys are co-located.

• Guarantees ordering per key: Using a key like userId keeps all events for that user in one partition and therefore ordered.

• No key provided: Records are distributed via the sticky/round-robin partitioner for even spread.

• Caveat: skew: A poorly chosen key (few distinct values) creates hot partitions and uneven load.

Yes, you can increase a topic's partition count at any time, but it has an important consequence: it breaks key-to-partition mapping for future messages.

• How: Use kafka-topics.sh --alter --partitions N or the Admin API; existing data is not moved or repartitioned.

• The main downside: broken ordering by key: Since routing is hash(key) % numPartitions, changing the partition count sends a given key to a different partition than before, so its history is now split across two partitions and global per-key order is no longer guaranteed.

• Practical guidance: Over-provision partitions up front if ordering-by-key matters, rather than growing later.

A Kafka producer is a client application that publishes (writes) records to Kafka topics, handling serialization, partition selection, batching, and delivery guarantees to the brokers.

• Core responsibilities:

  • Serialize the key and value with configured serializers.

  • Choose a partition (via key hash, custom partitioner, or sticky partitioner).

  • Batch records per partition and send them asynchronously to the leader broker.

• Asynchronous by design: Records go into an in-memory buffer and are sent in the background; you get a Future / callback for the result.

• Delivery control: Settings like acks, retries, and enable.idempotence determine durability and exactly-once semantics.

A Kafka consumer is a client application that subscribes to one or more topics and reads records from partitions in order, tracking its position via offsets.

• Core responsibilities:

  • Polls brokers for records and processes them in offset order per partition.

  • Commits offsets so it can resume after restarts (auto or manual commit).

• Offset semantics:

  • Commit-then-process risks at-most-once; process-then-commit risks at-least-once.

  • Offsets are stored in the __consumer_offsets topic keyed by group.

• Group membership: A consumer usually joins a group (via group.id) to share work; it maintains a heartbeat so the coordinator knows it's alive.

• Reading is non-destructive: Consuming does not delete data; retention is time/size based, so many consumers can read the same records.

If a consumer group has more consumers than partitions, the extra consumers sit idle: because each partition is assigned to only one consumer, there's nothing left for them to read.

• Idle consumers: With N partitions and N+M consumers, M consumers get no partitions and do no work.

• Parallelism is capped by partitions: Partition count is the ceiling on active consumers in a group; adding more won't increase throughput.

• They still provide value: Idle members act as hot standbys: if an active consumer fails, a rebalance hands its partitions to a spare, improving availability.

• Design implication: Provision enough partitions upfront to allow future scaling, since increasing partitions later can disrupt key-based ordering.

An offset is a monotonically increasing integer that uniquely identifies the position of a record within a specific partition: it is Kafka's per-partition sequence number, not a global one.

• Scope is per partition: Offset 5 in partition 0 is unrelated to offset 5 in partition 1; ordering is guaranteed only within a partition.

• Assigned by the broker on append: Once written, a record's offset never changes; it's stable and immutable.

• Used for reads: Consumers fetch by offset and track how far they've read, enabling replay by seeking to an earlier offset.

Beyond key and value, a Kafka record carries a timestamp, headers, and (implicitly) its partition and offset metadata. Headers are optional key-value pairs that attach metadata to a message without embedding it in the payload.

• Core fields:

  • Key: used for partitioning and compaction.

  • Value: the payload.

  • Timestamp: CreateTime or LogAppendTime.

  • Assigned by broker: partition and offset.

• Headers: list of (String key, byte[] value) entries:

  • Carry cross-cutting metadata: tracing/correlation IDs, schema version, content type, source system.

  • Let routing, filtering, or dead-letter logic read metadata without deserializing the value.

  • Multiple headers can share the same key (they're a list, not a map).

A Kafka broker is a single server in the Kafka cluster that stores partition data on disk, serves producer writes and consumer reads, and participates in replication. Multiple brokers together form the cluster that shares the load of all topics.

• Stores data: Persists partition logs (segments) to disk and enforces retention policies.

• Serves clients: Handles produce and fetch requests; a broker is the leader for some partitions and a follower for others.

• Participates in replication: As a follower it fetches from leaders to stay in the ISR; as a leader it commits records once replicated.

• Identity and coordination: Each broker has a unique broker.id; one broker also acts as the controller.

Bootstrap servers are an initial list of broker addresses a client uses only to make first contact; from there the client fetches full cluster metadata and connects directly to the right brokers. You don't need to list every broker, just enough to reliably reach the cluster.

• Configured via bootstrap.servers: A comma-separated host:port list; list a few brokers for resilience if one is down at startup.

• Metadata discovery: The client sends a metadata request to a bootstrap broker and receives the full broker list plus partition leaders.

• Direct connections afterward: The client then connects to the specific leader broker for each partition it produces to or consumes from.

• Advertised listeners matter: Brokers return the addresses set in advertised.listeners, so those must be reachable by clients (a common networking gotcha).

• Metadata refresh: Clients periodically refresh metadata so they follow leadership changes without reconfiguring bootstrap servers.

The difference is direction of data flow: a source connector imports data into Kafka from an external system, while a sink connector exports data out of Kafka into an external system.

• Source connector:

  • Reads from a system (database, file, queue) and produces records to Kafka topics.

  • Tracks its position with custom source offsets in connect-offsets.

  • Example: Debezium capturing MySQL change events.

• Sink connector:

  • Consumes from Kafka topics and writes records into an external system (S3, Elasticsearch, JDBC).

  • Behaves as a consumer group, committing offsets to __consumer_offsets.

• Shared machinery: Both split into tasks, run on workers, and use converters and SMTs; only the direction and offset mechanism differ.

Kafka Connect is format-agnostic at its core: it uses pluggable Converters that translate between Connect's internal data structures and the bytes on the wire, so the supported formats depend on which converter you configure.

• JSON: JsonConverter: human-readable, optionally embeds a schema in each message.

• Avro: AvroConverter: compact binary, requires Schema Registry for schema storage and evolution.

• Protobuf and JSON Schema: ProtobufConverter and JsonSchemaConverter, also backed by Schema Registry.

• Raw bytes and strings: ByteArrayConverter (pass-through) and StringConverter for simple text or binary payloads.

• Key and value are configured independently: key.converter and value.converter can differ (e.g. String key, Avro value).

Replication factor is the number of copies of a topic's partition kept across different brokers in the cluster: one leader and the rest followers.

• Set per topic: A factor of 3 means each partition has 3 total copies on 3 brokers.

• Purpose is fault tolerance:

  • If a broker fails, a follower is promoted to leader so data stays available.

  • A factor of N tolerates N-1 broker failures without data loss.

• Cannot exceed the number of brokers: Each replica must live on a distinct broker.

• Trade-off: Higher factor means more durability but more disk and network overhead. Production commonly uses 3.

At-least-once means every message is guaranteed to be delivered and processed, but under failure it may be delivered more than once. It is Kafka's default and most common semantic.

• How it is achieved:

  • Producer: acks=all with retries, so a message is resent if an ack is not received.

  • Consumer: commit offsets after processing, so a crash before commit causes reprocessing rather than loss.

• Why duplicates occur:

  • A producer retry after a lost ack writes the message twice.

  • A consumer that processes then dies before committing re-reads the same records.

• Handling duplicates: Make downstream processing idempotent (e.g. keyed upserts) or use exactly-once with transactions to eliminate them.

Message compression shrinks record batches before they leave the producer, reducing network bandwidth and disk usage at the cost of some CPU. It is configured per producer via compression.type.

• How it works:

  • The producer compresses an entire batch, not individual messages, so larger batches compress better.

  • Data is stored compressed on the broker and sent compressed to consumers, which decompress on read: end-to-end savings.

• Codec choices:

  • gzip: high ratio, higher CPU.

  • snappy and lz4: fast, moderate ratio, common for throughput.

  • zstd: strong ratio with good speed, often the modern default choice.

• Why use it:

  • Lower network cost and higher effective throughput, plus reduced storage.

  • Pairs well with batching (linger.ms) since bigger batches yield better compression.

The acks setting controls how many broker replicas must acknowledge a write before the producer considers it successful, trading durability against latency.

• acks=0: Fire and forget: producer doesn't wait for any ack. Fastest, but messages can be silently lost.

• acks=1: Leader writes and acks; followers may not have replicated yet. A leader crash right after can lose data.

• acks=all (or -1):

  • Leader waits until all in-sync replicas (ISR) have the record. Strongest durability, higher latency.

  • Only meaningful with min.insync.replicas set (e.g. 2), so writes fail if too few replicas are in sync.

• Rule of thumb: Use acks=all with idempotence for no-loss pipelines; relax only when occasional loss is acceptable for speed.

A Kafka log is the ordered, immutable, append-only sequence of records for a single partition: it's the fundamental storage abstraction. Every partition is its own log, and brokers manage these logs through segmenting and retention.

• What a log is:

  • Records are appended in order and assigned a monotonically increasing offset.

  • Once written, records are immutable; consumers read by offset and don't remove data.

• How logs are managed:

  • Split into segments so old data can be dropped by deleting whole files.

  • A retention policy (delete or compact) controls what stays.

  • The log cleaner (compaction) and log retention threads run in the background per broker.

• Key markers:

  • Log end offset: next offset to be written.

  • High watermark: highest offset fully replicated and safe for consumers to read.

  • Log start offset: earliest offset still retained.

Yes: since Kafka 2.8 (early access) and production-ready in 3.3+, Kafka can run in KRaft mode where dedicated controller nodes replace ZooKeeper entirely.

• KRaft mode:

  • Controllers form a Raft quorum and hold metadata in an internal __cluster_metadata topic.

  • No ZooKeeper process is deployed at all.

• Benefits:

  • Simpler operations, one system to secure and monitor.

  • Faster controller failover and much higher partition scalability.

• Practical notes:

  • New clusters should start in KRaft; existing ZooKeeper clusters can migrate.

  • ZooKeeper mode is deprecated and removed in Kafka 4.0, so KRaft is the path forward.

Kafka Streams is a lightweight Java/Scala client library for building real-time stream-processing applications that read from Kafka topics, transform data, and write results back to Kafka, all with no separate processing cluster.

• Core abstractions:

  • KStream: an unbounded stream of immutable events.

  • KTable: a changelog view representing the latest value per key.

  • GlobalKTable: fully replicated lookup table for non-co-partitioned joins.

• Capabilities:

  • Stateless ops (map, filter) and stateful ops (aggregations, joins, windowing).

  • Fault-tolerant local state via RocksDB plus changelog topics.

• Operational model:

  • Just a library in your app: scale by adding instances; parallelism follows partition count.

  • Supports event-time semantics and exactly-once processing.

A stream is an unbounded, continuously updating, ordered, replayable, and fault-tolerant sequence of immutable data records, where each record is a key-value pair. It is the fundamental data model that maps directly onto one or more Kafka topic partitions.

• Unbounded: There's no fixed end; records arrive forever, so processing is continuous rather than batch.

• Ordered and immutable: Records are ordered per partition and never change once written; you append new records instead of editing.

• Replayable and fault-tolerant: Because it's backed by Kafka's durable log, you can reprocess from any offset.

• Stream-table duality: A stream can be aggregated into a table, and a table's changelog is itself a stream: the two are interchangeable views of the same data.

Kafka Streams is a client library for building real-time processing applications directly on top of Kafka topics, so its use cases center on continuous, event-driven transformation and enrichment of streaming data.

• Stateless transformations: Filtering, mapping, and routing events (e.g. cleansing or reformatting records with filter / mapValues).

• Real-time aggregations: Counts, sums, and windowed metrics like clicks per minute or running totals per key.

• Joins and enrichment: Enrich an event stream against a KTable of reference data (e.g. attach customer profile to an order).

• Event-driven microservices: Services that react to events and produce derived events, with local state instead of an external DB.

• Monitoring and anomaly detection: Detect fraud, threshold breaches, or session patterns in near real time.

• Materialized views via interactive queries: Maintain queryable state stores that serve up-to-date aggregates without a separate database.

Kafka is used in microservices to enable asynchronous, event-driven communication: services publish events and react to them instead of calling each other directly, which reduces coupling and improves resilience.

• Loose coupling: Services communicate through topics, not direct endpoints, so they evolve and deploy independently.

• Resilience: A down service doesn't break callers; events accumulate and are processed on recovery.

• Data consistency patterns: Supports the Saga and outbox patterns for distributed transactions across services.

• Event sourcing / CQRS: State changes become events other services materialize into their own read models.

• Scalable fan-out: One domain event drives many downstream services without the source knowing them.

By 2026 Kafka usage has shifted toward simpler, cloud-native operations: ZooKeeper is gone in favor of KRaft, tiered storage decouples compute from long-term retention, and Kafka increasingly serves as the backbone for streaming data into analytics and AI/ML systems.

• KRaft replaces ZooKeeper: Metadata is managed by Kafka's own Raft quorum, simplifying deployment and improving scalability of partitions.

• Tiered storage: Older segments offload to object storage (S3, etc.), so brokers keep less local disk and retention becomes near-infinite and cheaper.

• Managed and serverless offerings: Cloud services (Confluent, MSK) and pay-per-use models reduce the operational burden of running clusters.

• Streaming for AI/ML and data lakes: Kafka feeds feature pipelines, real-time inference, and lakehouse ingestion as a standard pattern.

• Ecosystem standardization: Wider adoption of the schema registry, data contracts, and formats like Iceberg-backed topics for governance.

Change Data Capture (CDC) is the technique of capturing every insert, update, and delete in a database and streaming those changes into Kafka as events, so downstream systems stay in sync in near-real-time without batch ETL.

• How it works: Tools like Debezium (running on Kafka Connect) read the database's transaction log (WAL/binlog) and publish row-level change events to topics.

• Why log-based: Reading the commit log is low-impact and captures every change in order, unlike polling or trigger-based approaches.

• Event shape: Each event carries before/after state and the operation type; primary key is used as the message key for ordering per row.

• Common uses: Cache invalidation, replicating to search indexes/data lakes, microservice sync, and the outbox pattern.

• Pairs with log compaction: Compacted topics retain the latest state per key, mirroring the source table.

Kafka is a distributed, durable log built for high-throughput streaming and replayable event storage, while SQS/SNS are fully-managed messaging services optimized for simple decoupling with minimal operational overhead. Choose Kafka for scale, ordering, and replay; choose SQS/SNS when you want zero infrastructure and simple queue/pub-sub semantics.

• Retention and replay:

  • Kafka retains messages by time/size and lets consumers re-read from any offset; a log stays after consumption.

  • SQS deletes a message once acknowledged; there is no replay of consumed messages.

• Delivery model:

  • Kafka is a pull-based log where many independent consumer groups read the same topic.

  • SQS is a point-to-point queue (one consumer group effectively), SNS is push-based pub/sub fan-out.

• Ordering and throughput:

  • Kafka gives per-partition ordering and scales to millions of messages/sec.

  • SQS standard queues are best-effort ordering; FIFO queues order but cap throughput (300 msg/s per group without batching).

• Operational cost:

  • Kafka you run (or pay for a managed service like MSK/Confluent) and tune.

  • SQS/SNS are serverless: no brokers, no partitions, pay-per-request.

• When to choose which:

  • Kafka: event streaming, stream processing, high volume, replay, multiple consumers of the same data.

  • SQS/SNS: simple task queues, decoupling microservices, fan-out notifications, low volume, minimal ops.

A partition is the unit of parallelism, ordering, and storage in Kafka: a topic is split into partitions, each an ordered, append-only log. Partitions let a topic scale across brokers and consumers, but ordering is guaranteed only within a partition, not across them.

• Scalability: Partitions distribute a topic's data and load across multiple brokers, so throughput scales horizontally by adding partitions and brokers.

• Parallelism:

  • Within a consumer group, each partition is consumed by exactly one consumer, so max consumer parallelism equals the partition count.

  • More partitions than consumers means some consumers handle multiple; more consumers than partitions means some sit idle.

• Ordering:

  • Kafka guarantees order only within a single partition, by offset.

  • Messages sharing a key are hashed to the same partition, preserving per-key order.

• Trade-off: Partitioning is the tension point: more partitions buy parallelism but sacrifice global ordering and add overhead (open file handles, metadata, rebalance time).

Kafka guarantees ordering only within a single partition: records in a partition are stored and delivered in the exact offset order they were appended. There is no ordering guarantee across partitions of a topic.

• Per-partition ordering: Each partition is an append-only log; consumers read it sequentially by increasing offset.

• Keys drive order:

  • Records with the same key hash to the same partition, so all events for that key stay ordered.

  • To keep related events ordered, choose a key (e.g. userId, orderId) that groups them.

• Producer pitfalls: With retries and `max.in.flight.requests.per.connection` > 1, retried batches can reorder; enable idempotence (`enable.idempotence=true`) to preserve order safely.

• No global order: If you need total ordering across a whole topic, you must use a single partition, sacrificing parallelism.

No, Kafka does not allow decreasing the partition count of a topic; it is a one-way operation because shrinking would require destroying or merging log data and would corrupt offsets and ordering.

• Why it's disallowed:

  • Each partition is an independent ordered log with its own offsets; merging logs would break offset continuity and consumer positions.

  • Data in a removed partition would have to be moved, breaking per-key ordering and the append-only guarantee.

• The workaround: Create a new topic with fewer partitions and migrate/reprocess data into it (for example with a stream copy or MirrorMaker).

Kafka only orders records within a single partition because each partition is an independent log written and consumed in parallel; there is no shared clock or coordination across partitions that could impose a total order without destroying the scalability that partitions provide.

• Partitions are independent logs:

  • Offsets are only meaningful within one partition; there is no cross-partition sequence number.

  • Different partitions may live on different brokers and are written concurrently.

• Global order would kill parallelism: A total order requires a single serialization point, which defeats the whole purpose of splitting a topic for throughput.

• The practical answer:

  • Route related events to the same partition with a shared key when you need ordering for that entity.

  • If you truly need global order, use a single partition (and accept a single-consumer throughput ceiling).

A producer picks a partition per record using a partitioner: if you specify a partition it uses that, otherwise it hashes the key when present, and if there is no key it spreads records across partitions using the sticky (formerly round-robin) strategy.

• Explicit partition: If the record specifies a partition number, that value is used directly.

• Key-based hashing (key present): Partition = murmur2(key) % numPartitions, so equal keys always map to the same partition, preserving per-key order.

• No key: sticky partitioner: Modern clients fill one partition's batch, then switch ("stick"), improving batching efficiency over pure round-robin while still balancing load over time.

• Custom partitioner: Implement the Partitioner interface for domain-specific routing (for example geography or tenant-based).

The most important producer configs govern durability, ordering, throughput, and correctness: chiefly acks, retries, enable.idempotence, and the batching/latency knobs.

• Durability:

  • acks: 0 (fire-and-forget), 1 (leader only), all (all in-sync replicas; strongest).

  • min.insync.replicas (broker/topic side) pairs with acks=all to define how many replicas must confirm.

• Correctness and ordering:

  • enable.idempotence=true prevents duplicates from retries and preserves order.

  • retries and max.in.flight.requests.per.connection (keep at 5 or less with idempotence to protect ordering).

• Throughput vs latency:

  • batch.size and linger.ms: larger batches / small wait improve throughput at the cost of latency.

  • compression.type (snappy, lz4, zstd) shrinks payloads and boosts effective throughput.

  • buffer.memory bounds the in-memory record buffer before the producer blocks.

Producers achieve fault tolerance by combining broker replication with acknowledgments, retries, and idempotence so records survive broker failures and aren't lost or duplicated.

• Acknowledgments (acks):

  • acks=0: fire and forget (fastest, no durability guarantee).

  • acks=1: leader confirms; data lost if leader fails before replication.

  • acks=all: leader plus all in-sync replicas confirm (strongest durability).

• Replication and min.insync.replicas: Each partition has replicas; min.insync.replicas with acks=all rejects writes if too few replicas are in sync, preventing silent loss.

• Retries and delivery.timeout.ms: The producer automatically retries transient failures (leader election, network blips) up to the delivery timeout.

• Idempotent producer: enable.idempotence=true gives each batch a producer id and sequence number so retries don't create duplicates (exactly-once per partition).

• Transactions: For atomic writes across partitions/topics, transactional.id enables all-or-nothing commits.

The sticky partitioner (default since Kafka 2.4 for keyless records) sends a batch of records to one partition until the batch fills or is sent, then switches to another partition, improving batching efficiency while still spreading load over time.

• Key-hash partitioning:

  • Used when a record has a key: partition = hash(key) % numPartitions.

  • Guarantees the same key always lands on the same partition (ordering per key).

• Round-robin (old keyless default):

  • Distributes each record evenly across partitions one at a time.

  • Downside: records for different partitions scatter across many small batches, hurting throughput.

• Sticky partitioner:

  • Only applies to records without a key.

  • Fills one partition's batch before moving on, producing fewer, larger batches (lower latency and higher throughput).

  • Over many batches the distribution is still roughly uniform, so no partition is starved.

A consumer group is a set of consumers sharing a group id that cooperatively read a topic: Kafka assigns each partition to exactly one consumer in the group, enabling horizontal scaling while preserving per-partition ordering.

• Partition-to-consumer assignment:

  • Each partition is consumed by only one member of the group, so no two consumers in the same group process the same record.

  • A single consumer can own multiple partitions if there are more partitions than consumers.

• Parallelism: Throughput scales by adding consumers up to the partition count; partitions are the unit of parallelism.

• Multiple groups, independent reads: Different groups each get a full copy of the stream (pub/sub), each tracking its own offsets.

• Rebalancing: When members join, leave, or fail, Kafka reassigns partitions so the group keeps making progress.

The high-level API abstracts away group coordination, offset tracking, and rebalancing, while the low-level (simple) API gives the developer full manual control over brokers, partitions, and offsets at the cost of complexity.

• High-level consumer (modern KafkaConsumer):

  • Handles group membership, automatic partition assignment, and rebalancing.

  • Manages offset commits (auto or manual) in the __consumer_offsets topic.

  • The default for almost all applications.

• Low-level / simple consumer:

  • You explicitly choose brokers, partitions, and start offsets and handle leader failover yourself.

  • No group coordination or automatic rebalancing.

  • Useful for niche cases needing precise replay/offset control; largely superseded by assign() and seek() on the high-level client.

• Takeaway: Modern Kafka collapses this distinction: use the high-level KafkaConsumer with subscribe() for groups or assign() for manual control.

In Kafka's pull model, consumers request (poll) data from brokers at their own pace rather than having brokers push data to them, which gives consumers control over throughput and lets them batch efficiently.

• How it works:

  • The consumer calls poll() specifying where it left off; the broker returns records from that offset.

  • The broker is stateless about consumer progress: offsets are tracked by the consumer/group.

• Why pull over push:

  • Consumers control their own rate, avoiding being overwhelmed (natural backpressure).

  • Enables efficient batching: a consumer can fetch many records per request.

  • Slow consumers don't force the broker to buffer per-consumer state.

• Long polling: To avoid busy-waiting on empty topics, fetch.min.bytes and fetch.max.wait.ms let the broker hold the request until enough data arrives.

They are two ways to decide which partitions a consumer reads: subscribe() hands partition assignment to Kafka's group protocol (dynamic, rebalanced, cooperative), while assign() pins the consumer to specific partitions manually with no group coordination.

• subscribe(): group-managed:

  • You name topics (or a pattern); the group coordinator distributes partitions across members and rebalances when membership changes.

  • Offsets are committed and tracked per consumer group, enabling scaling and failover.

• assign(): manual:

  • You explicitly list TopicPartition objects; no rebalances, no group coordination.

  • Useful for exact control (e.g., reading a specific partition, static partitioning) but you handle failover yourself.

• You cannot mix them on the same consumer instance: it's one or the other.

A consumer offset is the position a consumer group has committed for each partition: it records the next offset the group should read, so consumers can resume after restart or rebalance without reprocessing everything.

• Stored in an internal topic: Committed offsets live in the compacted topic __consumer_offsets, keyed by (group, topic, partition).

• Commit modes:

  • Auto-commit (enable.auto.commit=true) periodically commits in the background; simple but can lose or duplicate on failure.

  • Manual commit (commitSync() / commitAsync()) gives control over exactly when a batch is marked processed.

• Delivery semantics depend on order: Commit after processing = at-least-once; commit before processing = at-most-once.

Both set the record's timestamp, but the difference is who assigns it: CreateTime uses the time the producer created the record, while LogAppendTime uses the time the broker appended it to the log. The topic config message.timestamp.type selects the mode.

• CreateTime (default): Reflects application/event time; useful for real event semantics but can be out of order or skewed by producer clocks.

• LogAppendTime: Broker overwrites the timestamp on append, so it is monotonic per partition but loses the original event time.

• Impact: Affects time-based indexing, offsetsForTimes() lookups, retention-by-time, and stream processing windows.

The log-end-offset (LEO) is the offset of the next record to be written to a partition replica, while the high watermark (HW) is the highest offset that has been replicated to all in-sync replicas and is therefore safe for consumers to read.

• Log-end-offset (LEO): Per-replica: where the next append goes; the leader's LEO advances as soon as it writes, before followers catch up.

• High watermark (HW):

  • The minimum of the LEOs across the ISR; records below it are fully replicated and durable.

  • Consumers can only fetch records up to the HW, guaranteeing they never read data that could be lost if the leader fails.

• The gap: HW ≤ LEO; the difference represents written-but-not-yet-fully-replicated records that remain invisible to consumers.

__consumer_offsets is an internal, compacted Kafka topic where consumer groups durably store their committed offsets, so consumers can resume from where they left off after a restart or rebalance.

• It is a regular Kafka topic:

  • Default 50 partitions, replicated, and uses log compaction so only the latest offset per key is retained.

  • Replaced the old ZooKeeper-based offset storage for scalability.

• How commits are stored:

  • The key is (group.id, topic, partition) and the value is the committed offset plus metadata.

  • The partition a group writes to is chosen by hashing group.id, which also decides the group's coordinator.

• Read on startup: The group coordinator loads offsets from this topic to know each group's position; compaction keeps only current values.

Kafka's core components are producers, brokers (organized into a cluster with topics and partitions), consumers grouped into consumer groups, and a metadata/coordination layer (KRaft controllers or, historically, ZooKeeper). Producers write records to partitions, brokers store and replicate them, and consumers read them.

• Producers: Publish records to topics, choosing a partition by key hash or round-robin.

• Brokers and the cluster: Each broker stores partition data on disk and serves reads/writes; a cluster is many brokers sharing partitions.

• Topics and partitions: A topic is split into ordered, append-only partitions; each partition has one leader and follower replicas for fault tolerance.

• Consumers and consumer groups: Consumers pull records; within a group each partition is consumed by exactly one member, enabling parallelism.

• Coordination layer: KRaft controllers (or older ZooKeeper) manage metadata, leader election, and cluster membership.

• How they interact: Producers send to the partition leader, followers replicate, consumers fetch from leaders (or followers) and commit offsets back to Kafka.

If a broker goes down, Kafka stays available as long as replicas exist: partitions whose leader was on that broker have new leaders elected from in-sync replicas, and clients transparently reconnect to the new leaders. Data on that broker is not lost if it was replicated.

• Leader re-election: The controller detects the failure and promotes an in-sync replica (ISR) to leader for each affected partition.

• Client failover: Producers and consumers refresh metadata and route to the new leaders; brief retriable errors may occur.

• Durability depends on replication:

  • With replication.factor > 1 and healthy ISR, no committed data is lost.

  • If replication.factor=1, that broker's partitions are offline until it returns.

• Availability risk: If all ISR replicas are down, the partition is unavailable unless unclean.leader.election is enabled (which risks data loss).

• On recovery: The broker rejoins, re-syncs its replicas from current leaders, and can be reassigned leadership.

Kafka is a distributed, partitioned, replicated commit log: messages are durably appended to partitioned logs and retained for a configurable time, and consumers pull at their own pace tracking their own offset. This differs from traditional brokers that push messages and typically delete them once acknowledged.

• Log-centric storage: Records are appended to an immutable, ordered log per partition and kept by retention, not deleted on consumption.

• Pull vs push: Consumers pull and control their offset, so they can replay history; traditional brokers push and track state server-side.

• Partitioning for scale: Topics are split across brokers, giving horizontal scalability and parallel consumption via consumer groups.

• Replication for durability: Each partition is replicated across brokers with leader/follower roles, unlike many single-node queue brokers.

• Multiple independent consumers: Different groups read the same topic independently; in a classic queue a message is consumed once and gone.

Kafka Connect is a framework for streaming data between Kafka and external systems using reusable, configuration-driven connectors instead of hand-written code. It was introduced to standardize the common, repetitive integration work (scaling, fault tolerance, offset tracking) so you configure a connector rather than build and operate a bespoke producer or consumer app.

• What it is: A distributed runtime plus a plugin ecosystem of connectors (JDBC, S3, Elasticsearch, Debezium, etc.) configured via JSON over a REST API.

• Why it was introduced: Integration code is largely the same everywhere: it needs parallelism, restart-safe offset management, delivery guarantees, and monitoring. Connect solves these once.

• vs custom producers/consumers:

  • Config over code: no application to build, test, and deploy for standard sources/sinks.

  • Built-in scaling and failover: work is split into tasks and rebalanced across workers automatically.

  • Managed offsets, converters, SMTs, and dead-letter queues out of the box.

• When to still write custom code: Complex business logic or joins belong in a stream processor (Kafka Streams/Flink); Connect is for moving data, not transforming it heavily.

Connect tracks progress so it can resume after restarts, but source and sink connectors do this differently: source connectors store custom offsets in an internal Kafka topic, while sink connectors reuse Kafka's ordinary consumer group offset mechanism.

• Source connectors:

  • Each record carries a source partition and source offset (e.g. a filename plus line number, or a DB table plus row/timestamp).

  • Connect commits these to the internal connect-offsets topic (compacted), so on restart the connector asks Connect where it left off.

• Sink connectors:

  • They are regular consumers, so Kafka tracks their committed offsets per topic-partition in the __consumer_offsets topic.

  • Connect commits offsets only after the task confirms the data was delivered to the external system.

• Commit control: Frequency is governed by offset.flush.interval.ms; other internal topics are connect-configs and connect-status.

Converters handle serialization: they translate between Connect's internal, format-agnostic data representation and the byte format stored in Kafka. They are the boundary that decides whether your messages are JSON, Avro, Protobuf, or raw bytes.

• Direction of work:

  • Source: connector produces internal SchemaAndValue objects, the converter serializes them to bytes for Kafka.

  • Sink: the converter deserializes Kafka bytes back into the internal representation for the connector.

• Configured separately for keys and values: key.converter and value.converter, e.g. AvroConverter, JsonConverter, StringConverter.

• Schema handling: Avro/Protobuf converters integrate with Schema Registry; JsonConverter can embed schemas via schemas.enable.

• Key point: Converters are independent of the connector: the same JDBC connector can write Avro or JSON just by swapping converters. SMTs run between the connector and the converter.

Single Message Transforms are lightweight, per-record modifications applied inline in the Connect pipeline, letting you reshape or route messages without a separate stream processor. They operate on one message at a time and are configured declaratively in the connector.

• Where they run: Between connector and converter: after a source connector produces a record (before serialization), or after a sink deserializes (before the connector writes).

• Common uses:

  • Rename/remove/mask fields, add metadata, cast types.

  • Route records to different topics (RegexRouter, TimestampRouter).

  • Extract a field as the message key (ValueToKey, ExtractField).

• When NOT to use them: They are stateless and single-record: no joins, aggregations, or cross-record logic. For that use Kafka Streams or ksqlDB.

These are the three layers of Connect's execution model: a connector is the logical job definition, tasks are the parallel units that actually copy data, and workers are the JVM processes that run those tasks.

• Connector:

  • A configured instance of a plugin that defines what to move and decides how to divide the work into tasks (tasks.max caps the count).

  • It does no data copying itself: it's the coordinator.

• Task: The unit that actually reads or writes records; parallelism comes from running multiple tasks (e.g. one per table or per partition group).

• Worker:

  • The process hosting connectors and tasks. Standalone mode is one worker; distributed mode is a cluster of workers sharing work.

  • Workers rebalance tasks among themselves and provide fault tolerance: if one dies, its tasks move elsewhere.

A Dead Letter Queue in Kafka Connect is a Kafka topic where a sink connector routes records it cannot process, so a single bad message doesn't stop the whole pipeline. It is a resiliency feature for handling poison-pill records.

• What triggers a DLQ:

  • Errors during conversion/deserialization or transformation of a record (e.g. a malformed message that fails the Converter or an SMT).

  • Sink-side processing errors, depending on how tolerance is configured.

• How you enable it:

  • Set errors.tolerance=all plus errors.deadletterqueue.topic.name to route failed records instead of failing the task.

  • Add errors.deadletterqueue.context.headers.enable=true to record failure metadata (original topic, partition, offset, exception) in message headers for debugging.

• When to use it:

  • When you'd rather keep the pipeline flowing and inspect/reprocess bad records later than halt on the first error.

  • Note: DLQ is a sink-connector feature; source connectors don't have it.

Kafka connectors are pluggable, reusable components that move data between Kafka and external systems without you writing custom producer/consumer code. They come in two flavors: source connectors pull data into Kafka, and sink connectors push data out.

• Two directions:

  • Source: imports from external systems into Kafka topics (e.g. a database via CDC, files, message queues).

  • Sink: exports from Kafka topics into external systems (e.g. Elasticsearch, S3, JDBC databases).

• How they run:

  • A connector defines the integration; the actual work is split into tasks that Kafka Connect distributes across workers for parallelism and fault tolerance.

  • Configured declaratively via JSON/properties (connector class, topics, credentials), typically posted to the Connect REST API.

• Converters and transforms:

  • Converters handle serialization (e.g. Avro, JSON, Protobuf), often with Schema Registry.

  • Single Message Transforms (SMTs) can tweak records in flight (rename fields, mask data, route topics).

• Why use them: reuse battle-tested connectors instead of building and maintaining bespoke integration code.

Both are part of the Kafka ecosystem but solve different problems: Kafka Connect moves data in and out of Kafka using configuration, while Kafka Streams is a library for processing and transforming data within Kafka using code.

• Kafka Connect:

  • A framework for integration: connecting Kafka to external systems (databases, object stores, search indexes) via source and sink connectors.

  • Configuration-driven, no code needed; runs as its own distributed worker cluster.

• Kafka Streams:

  • A Java/Scala client library for stream processing: filtering, joining, aggregating, windowing data between Kafka topics.

  • Runs inside your application (no separate cluster); supports stateful processing with local state stores and exactly-once semantics.

• Rule of thumb: Use Connect to get data into/out of Kafka; use Streams to transform data already in Kafka.

Kafka Connect runs as a separate service so integration logic is decoupled from both the brokers and your applications, giving you a dedicated, scalable, fault-tolerant layer for data movement that is managed by configuration rather than code.

• Separation of concerns:

  • Brokers stay focused on storage and messaging; connector CPU/memory load doesn't compete with broker resources.

  • Applications don't embed producer/consumer glue code for every external system.

• Scalability and fault tolerance:

  • In distributed mode, a cluster of workers shares connectors and tasks; if a worker dies, tasks rebalance to survivors.

  • You scale the Connect cluster independently of brokers and apps.

• Centralized operations:

  • A single REST API to deploy, configure, pause, and monitor connectors.

  • Offsets, config, and status are stored in internal Kafka topics, so state survives restarts.

Yes. Because Kafka Connect defaults to at-least-once delivery, it can produce duplicates whenever a task fails after processing records but before its offsets are committed, so the work is replayed on recovery.

• Why duplicates happen:

  • A worker crashes or rebalances between writing data and committing offsets, so the last batch is re-read and re-emitted.

  • Retries after transient errors can re-send records that actually succeeded.

• How to mitigate:

  • Enable exactly-once source support (Kafka 3.3+) for connectors that implement it.

  • Use idempotent sinks: upserts keyed on a unique field or on topic/partition/offset.

  • Design downstream consumers to be idempotent or to deduplicate.

• Bottom line: treat duplicates as expected unless exactly-once is explicitly configured and supported end to end.

Both convert Connect's internal records to and from wire bytes, but JsonConverter produces self-contained JSON while AvroConverter produces compact binary and offloads schemas to the Schema Registry.

• Encoding and size: JSON is verbose text; Avro is compact binary, so Avro is smaller and faster on the wire.

• Where the schema lives:

  • JsonConverter can embed schema in every message (schemas.enable=true), bloating payloads.

  • AvroConverter stores the schema once in Schema Registry and writes only a small schema ID per message.

• Schema evolution and validation: Avro enforces compatibility rules via the registry; JSON has no central enforcement.

• Dependencies and readability: JSON is human-readable and needs no registry; Avro needs Schema Registry running but is more robust at scale.

Yes: Schema Registry is only required by converters that externalize schemas (Avro, Protobuf, JSON Schema). With JSON, String, or ByteArray converters, Connect runs fully without it.

• No registry needed: JsonConverter (with schemas.enable true or false), StringConverter, ByteArrayConverter.

• Registry required: AvroConverter, ProtobufConverter, JsonSchemaConverter all resolve schema IDs against the registry.

• Trade-off without a registry: You lose centralized schema storage, compatibility enforcement, and compact ID-based encoding; JSON with embedded schemas is larger and unvalidated.

A tombstone is a record with a non-null key and a null value; in Connect it signals a deletion, both for log-compacted topics and for sink connectors that need to remove the corresponding row.

• Represents a delete: Source connectors (e.g. Debezium) emit a null value keyed by the primary key when a source row is deleted.

• Drives sink deletes: A sink like JDBC, with delete.enabled=true, deletes the matching record when it sees a tombstone.

• Log compaction: On compacted topics, a tombstone tells Kafka to purge all prior values for that key after a retention period.

• Converter caveat: The value converter must pass a null through as a tombstone rather than erroring on an empty payload.

Standalone mode runs a single worker process with local config and no fault tolerance; distributed mode runs a cluster of workers that share connector state via Kafka topics, giving scalability and automatic failover.

• Standalone:

  • Single JVM process; connector configs come from local properties files.

  • Offsets stored in a local file; if the process dies, work stops (no failover).

  • Good for development, testing, or a source that must run on one specific machine.

• Distributed:

  • Multiple workers form a group; tasks are balanced across them and rebalanced on failure.

  • Config, offsets, and status are stored in internal Kafka topics, not local files.

  • Managed via the REST API rather than files; this is the production standard.

• Key distinction: Both run the same connectors; the difference is how state is stored and whether the deployment is scalable and fault-tolerant.

Change Data Capture (CDC) reads the database's transaction log (the write-ahead log the DB uses for its own recovery and replication), so it captures committed row changes in commit order without polling tables.

• Per-database log names differ:

  • MySQL: the binlog (binary log), ideally in ROW format.

  • PostgreSQL: the WAL (write-ahead log), consumed via logical decoding / replication slots.

  • Oracle: redo logs (via LogMiner or similar).

  • SQL Server: the transaction log, exposed through its CDC tables.

  • MongoDB: the oplog / change streams.

• Why the log and not the tables: It records every committed change in order, so CDC sees inserts, updates, and deletes with no missed intermediate states and minimal load.

• Tools: Debezium connectors on Kafka Connect are the common way to stream these logs into Kafka topics.

An update is emitted as a single change event carrying both the before image (row state prior to the change) and the after image (new state), along with metadata identifying the operation as an update.

• Envelope structure (e.g. Debezium):

  • An op field marks the type: c (create), u (update), d (delete), r (snapshot read).

  • For an update, before holds the old values and after holds the new ones.

• Keying and ordering: The event key is the row's primary key, so all changes to the same row land on the same partition and stay ordered.

• before availability: Full before images may require extra DB config (e.g. MySQL binlog_row_image=FULL or Postgres REPLICA IDENTITY FULL); otherwise only the key/changed columns appear.

A delete becomes a change event with the before image populated and the after set to null, plus a delete op marker; for compacted topics it is typically followed by a tombstone so the key can eventually be removed.

• The delete event: Has op = d, before containing the last known row state, and after = null.

• The tombstone:

  • A follow-up record with the same key and a null value.

  • On a log-compacted topic, compaction uses the tombstone to delete all prior records for that key, keeping downstream state (and the topic) clean.

• Consumer handling: Sinks must interpret a null value as "remove this key" rather than as a corrupt record; Debezium's tombstone emission is configurable.

Kafka achieves durability by persisting every message to an append-only, replicated commit log on disk, and fault tolerance by replicating each partition across multiple brokers so the cluster survives broker failures without data loss.

• Persistence to disk: Messages are written to a segmented log and retained by time/size, not deleted on consume, so they can be re-read.

• Replication: Each partition has a replication factor of N; leaders serve traffic and followers copy the log, so up to N-1 broker losses can be tolerated.

• Producer acknowledgements:

  • acks=all waits until all in-sync replicas have the record, the strongest durability setting.

  • min.insync.replicas forces a minimum number of replicas present or the write is rejected.

• Automatic recovery: If a leader fails, the controller elects a new leader from the ISR, and clients transparently reconnect.

• Idempotence/transactions: Idempotent producers prevent duplicates and transactions enable exactly-once semantics.

For every partition, one replica is elected leader and the rest are followers: the leader handles all reads and writes for that partition, while followers passively replicate the leader's log to stay ready to take over.

• Leader:

  • The single source of truth for the partition; all produce and consume requests go through it.

  • Tracks which followers are caught up (the ISR) and advances the high watermark.

• Followers:

  • Continuously fetch records from the leader to mirror its log; they normally do not serve clients (though follower fetch for locality exists).

  • A follower that keeps up remains in the ISR and is eligible to become leader.

• Failover: If the leader dies, the controller promotes an in-sync follower to leader, preserving availability.

• Balance: Leadership is spread across brokers so no single broker carries all the load.

Kafka replicates at the partition level: each partition has one leader and a configured number of follower replicas that pull records from the leader; once all in-sync replicas have a record it is committed, and the leader tracks progress via the high watermark.

• Pull-based replication: Followers issue fetch requests to the leader (like consumers do), copying records in offset order into an identical log.

• Commit and acknowledgement: The leader advances the high watermark once all ISR members have the record; with acks=all the producer is only acked at that point.

• ISR management: Lagging or dead followers are dropped from the ISR; rejoined followers must catch up before being re-added.

• Leader election: On leader failure the controller (in KRaft or via ZooKeeper historically) picks a new leader from the ISR, preserving all committed records.

• Replication factor: Set per topic; RF=3 is a common choice, tolerating two broker failures while retaining committed data.

You increase replication factor by defining a new replica assignment in a JSON file and applying it with kafka-reassign-partitions.sh: Kafka copies the partition data to the new brokers and adds them as followers.

• Write a reassignment JSON: For each partition, list the broker IDs that should hold replicas (more IDs than before = higher factor).

• Execute the reassignment: Run kafka-reassign-partitions.sh --execute with the JSON; new followers replicate from the leader until they join the ISR.

• Verify: Use --verify to confirm completion and check with kafka-topics.sh --describe.

• Note: You cannot change the factor via --alter directly; reassignment is the supported path, and it adds network/disk load during the copy.

Under-replicated partitions have fewer in-sync replicas than the configured replication factor; offline partitions have no available leader at all. Both are key health indicators: the first signals stress, the second signals unavailability.

• Under-replicated partitions:

  • ISR count is below replication factor, so some followers have fallen behind or their broker is down.

  • Causes: broker overload, network issues, slow disks, or a broker restart.

  • Still serving reads/writes but with reduced fault tolerance.

• Offline partitions:

  • No leader is available, so the partition cannot serve produce or consume requests.

  • Usually means all replicas are down or unclean election is disabled with no ISR left.

• Monitoring: Watch the JMX metrics UnderReplicatedPartitions and OfflinePartitionsCount: both should be 0 in a healthy cluster.

Kafka's own guarantees are at-least-once by default, so duplicates are expected; you eliminate their effect by making processing idempotent or by using Kafka's transactional exactly-once features.

• Idempotent producer: Set enable.idempotence=true: the broker deduplicates producer retries using a producer ID and sequence numbers.

• Idempotent consumer logic: Deduplicate by a business/message key: upsert into a DB or check a processed-ID store so reprocessing the same record is harmless.

• Transactional exactly-once: Use Kafka transactions to atomically commit output records and consumer offsets for Kafka-to-Kafka pipelines.

• Manual offset control: Commit offsets only after successful processing so you know exactly what has been handled.

Exactly-once semantics (EOS) means each record is processed and reflected in the output exactly one time, with no duplicates and no loss, even across retries and failures. In Kafka it's achieved by combining the idempotent producer with transactions.

• Two building blocks:

  • Idempotent producer: prevents duplicate writes from producer retries within a partition.

  • Transactions: atomically write records to multiple partitions and commit consumer offsets together.

• Scope: EOS holds inside Kafka (read-process-write). Extending it to external systems requires idempotent sinks or two-phase coordination.

• Read side: Consumers set isolation.level=read_committed so they only see committed transactional messages.

• Kafka Streams: Enables it with one setting: processing.guarantee=exactly_once_v2.

At-least-once guarantees no message is lost but allows duplicates on retries; exactly-once guarantees each message is reflected once with no loss and no duplicates. Exactly-once costs more overhead and is Kafka's default only when explicitly enabled.

• At-least-once (common default):

  • Producer retries a send it isn't sure landed, and consumers commit offsets after processing, so a crash can cause reprocessing.

  • Cheap and simple; requires idempotent handling downstream.

• Exactly-once:

  • Idempotent producer + transactions ensure no duplicates even under retries and failovers.

  • Higher latency/complexity and only within Kafka's boundary.

• Also note at-most-once: Commit offsets before processing: no duplicates but possible loss. Rarely wanted.

The acks setting controls how many broker replicas must acknowledge a write before the producer considers it successful, trading durability against latency and throughput.

• acks=0 (fire and forget): Producer doesn't wait for any acknowledgment: lowest latency, highest throughput, but messages can be lost silently.

• acks=1 (leader only): Leader acknowledges after writing to its log. Fast, but if the leader fails before followers replicate, data can be lost.

• acks=all / -1 (full ISR):

  • Leader waits for all in-sync replicas to acknowledge: strongest durability, higher latency.

  • Pair with min.insync.replicas > 1 so a write fails rather than committing to too few replicas.

• Interplay with EOS: Idempotent and transactional producers require acks=all for their guarantees to hold.

An idempotent producer guarantees that retries of the same message don't create duplicates on the broker, giving exactly-once semantics per partition for produced records. It's enabled with enable.idempotence=true.

• The duplicate problem it solves: A producer sends a batch, the broker writes it, but the ack is lost; the producer retries and the same data is written twice.

• How it deduplicates:

  • Each producer gets a unique producer ID (PID), and each message carries a per-partition monotonic sequence number.

  • The broker tracks the last sequence number per (PID, partition) and rejects any record whose sequence it has already seen, treating it as a duplicate.

• What it requires: It sets safe defaults: acks=all, retries > 0, and max.in.flight.requests.per.connection <= 5.

• Scope: Guarantees no duplicates within a single producer session and partition; cross-session exactly-once needs transactions with a transactional.id.

Producers accumulate records into per-partition batches in memory, optionally compress each batch, then send it; this amortizes network round-trips and improves throughput. Batching and compression are controlled by a small set of related configs.

• Batching flow:

  • Records are grouped by topic-partition into batches; a batch is sent when it fills or its linger time expires.

  • Compression is applied to the whole batch, so bigger batches compress better and cut network cost.

• Key configs:

  • batch.size: max bytes per partition batch (a size trigger, not a hard limit per record).

  • linger.ms: how long to wait for more records before sending, even if the batch isn't full.

  • compression.type: codec (none, gzip, snappy, lz4, zstd).

  • buffer.memory: total memory available for unsent batches.

• Tuning intuition: Raise batch.size and linger.ms for higher throughput and better compression at the cost of latency.

Together, batch.size and linger.ms govern the throughput vs. latency tradeoff: they decide how big batches get and how long the producer waits to fill them before sending.

• batch.size:

  • Upper bound (in bytes) on a per-partition batch. When a batch hits this size, it's sent immediately.

  • Larger values mean fewer, bigger requests: better throughput and compression, more memory used.

• linger.ms:

  • Artificial delay before sending, letting more records accumulate. Default 0 sends as soon as possible.

  • A small value (e.g. 5 to 20 ms) can dramatically increase batching under load with minimal latency cost.

• How they interact:

  • A batch is sent when whichever happens first: it reaches batch.size or linger.ms elapses.

  • Under high load batches fill fast so linger rarely matters; under light load linger is what enables batching at all.

delivery.timeout.ms is the total time budget for a record from the moment send() returns until success or permanent failure. It caps the entire lifecycle, including batching, transmission, and all retries.

• What it bounds:

  • Time waiting in the send buffer plus request.timeout.ms per attempt plus retry.backoff.ms between attempts.

  • Once exceeded, the record's callback fails with a TimeoutException, regardless of retries remaining.

• Relationship to retries:

  • It is the real limit on retrying, not the retries count: retries stop when this deadline is hit.

  • Modern practice is to leave retries high (or default) and control failure behavior through this timeout.

• Constraint: Must be >= linger.ms + request.timeout.ms; default is 120000 (2 minutes).

buffer.memory is the total amount of memory the producer can use to buffer records waiting to be sent to brokers. When it fills, sends block or fail rather than growing memory unbounded.

• What it holds:

  • All unsent record batches across partitions, plus buffers awaiting broker acks.

  • Fills up when the application produces faster than the network/brokers can drain (backpressure).

• What happens when it's full:

  • send() blocks up to max.block.ms waiting for space.

  • If space isn't freed in time, it throws a TimeoutException (or BufferExhaustedException).

• Tuning:

  • Increase it to absorb bursts and slow brokers; it effectively bounds how much in-flight/pending data the producer holds.

  • A frequently-full buffer signals the producer is a bottleneck: scale producers, tune batching, or add brokers.

An offset marks the next record a consumer will read in a partition; consumers commit offsets to a special topic (__consumer_offsets) so progress survives restarts and rebalances. Auto-commit periodically commits in the background, while manual commit lets you control exactly when (and whether sync or async).

• Auto-commit:

  • Enabled with enable.auto.commit=true; commits the last polled offsets every auto.commit.interval.ms during poll().

  • Simple, but risks data loss (offset committed before processing finishes) or duplicates on crash.

• Manual sync commit: commitSync() blocks until the broker acks and retries on retriable errors; strong guarantees but adds latency.

• Manual async commit: commitAsync() is non-blocking with a callback; higher throughput but no automatic retry (a later commit supersedes it).

• Common pattern:

  • Use commitAsync() in the steady loop for speed, and a final commitSync() on shutdown/rebalance for correctness.

  • Committing after processing gives at-least-once; committing before gives at-most-once.

auto.offset.reset decides where a consumer starts when it has no valid committed offset (new group, or the committed offset has aged out). The three values trade off completeness versus recency versus strictness.

• earliest: Start from the beginning of the partition: use when you must process all historical data (ETL, reprocessing, building state).

• latest:

  • Start from new records only: use for real-time consumers that don't care about the backlog (live dashboards, alerting).

  • This is the default and can silently skip messages if a group starts fresh mid-stream.

• none: Throw an exception if no committed offset exists: use when starting position must be explicit and you'd rather fail loudly than guess.

• Key point: this setting only applies when there is no valid offset; once a group has committed offsets, it resumes from them regardless.

max.poll.records caps how many records a single poll() call returns: it controls the batch size your application processes per loop, and indirectly affects rebalancing because you must finish that batch before the next poll() within max.poll.interval.ms.

• Controls batch size, not fetch size:

  • It limits records handed to the app per poll(); the network fetch is governed separately by fetch.max.bytes / max.partition.fetch.bytes.

  • The consumer buffers fetched records and drains them across polls up to this cap.

• Impact on rebalancing:

  • If per-record processing is slow, a large max.poll.records can make one batch exceed max.poll.interval.ms, so the consumer misses its next poll and is kicked from the group, triggering a rebalance.

  • Lowering it shortens the time between polls, keeping the consumer alive.

• Tuning trade-off:

  • Higher values improve throughput (fewer loop iterations) but risk poll timeouts.

  • Lower values reduce rebalance risk and latency per batch but add overhead.

• Note: heartbeats are sent on a background thread, so liveness is really about calling poll() often enough, which this setting influences.

The kafka-consumer-groups.sh tool inspects consumer group membership and lag, and can reset committed offsets for a group that is not actively running.

• Inspect:

  • --list shows all groups; --describe shows per-partition current offset, log-end offset, and LAG.

  • --describe --members or --state show assignment and group state (Stable, Rebalancing, Empty).

• Reset offsets (requires --reset-offsets):

  • Targets: --to-earliest, --to-latest, --to-offset, --to-datetime, --shift-by, --by-duration.

  • Use --dry-run (default) to preview, then --execute to apply.

  • Scope with --topic or --all-topics.

• Important constraint: Offsets can only be reset when the group has no active members: stop consumers first, otherwise the command fails.

• Also supports --delete to remove a group and --delete-offsets for specific topic offsets.

Consumer lag is the difference between the partition's log-end offset (latest produced) and the consumer group's committed offset: it measures how far behind consumers are, in messages. Sustained or growing lag means consumers can't keep up with producers.

• Definition:

  • lag = log-end offset - committed offset, computed per partition and summed per group.

  • Constant small lag is normal; steadily rising lag signals a problem.

• How to monitor:

  • kafka-consumer-groups --describe shows the LAG column ad hoc.

  • For continuous monitoring use Burrow, Kafka Lag Exporter, or the records-lag-max client JMX metric into Prometheus/Grafana with alerts.

• How to remediate:

  • Scale consumers: add instances up to the partition count to increase parallelism.

  • Speed up processing: optimize handler logic, batch writes, or offload slow work.

  • Tune max.poll.records and fetch settings for better throughput.

  • If topic is under-partitioned, increase partitions to raise the parallelism ceiling.

Monitor a small set of high-signal metrics across three layers: broker health and throughput, producer send behavior, and consumer lag/rebalancing. Most come from JMX.

• Broker:

  • UnderReplicatedPartitions and OfflinePartitionsCount: availability/replication health (should be 0).

  • ActiveControllerCount: exactly 1 across the cluster.

  • RequestHandlerAvgIdlePercent and request/response latency: saturation and slowness.

  • BytesInPerSec / BytesOutPerSec and ISR shrink/expand rate.

• Producer:

  • record-send-rate, request-latency-avg, record-error-rate, and record-retry-rate.

  • batch-size-avg and buffer availability to spot backpressure.

• Consumer:

  • records-lag-max: the headline health metric.

  • fetch-latency, records-consumed-rate, and rebalance rate/frequency.

• System (underlying): Disk usage/I/O, page cache, network, and JVM GC pause times.

High lag means production outpaces consumption. Investigate systematically: confirm it's real and growing, determine whether the bottleneck is the consumer, the topic layout, or the broker, then apply the matching fix.

• Confirm and localize:

  • Run kafka-consumer-groups --describe to see per-partition lag: is it all partitions or a few hot ones?

  • Skewed lag on specific partitions points to a partitioning/key-hotspot problem.

• Check consumer health:

  • Are all consumer instances alive and assigned partitions? Idle consumers waste parallelism.

  • Frequent rebalances (from max.poll.interval.ms timeouts) stall progress: look at slow processing per record.

  • Check for a poison message or downstream dependency (DB, API) slowing the handler.

• Check parallelism limits: Consumers cannot exceed partition count: if instances = partitions, adding more won't help.

• Check the broker/producer side: A sudden producer traffic spike or broker latency/URP issues can drive lag.

• Remediate:

  • Scale out consumers, optimize/parallelize processing, tune max.poll.records and fetch sizes.

  • Increase partitions if the topic is the parallelism ceiling.

The AdminClient is Kafka's programmatic administration API for managing cluster metadata and resources without shelling out to CLI scripts, so tooling and automation can create topics, alter configs, and inspect the cluster.

• Topic management: createTopics(), deleteTopics(), listTopics(), describeTopics(), and adding partitions with createPartitions().

• Configuration: describeConfigs() and alterConfigs() / incrementalAlterConfigs() for topic and broker settings at runtime.

• Consumer groups & offsets: listConsumerGroups(), describeConsumerGroups(), listConsumerGroupOffsets(), and alterConsumerGroupOffsets() for lag inspection and offset resets.

• Cluster & security: describeCluster() for broker/controller info, plus ACL operations (createAcls(), describeAcls()) and reassignment/log-dir queries.

• Nature of the API: Asynchronous: methods return result objects holding KafkaFutures you await for completion or errors.

The Schema Registry is a separate service that stores and versions the schemas (Avro, Protobuf, JSON Schema) used by Kafka messages, letting producers and consumers agree on data structure and enforce compatibility as it evolves.

• Central schema store: Schemas are registered under subjects and assigned globally unique ids; it typically stores its data in a Kafka topic (_schemas).

• Decouples payload from schema: Messages carry a small schema id instead of the full schema, shrinking payloads while still being self-describing.

• Enforces compatibility: On registration it rejects incompatible schema changes per the configured mode, preventing producers from breaking consumers.

• Why it matters:

  • Kafka itself treats values as opaque bytes; without a registry, schema drift silently corrupts downstream systems.

  • Enables safe independent deployment of producers and consumers as data evolves.

A subject is a named scope under which schemas are registered and evolved, and each registration creates a new version within that subject; subjects map to topics through a configurable naming strategy but are not the topic itself.

• Subject: The unit that compatibility rules are applied to; each subject has its own version history and compatibility mode.

• Version: An incrementing number per subject; registering a new (compatible) schema adds a version, while the schema id is global across all subjects.

• Relation to topics via naming strategy:

  • TopicNameStrategy (default): subjects are <topic>-key and <topic>-value, so key and value evolve independently.

  • RecordNameStrategy: subject is the record's fully qualified name, allowing multiple event types per topic.

  • TopicRecordNameStrategy: combines both, scoping a record type to a topic.

• Key point: A single schema id can be referenced by many subjects/versions; the subject controls governance, the id identifies the concrete schema bytes.

Log compaction is a retention mode that keeps at least the latest value for each message key, rather than deleting by age or size. It turns a topic into a changelog where the log holds the most recent state per key.

• How it works:

  • A background log cleaner rewrites segments, discarding older records that share a key with a newer record.

  • Compaction runs on the inactive (already-rolled) part of the log; the active segment is untouched.

  • Keys are required: records with null keys can't be compacted.

• Guarantees:

  • Offsets never change; consumers reading from the beginning always see the final value per key.

  • A tombstone (null value) marks a key for deletion.

• Typical use cases:

  • Changelog / state stores (Kafka Streams uses compacted topics to restore state).

  • Database CDC and cache warming where only the latest row state matters.

  • Internal topics like __consumer_offsets.

• Config: set cleanup.policy=compact (can combine as compact,delete).

Kafka retains messages for a configurable window based on time or size and deletes whole log segments once they exceed those limits: retention is per topic, not per consumer, and consumption never deletes data.

• Retention triggers (delete policy):

  • retention.ms: delete segments older than this age.

  • retention.bytes: cap total partition size, deleting oldest segments first.

  • Whichever limit is hit first triggers deletion.

• Deletion is segment-granular:

  • Kafka never deletes individual messages under delete policy; it drops entire closed segments.

  • The active segment is never deleted, so real retention can slightly exceed the configured limit.

• Independent of consumers: Data survives whether or not it's been read; a slow consumer can lose data if retention expires first.

• Alternative: cleanup.policy=compact retains latest-per-key instead of deleting by age.

They are the two cleanup policies that decide how Kafka reclaims log space. delete removes old data by age or size; compact keeps the latest value per key indefinitely.

• cleanup.policy=delete:

  • Discards whole segments older than retention.ms or beyond retention.bytes.

  • Time/size-based; doesn't care about keys. The default for most topics.

  • Use for event streams where old events lose value.

• cleanup.policy=compact:

  • Retains at least the newest record per key; older duplicates are removed by the log cleaner.

  • Requires keyed records; tombstones (null value) delete a key.

  • Use for state/changelog topics where you want the latest snapshot per key.

• Combine them: compact,delete compacts by key but also drops data past the retention window.

A tombstone is a record with a valid key but a null value written to a compacted topic. It signals that the key should be deleted, and after compaction it removes all prior records for that key from the log.

• How it triggers deletion:

  • The log cleaner sees the null-value record, treats it as the latest state, and purges earlier values for that key.

  • Consumers reading the topic see the tombstone, so they know to remove the key from their own state/cache.

• Why it's kept for a while:

  • The tombstone itself is retained for delete.retention.ms so lagging consumers still observe the delete.

  • After that window, the tombstone is also removed and the key vanishes entirely.

• Note: tombstones only make sense with cleanup.policy=compact; keys are mandatory.

SSL/TLS provides encryption (and optionally certificate-based authentication) of the network connection, while SASL is a pluggable framework for authenticating client and broker identities; they are orthogonal and often used together.

• SSL/TLS:

  • Primarily encrypts traffic so data on the wire is confidential and tamper-resistant.

  • Can also authenticate: mutual TLS (mTLS) uses client certificates as identity.

  • Needs a keystore (own cert/key) and truststore (trusted CAs) on brokers and clients.

• SASL:

  • An authentication framework only: it verifies identity but does not encrypt.

  • Supports mechanisms: PLAIN, SCRAM, GSSAPI, OAUTHBEARER.

• How they combine (listener protocols):

  • SSL: TLS encryption, optionally mTLS auth.

  • SASL_SSL: SASL authentication over TLS-encrypted connections (recommended, since plain SASL sends credentials unencrypted).

  • SASL_PLAINTEXT: auth without encryption, avoid outside trusted networks.

Kafka is a distributed, log-based streaming platform that retains messages and lets consumers pull and re-read them, whereas RabbitMQ is a traditional broker that pushes messages and typically deletes them once acknowledged.

• Storage model:

  • Kafka is an append-only, replayable log: messages persist by retention policy, so many consumers read the same data independently.

  • RabbitMQ is queue-based: a message is generally removed after being delivered and acked.

• Delivery direction:

  • Kafka consumers pull and track their own offset.

  • RabbitMQ pushes to consumers and manages per-message ack/routing.

• Ordering and scale:

  • Kafka scales horizontally via partitions and guarantees order within a partition.

  • RabbitMQ excels at flexible routing (exchanges, bindings) but ordering and throughput are more limited.

• Use cases:

  • Kafka: high-throughput event streaming, replay, stream processing, event sourcing.

  • RabbitMQ: complex routing, task queues, request/reply, low-latency per-message workflows.

Kafka Streams is a client-side Java/Scala library for building stateful stream-processing applications on top of Kafka: it consumes, transforms, aggregates, and produces back to topics, going beyond simple message-in/message-out consumption.

• It's a library, not a cluster:

  • Runs inside your app; no separate processing cluster to deploy (unlike Spark/Flink).

  • Scales by running more instances of the same app in a consumer group.

• Stateful vs. simple consumption:

  • Traditional consumers process one message at a time with app-managed state.

  • Kafka Streams manages local state stores (RocksDB) backed by changelog topics for fault-tolerant aggregations, joins, and windows.

• Rich abstractions: Models data as a KStream (event stream) or KTable (changelog/table), enabling the stream-table duality.

• Guarantees: Supports exactly-once processing and event-time semantics with windowing.

Windowing groups records into time-bounded buckets so aggregations (counts, sums, joins) apply over finite intervals rather than an unbounded stream; Kafka Streams offers four window types differing in how boundaries and overlap work.

• Tumbling windows: Fixed-size, non-overlapping, gapless (e.g. every 1 minute); each record falls in exactly one window.

• Hopping windows: Fixed-size but advance by a smaller hop, so they overlap; a record can belong to multiple windows (e.g. size 5 min, hop 1 min).

• Sliding windows: Windows defined by the time difference between record timestamps; boundaries move with the data and are used for join/aggregation over a time distance.

• Session windows: Dynamic, data-driven: group records separated by less than an inactivity gap; a new gap closes the session (great for user-activity sessions).

• Related concept: grace period: Late-arriving records within the grace period still update their window; after that they're dropped.

Kafka Streams is a client-side Java/Scala library for building stream processing applications that read from and write to Kafka topics: no separate cluster is needed, you just run your app as normal JVM processes. Its core abstractions model data either as an unbounded event stream or as a changing table.

• It's a library, not a framework/cluster: You embed it in your app and scale by running more instances; Kafka handles partitioning and coordination.

• KStream: A record stream: each record is an independent, append-only event (an insert). Two records with the same key are both retained.

• KTable: A changelog stream interpreted as an upsert table: the latest value per key wins, and a null value is a delete (tombstone). Partitioned across instances.

• GlobalKTable: Like a KTable but each instance holds a full copy of all partitions, enabling joins on any key (not just co-partitioned data) at the cost of more storage.

• Two DSL levels: A high-level DSL (map, filter, join, aggregate) and a low-level Processor API for custom logic.

The Consumer API is a low-level primitive for polling records; Kafka Streams is a higher-level processing library built on top of the producer and consumer that adds stateful processing, transformations, joins, windowing, and exactly-once semantics so you don't build them yourself.

• Level of abstraction:

  • Consumer API: you call poll(), track offsets, and write all processing/output logic manually.

  • Kafka Streams: you declare a topology (map, join, aggregate) and the library runs it.

• State management: Streams gives built-in local state stores plus changelog-backed fault tolerance; with the plain consumer you'd build this yourself.

• Semantics and time: Streams offers exactly-once processing, event-time windowing, and out-of-order handling out of the box.

• What they share: Streams uses consumer groups underneath, so scaling and partition assignment work the same way; it is not a separate cluster.

A topology is the directed acyclic graph (DAG) of processing nodes that defines your application's computational logic: it describes how data flows from source topics, through processors that transform it, to sink topics. You build it once at startup and Kafka Streams executes it against the data.

• Three kinds of nodes:

  1. Source processors: read records from input topics.

  2. Stream processors: apply logic like filter, map, join, or aggregations.

  3. Sink processors: write results back to output topics.

• How you define it: With the DSL via StreamsBuilder, or explicitly with the low-level Topology (Processor API).

• Sub-topologies and tasks: Kafka Streams splits the topology into sub-topologies and instantiates tasks (one per input partition) that run in parallel across threads and instances.

A KTable represents a changelog: it models the latest state per key (an upsert view), whereas a KStream models an append-only sequence of independent events. Same-key records in a KStream are separate facts; in a KTable a new value replaces the old one and a null value deletes the key.

• Interpretation of a record:

  • KStream: "something happened" (an event/insert).

  • KTable: "this is the current value for this key" (an update/upsert).

• Handling duplicate keys: KStream keeps all of them; KTable keeps only the most recent.

• Backing state: A KTable is materialized in a state store with a compacted changelog, so it's stateful; a plain KStream is stateless.

• Example that clarifies the difference: For key user1 with values 100 then 200: a KStream sums to 300, a KTable holds 200.

Transformations are the operators that build your topology, split into stateless ones (each record processed independently) and stateful ones (needing accumulated state in a store). The distinction matters because stateful operations require state stores, changelog topics, and co-partitioned data, which affects correctness and performance.

• Stateless transformations:

  • Examples: map, filter, flatMap, branch.

  • No memory of past records; easy to scale and recover since there's nothing to restore.

• Stateful transformations:

  • Examples: aggregate, count, reduce, join, and windowed operations.

  • Maintain state in a local store backed by a changelog topic for fault tolerance.

  • Often require repartitioning so records with the same key land on the same task (co-partitioning).

• Practical implication: Stateful ops add disk, changelog traffic, and recovery time; prefer stateless where possible.

Kafka Streams scales horizontally by relying on Kafka's consumer group protocol: you run more instances (or threads) sharing the same application ID, and tasks are rebalanced across them, with partition count setting the upper limit.

• Same application ID = one consumer group: All instances with the same application.id cooperate; Kafka assigns partitions/tasks among them.

• Add instances to scale out:

  • Start a new instance and a rebalance redistributes tasks, moving some load off existing instances.

  • No code change or coordination service needed: it's just another group member.

• Bounded by partition count:

  • Max useful instances/threads equals the number of input partitions; extras stay idle.

  • Partition topics generously up front to leave headroom for scaling.

• State moves with tasks: When a task migrates, its local state is rebuilt from the changelog topic on the new instance (standby replicas speed this up).

The application ID (application.id) uniquely identifies a Kafka Streams application and is the single most important config: it defines the consumer group, names internal topics, and scopes local state.

• Defines the consumer group: All instances sharing the ID form one group and split the work; it's how Kafka Streams achieves scaling and load balancing.

• Names internal topics: Changelog and repartition topics are prefixed with the application ID, so two apps must never share it or they'll corrupt each other's state.

• Scopes local state directories: State stores on disk are organized under the application ID.

• Practical implications:

  • Changing the ID makes the app a brand-new group: it reprocesses from the configured offset reset and rebuilds all state.

  • Must be unique per logical application across the cluster.

ksqlDB is a streaming SQL engine that lets you build stream processing applications with SQL statements instead of Java/Scala code. Under the hood it compiles those queries into Kafka Streams topologies, so it is essentially a higher-level abstraction over Kafka Streams.

• SQL over streams: You declare STREAM and TABLE objects and run continuous queries (filters, joins, aggregations, windows) with familiar SQL.

• Built on Kafka Streams:

  • Each persistent query becomes a Kafka Streams app, inheriting its scaling, fault tolerance, and state store model.

  • Runs as its own server cluster, separate from brokers.

• Kafka Streams vs ksqlDB trade-off:

  • Kafka Streams: a library embedded in your JVM app, giving full programmatic control and custom logic.

  • ksqlDB: less flexible but far faster to build with, and accessible to non-Java teams.

• Extras: pull queries against materialized state, plus connector management for ingest/egress.

The most impactful parameters cluster around producer batching, durability, consumer fetching, and broker replication. Knowing what each trades off is the key interview point.

• Producer:

  • batch.size and linger.ms: bigger batches / small wait improve throughput at slight latency cost.

  • acks: 0/1/all trades durability for latency/throughput.

  • compression.type, buffer.memory, max.in.flight.requests.per.connection.

• Consumer:

  • fetch.min.bytes and fetch.max.wait.ms: batch fetches for efficiency.

  • max.poll.records and max.partition.fetch.bytes: control processing chunk size.

• Broker / topic:

  • num.io.threads, num.network.threads: broker concurrency.

  • min.insync.replicas with acks=all: durability guarantee.

  • num.partitions and replication.factor: parallelism vs overhead.

Kafka deliberately avoids maintaining its own in-process cache and random disk access. Instead it appends to log files sequentially and relies on the OS page cache to buffer reads and writes, which turns disk into a fast, RAM-backed streaming medium.

• Sequential disk I/O:

  • Each partition is an append-only log, so writes are sequential, avoiding costly disk seeks.

  • Sequential throughput on modern disks is orders of magnitude higher than random access.

• OS page cache as the buffer:

  • Writes land in page cache and are flushed to disk asynchronously by the OS.

  • Consumers reading recent data (the common case) are served from cache in RAM, never hitting disk.

• Why delegate to the OS:

  • Avoids duplicating data in a JVM heap cache and dodges GC pressure from large heaps.

  • Cache survives broker process restarts since it lives in the kernel.

• This pairs naturally with zero-copy: cached log segments go straight from page cache to the socket via sendfile().

All four reduce network and disk usage but differ in compression ratio versus CPU cost. Roughly: gzip compresses hardest but slowest, snappy and lz4 favor speed, and zstd gives the best overall balance.

• gzip:

  • Highest compression ratio, so smallest payloads and lowest bandwidth/storage.

  • Highest CPU cost and latency; best when network/storage is the bottleneck, not CPU.

• snappy:

  • Fast compress/decompress with modest ratio.

  • Good general-purpose choice when CPU is limited.

• lz4:

  • Very fast, similar ratio to snappy, often even lower latency.

  • Popular default for high-throughput, latency-sensitive pipelines.

• zstd:

  • Compression near gzip levels at much better speed, and tunable levels.

  • Increasingly the recommended default (needs newer Kafka/clients).

• Rule of thumb:

  • CPU-bound: lz4/snappy. Bandwidth/storage-bound: zstd (or gzip for max shrink).

  • Note batches are compressed together, so larger batches improve ratio for all codecs.

You reprocess historical data by repositioning the consumer to an earlier offset, either programmatically with the seek() family of methods or administratively by resetting a consumer group's committed offsets.

• Programmatic seek in the consumer:

  • seek(partition, offset) jumps to an exact offset; seekToBeginning() and seekToEnd() jump to the log ends.

  • offsetsForTimes() finds the offset for a timestamp, enabling time-based replay.

• auto.offset.reset: Only applies when there is no committed offset for the group; earliest replays from the log start, latest reads only new data. It is not a manual replay switch.

• Reset with the CLI tool: kafka-consumer-groups.sh --reset-offsets (with the group stopped) supports --to-earliest, --to-datetime, --shift-by, or --to-offset.

• Caveats:

  • You can only replay data still retained (bounded by retention/compaction).

  • Consider replaying with a fresh group id or into a separate output to avoid disturbing live consumers, and ensure downstream processing is idempotent.

In event sourcing, state is derived by replaying an ordered log of immutable events, and Kafka's append-only, durable, replayable partitions make it a natural backbone for that log, though it is used as a transport/store rather than a full event-store database.

• Events as the source of truth:

  • Each state change is published as an immutable event; current state is a projection built by consuming the log.

  • Kafka's immutability and offset-based replay let you rebuild state or new read models from scratch.

• Ordering per aggregate: Key events by aggregate/entity id so all events for one entity land in the same partition and stay ordered.

• Compaction for latest state: Use log compaction (`cleanup.policy=compact`) on keyed topics to retain the latest event per key indefinitely as a durable snapshot.

• Projections and CQRS: Consumers (often Kafka Streams) materialize read models/state stores from the event log, separating write and read sides.

• Caveats:

  • Kafka has no rich query by id or per-event transactional guarantees of a dedicated event store; you often pair it with a database or state store.

  • Global ordering across partitions isn't guaranteed, so design aggregates to fit within a partition.

Choose partition count from your target throughput and required consumer parallelism, then round up for headroom: estimate partitions as max(target-throughput / per-partition-throughput, desired-consumer-count). Over-partitioning wastes resources and hurts latency, while under-partitioning caps your parallelism.

• Sizing approach:

  • Measure per-partition producer and consumer throughput (e.g. tp), then partitions ≥ max(target/tp_producer, target/tp_consumer).

  • Ensure at least as many partitions as the peak number of consumers you want in a group.

• Costs of too many partitions: More open files, more replication and controller metadata, longer leader-election and rebalance times, higher end-to-end latency.

• Increasing partitions is easy but disruptive: You can add partitions online, but key-to-partition mapping changes, breaking per-key ordering for existing keys.

• Decreasing is effectively impossible: Kafka does not support reducing partitions; you must create a new topic and migrate, so start conservative but with room to grow.

• Rule of thumb: over-provision modestly for future growth, but avoid thousands of tiny partitions per broker.

Group management centers on how partitions get assigned during rebalances and how failures are detected: you tune the assignment strategy, the rebalance protocol, and the liveness/timeout settings.

• Partition assignment strategies (partition.assignment.strategy):

  • RangeAssignor: assigns contiguous ranges per topic (can imbalance across many topics).

  • RoundRobinAssignor: spreads all partitions evenly across members.

  • StickyAssignor: balances while minimizing partition movement between rebalances.

  • CooperativeStickyAssignor: incremental rebalancing that avoids a full stop-the-world reassignment.

• Rebalance protocol: Eager rebalancing revokes all partitions then reassigns; cooperative rebalancing only moves what's needed, reducing pauses.

• Liveness and timeouts:

  • session.timeout.ms and heartbeat.interval.ms govern failure detection.

  • max.poll.interval.ms evicts a consumer that stops calling poll() (stuck processing).

• Static membership: group.instance.id lets a restarting consumer rejoin without triggering a rebalance.

The consumer's isolation.level controls whether it can see records from transactions that haven't committed yet, which matters when producers use Kafka transactions.

• read_uncommitted (default): Returns all records up to the high watermark, including messages from open or aborted transactions.

• read_committed:

  • Returns only records from committed transactions plus non-transactional records; aborted records are filtered out.

  • Such a consumer reads only up to the Last Stable Offset (LSO), so uncommitted data past an in-flight transaction stays invisible.

• Use read_committed for exactly-once / atomic read-process-write pipelines where you must not act on data that later aborts.

The Last Stable Offset (LSO) is the offset up to which all transactional messages are known to be resolved (committed or aborted): a read_committed consumer can only read up to the LSO, so it never sees data from a still-open transaction.

• Relation to transactions: When a transaction is in flight, the LSO sits just before its first record and cannot advance until that transaction commits or aborts.

• Relation to the high watermark: The LSO is always ≤ the high watermark; an open transaction holds the LSO back even though the high watermark may be higher.

• Consumer visibility: read_committed reads up to the LSO and skips aborted records; read_uncommitted reads up to the high watermark and ignores the LSO.

Each log segment has two companion index files that let Kafka locate messages quickly without scanning the whole .log file: .index maps offsets to physical byte positions, and .timeindex maps timestamps to offsets.

• A segment is a trio of files:

  • .log holds the actual records; .index and .timeindex are sparse indexes over it.

  • Each file is named after the base (first) offset of the segment.

• .index (offset index): maps a relative offset to the byte position in the .log file.: Used to answer "where does offset N live?" via a binary search, then a short sequential scan.

• .timeindex: maps a timestamp to the earliest offset with that time or later.: Powers offsetsForTimes() and time-based lookups, plus time-based retention decisions.

• Sparse by design: An entry is added only every index.interval.bytes (default 4KB), keeping indexes small and often memory-mapped.

The Kafka Controller is a special broker responsible for cluster-wide administrative decisions: managing partition leadership, tracking broker liveness, and propagating metadata to all brokers. Exactly one broker acts as controller at a time.

• Leader election: Chooses partition leaders and reassigns them when brokers fail or join.

• Membership and ISR management: Detects broker failures and updates the in-sync replica sets accordingly.

• Metadata propagation: Pushes updated partition state and leadership info to other brokers so they route correctly.

• How the controller itself is chosen:

  • With ZooKeeper: the first broker to create the controller znode wins.

  • With KRaft: a quorum of controller nodes uses a Raft-based election, and the metadata lives in a replicated log rather than ZooKeeper.

Preferred leader election is Kafka restoring each partition's leadership to its preferred replica (the first broker in the partition's replica assignment list). Kafka does automatic rebalancing to keep leadership evenly spread across brokers, avoiding a situation where one broker ends up hosting most leaders after failures and recoveries.

• Preferred replica: For each partition, the first entry in the replica list is the preferred leader; under normal conditions it should be the active leader.

• Why leadership drifts: When a broker dies, its leaderships move to surviving replicas. When it rejoins it becomes a follower, so leaders pile up on the brokers that stayed alive.

• Automatic rebalancing:

  • Controlled by auto.leader.rebalance.enable (default true); the controller periodically checks imbalance against leader.imbalance.per.broker.percentage and triggers election.

  • Balances client load: producers/consumers talk to the leader, so even leader distribution means even network and I/O load.

• Manual trigger: Can be run with kafka-leader-election.sh using the PREFERRED election type.

• Caveat: Election briefly interrupts produce/consume for affected partitions, so rebalancing has a small cost worth timing sensibly.

Yes, but with conditions: sink connectors can achieve effectively exactly-once, and since Kafka 3.3 source connectors support true exactly-once via transactions, provided both the connector and the external system cooperate.

• Sink side: Achieved by idempotent writes or by writing data and committing offsets atomically in the target system; otherwise defaults to at-least-once.

• Source side:

  • Framework-level EOS added in KIP-618: enable with exactly.once.source.support on the worker so records and source offsets commit in one Kafka transaction.

  • The connector must define transaction boundaries and support offset APIs.

• Requirements: Distributed mode, and the connector plugin must be written to support it: not every connector qualifies.

• Default guarantee: Without these features Connect gives at-least-once, so consumers must tolerate duplicates.

Kafka Connect provides at-least-once delivery by default, and can achieve exactly-once for supported connectors: source connectors via transactional offsets, and sink connectors depending on the target system's ability to deduplicate or write idempotently.

• At-least-once (default): Records are committed after they're written, so on failure/restart Connect may reprocess and produce duplicates, but never silently drops data.

• Exactly-once source: Supported since Kafka 3.3 for source connectors that implement it, by writing records and source offsets in a single transaction (exactly.once.source.support).

• Exactly-once sink: Not guaranteed by the framework alone; depends on the sink system supporting idempotent upserts or transactional writes keyed on Kafka offsets.

• Practical takeaway: assume at-least-once and design downstream consumers/sinks to tolerate duplicates unless you've explicitly enabled exactly-once.

Kafka Connect avoids data loss by tracking progress through committed offsets and only advancing them after data is durably handled, so a crash resumes from the last safe point rather than skipping records.

• Offset tracking:

  • Source connectors persist source offsets (position in the external system) to an internal Kafka topic; sink connectors commit consumer offsets only after records are delivered.

  • On restart, tasks resume from committed offsets, replaying anything not confirmed.

• Durable internal state: Config, offsets, and status live in replicated Kafka topics, so worker failures don't lose progress.

• Producer durability: Source producers rely on Kafka's replication and acknowledgements (acks=all) so committed records survive broker failures.

• Error handling: Bad records can be routed to a DLQ instead of being dropped, preserving them for inspection.

• Trade-off: this safety is why the default is at-least-once, replay to avoid loss can mean duplicates.

You make sink writes idempotent so that reprocessing the same record produces the same result: typically by using deterministic keys and upsert semantics in the target system, since Kafka Connect's at-least-once delivery means the same record may arrive more than once.

• Deterministic document/row keys: Derive the primary key from the record (business key, or Kafka topic/partition/offset) so a replayed record overwrites rather than inserts a duplicate.

• Upsert instead of insert: Configure the connector for upsert mode where available (e.g. JDBC sink insert.mode=upsert with a defined pk.mode; Elasticsearch uses the doc ID for idempotent indexing).

• Leverage the sink's guarantees: Use conditional writes, unique constraints, or transactions in the target to reject or absorb duplicates.

• Track offsets in the sink: Store the last processed Kafka offset alongside the data so the sink can skip already-applied records after a replay.

Single Message Transforms (SMTs) are meant for lightweight, per-record, stateless tweaks: avoid them when transformation logic gets heavy, stateful, or spans multiple records.

• Aggregations or joins across records: SMTs see one record at a time with no state, so windowing, joins, or counts belong in Kafka Streams or ksqlDB.

• Expensive external lookups: Calling a DB or REST service per record inside an SMT adds latency to the pipeline and hurts throughput.

• Complex or branching business logic: Long chains of SMTs become hard to test and reason about: a real stream-processing app is clearer.

• Splitting or fanning out records: An SMT transforms one record into one record; it cannot emit multiple outputs from a single input.

• Rule of thumb: use SMTs for renaming, routing, masking, adding/removing fields, and format tweaks; anything stateful or multi-record goes elsewhere.

Schema evolution in Connect flows through the converter and (usually) the Schema Registry: sources emit records carrying a schema, the registry checks the new schema against compatibility rules, and sinks map the evolving schema onto their target.

• Connect records carry a schema: Each record has a Schema object; the converter serializes it and registers/looks up the version.

• Compatibility is enforced by the registry: Modes like BACKWARD, FORWARD, and FULL decide whether adding/removing fields is allowed; incompatible changes are rejected.

• Safe changes: Adding an optional field with a default is the classic backward-compatible evolution.

• Sink side adapts: Connectors like the JDBC sink can auto-evolve target tables (e.g. add a column) when auto.evolve is enabled.

• With plain JSON: No registry means no automatic enforcement; you manage compatibility yourself.

Debezium is a set of source connectors that run inside Kafka Connect to perform Change Data Capture (CDC): they read a database's transaction log and stream every insert, update, and delete into Kafka topics.

• Log-based CDC: Reads the DB's write-ahead/binlog (e.g. Postgres WAL, MySQL binlog) instead of polling tables, capturing changes with low impact.

• Runs as a Connect source connector: Deployed like any connector (usually distributed mode); Connect handles offsets, scaling, and restarts.

• Change event structure: Each event carries before and after states plus metadata (op type, source, timestamp).

• Offset tracking and snapshots: On first start it snapshots existing data, then streams incremental changes, storing log position as Connect offsets for resumability.

• Deletes as tombstones: A delete produces a change event followed by a null-value tombstone keyed by the primary key, enabling compaction and downstream deletes.

The In-Sync Replica set is the group of replicas (including the leader) that are fully caught up with the leader's log within a lag threshold; only ISR members are eligible for leader election, which makes the ISR the pivot between durability and availability.

• Membership: A follower stays in the ISR while it fetches within replica.lag.time.max.ms; if it falls behind or dies, the leader shrinks the ISR.

• Durability role:

  • With acks=all, a write is only acknowledged once all ISR members have it, so committed data survives as long as one ISR replica does.

  • min.insync.replicas sets the floor: if the ISR shrinks below it, writes are rejected rather than risk under-replicated data.

• Availability tension: Requiring a large ISR is safer but more likely to block writes; unclean.leader.election.enable=true trades durability for availability by allowing an out-of-sync replica to become leader (losing data).

The High Watermark is the offset up to which a message has been replicated to all in-sync replicas; it marks the boundary of committed data and consumers can only read messages below it.

• Definition: It is the minimum of the log-end offsets across the ISR, advanced by the leader as followers catch up.

• Message visibility: Records above the high watermark exist on the leader but are not yet committed, so they are invisible to consumers until fully replicated.

• Durability guarantee: Because everything up to the high watermark is on all ISR replicas, those messages survive a leader failover; a new leader truncates its log to the high watermark to stay consistent.

• Contrast: The Log End Offset (LEO) is the next offset to be written; the gap between LEO and high watermark is unreplicated, at-risk data.

A replication tool in Kafka is a utility that manages copying data between clusters or manages partition/replica placement, used for disaster recovery, migration, and multi-datacenter setups.

• MirrorMaker (and MirrorMaker 2): Replicates topics between separate clusters. MM2 (built on Kafka Connect) also syncs consumer offsets and topic configs.

• Cluster/replica management tools:

  • kafka-reassign-partitions.sh: moves replicas across brokers and changes replica assignments.

  • kafka-preferred-replica-election.sh: restores leadership to the preferred (balanced) leaders.

• Replica verification: kafka-replica-verification.sh: checks that replicas hold consistent data.

• Confluent Replicator: Commercial tool for cross-cluster replication with additional features over MirrorMaker.

min.insync.replicas is the minimum number of in-sync replicas that must acknowledge a write for it to succeed. Combined with acks=all, it guarantees a message is durably stored on that many replicas before the producer gets a success.

• How they combine:

  • acks=all tells the leader to wait for all current ISR members to replicate the message.

  • min.insync.replicas defines how many replicas the ISR must contain; if fewer are in sync, the leader rejects the write with NotEnoughReplicas.

• Typical durable setup: Replication factor 3 with min.insync.replicas=2 and acks=all: tolerates one broker failure while still accepting writes and never losing acknowledged data.

• The trade-off: Setting it equal to replication factor maximizes durability but loses availability if any replica drops.

• acks=all alone is not enough: Without min.insync.replicas, if the ISR shrinks to just the leader, acks=all means acknowledgment from one replica only.

Unclean leader election lets an out-of-sync replica become leader when no in-sync replica is available. It keeps the partition available but can lose committed messages the new leader never received.

• When it happens: All ISR replicas are down and only a lagging follower remains; Kafka must choose between waiting or electing that stale replica.

• The trade-off:

  • unclean.leader.election.enable=true: favors availability, the partition stays writable but data after the stale replica's last offset is lost.

  • unclean.leader.election.enable=false (default): favors durability, the partition stays offline until an in-sync replica returns.

• Recommendation: Keep it disabled for data-critical topics; enable only when uptime matters more than occasional loss.

Kafka supports three delivery semantics that trade off duplication against loss: at-most-once (may lose, never duplicate), at-least-once (never loses, may duplicate), and exactly-once (no loss, no duplicate). Each is achieved through producer, broker, and consumer configuration.

• At-most-once:

  • Producer uses low acks (0) and does not retry; consumer commits offsets before processing.

  • A failure loses the message but never reprocesses it.

• At-least-once:

  • Producer uses acks=all with retries; consumer commits offsets only after processing.

  • Guarantees delivery but a retry or reprocessing after failure can duplicate.

• Exactly-once (EOS):

  • Idempotent producer (enable.idempotence=true) dedupes retries using a producer ID and sequence numbers.

  • Transactions tie produce and offset commits into one atomic unit (transactional.id, isolation.level=read_committed).

  • Achievable end-to-end within Kafka, notably in Kafka Streams with processing.guarantee=exactly_once_v2.

Kafka transactions let a producer write to multiple partitions and commit its consumer offsets as a single atomic unit, so a read-process-write loop either fully commits or fully aborts. This is what makes consume-transform-produce pipelines exactly-once.

• The atomic bundle: The output records AND the source offsets (via sendOffsetsToTransaction()) are committed together, so offsets advance only if output was written.

• Setup: Configure a unique transactional.id, call initTransactions() once, then wrap each cycle in begin/commit.

• Consumer must read committed: Downstream consumers use read_committed to skip aborted records.

Kafka's EOS only guarantees exactly-once within Kafka. The moment your processing writes to or reads from an external system that isn't part of Kafka's transaction, duplicates can reappear because the two systems can't commit atomically together.

• Transaction boundary ends at Kafka: A write to a database or REST call is not covered by the Kafka transaction, so it may succeed while the Kafka commit later aborts (or vice versa).

• Crash between two commits: If the app writes to the external system, then crashes before committing offsets, it reprocesses and writes again.

• Non-idempotent side effects: Sending an email or charging a card twice can't be rolled back by Kafka.

• Fixes: Make the external sink idempotent (upserts keyed by record ID), or use transactional outbox / two-phase patterns to align the external commit with Kafka.