Kafka Architecture Interview Guide

Core Components

1. Producer

A Producer sends messages to Kafka topics.

Kafka can determine the target partition based on the message key, or the producer can specify the partition explicitly.

kafkaTemplate.send("orders-topic", orderId, order);

Example:

• Order Service publishes order events to the orders-topic.

---

2. Topic

A Topic is a logical channel where messages are stored.

Example:

orders-topic
payments-topic
notifications-topic

Think of a Topic like a table in a database.

---

3. Partition

A topic is divided into one or more partitions.

Orders Topic

Partition 0
Partition 1
Partition 2

Benefits:

• Parallel processing
• High throughput
• Scalability

Messages having the same key are always written to the same partition.

Example:

OrderId = 101 -> Partition 1
OrderId = 102 -> Partition 2
OrderId = 101 -> Partition 1

This guarantees ordering for a particular key.

---

4. Broker

A Broker is a Kafka server.

Example:

Broker 1
Broker 2
Broker 3

Responsibilities:

• Stores messages
• Serves producers and consumers
• Maintains partitions
• Handles replication

---

5. Replication

Each partition has:

• One Leader
• One or more Followers

Example:

Partition 0

Leader   -> Broker 1
Follower -> Broker 2
Follower -> Broker 3

Rules:

• Producers always write to the Leader.
• Consumers read from the Leader.
• Followers continuously replicate data from the Leader.

If the leader crashes, Kafka elects one of the In-Sync Replicas (ISR) as the new leader.

---

6. Consumer

Consumers read messages from Kafka.

Example:

@KafkaListener(topics = "orders-topic")
public void consume(Order order) {
    System.out.println(order);
}

---

7. Consumer Group

Consumers can belong to the same consumer group.

Example:

Orders Topic

Partition 0 -> Consumer 1
Partition 1 -> Consumer 2
Partition 2 -> Consumer 3

Rules:

• One partition can be consumed by only one consumer in the same group.
• Multiple consumer groups can consume the same topic independently.

Example:

Group A -> Order Processing

Group B -> Analytics

Group C -> Audit

All three groups receive the same messages.

---

8. Offset

Each message inside a partition gets a unique offset.

Offset 0
Offset 1
Offset 2
Offset 3

Kafka stores the last committed offset for every consumer group.

When the consumer restarts, it resumes from the last committed offset.

---

Kafka Architecture Diagram

flowchart LR

P[Producer]
T[Topic]
PT[Partitions]
L[Leader Broker]
R[Follower Replicas]
CG[Consumer Group]
C[Consumers]

P --> T
T --> PT
PT --> L
L --> R
L --> CG
CG --> C

---

Message Flow

Producer
    ↓
Topic
    ↓
Partition
    ↓
Leader Broker
    ↓
Follower Replicas
    ↓
Consumer Group
    ↓
Consumers

---

Delivery Guarantees

Type	Description	Drawback
At Most Once	Offset committed before processing	Message loss possible
At Least Once	Offset committed after processing	Duplicate messages possible
Exactly Once	Kafka Transactions + Idempotent Producer	More complex

---

Common Kafka Interview Questions

Why do we need partitions?

• Parallelism
• Scalability
• High throughput

---

Why do we need replication?

• Fault tolerance
• High availability

---

Can two consumers read the same partition in one consumer group?

No.

Only one consumer in a consumer group can consume a partition.

---

What happens if consumers are more than partitions?

Example:

Partitions = 3

Consumers = 5

Consumer 4 -> Idle

Consumer 5 -> Idle

Extra consumers remain idle until more partitions become available.

---

What is ISR?

ISR stands for In-Sync Replicas.

These replicas are fully synchronized with the leader and are eligible to become the new leader if the current leader fails.

---

What Happens If a Kafka Consumer Crashes?

A consumer crash may occur due to:

• Application crash
• Server shutdown
• OutOfMemoryError
• Network failure
• Consumer stops sending heartbeats

---

Example

Partition 0

Offset 0 ✓

Offset 1 ✓

Offset 2 (Currently Processing)

The consumer crashes before committing Offset 2.

---

What Kafka Does

1. Kafka detects that the consumer has stopped sending heartbeats.
2. After session.timeout.ms, Kafka considers the consumer dead.
3. Kafka triggers a rebalance.
4. Another consumer in the same consumer group receives the partition.
5. The new consumer starts reading from the last committed offset.

Example:

Last committed offset = 1

New consumer starts from Offset 2

---

Consumer Crash Flow

sequenceDiagram

participant C1 as Consumer 1

participant Kafka

participant C2 as Consumer 2

C1->>Kafka: Read Offset 2

Note over C1: Consumer crashes before commit

Kafka->>Kafka: Heartbeats stop

Kafka->>Kafka: Detect timeout

Kafka->>Kafka: Trigger rebalance

Kafka->>C2: Assign partition

C2->>Kafka: Resume from last committed offset

---

Result

• No message loss
• Offset 2 is processed again
• Duplicate processing is possible

This is called At-Least-Once Delivery.

---

Prevent Duplicate Processing

Use idempotent processing.

if (!processedEvents.contains(eventId)) {
    processOrder();
    processedEvents.add(eventId);
}

A database unique constraint enforces this at the storage layer too:

CREATE UNIQUE INDEX uk_order_id ON orders(order_id);

Kafka also supports exactly-once semantics (EOS) at the producer/transaction level:

enable.idempotence=true
transactional.id=my-transaction

This guarantees exactly-once for Kafka-internal read-process-write chains (e.g. consuming from one topic and producing to another within a transaction). It does not make external side effects — a database write, an API call, sending an email — idempotent. Any consumer that has an external side effect still needs the idempotent-consumer pattern (unique constraint or processed-event tracking) above, regardless of EOS settings on the Kafka client.

---

Key Takeaways

• Kafka architecture consists of Producers, Topics, Partitions, Brokers, Replication, Consumers, Consumer Groups, and Offsets.
• Partitions enable scalability and parallel processing while preserving ordering for the same key.
• Replication provides fault tolerance and high availability through leader-follower architecture.
• When a consumer crashes, Kafka detects missing heartbeats, rebalances the consumer group, and resumes consumption from the last committed offset.
• At-Least-Once Delivery ensures no message loss but may cause duplicate processing; idempotent processing helps avoid duplicate business operations.