Log Aggregation System Design (HLD & LLD)

A Log Aggregation System collects logs from multiple services and servers, stores them centrally, and enables searching, monitoring, visualization, and alerting. It is a core component of modern distributed systems and microservices architectures.

Typical implementations include:

• ELK Stack (Elasticsearch + Logstash + Kibana)
• Splunk
• Datadog
• Grafana Loki

Functional Requirements

1. Applications send logs to the system.
2. Aggregate logs from multiple sources.
3. Support log search.
4. Filter by:
   • Service
   • Timestamp
   • Severity
5. Dashboards and visualization.
6. Alerting based on log patterns.

Non-Functional Requirements

• High throughput (millions of logs/sec)
• Fault tolerance
• Near real-time search
• Scalable storage
• Durable persistence

High-Level Design

Components

1. Log Producers (Applications / Servers)
2. Log Agents
3. Log Ingestion Layer
4. Message Queue / Buffer
5. Log Processing
6. Storage
7. Query Service
8. Visualization Dashboard

Architecture

flowchart TD
    A[Application Servers] --> B["Log Agents<br/>Fluentd/Filebeat"]
    B --> C[Log Ingestion API]
    C --> D[Kafka]
    D --> E[Log Processing Service]
    E --> F["Storage<br/>Elasticsearch / S3"]
    F --> G[Query API]
    G --> H["Dashboard<br/>Kibana / Grafana"]

Component Details

Log Producers

Applications generate logs.

INFO 2026-03-16 Payment processed
ERROR 2026-03-16 Payment failed

Log Agents

Examples:

• Fluentd
• Logstash
• Filebeat

Responsibilities:

• Read log files
• Batch logs
• Forward to ingestion service

Log Ingestion Service

Receives logs via REST.

POST /logs

Responsibilities:

• Validate
• Enrich
• Publish to Kafka

Kafka

Decouples ingestion from processing.

Benefits:

• Handles spikes
• Prevents log loss
• Multiple consumers

Topic:

logs-topic

Partition strategy:

service-name

Log Processing

Responsibilities:

• Parse
• Extract fields
• Enrich metadata
• Normalize

Raw log:

ERROR Payment failed for user 123

Structured log:

{
  "timestamp": "...",
  "service": "payment-service",
  "level": "ERROR",
  "userId": 123,
  "message": "Payment failed"
}

Storage

Hot Storage

• Elasticsearch
• OpenSearch

Recent searchable logs (7-30 days).

Cold Storage

• S3
• HDFS

Long-term archival.

Query Service

GET /logs?service=payment&level=ERROR
GET /logs?startTime=...&endTime=...

Dashboard

Examples:

• Kibana
• Grafana

Capabilities:

• Search
• Dashboards
• Error monitoring

Scaling

Horizontal Scaling

Scale independently:

• Agents
• Kafka partitions
• Processing workers
• Elasticsearch nodes

Kafka Partitioning

service-name

or

timestamp

Batch Processing

batch size = 500 logs

Improves throughput.

Data Model

Fields:

LogEntry
---------
logId
timestamp
serviceName
host
logLevel
message
traceId
spanId
metadata

Example:

{
  "timestamp": "2026-03-16T10:30:00",
  "service": "order-service",
  "level": "ERROR",
  "host": "server-1",
  "traceId": "abc123",
  "message": "Order failed"
}

Low-Level Design

LogEntry

class LogEntry {

    String id;
    String serviceName;
    String host;
    String level;
    String message;
    long timestamp;
    String traceId;
}

Controller

@RestController
public class LogController {

    @PostMapping("/logs")
    public void ingest(@RequestBody List<LogEntry> logs){
        logService.publish(logs);
    }
}

Service

@Service
public class LogService {

   @Autowired
   KafkaProducer producer;

   public void publish(List<LogEntry> logs){
       producer.send("logs-topic", logs);
   }
}

Kafka Consumer

@KafkaListener(topics="logs-topic")
public void process(List<LogEntry> logs){
     logProcessor.process(logs);
}

Processor

public void process(LogEntry log){
     logRepository.save(log);
}

Repository

public interface LogRepository {

     void save(LogEntry log);

     List<LogEntry> search(Query query);
}

Elasticsearch Indexing

Daily indices:

logs-2026-03-16
logs-2026-03-17

Benefits:

• Faster search
• Easier retention cleanup

Handling Massive Volume

Sampling

1 of every 10 INFO logs

Prioritize Log Levels

ERROR > WARN > INFO > DEBUG

Compression

Compress before storage.

Failure Handling

Agent Failure

Buffer locally.

Kafka Failure

RF = 3

Elasticsearch Failure

Deploy replicas.

Alerting

Example rule:

If ERROR logs > 1000 in 5 minutes
Trigger alert

Common tools:

• Prometheus
• Alertmanager

Example Production Pipeline

flowchart LR
    A[Applications] --> B[Filebeat]
    B --> C[Kafka]
    C --> D[Logstash]
    D --> E[Elasticsearch]
    E --> F[Kibana]

Common Interview Questions

• How do you prevent log loss?
• How do you support 10 million logs/sec?
• How do you implement efficient log search?
• Difference between logs, metrics, and traces?
• How do you correlate logs across microservices?

Key Takeaways

• Kafka provides durable buffering and decouples producers from consumers.
• Elasticsearch/OpenSearch is optimized for recent searchable logs, while S3/HDFS stores long-term archives.
• Daily index partitioning simplifies retention and improves query performance.
• Horizontal scaling across ingestion, processing, messaging, and storage enables very high throughput.
• Structured logs with trace IDs simplify distributed debugging.