API Gateway System Design (HLD and LLD)

What is an API Gateway?

An API Gateway is a single entry point for all client requests that routes them to the appropriate microservices while handling common cross-cutting concerns.

Typical responsibilities include:

• Authentication and authorization
• Request routing
• Rate limiting
• Load balancing
• Request/response transformation
• Caching
• Logging and monitoring

Conceptual Architecture

flowchart LR
    C[Client] --> G[API Gateway]
    G --> U[User Service]
    G --> O[Order Service]
    G --> P[Payment Service]

Popular implementations:

• Kong
• Netflix Zuul
• Spring Cloud Gateway
• AWS API Gateway
• NGINX

---

Functional Requirements

1. Route requests to the correct microservice.
2. Authenticate users (JWT/OAuth).
3. Rate limit requests.
4. Load balance traffic.
5. Aggregate responses (optional).
6. Cache responses.
7. Log requests and expose monitoring metrics.

---

Non-Functional Requirements

• High availability
• Low latency
• High throughput
• Horizontal scalability
• Security
• Fault tolerance

---

High-Level Design (HLD)

Overall Architecture

flowchart TD
    Client --> Gateway[API Gateway]
    Gateway --> Auth[Authentication]
    Gateway --> RL[Rate Limiter]
    Gateway --> Router[Router]
    Gateway --> Cache[Cache]
    Gateway --> Logs[Logging]
    Router --> UserSvc[User Service]
    Router --> OrderSvc[Order Service]
    Router --> PaymentSvc[Payment Service]

---

Core Components

Request Router

Determines which service should handle the request.

Example mappings:

/api/users/**     -> User Service
/api/orders/**    -> Order Service
/api/payments/**  -> Payment Service

Routing information may be stored in:

• Configuration
• Service Registry (Eureka/Consul)

Authentication Module

Validates:

• JWT tokens
• OAuth tokens
• API keys

sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant A as Auth
    participant S as Service

    C->>G: Request + JWT
    G->>A: Validate token
    A-->>G: Valid
    G->>S: Forward request
    S-->>G: Response
    G-->>C: Response

Rate Limiter

Protects backend services.

Common algorithms:

• Token Bucket
• Leaky Bucket
• Sliding Window
• Fixed Window

Example policy:

100 requests/minute/user

Storage:

• Redis
• In-memory cache

See the Rate Limiter system design case study for the full algorithm comparison and implementation.

Load Balancer

Routes traffic across service instances.

Order Service
- Instance 1
- Instance 2
- Instance 3

Algorithms:

• Round Robin
• Least Connections
• Weighted Routing

Caching Layer

Used primarily for:

• GET requests
• Static responses

Example:

GET /products

Redis TTL: 30 seconds

Logging and Monitoring

Track:

• Request latency
• Error rates
• Traffic volume

Typical tools:

• Prometheus
• Grafana
• ELK Stack

---

Request Flow

flowchart LR
    A[Client]
    B[Gateway]
    C[Authenticate]
    D[Rate Limit]
    E[Route]
    F[Load Balance]
    G[Microservice]
    H[Response]

    A-->B-->C-->D-->E-->F-->G-->H

Flow:

1. Client sends request.
2. Gateway receives request.
3. Authentication filter validates token.
4. Rate limiter checks quota.
5. Router determines destination.
6. Load balancer selects an instance.
7. Request is forwarded.
8. Response is returned through the gateway.

---

Low-Level Design (LLD)

Class Structure

classDiagram
    class ApiGateway {
        Router
        AuthFilter
        RateLimiter
        LoadBalancer
        CacheManager
    }

    class Router
    class AuthFilter
    class RateLimiter
    class LoadBalancer
    class CacheManager

    ApiGateway --> Router
    ApiGateway --> AuthFilter
    ApiGateway --> RateLimiter
    ApiGateway --> LoadBalancer
    ApiGateway --> CacheManager

Router

class Router {

    Map<String, String> routeMap;

    public String getService(String path) {
        if(path.startsWith("/users"))
            return "USER_SERVICE";

        if(path.startsWith("/orders"))
            return "ORDER_SERVICE";

        return null;
    }
}

RateLimiter Interface

interface RateLimiter {

    boolean allowRequest(String clientId);

}

Token Bucket Implementation

class TokenBucketRateLimiter implements RateLimiter {

    private int capacity;
    private int tokens;
    private long lastRefillTime;

    public synchronized boolean allowRequest(String clientId) {

        refill();

        if(tokens > 0) {
            tokens--;
            return true;
        }

        return false;
    }

    private void refill() {
        // refill logic
    }
}

Authentication Filter

class AuthFilter {

    public boolean authenticate(String token) {

        if(token == null)
            return false;

        return JwtUtil.validate(token);
    }
}

Round Robin Load Balancer

class RoundRobinLoadBalancer {

    private List<String> instances;
    private AtomicInteger index = new AtomicInteger(0);

    public String getInstance() {

        int i = index.getAndIncrement();
        return instances.get(i % instances.size());
    }
}

Cache Manager

class CacheManager {

    RedisClient redis;

    public String get(String key) {
        return redis.get(key);
    }

    public void put(String key, String value) {
        redis.set(key, value);
    }
}

---

Distributed API Gateway

flowchart TD
    Clients --> LB[Load Balancer]
    LB --> G1[Gateway 1]
    LB --> G2[Gateway 2]
    LB --> G3[Gateway 3]
    G1 --> Services
    G2 --> Services
    G3 --> Services

Gateway instances should remain stateless.

Shared infrastructure:

• Redis
• Configuration Server

---

Storage

Purpose	Storage
Rate limiting	Redis
Cache	Redis
Configuration	Config Server

---

Scaling Strategy

Horizontal Scaling

Add additional gateway instances behind a load balancer.

Stateless Design

Persist shared state in:

• Redis
• Kafka
• Configuration Server

---

Failure Handling

Use circuit breakers such as:

• Resilience4j
• Hystrix

Flow:

Gateway -> Service failure -> Circuit opens -> Fallback response

---

Security

Typical responsibilities:

• SSL termination
• JWT validation
• OAuth2
• API key validation
• IP whitelisting
• WAF integration

---

Spring Cloud Gateway Example

Architecture:

Client
  |
Spring Cloud Gateway
  |
Eureka
  |
Microservices

Example route:

@Bean
RouteLocator routes(RouteLocatorBuilder builder) {
    return builder.routes()
        .route("user-service",
            r -> r.path("/users/**")
            .uri("lb://USER-SERVICE"))
        .build();
}

---

API Gateway vs Load Balancer

Feature	Load Balancer	API Gateway
Layer	L4/L7	L7
Routing	Basic	Advanced
Authentication	No	Yes
Rate Limiting	No	Yes
Request Transformation	No	Yes

---

Common Interview Questions

1. How do you implement distributed rate limiting?
2. How do you avoid the gateway becoming a bottleneck?
3. How do you implement API aggregation?
4. How do you handle gateway failures?
5. API Gateway vs Service Mesh?
6. Why Spring Cloud Gateway instead of Zuul?

---

Key Takeaways

• API Gateway centralizes routing, security, rate limiting, caching, logging, and monitoring.
• Gateways should remain stateless to enable horizontal scaling.
• Redis is commonly used for distributed caching and rate limiting.
• Spring Cloud Gateway is a popular Java implementation for microservice ecosystems.
• Circuit breakers and load balancing improve resilience and availability.