Java Backend Production Scenario Interview Guide

1. API Response Time Suddenly Increases (200 ms → 5 seconds)

Investigation Strategy

Never guess. Start with metrics.

flowchart TD
    C[Client] --> LB[Load Balancer]
    LB --> G[API Gateway]
    G --> S[Spring Boot Service]
    S --> DB[(Database)]
    S --> R[(Redis)]
    S --> M[Other Microservices]

Investigation Steps

Check dashboards (Prometheus, Grafana, Datadog, New Relic)
- CPU
- Memory
- Thread count
- GC time
- Database connections
- Request rate
Use distributed tracing (Zipkin, Jaeger, OpenTelemetry)

flowchart TD
    API --> UserService
    UserService --> OrderService
    OrderService --> PaymentService

Check database:
- Slow query log
- EXPLAIN
- Missing indexes
- Lock waits
Check downstream service latency.
Capture thread dumps.

jstack <pid>

Look for blocked threads, deadlocks, and waiting threads.

Review GC logs for Full GC pauses.

Root Causes

Database bottleneck
External API latency
Network issues
JVM/GC pauses
Thread pool exhaustion
Cache failures

2. Slow Downstream Microservice

Prevent cascading failures using:

flowchart LR
    Timeout --> Retry --> CircuitBreaker --> Fallback

Use Resilience4j:

@CircuitBreaker
@Retry
@TimeLimiter
@Bulkhead

Example usage with a fallback method:

@CircuitBreaker(name = "paymentService", fallbackMethod = "fallback")
public Payment getPayment() {
    return paymentClient.getPayment();
}

3. Circuit Breaker vs Retry vs Timeout vs Bulkhead

Pattern	Use When
Timeout	Prevent waiting forever
Retry	Temporary/transient failures
Circuit Breaker	Service repeatedly fails
Bulkhead	Prevent one dependency from exhausting all threads

Example:

Timeout: 2 seconds
Retry: 3 attempts
Circuit opens after 50% failures
Bulkhead: limit to 20 threads

4. Database Connection Pool Exhausted

Possible Causes

Long-running transactions
Connection leaks
Slow SQL
Small connection pool
Deadlocks
Lock contention

Example:

@Transactional
public void process() {
    // Avoid long-running transactions
}

Check HikariCP metrics:

Active connections
Idle connections
Waiting threads

5. Handling Duplicate Payment Requests

Implement idempotency.

Request:

POST /pay
Idempotency-Key: abc123

If the same request arrives again, return the previously stored response.

flowchart TD
A[Receive Request] --> B{Already Processed?}
B -->|Yes| C[Return Existing Response]
B -->|Processing| D[Reject Duplicate]
B -->|No| E[Process Payment]
E --> F[Store Result]

6. Idempotency Key Storage

Store the key in Redis or the database.

Structure:

Key: abc123
Status: SUCCESS
Response: PaymentId

Use SETNX (Redis) or a unique database constraint so only the first request succeeds.

7. Redis Is Down

The application should continue functioning.

Fallback:

Redis unavailable
        →
      Database

Performance may degrade, but availability should remain.

8. Cache Invalidation Across Multiple Instances

flowchart LR
    U[User Update] --> P[Redis Pub/Sub or Kafka Event]
    P --> A1[App Instance 1]
    P --> A2[App Instance 2]
    P --> A3[App Instance 3]

Possible solutions:

Redis Pub/Sub
Kafka events
Spring Cache

@CacheEvict

9. Works for 100 Users but Fails at 10,000

Check:

Connection pools
Thread pools
Database bottlenecks
CPU
Garbage collection
Rate limiting
Network
Load balancer

Most common bottlenecks:

Database
Connection pool
External APIs

10. Processing a 5 GB CSV Without OutOfMemoryError

Never load the entire file.

Avoid:

Files.readAllLines(path);

Instead:

BufferedReader reader = Files.newBufferedReader(path);
// or Spring Batch

Process in chunks:

Read 1000 rows
Process
Write
Clear memory

11. Synchronous vs Asynchronous Communication

Synchronous

Use for:

Login
Payment validation
OTP verification

Asynchronous

Use for:

Email
Notifications
Auditing
Analytics
Invoice generation

12. Kafka Consumer Crashes Before Offset Commit

Scenario:

sequenceDiagram
    Kafka->>Consumer: Deliver message
    Consumer->>Consumer: Process successfully
    Consumer--XKafka: Crashes before commit
    Kafka->>Consumer: Redelivers message

Solution:

Design consumers to be idempotent.

13. Preventing Duplicate Kafka Processing

Store processed event IDs in:

Redis
Database

If an event ID already exists, skip processing.

14. Kubernetes Pod Keeps Restarting

Commands:

kubectl describe pod <pod-name>
kubectl logs <pod-name> --previous

Check for:

OOMKilled
CrashLoopBackOff
Probe failures
Restart count
Kubernetes events

15. CPU Normal but API Latency High

Possible causes:

Database locks
Network latency
Slow downstream service
Thread starvation
GC pauses
Blocking I/O

16. Identifying the Bottleneck

Use distributed tracing.

flowchart LR
    API --> Service --> Database
    Service --> ExternalAPI

Measure:

API latency
Database execution time
Network RTT
External service latency

Tools:

Zipkin
Jaeger
OpenTelemetry

17. SQL Slow Only in Production

Investigate:

Execution plans
Indexes
Statistics
Data volume
Locks
Parameter sniffing
Database version
Hardware differences

18. Distributed Transactions

Avoid XA transactions.

Prefer:

Saga Pattern
Outbox Pattern

Each service commits locally and compensates on failure.

19. Saga Pattern and Compensation Failures

flowchart LR
    Order --> Inventory --> Payment --> Shipping

If payment fails:

Undo inventory reservation
Undo order creation

If compensation also fails:

Retry
Dead Letter Queue
Manual intervention

Choreography vs Orchestration

Two ways to implement a Saga:

Choreography — each service publishes events; the next service reacts, with no central coordinator.
Orchestration — a central orchestrator explicitly tells each service what to do next.

flowchart LR
    Order --> Payment --> Shipping
    Payment -->|Failure| Cancel["Compensating Transaction: Cancel Order"]

Popular Saga orchestration tools: Temporal, Camunda, Axon, Eventuate Tram.

Why Not Use One Transaction Across Multiple Microservices?

Reasons:

Independent databases per service
Two-Phase Commit (2PC) introduces latency and tight coupling
Reduced scalability and availability

Preferred approach: Saga pattern + Kafka/RabbitMQ + eventual consistency, instead of distributed ACID transactions.

20. Duplicate Requests Due to Network Retries

Use:

Idempotency keys
Unique business keys
Database unique constraints

Return the same response for duplicate requests.

21. Zero-Downtime Deployment

Strategies:

Rolling update
Blue-Green deployment
Canary deployment

Kubernetes:

maxUnavailable: 0
maxSurge: 1

Use readiness probes before routing traffic.

22. Graceful Shutdown

During pod termination:

SIGTERM

Configure:

Finish in-flight requests
Reject new requests
terminationGracePeriodSeconds
Readiness probe removal before shutdown

23. API Rate Limiting

Algorithms:

Token Bucket
Leaky Bucket
Sliding Window

Store counters in Redis.

Reject excess requests with:

HTTP/1.1 429 Too Many Requests

Concrete implementation with Bucket4j:

<dependency>
 <groupId>com.bucket4j</groupId>
 <artifactId>bucket4j-core</artifactId>
</dependency>

Bucket bucket = Bucket4j.builder()
    .addLimit(Bandwidth.simple(10, Duration.ofMinutes(1)))
    .build();

Or at the gateway layer with Spring Cloud Gateway:

filters:
  - name: RequestRateLimiter
    args:
      redis-rate-limiter.replenishRate: 10
      redis-rate-limiter.burstCapacity: 20

For a full HLD/LLD treatment (algorithm comparison, distributed design, custom Token Bucket implementation, edge cases), see the Rate Limiter System Design case study.

24. Memory Leak Investigation

Monitor:

Heap usage
Heap dumps
GC logs

Tools:

Eclipse MAT
VisualVM
YourKit

Look for:

Large collections
ThreadLocal leaks
Static references
Cache growth
Listener leaks

25. Production Issue Only Under Heavy Traffic

Approach:

Metrics
Logs
Distributed tracing
Thread dump
Heap dump
GC logs
Database metrics
Load testing

Tools:

JMeter
Gatling
Prometheus
Grafana
OpenTelemetry
Zipkin
Jaeger

Reproduce using production-like traffic rather than local environments.

Interview Mindset

A senior engineer should:

Observe first (metrics, logs, traces)
Isolate the bottleneck
Mitigate quickly (timeouts, circuit breakers, scaling)
Fix the root cause
Prevent recurrence (monitoring, alerts, automation)

Additional Topics

Spring Boot production tuning
Kafka partitioning, ordering, retries, DLQs
Redis caching patterns
Kubernetes probes and autoscaling
Observability
Resilience4j
Concurrency control
Database optimization

Summary

Always diagnose production issues using metrics, logs, and tracing before making changes.
Build resilient microservices with timeouts, retries, circuit breakers, bulkheads, and idempotency.
Design for scalability through efficient resource management, streaming, caching, and rate limiting.
Prefer observability and automation to quickly identify and resolve production issues.
Demonstrate a structured troubleshooting approach in interviews to reflect real-world production experience.