File Upload Service Design (HLD and LLD)

Functional Requirements

Users can upload files.
System stores files reliably.
Users can retrieve/download files.
Support large files (GBs).
Generate shareable URLs.
Support multiple file types (image, video, documents).
Track file metadata.

Non-Functional Requirements

Highly scalable
Highly available
Fault tolerant
Secure uploads
Fast upload/download
Handle large files
Cost optimized storage

High-Level Architecture

flowchart TD
    C[Client] --> G[API Gateway]
    G --> U[Upload Service]
    U --> M[(Metadata DB)]
    U --> O[(Object Storage)]
    O --> CDN[CDN]

Components

Client

Uploads files using:

POST /files/upload

Supports direct upload or pre-signed URL upload.

API Gateway

Responsibilities:

Authentication
Rate limiting
Request routing
Logging

Examples:

Kong
Nginx
AWS API Gateway

Upload Service

Responsible for:

Validating uploads
Generating upload URLs
Saving metadata
Managing lifecycle

Object Storage

Examples:

Amazon S3
Google Cloud Storage
Azure Blob

Benefits:

Highly scalable
Durable
Cost effective

Metadata Database

Schema:

FileMetadata
------------
id
user_id
file_name
file_size
content_type
storage_path
upload_time
status
checksum

Options:

MySQL
PostgreSQL
DynamoDB

CDN

Examples:

Cloudflare
CloudFront
Akamai

Provides fast downloads.

Upload Flows

Upload Through Service

sequenceDiagram
    participant C as Client
    participant S as Upload Service
    participant O as Object Storage
    C->>S: Upload file
    S->>O: Store file
    O-->>S: Success
    S-->>C: Upload complete

Drawback: the upload service becomes the bottleneck.

Pre-Signed URL Upload (Recommended)

sequenceDiagram
    participant C as Client
    participant S as Upload Service
    participant O as Object Storage
    C->>S: Request upload URL
    S-->>C: Pre-signed URL
    C->>O: Upload directly
    C->>S: Confirm upload
    S->>S: Save metadata

Advantages:

Reduces backend load
Scales efficiently

Large File Uploads

Use multipart upload.

File
 - Part 1
 - Part 2
 - Part 3
 - Part 4

Benefits:

Resume uploads
Parallel uploads
Retry individual parts

Low-Level Design

APIs

Generate Upload URL

POST /files/upload-url

Request:

{
  "fileName": "video.mp4",
  "contentType": "video/mp4",
  "fileSize": 50000000
}

Response:

{
  "uploadUrl": "https://s3....",
  "fileId": "12345"
}

Complete Upload

POST /files/{fileId}/complete

Get File

GET /files/{fileId}

Returns download URL and metadata.

Delete File

DELETE /files/{fileId}

Database

files
------
id (PK)
user_id
file_name
file_size
content_type
storage_path
checksum
status
created_at
updated_at

Indexes:

CREATE INDEX idx_files_user_id ON files(user_id);
CREATE INDEX idx_files_created_at ON files(created_at);

Java Spring Boot Example

Entity

@Entity
public class FileMetadata {

    @Id
    private String id;

    private String userId;
    private String fileName;
    private Long fileSize;
    private String contentType;
    private String storagePath;
    private String checksum;
    private String status;
}

Controller

@RestController
@RequestMapping("/files")
public class FileController {

    @PostMapping("/upload-url")
    public UploadResponse generateUploadUrl(
            @RequestBody UploadRequest request) {
        return fileService.generateUploadUrl(request);
    }

    @PostMapping("/{id}/complete")
    public void completeUpload(@PathVariable String id) {
        fileService.completeUpload(id);
    }
}

Service

@Service
public class FileService {

    public UploadResponse generateUploadUrl(UploadRequest request) {

        String fileId = UUID.randomUUID().toString();

        String presignedUrl = s3Service.generateUrl(fileId);

        return new UploadResponse(fileId, presignedUrl);
    }
}

Security

Validate file types.
Block executable uploads where appropriate.
Scan uploaded files (e.g. ClamAV).
Require authentication.
Use expiring signed URLs.

Scaling

Stateless upload service
Kubernetes + autoscaling
Load balancer
Queue-based background processing

flowchart TD
    U[Upload] --> K[Kafka/Queue]
    K --> V[Virus Scanner]
    K --> T[Thumbnail Generator]
    K --> P[Image Processor]

Handling Millions of Uploads

Direct uploads to object storage
CDN for downloads
Chunked uploads
Storage lifecycle policies

Example:

S3 Standard -> Glacier

Advanced Topics

Resumable Uploads

upload_session
--------------
file_id
part_number
status

File Deduplication

Use checksums (MD5/SHA256) to store identical content once and reference it from multiple users.

Versioning

file v1
file v2
file v3

Production Architecture

flowchart TD
    C[Client] --> G[API Gateway]
    G --> U[Upload Service]
    U --> D[(Metadata DB)]
    U --> S[(Object Storage)]
    S --> CDN[CDN]
    U --> Q[Message Queue]
    Q --> W[Workers: Virus Scan, Thumbnails]

Key Takeaways

Prefer pre-signed URLs for scalability.
Store metadata separately from file contents.
Use multipart uploads for large files.
Offload post-processing through asynchronous workers.
Secure uploads with authentication, validation, malware scanning, and expiring signed URLs.