Handbook

📚 This is Part 1 of the “Scaling Your API” Series

• Part 1 (this page): Performance & Infrastructure - Technical techniques to handle millions of requests
• Part 2: Design & Architecture → - Organizational strategies and API design patterns for large-scale systems
• Part 3: Choosing the Right Database → - Database selection for your API
• Part 4: Load Balancing & High Availability → - Keeping your API always available
• Part 5: Monitoring & Performance → - Tracking and improving API performance

The Journey: 1 RPS → 1,000,000 RPS

Imagine your API is a restaurant. At first, you have one chef (server) handling one order at a time. But what happens when you go viral and suddenly have a million customers waiting?

Overview: The 14 Key Scaling Techniques

Scaling an API isn’t about implementing all techniques at once—it’s about choosing the right tools for your current stage. This guide covers 14 proven techniques in order of complexity:

Foundation (Must-Haves for Every API)

1. Vertical Scaling - Start here: upgrade your single server’s resources
2. Connection Pooling - Reuse database connections instead of opening new ones
3. Avoid N+1 Queries - Fetch related data efficiently with joins

Essential Optimizations (1-10K RPS)

1. Horizontal Scaling - Add more servers to distribute the load
2. Load Balancing - Intelligently route traffic across servers
3. Caching - Store frequently accessed data in memory (Redis/Memcached)
4. Pagination - Send data in manageable chunks instead of all at once

Performance Enhancements (10K-100K RPS)

1. Fast JSON Serializers - Use optimized libraries for data serialization
2. Compression - Reduce payload sizes with Gzip or Brotli
3. CDN - Serve content from edge servers close to users
4. Async Processing - Move slow tasks to background workers

Advanced Scaling (100K+ RPS)

1. Async Logging - Make logging non-blocking
2. Database Sharding - Split your database horizontally
3. Rate Limiting - Protect your system from abuse

The Three Critical Insights

1. Start Simple, Scale Progressively Don’t build for 1M RPS on day one. A single well-optimized server can handle 10,000+ RPS with proper caching and database optimization.

2. Fix the Bottleneck, Not Everything Measure your system to identify the actual bottleneck (CPU? Memory? Database? Network?) and address that specific issue first.

3. Some Techniques Are Always Worth It Connection pooling, avoiding N+1 queries, basic caching, pagination, and compression should be implemented from the start—they’re simple and have massive impact.

Let’s dive into each technique in detail, starting from the simplest to the most advanced.

Level 1: Vertical Scaling (The Easy Start)

What is it?

Make your single server bigger and more powerful — more CPU, more RAM, faster disk.

Real-World Analogy

Instead of hiring more chefs, you give your one chef a bigger kitchen with better equipment.

Limits

• ❌ There’s a ceiling — servers can only get so big
• ❌ Single point of failure — if it crashes, everything goes down
• ❌ Expensive at the top end

Good for: 1 → 100 RPS

Level 2: Horizontal Scaling (Add More Servers)

What is it?

Instead of one big server, use many smaller servers working together.

Real-World Analogy

Hire more chefs! Each chef can work independently, handling their own orders.

Benefits

• ✅ No ceiling — just add more servers
• ✅ No single point of failure — if one dies, others keep working
• ✅ Cost-effective — use cheaper commodity hardware

Good for: 100 → 10,000 RPS

Level 3: Load Balancing (Traffic Cop)

What is it?

A load balancer distributes incoming requests across multiple servers evenly.

Common Algorithms

Real-World Analogy

A host at a restaurant who seats customers at different tables to keep all waiters equally busy.

Level 4: Database Connection Pooling

What is it?

Instead of opening a new database connection for every request, maintain a pool of ready-to-use connections that can be reused.

Why It Matters

Opening a database connection is expensive:

With 1000 requests/sec, you save 50-100 seconds of work!

Without vs With Connection Pooling

Real-World Analogy

Instead of getting a new fork for every bite of food, you keep your fork and reuse it throughout the meal.

Implementation Tips

# Python example with connection pooling
from sqlalchemy import create_engine

# Create engine with connection pool
engine = create_engine(
    'postgresql://user:pass@localhost/db',
    pool_size=20,           # Keep 20 connections ready
    max_overflow=10,        # Allow 10 more if needed
    pool_pre_ping=True      # Check if connection is alive
)

Good for: All production APIs with databases

Level 5: Avoiding N+1 Query Problems

What is it?

The N+1 problem happens when you fetch a list of items, then make a separate query for each item’s related data. This turns 1 query into N+1 queries!

The Better Way: Join or Batch Queries

Problem Example

# ❌ BAD: N+1 Problem
users = User.query.all()  # 1 query

for user in users:  # 100 users
    posts = user.posts  # 100 more queries!
    # Total: 101 queries

Solution Example

# ✅ GOOD: Eager Loading
users = User.query.options(
    joinedload(User.posts)
).all()  # Just 1 query with JOIN!

for user in users:
    posts = user.posts  # No additional query!

Performance Impact

Real-World Analogy

Instead of making 100 trips to the store to buy 100 items, you make one trip with a shopping list and get everything at once.

Good for: Any API that returns lists with related data

Level 6: Caching (Remember & Reuse)

What is it?

Store frequently accessed data in fast memory so you don’t have to compute or fetch it again.

Types of Caching

What to Cache

• ✅ API responses that don’t change often
• ✅ Database query results
• ✅ Session data
• ✅ Computed values (totals, aggregations)

Real-World Analogy

A chef who preps ingredients in advance. Instead of chopping onions for every order, they chop a big batch and grab from it.

Good for: 1,000 → 100,000 RPS

Level 7: Pagination (Send Data in Chunks)

What is it?

Instead of returning thousands of records at once, break them into small pages that load quickly.

Common Pagination Approaches

Implementation

// Offset-based pagination (simple but slower for large offsets)
GET /api/users?page=1&limit=20

// Cursor-based pagination (faster, handles real-time changes)
GET /api/users?cursor=abc123&limit=20

Response Format

{
  "data": [...],
  "pagination": {
    "total": 10000,
    "page": 1,
    "limit": 20,
    "total_pages": 500,
    "next_cursor": "xyz789"
  }
}

Real-World Analogy

Instead of downloading an entire encyclopedia, you download one page at a time as you read it.

Good for: Any endpoint that returns lists or collections

Level 8: Lightweight JSON Serializers

What is it?

JSON serialization (converting data structures to JSON) can be a bottleneck. Fast serializers can be 2-10x faster than default ones.

Performance Comparison

Example Implementation

# ❌ Default (slower)
import json
response = json.dumps(data)

# ✅ Fast serializer
import orjson
response = orjson.dumps(data)  # 5-10x faster!

// ❌ Default (slower)
const json = JSON.stringify(data);

// ✅ Fast serializer with schema
const fastJson = require('fast-json-stringify');
const stringify = fastJson({
  type: 'object',
  properties: {
    name: { type: 'string' },
    age: { type: 'integer' }
  }
});
const json = stringify(data);  // 2-3x faster!

Real-World Analogy

Using a high-speed printer instead of an old dot-matrix printer to print the same document.

Good for: High-traffic APIs with large response payloads

Level 9: Compression (Shrink the Data)

What is it?

Compress API responses before sending them over the network, reducing bandwidth and transfer time.

Compression Algorithms

When to Use Compression

• ✅ Text responses (JSON, HTML, CSS, JS)
• ✅ Responses larger than 1KB
• ❌ Already compressed data (images, videos)
• ❌ Very small responses (< 1KB)

Implementation

// Express.js (Node.js)
const compression = require('compression');
app.use(compression());  // Automatically compress all responses

# Flask (Python)
from flask_compress import Compress
Compress(app)  # Enable compression

# Nginx configuration
gzip on;
gzip_types application/json text/css application/javascript;
gzip_min_length 1000;

Real-World Analogy

Vacuum-sealing clothes before packing them in a suitcase — same clothes, much less space.

Good for: All text-based API responses

Level 10: Content Delivery Network (CDN)

What is it?

A network of servers spread around the world that cache your content closer to users.

Without CDN vs With CDN

Real-World Analogy

Instead of one restaurant in New York, you open franchises worldwide. Customers get the same food from their local branch.

Best For

• Static assets (images, CSS, JS)
• API responses that don’t change per user
• Video streaming

Level 11: Asynchronous Processing

What is it?

For slow operations, accept the request immediately and process it in the background.

Common Use Cases

Tools

• Message Queues: RabbitMQ, Amazon SQS, Redis
• Task Processors: Celery (Python), Sidekiq (Ruby), Bull (Node.js)

Real-World Analogy

At a busy restaurant, the waiter takes your order and immediately moves to the next table. The kitchen processes orders in the background.

Level 12: Asynchronous Logging

What is it?

Logging can slow down your API if it writes to disk synchronously. Async logging writes to a buffer first, then a background thread handles the disk writes.

Performance Impact

Implementation

# Python with async logging
import logging
from logging.handlers import QueueHandler, QueueListener
from queue import Queue

# Create a queue for async logging
log_queue = Queue()

# Main thread uses QueueHandler (non-blocking)
handler = QueueHandler(log_queue)
logger = logging.getLogger()
logger.addHandler(handler)

# Background thread processes the queue
listener = QueueListener(log_queue, logging.FileHandler('app.log'))
listener.start()

# Now logging is async!
logger.info('This is non-blocking')  # < 1ms

// Node.js with async logging (pino)
const pino = require('pino');
const logger = pino({
  transport: {
    target: 'pino/file',
    options: { destination: 'app.log' }
  }
});

// Logs are buffered and written asynchronously
logger.info('This is non-blocking');

Real-World Analogy

Instead of stopping to write down every detail in a notebook, you jot quick notes on sticky notes and organize them into the notebook later.

Good for: High-traffic APIs with extensive logging

Level 13: Database Sharding (Split Your Data)

What is it?

Divide your database into smaller pieces (shards), each handling a portion of the data.

Sharding Strategies

Trade-offs

Real-World Analogy

Instead of one giant warehouse, you have regional warehouses. Each handles orders for their region.

Good for: 100,000 → 1,000,000+ RPS

Level 14: Rate Limiting (Protect Your System)

What is it?

Limit how many requests a user or client can make in a given time period.

Common Algorithms

Rate Limit Headers

HTTP/1.1 200 OK
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 67
X-RateLimit-Reset: 1640995200

Real-World Analogy

A bouncer at a club who only lets in 100 people per hour, no matter how long the line is.

The Complete Picture

Here’s how all these techniques work together at scale:

Scaling Roadmap

Quick Reference: What to Use When

Complete Summary

Bringing It All Together: Your Scaling Journey

Scaling from 1 to 1 million RPS is not a single leap—it’s a series of strategic steps. Here’s how to think about your progression:

Phase 1: Foundation (1-1K RPS)

Focus: Get the basics right

• ✅ Start with vertical scaling (bigger server)
• ✅ Implement connection pooling immediately
• ✅ Fix N+1 query problems
• ✅ Add basic caching for read-heavy operations
• ✅ Enable compression

Mindset: At this stage, a single well-configured server is sufficient. Don’t over-engineer.

Phase 2: Growth (1K-10K RPS)

Focus: Distribute and optimize

• ✅ Add horizontal scaling (multiple servers)
• ✅ Set up a load balancer
• ✅ Implement pagination for all list endpoints
• ✅ Use fast JSON serializers
• ✅ Expand caching strategy

Mindset: Your single server is maxed out. Time to distribute the load.

Phase 3: Scale (10K-100K RPS)

Focus: Edge optimization and async patterns

• ✅ Deploy a CDN for static content and cacheable responses
• ✅ Move slow operations to async workers
• ✅ Implement async logging
• ✅ Optimize database with better indexing and query patterns

Mindset: Every millisecond counts. Optimize the request path and offload work.

Phase 4: Massive Scale (100K-1M+ RPS)

Focus: Database distribution and protection

• ✅ Implement database sharding
• ✅ Add rate limiting to protect against abuse
• ✅ Consider multi-region deployment
• ✅ Implement circuit breakers and failover mechanisms

Mindset: You’re operating at web scale. Focus on reliability, distribution, and resilience.

Critical Success Factors

1. Measure Before You Optimize

The bottleneck you imagine is rarely the real one. Use monitoring and profiling tools:

• Application Performance Monitoring (APM): New Relic, Datadog, or Grafana
• Database Profiling: Identify slow queries with EXPLAIN plans
• Load Testing: Use tools like k6, Gatling, or Apache JMeter

Don’t guess—measure, identify the bottleneck, then fix it.

2. Implement Quick Wins First

Some techniques have massive impact with minimal complexity:

• Connection pooling: 5-10x faster database operations
• Avoid N+1 queries: 40x faster response times
• Compression: 70-85% smaller payloads
• Basic caching: 100x faster for repeated queries

Start here before jumping to complex solutions.

3. Progressive Enhancement Over Big Rewrites

Add capabilities incrementally rather than rewriting everything:

• Add caching without changing your API
• Add horizontal scaling without modifying application code
• Add async processing for new features first

This reduces risk and delivers value continuously.

4. Know When NOT to Scale

• Traffic patterns: Don’t optimize for traffic spikes that happen once a year
• Cost vs. benefit: Sometimes accepting slower responses is cheaper than scaling
• Product stage: Pre-product-market fit? Focus on features, not scale

Premature optimization wastes time and money.

Common Scaling Mistakes to Avoid

❌ Mistake 1: Jumping to Microservices Too Early A well-optimized monolith can handle 10,000+ RPS. Microservices add complexity—only adopt them when you have clear organizational or technical reasons.

❌ Mistake 2: Ignoring Database Optimization Adding more application servers won’t help if your database is the bottleneck. Fix queries first.

❌ Mistake 3: Not Testing Under Load Load test before you have a problem, not during an outage. Know your limits.

❌ Mistake 4: Caching Everything Cache only what’s expensive to compute and read frequently. Over-caching adds complexity.

❌ Mistake 5: Forgetting About Cache Invalidation As Phil Karlton said: “There are only two hard things in Computer Science: cache invalidation and naming things.” Plan your cache invalidation strategy from day one.

Key Takeaways

1. Start Simple: A single optimized server with connection pooling, query optimization, and basic caching can handle 10,000+ RPS. Don’t over-engineer early.

2. Measure First, Optimize Second: Use APM tools and profiling to find your actual bottlenecks. The problem is rarely where you think it is.

3. Low-Hanging Fruit Matters Most:
   • Connection pooling (implement immediately)
   • Avoid N+1 queries (implement immediately)
   • Basic caching (implement early)
   • Pagination (implement for all list endpoints)
   • Compression (enable by default)
4. Progressive Enhancement: Your architecture should evolve with your traffic:
   • 1-1K RPS: Single server + optimizations
   • 1K-10K RPS: Horizontal scaling + load balancing
   • 10K-100K RPS: CDN + async patterns
   • 100K-1M+ RPS: Sharding + multi-region
5. Always Profile: The bottleneck you imagine is rarely the real one. Measure, don’t guess!

Final Thoughts

Scaling is a journey, not a destination. The techniques in this guide will take you from handling your first user to serving millions. The key is to:

• Start with fundamentals (connection pooling, query optimization)
• Add capabilities progressively (don’t jump to advanced techniques)
• Measure constantly (know your bottlenecks)
• Optimize deliberately (focus on impact, not complexity)

Remember: Twitter started as a Ruby on Rails monolith. Facebook started on a single server. Instagram scaled to 30+ million users with just 3 engineers. Good architecture and simple optimizations can take you incredibly far.

Build for today’s needs with an eye toward tomorrow’s scale. When you need the next level, you’ll know—your monitoring will tell you. Until then, keep it simple and keep shipping. 🚀

Continue Your Journey

This guide covered the technical and infrastructure aspects of scaling APIs. But as your organization grows, you’ll face a different kind of scaling challenge: organizational and design complexity.

→ Continue to Part 2: Design & Architecture Strategies to learn about:

• API portfolio management
• Design-first methodologies
• Organizational patterns for large-scale API development
• Versioning strategies
• Governance and evangelism at scale