Handbook

Uber's data infrastructure in 2020:

System 1: Apache Kafka
- 100+ servers
- 3 engineers dedicated to it
- Upgrade every 6 months

System 2: HDFS (Storage)
- 1,000+ servers
- 10 engineers managing
- Constant disk failures

System 3: Apache Hudi
- Built by Uber, maintained by Uber
- 5 engineers full-time

System 4: Apache Spark
- 500+ servers
- 5 engineers managing

System 5: Apache Flink
- 200+ servers
- 4 engineers managing

System 6: Apache Pinot
- 100+ servers
- 3 engineers managing

System 7: Presto
- 200+ servers
- 3 engineers managing

TOTAL:
- 2,000+ servers to manage
- 35+ engineers JUST for infrastructure
- Plus: networking, security, monitoring teams
- Grand total: 300+ engineers

That's an entire company just to run infrastructure!

Engineer costs alone:
- 300 engineers × $200K average salary
- = $60 million/year just in salaries!
- Plus infrastructure costs: $100M/year
- Total: $160M/year

And engineers could be building features instead!

Scenario: Expanding to India (2019)

Traditional approach (what Uber did):
Month 1-2: Find data center in India
Month 3-4: Order servers (long lead times)
Month 5-6: Ship and install hardware
Month 7-8: Configure networking
Month 9-10: Install and test software
Month 11-12: Train local team
Result: 12 months, $20 million

Cloud approach (what they wanted):
Week 1: Enable GCP region in India
Week 2: Deploy infrastructure (copy config)
Week 3: Test
Week 4: Launch
Result: 1 month, $2 million

10x faster, 10x cheaper!

Traditional infrastructure:
- Predict next year's growth: "We'll grow 50%"
- Order servers: 3-6 months lead time
- Install: 2-3 months
- Results: 6-12 months later

What actually happens:
❌ Grow slower than expected? Wasted money on unused servers
❌ Grow faster than expected? Run out of capacity, can't serve users
❌ Seasonal spikes? Must provision for peak (waste 70% of capacity off-peak)

Example:
New Year's Eve: 10x normal traffic
- Need: 10,000 servers for 1 day
- Reality: Must buy 10,000 servers permanently
- Cost: $50M for servers used 1 day/year!

Uber operates in 70+ countries

Regulations:
- EU data must stay in EU (GDPR)
- China data must stay in China
- Brazil data must stay in Brazil
- Australia data must stay in Australia
- Etc.

Uber's challenge:
Need to replicate ENTIRE infrastructure in each region
- 2,000 servers × 10 regions = 20,000 servers
- 300 engineers × 10 regions = 3,000 engineers (impossible!)
- Cost: $160M × 10 = $1.6 billion/year (impossible!)

Reality: Could only expand to new countries very slowly

Before (2020):
- 300 engineers managing infrastructure
- 500 engineers building features
- Ratio: 37% on infrastructure!

What Uber wanted:
- 100 engineers managing infrastructure (cloud)
- 700 engineers building features
- Ratio: 12% on infrastructure

Result: 200 more engineers building features!

Phase 1: Build parallel infrastructure
├─ Old system: Serves production (100% of traffic)
├─ New system: Built in parallel (0% of traffic)
└─ Duration: 6 months

Phase 2: Shadow testing
├─ Old system: Serves production (100% of traffic)
├─ New system: Receives copy of data (0% of traffic)
├─ Compare results: Are they identical?
└─ Duration: 6 months

Phase 3: Gradual migration
├─ Old system: 90% of traffic
├─ New system: 10% of traffic (test users)
├─ Monitor: Any issues?
└─ Duration: 12 months

Phase 4: Full migration
├─ Old system: 0% of traffic (shut down)
├─ New system: 100% of traffic
└─ Duration: 6 months

Total time: 30 months (2.5 years)

Problem: Engineers had thousands of Spark jobs

Bad approach:
"Rewrite all your code for GCP!"
Result: Years of work, high risk of errors

Uber's approach:
Keep the same interfaces:
- Spark jobs run on Dataproc (managed Spark)
- Same code, different infrastructure
- Engineers don't even notice!

Migration complexity:
- Infrastructure team: High (rebuilding everything)
- Data engineers: Low (minimal code changes)
- Data scientists: Zero (no changes needed)

Phase 1: Analytics & Reporting (Non-Critical)
├─ Risk: Low (if it breaks, business continues)
├─ Complexity: Medium
├─ Migrated: Pinot → BigQuery
├─ Duration: 6 months
└─ Learning: How to use BigQuery effectively

Phase 2: Batch Processing (Medium Critical)
├─ Risk: Medium (affects daily reports)
├─ Complexity: High
├─ Migrated: Spark → Dataproc
├─ Duration: 12 months
└─ Learning: Auto-scaling, cost optimization

Phase 3: Storage (Critical)
├─ Risk: High (can't lose data!)
├─ Complexity: Very High
├─ Migrated: HDFS → Cloud Storage + BigQuery
├─ Duration: 12 months
└─ Learning: Data governance, security

Phase 4: Real-Time Systems (Most Critical)
├─ Risk: Very High (affects user experience)
├─ Complexity: Extreme
├─ Migrated: Kafka → Pub/Sub, Flink → Dataflow
├─ Duration: 6 months
└─ Learning: Performance tuning, failover

Total duration: 36 months (3 years)

Before (Kafka):
✅ Performance: 1M events/second
❌ Management: 100+ servers to manage
❌ Scaling: Manual (add servers)
❌ Monitoring: Build custom dashboards
❌ Upgrades: Quarterly maintenance windows
❌ Team: 3 engineers full-time
❌ Cost: $2M/year (servers + engineers)

After (Pub/Sub):
✅ Performance: 1M+ events/second (same or better)
✅ Management: Zero servers
✅ Scaling: Automatic
✅ Monitoring: Built-in dashboards
✅ Upgrades: Google handles it
✅ Team: 0 engineers (Google manages)
✅ Cost: $1M/year (pay per message)

Savings: $1M/year + 3 engineers freed up

Old Kafka code (before):
Properties props = new Properties();
props.put("bootstrap.servers", "kafka1:9092,kafka2:9092,kafka3:9092");
props.put("group.id", "uber-analytics");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

New Pub/Sub code (after):
// Almost identical!
ProjectSubscriptionName subscription = 
  ProjectSubscriptionName.of("uber-project", "uber-analytics");
Subscriber subscriber = Subscriber.newBuilder(subscription, receiver).build();

Migration time: Hours (not months!)

Before (HDFS):
✅ Performance: Good
❌ Capacity: Limited (must buy more disks)
❌ Servers: 1,000+ servers to manage
❌ Disk failures: Weekly (replace failed disks)
❌ Expansion: Order disks 3 months in advance
❌ Team: 10 engineers full-time
❌ Cost: $50M/year (hardware + data centers + engineers)

After (Cloud Storage):
✅ Performance: Excellent (Google's network)
✅ Capacity: Infinite (no limits)
✅ Servers: Zero
✅ Disk failures: Never (Google handles it)
✅ Expansion: Instant (just write more data)
✅ Team: 0 engineers (Google manages)
✅ Cost: $20M/year (pay per GB stored)

Savings: $30M/year + 10 engineers freed up

HDFS (self-managed):
- Servers: 50 servers × $10K each = $500K
- Data center space: $100K/year
- Power & cooling: $200K/year
- Network: $50K/year
- Disk replacements: $100K/year
- Engineers: 1 engineer × $200K = $200K/year
Total: $1.15M/year

Cloud Storage:
- Storage: 1 PB × $20/TB/month = $20K/month
- Data transfer: ~$50K/month
Total: $840K/year

Savings: $310K/year per petabyte!
Plus: No engineers needed, scales instantly

Old system needed 3 tools:
1. Hudi: Store and update data
2. Pinot: Fast analytics queries
3. Presto: SQL queries across systems

BigQuery does all three:
1. Stores petabytes of data ✓
2. Queries in milliseconds ✓
3. Uses standard SQL ✓
Plus:
4. Auto-scales ✓
5. Zero management ✓
6. Integrates with everything ✓

Before:
❌ 3 systems to learn
❌ 3 systems to maintain
❌ Data duplicated 3 times (storage waste)
❌ Complex data syncing
❌ 400+ servers across 3 systems
❌ 11 engineers managing
❌ Query time: 100-500ms (Pinot), 1-10s (Presto)
❌ Cost: $30M/year

After:
✅ 1 system to learn
✅ 0 systems to maintain (Google manages)
✅ Data stored once
✅ No syncing needed
✅ 0 servers
✅ 0 engineers managing
✅ Query time: 100ms - 5s (depending on query)
✅ Cost: $15M/year (pay per query)

Savings: $15M/year + 11 engineers freed up
Complexity reduction: 66% (3 systems → 1)

Stores data in columns (not rows)
Query: "Average fare in NYC"
- Only reads "fare" column
- Ignores other columns
- 10-100x faster than reading full rows

Traditional database:
- Storage and compute tied together
- Must pay for compute 24/7
- Can't scale independently

BigQuery:
- Storage separate from compute
- Pay for compute only when querying
- Auto-scales compute for each query

Example:
Small query: Uses 10 nodes, costs $0.10
Huge query: Uses 1,000 nodes, costs $10
Both finish in ~same time!

BigQuery automatically:
- Chooses best indexes
- Partitions data optimally
- Compresses data
- Caches frequent queries
- Parallelizes queries

Engineer's job:
- Just write SQL
- BigQuery optimizes everything

Before (Self-Managed Spark):
❌ Cluster setup: 2-4 weeks
❌ Always-on clusters: Waste money during idle time
❌ Scaling: Manual (provision more nodes)
❌ Monitoring: Build custom tools
❌ Upgrades: Complex, risky
❌ Cost: $40M/year (servers + engineers)
❌ Team: 5 engineers managing

After (Dataproc):
✅ Cluster setup: 90 seconds!
✅ Ephemeral clusters: Spin up for job, shut down after
✅ Scaling: Auto-scales based on job size
✅ Monitoring: Built-in dashboards
✅ Upgrades: Click button (or automatic)
✅ Cost: $20M/year (pay per job)
✅ Team: 1 engineer managing

Savings: $20M/year + 4 engineers freed up

Morning (8am): 
- Cluster: Running (100 nodes)
- Utilization: 80%
- Cost: $100/hour

Midday (12pm):
- Cluster: Running (100 nodes)
- Utilization: 90%
- Cost: $100/hour

Night (2am):
- Cluster: Running (100 nodes)
- Utilization: 5% (mostly idle!)
- Cost: $100/hour (wasting $95/hour!)

Daily cost: $2,400
Wasted: ~$1,000/day (idle time)
Monthly waste: $30,000!

Job 1 (8:00am): Need 50 nodes for 30 minutes
- Spin up 50-node cluster
- Run job
- Shut down cluster
- Cost: $25

Job 2 (9:00am): Need 200 nodes for 15 minutes
- Spin up 200-node cluster
- Run job
- Shut down cluster
- Cost: $50

Job 3 (10:00am): Need 20 nodes for 1 hour
- Spin up 20-node cluster
- Run job
- Shut down cluster
- Cost: $20

Night (2am): No jobs
- No clusters running
- Cost: $0 ✅

Daily cost: ~$1,200
Savings: 50% compared to always-on!

Before (Flink):
✅ Performance: Excellent (sub-100ms)
❌ Complexity: Very high (hardest system to manage)
❌ Scaling: Manual, tricky
❌ State management: Complex (where to store state?)
❌ Failover: Build custom logic
❌ Team: 4 engineers full-time
❌ Cost: $15M/year

After (Dataflow):
✅ Performance: Excellent (sub-100ms, same as Flink)
✅ Complexity: Low (Google manages)
✅ Scaling: Auto-scales automatically
✅ State management: Handled by Google
✅ Failover: Automatic
✅ Team: 0.5 engineers (part-time)
✅ Cost: $10M/year

Savings: $5M/year + 3.5 engineers freed up

HDFS (on-premises):
- Local disk access: 1-5ms latency
- Network within data center: <1ms

Cloud Storage (GCP):
- Network access: 10-50ms latency
- Across internet: Can spike higher

Result: Some jobs 2-5x slower!

1. Use persistent disks for hot data
   - Frequently accessed data on fast disks
   - Cold data in Cloud Storage

2. Optimize data layout
   - Partition data effectively
   - Use compressed formats (Parquet)

3. Leverage BigQuery
   - Instead of reading files, query BigQuery
   - BigQuery optimized for cloud access

Result: Performance back to baseline (or better!)

First month on GCP:
- Bill: $8M (expected: $5M)
- Overage: $3M!

Why?
- Data egress charges (moving data out)
- Inefficient query patterns
- Always-on development clusters
- Not using reserved instances

1. Reserved instances
   - Commit to baseline usage: 40% discount

2. Committed use discounts
   - Commit to 1-3 years: 50-70% discount

3. Query optimization
   - Use partitioned tables
   - Limit SELECT * queries
   - Cache frequent queries

4. Ephemeral dev clusters
   - Auto-shut down after 2 hours idle
   - Saved $500K/month!

5. Data lifecycle policies
   - Move old data to cold storage
   - Delete temp data automatically

Result: Reduced costs to $5M/month target

GDPR compliance:
- EU users' data must stay in EU
- Can't accidentally move to US

Risk:
- Data engineer runs query
- BigQuery job in US region
- Reads EU data
- Violation! Huge fines!

1. Regional isolation
   - EU data: Only in EU regions
   - US data: Only in US regions
   - Enforced with IAM policies

2. Data access proxies
   - Engineers don't access data directly
   - Go through proxy that enforces rules
   - Proxy logs all access

3. Automated compliance checks
   - Daily scans for violations
   - Alert security team immediately
   - Auto-remediation where possible

Result: Zero compliance violations!

300 data engineers used to:
- Spark on YARN
- Hive SQL
- Custom tools

Now need to learn:
- BigQuery SQL (different dialect)
- Dataproc
- Cloud Console
- New monitoring tools

1. Gradual transition
   - Keep old system running during training
   - Learn at own pace

2. Internal training program
   - "GCP Boot Camp": 2-week intensive course
   - Hands-on labs
   - Office hours with experts

3. Internal documentation
   - Migration guides
   - Best practices
   - Common pitfalls

4. Champions program
   - Early adopters help teammates
   - Share lessons learned

5. Abstraction layers
   - Build wrappers that hide complexity
   - Engineers write same code, runs on GCP

Result: 90% of engineers productive in 3 months

Before (2020):
Infrastructure: $100M/year
- Servers: $50M
- Data centers: $30M
- Network: $10M
- Maintenance: $10M

Engineers: $60M/year
- 300 engineers × $200K average

Total: $160M/year

After (2024):
Infrastructure: $60M/year
- GCP services: $60M
- (No servers, data centers, etc.)

Engineers: $20M/year
- 100 engineers × $200K average
- 200 engineers redeployed to product work

Total: $80M/year

SAVINGS: $80M/year (50% reduction!)

Development time:
Before: 3-6 months to launch new data product
- Provision infrastructure: 1-2 months
- Setup and configure: 1 month
- Develop: 1-2 months
- Test and deploy: 1 month

After: 2-4 weeks
- Provision infrastructure: 0 days (instant!)
- Setup and configure: 1 day
- Develop: 1-2 weeks
- Test and deploy: 1 week

Improvement: 5-10x faster!

Scaling time:
Before: 3-6 months
- Plan capacity
- Order servers
- Install and configure
- Deploy

After: Automatic
- Traffic spike? Auto-scales in minutes
- New region? Enable in days
- Seasonal demand? Scales up/down automatically

Improvement: 100x faster (months → minutes)

System uptime:
Before: 99.9% (9 hours downtime/year)
- Hardware failures
- Human errors
- Upgrade issues

After: 99.99% (52 minutes downtime/year)
- Google's infrastructure
- Automatic failover
- No manual upgrades

Improvement: 10x fewer outages

Data durability:
Before: 99.99% (risk of data loss)
- 3 copies of data
- But: Data center fire = could lose region

After: 99.999999999% (11 nines!)
- Google replicates across regions
- Virtually impossible to lose data

Improvement: 10,000x more durable

New region setup:
Before: 6-12 months, $10-20M
- Find/build data center
- Order and install hardware
- Hire local team
- Deploy and configure

After: 1-2 weeks, $1-2M
- Enable GCP region
- Copy configuration
- Test
- Launch

Improvement: 10-20x faster, 10x cheaper

Uber's expansion since migration (2020-2024):
- Launched in 20 new countries
- Doubled total markets (35 → 70 countries)
- Total cost: $30M (would have been $300M before!)

The GCP migration enabled Uber's global expansion!

Before migration:
- 300 engineers on infrastructure
- 500 engineers on product
- Ratio: 37% infrastructure

After migration:
- 100 engineers on infrastructure
- 700 engineers on product
- Ratio: 12% infrastructure

Impact:
- 200 engineers freed up
- Building features, not managing servers
- Faster innovation
- More products shipped

1. Real-time fraud detection
   - Needed: Scale to analyze 10M transactions/second
   - Before: Impossible (limited capacity)
   - After: Built on Dataflow (auto-scales)

2. Global demand forecasting
   - Needed: Process 1 PB of data daily
   - Before: Would take 12 hours (too slow)
   - After: BigQuery processes in 1 hour

3. AI-powered pricing
   - Needed: Train models on 5 years of data
   - Before: Would take weeks, huge compute cost
   - After: Dataproc spins up 10,000 nodes, done in hours

4. Real-time driver bonuses
   - Needed: Calculate bonuses in real-time
   - Before: Batch processing (next day)
   - After: Dataflow processes in milliseconds

These features wouldn't exist without GCP migration!

Uber's approach:
Year 1: Migrate 10% of workloads
- Learn the platform
- Discover issues
- Build confidence

Year 2: Migrate 50% of workloads
- Apply lessons learned
- Optimize costs
- Train teams

Year 3: Migrate remaining 40%
- Tackle critical systems last
- After gaining experience

What NOT to do:
❌ "Big bang" migration (switch everything overnight)
❌ Migrate critical systems first
❌ Rush to meet arbitrary deadline

Result: Zero major incidents!

The "shadow mode" approach:
1. New system receives copy of all data
2. Both systems process in parallel
3. Compare results continuously
4. Only switch when identical for 30 days

Benefits:
✅ No risk to production
✅ Discover issues before they impact users
✅ Can roll back instantly if needed
✅ Build confidence gradually

Cost:
❌ Running two systems = 2x cost temporarily
✅ But: Worth it for zero-risk migration

Uber's timeline:
- Parallel run: 6 months per system
- Extra cost: $30M
- Value: Priceless (zero production incidents)

Uber's cost journey:
Month 1: $8M (60% over budget!)
- Haven't optimized yet
- Learning platform

Month 3: $7M (40% over budget)
- Applied some optimizations
- Still learning

Month 6: $6M (20% over budget)
- Most optimizations done
- Getting there

Month 12: $5M (on budget!)
- Fully optimized
- Using reserved instances
- Efficient query patterns

Key insight: Budget for learning curve!

Uber's training investment:
- 2-week boot camps: $500K
- Internal documentation: $200K
- Office hours (expert time): $300K
- External trainers: $100K
Total: $1.1M

Return on investment:
- Faster adoption: Saved 6 months
- Fewer errors: Prevented costly mistakes
- Better optimization: Saved $3M/year
ROI: 300% in first year!

Lesson: Training pays for itself quickly

Uber's two-phase approach:

Phase 1: Lift-and-Shift (Year 1-2)
- Move to GCP IaaS
- Minimal changes to code
- Fast migration
- Benefits: Faster scaling, less management

Phase 2: Modernization (Year 2-4)
- Adopt PaaS services (BigQuery, Dataflow)
- Refactor applications
- Slower, requires code changes
- Benefits: Lower costs, better performance

Why two phases?
✅ Phase 1: Quick wins, low risk
✅ Phase 2: Maximize benefits
❌ Trying to do both at once: Too complex, high risk

Current status (2024):
- 70% modernized to PaaS
- 30% still on IaaS
- Target: 90% PaaS by 2026

Infrastructure:
✅ 100% on GCP
✅ 0 self-managed servers
✅ 70% using PaaS services
✅ 30% using IaaS (still migrating)

Cost:
✅ $80M/year (down from $160M)
✅ 50% cost reduction
✅ Still optimizing

Team:
✅ 100 engineers on infrastructure (down from 300)
✅ 200 engineers redeployed to product
✅ Faster innovation

Goal: Move remaining 30% from IaaS to PaaS
- Replace remaining self-managed Spark → Dataproc
- Migrate all storage to BigQuery/Cloud Storage
- Adopt Vertex AI for ML workloads

Expected benefits:
- Additional 20% cost savings ($16M/year)
- Reduced complexity

Goal: Don't depend on single cloud provider
- Primary: GCP (current)
- Secondary: AWS (for specific workloads)
- Disaster recovery: Azure

Benefits:
- Avoid vendor lock-in
- Negotiate better pricing
- Resilience (if one cloud goes down)

Challenge:
- Increased complexity
- Team needs to learn multiple clouds

Goal: Move from batch to real-time processing
- Real-time earnings (not next-day)
- Real-time fraud detection (not minutes)
- Real-time analytics (not hourly)

Enabled by:
- Dataflow's streaming capabilities
- BigQuery's streaming inserts
- Pub/Sub's low latency

Expected benefits:
- Better user experience
- Faster fraud detection
- More accurate pricing

Goal: Leverage AI for everything
- Demand forecasting
- Dynamic pricing optimization
- Route optimization
- Driver-rider matching
- Fraud detection
- Customer support automation

Enabled by:
- Vertex AI (managed ML platform)
- BigQuery ML (SQL-based ML)
- TensorFlow on Dataproc

Investment: $50M over 3 years
Expected ROI: $200M in increased efficiency

2015-2020: Built Custom Infrastructure
- 7 specialized systems
- 2,000+ servers
- 300 engineers
- $160M/year
- 99.9% uptime

2020-2023: Migrated to GCP
- 3-year journey
- $30M migration cost
- Zero production incidents
- Phased approach

2024: Fully on Cloud
- 5 main GCP services
- 0 servers
- 100 engineers
- $80M/year
- 99.99% uptime

Transformation:
- 50% cost reduction
- 10x faster to scale
- 5-10x faster development
- 10x better reliability
- 200 engineers freed for product work

DON'T build like old Uber:
❌ Self-managed infrastructure
❌ Custom systems
❌ Years of engineering effort

DO start with cloud:
✅ Use managed services (BigQuery, etc.)
✅ Scale automatically
✅ Focus on product, not infrastructure

Your tech stack should be:
Phase 1 (0-100K users): $100-1K/month
- PostgreSQL (managed)
- Redis (managed)
- Simple analytics

Phase 2 (100K-1M users): $1K-10K/month
- Cloud SQL or Aurora
- Redis/Memcached
- BigQuery for analytics

Phase 3 (1M-10M users): $10K-100K/month
- BigQuery for data warehouse
- Pub/Sub for events
- Dataflow for processing

Only at 10M+ users:
- Consider what Uber does
- But still on managed services!

Migration timeline:
Year 1: Plan and pilot
- Assess current infrastructure
- Calculate TCO (total cost of ownership)
- Run pilot projects
- Build business case

Year 2-3: Migrate
- Start with non-critical systems
- Run in parallel
- Train team
- Optimize costs

Year 4: Complete and optimize
- Finish migration
- Shut down old infrastructure
- Celebrate savings!

Expected savings:
- 30-50% cost reduction
- 5-10x faster development
- Fewer engineers on infrastructure

Uber's proven approach:
1. Phased migration (3 years)
2. Run in parallel (shadow mode)
3. Migrate by risk level (low to high)
4. Invest in training ($1M+)
5. Start with lift-and-shift
6. Then modernize to PaaS
7. Optimize costs continuously

Expected outcomes:
- 40-60% cost savings (long-term)
- Faster innovation
- Global expansion enabled
- Better reliability

Investment required:
- Migration cost: $10-100M (depending on size)
- Time: 2-5 years
- ROI: 2-3 years