Handbook

Modernizing Data Infrastructure: From Legacy EDW to Unified Platform

Your data warehouse is becoming your biggest bottleneck. The on-prem EDW that powered your business for the last decade can’t keep up with AI, real-time analytics, and cloud-scale workloads. It’s expensive to scale, slow to adapt, and wasn’t built for the demands of modern data and ML.

AI needs a unified data foundation. You can’t train models on fragmented data. You can’t govern data across three different systems. You can’t move fast when teams spend days reconciling numbers and searching for datasets.

The reality:

You have a legacy on-prem EDW built over 10-15 years. It holds critical historical data, runs finance and compliance workloads, and has hundreds of reports depending on it. You can’t just “lift and shift” it to the cloud — the risk is too high, the cost is unclear, and the business won’t tolerate downtime.

You also have an on-prem Enterprise Data Lake (EDL) — typically Cloudera Data Platform (CDP) with Hadoop and Spark. It was built to handle big data workloads that the EDW couldn’t, but it’s become another silo with its own data, governance, and access patterns.

But you also can’t scale on-prem anymore. Storage is expensive, compute is limited, and your team wants cloud-native tools and flexibility. So you’ve added cloud platforms — Snowflake for analytics, Databricks for engineering and ML.

Now you have a hybrid mess:

• Your new cloud infrastructure needs to fetch data from both EDW and CDP/EDL
• Data exists in three places: EDW (structured), CDP/Hadoop (big data), and cloud (Snowflake/Databricks)
• Teams build the same pipelines multiple times (EDW, CDP, cloud)
• Data is duplicated, governance diverges, and nobody knows which system has the “truth”
• You’re paying for four platforms (EDW + CDP + Snowflake + Databricks) but getting one fragmented experience

You can’t put all data in the cloud. Compliance, latency, and legacy dependencies keep some data on-prem.

You can’t stay fully on-prem. You need cloud scale, modern tools, and flexibility to power AI and modern analytics.

The solution: A unified data infrastructure layer built on on-prem object storage + Iceberg + Spark + federated query that makes EDW + Snowflake + Databricks feel like one platform — giving you the strong data foundation needed to scale AI, without forcing a big-bang migration.

The core layer:

• On-prem object storage (MinIO, Ceph, Dell ECS) as your lakehouse foundation
• Apache Iceberg for open table format (works everywhere)
• Spark for transformations (on-prem or cloud)
• Federated query (Trino/Presto) to query across on-prem and cloud seamlessly

Think of it like a city with old and new infrastructure. You don’t tear down the old water system overnight. You build new pipes (object storage + Iceberg) that connect to the old ones (EDW), standardize the quality checks (Spark), and give everyone one map (federated query) to find what they need — even if data lives both on-prem and in cloud.

The Pain: Which Platform Should We Use?

Problem 1: Platform Paralysis

Data Engineer: "Where should I build this new pipeline?"
  Option A: EDW (legacy, but finance depends on it)
  Option B: Databricks (modern, but not everyone has access)
  Option C: Snowflake (fast, but expensive for large transforms)

Result: 3-day debate, no decision, project delayed

The real issue: You don’t have a clear answer to “which platform for what workload?”

Problem 2: Every Team Picks a Different Platform

Finance team: "We only trust the EDW"
Data Science team: "We only use Databricks"
BI team: "We only use Snowflake"
Analytics Engineering: "We use dbt... but where do we run it?"

Result: Data scattered across 3 platforms, no one can find anything

The real issue: No unified strategy, every team optimizes locally.

Problem 3: Same Dataset, Three Different Versions

Customer data exists in:
- EDW: customer_master (legacy schema)
- Databricks: silver.customers (Delta Lake)
- Snowflake: analytics.dim_customer (star schema)

Which one is the source of truth? Nobody knows.

Result: Teams build their own versions, making it worse

The real issue: No single system of record for curated data.

Problem 4: Platform Lock-In Fears

Executive: "If we go all-in on Snowflake, what if we need to move?"
Architect: "If we standardize on Databricks, can BI teams still use Tableau?"
CFO: "Why are we paying for three platforms?"

Result: Analysis paralysis, no modernization happens

The real issue: Fear of vendor lock-in prevents any decision.

Problem 5: New Workloads Don’t Know Where to Go

ML Engineer: "I need to train a model on customer data"
  - EDW has historical data (but no GPU, no Spark)
  - Databricks has ML tools (but data is in EDW)
  - Snowflake has some data (but expensive for training)

Result: Copy data everywhere, waste time, duplicate storage costs

The real issue: No clear platform strategy for different workload types.

The root cause? You have multiple platforms, but no unified strategy or clear decision framework for which platform to use for what.

The Solution: One Logical Platform (Not One Physical System)

Key insight: You don’t need to replace everything with one tool. You need to standardize how things work together — including on-prem platforms like EDL/CDP, EDW, and cloud platforms like Snowflake and Databricks.

What makes it feel like “one platform”? You standardize:

• How data lands (ingestion contracts)
• How it’s stored (one open table format + consistent zones)
• How it’s governed (one catalog + one policy model)
• How it’s observed (lineage + data quality)
• How it’s consumed (semantic layer + APIs)

If you do just that, you can modernize without a big-bang migration.

The 2-Plane Model (Control Plane vs Data Plane)

Think of it like air traffic control. The control tower (Control Plane) coordinates all the planes (Data Plane), but the planes still fly independently. The tower ensures they don’t crash into each other and follow the same rules.

Most enterprises end up with this split:

• Control Plane: the unified experience (catalog, policies, contracts, quality, lineage, portal)
• Data Plane: where data + compute actually live (EDW / Snowflake / Databricks / object storage)

The Control Plane is what makes teams say: “This feels unified.”

The Data Plane is where the actual work happens (storage + compute).

How Data Flows Through the System

Real example: A customer places an order on your website. Let’s trace that data through the entire platform.

Step-by-step:

1. Order placed → captured by CDC (Change Data Capture)
2. Ingestion → validated against contract, published to Landing zone
3. Curation → cleaned, joined with customer data, published to Curated zone
4. Data Products → domain team builds “Order Analytics” product
5. Consumption → served to Snowflake (BI), Databricks (ML), EDW (Finance)

Key point: Data flows through one path, but can be consumed by multiple engines.

The Lakehouse Zones (Bronze → Silver → Gold)

Think of it like a factory assembly line. Raw materials come in (Bronze), get processed (Silver), and become finished products (Gold).

Bronze Zone (Landing)

Purpose: Store raw data exactly as it arrived

Characteristics:

• Immutable (never modified)
• Source-aligned (one table per source table)
• No transformations
• Keep everything (even bad data goes to quarantine)

Example:

s3://lakehouse/bronze/orders/
  - raw_orders_2024_01_11.parquet
  - raw_orders_2024_01_12.parquet

Silver Zone (Curated)

Purpose: Cleaned, standardized, ready for analytics

Characteristics:

• Data quality checks passed
• Standardized formats (dates, currencies, etc.)
• Conformed dimensions (one customer table, not five)
• Deduplicated

Example:

s3://lakehouse/silver/orders/
  - clean_orders/ (Iceberg table)
  - clean_customers/ (Iceberg table)

Gold Zone (Products)

Purpose: Business-ready datasets for specific use cases

Characteristics:

• Domain-owned (Finance owns revenue tables)
• Documented + versioned
• Optimized for consumption
• SLAs enforced

Example:

s3://lakehouse/gold/finance/
  - revenue_daily/ (dimensional model)
  - customer_lifetime_value/ (aggregate)

Reference Architecture: The 7 Layers

Each layer solves a specific problem. You can implement them incrementally (you don’t need all 7 on day one).

Layer A: Connectivity and Ingestion

Goal: Move data reliably between on-prem and cloud, batch + streaming

Typical sources:

• On-prem databases (CDC / log-based replication)
• Legacy file drops (SFTP, NFS)
• Application events / streams (Kafka, Kinesis)
• SaaS and cloud databases (Salesforce, Workday, etc.)

Standard pattern: Treat ingestion as pipelines with contracts

A contract should include:

Pipeline shape (standardize this everywhere):

ingest → validate → quarantine → publish

Why this matters: This single pattern prevents “every team reinvents ingestion differently.”

Layer B: Unified Storage (Lakehouse Foundation)

Goal: One consistent data representation across platforms using object storage + Iceberg

The core architecture: On-prem object storage as your lakehouse

🏢 On-Prem Object Storage (Primary Option)
├── MinIO (S3-compatible, open source)
├── Ceph (enterprise-grade, scalable)
├── Dell ECS (enterprise storage)
└── NetApp StorageGRID (hybrid cloud)

+ Apache Iceberg tables
+ Spark for transformations
+ Trino/Presto for federated query

Why on-prem object storage?

• ✅ Data stays on-prem (compliance, latency, governance)
• ✅ S3-compatible API (works with all modern tools)
• ✅ No cloud egress costs (data doesn’t leave your datacenter)
• ✅ Scales like cloud (petabyte-scale, elastic)
• ✅ Works with Iceberg (same as S3/ADLS/GCS)

Use a single “platform default” open table format:

• Apache Iceberg (recommended - works everywhere: on-prem, Snowflake, Databricks, Trino)
• Delta Lake or Hudi can work too, but pick one to reduce fragmentation

Why open format (Iceberg)?

Proprietary format (Snowflake-only):
- Data locked in Snowflake
- Can't use Databricks without copying
- Can't query from on-prem Spark
- Vendor lock-in

Open format (Iceberg on object storage):
- On-prem Spark can read it ✅
- Snowflake can read it (external tables) ✅
- Databricks can read it ✅
- Trino/Presto can read it ✅
- Future tools can read it ✅

Key rule: Object storage (on-prem or cloud) becomes your system-of-record for analytical data going forward.

Deployment options:

1. On-prem only: Object storage + Iceberg on-prem, federated query to cloud when needed
2. Hybrid: Primary data on-prem, replicate to cloud for specific workloads
3. Cloud primary: S3/ADLS/GCS, replicate back to on-prem if needed

The directional goal:

• Publish curated datasets to object storage (on-prem or cloud) in Iceberg format
• Let Spark, Snowflake, Databricks, Trino all read from that same truth
• Use federated query (Trino/Presto) to query across locations seamlessly

Detailed Hybrid Architecture: Object Storage + Iceberg Internals

This shows how on-prem and cloud connect, and what’s inside the lakehouse storage layer.

Key components explained:

1. On-prem object storage (Primary): MinIO/Ceph/Dell ECS as your lakehouse foundation
   • S3-compatible API (works with all modern tools)
   • Data stays on-prem (compliance, latency, governance)
   • Holds Bronze/Silver/Gold zones in Iceberg format
2. Iceberg structure:
   • Metadata files: Track table schema, snapshots, and versions
   • Manifest files: Point to actual data files (like an index)
   • Data files: Immutable Parquet files with the actual data

3. On-prem Spark: Transformations run on-prem, read/write Iceberg tables directly

4. Trino/Presto (Federated Query): Query across on-prem and cloud seamlessly
   • Single SQL query can join on-prem Iceberg tables with cloud Snowflake tables
   • No data movement required
   • Users don’t need to know where data lives

5. Catalog: Registers all tables (on-prem and cloud) and tracks lineage

6. Cloud compute engines: Databricks and Snowflake can read Iceberg tables (via replication or federated query)

7. Optional cloud replication: Replicate specific datasets to cloud for cloud-native workloads

Why this works:

• Primary data stays on-prem (compliance, cost, control)
• Multiple engines can read it (Spark, Trino, Snowflake, Databricks via Iceberg)
• Federated query eliminates need to copy data everywhere
• Catalog knows about everything (unified discovery across on-prem and cloud)
• Gradual cloud adoption (replicate only what needs to be in cloud)

Layer C: Compute (Multi-Engine, Same Rules)

Goal: Allow Snowflake and Databricks to coexist without duplicated logic or governance

Use each engine for what it’s good at:

Important rule: Avoid building two different curated layers (one in Snowflake and one in Databricks) that drift.

Prefer:

• Curate once in open format (Iceberg on on-prem object storage)
• Serve through either engine (via VPN or replication)
• Build engine-specific marts only when there’s a strong reason (cost, performance, concurrency)

Layer C+: Federated Query (Query Anywhere, Data Stays Put)

Goal: Query data across on-prem and cloud without moving it

The problem: You have data in on-prem object storage (Iceberg), Snowflake, Databricks, and legacy EDW. Users need to join data across all of them. Copying data everywhere is expensive and slow.

The solution: Federated query engine (Trino or Presto)

What is federated query?

A single SQL query that can access multiple data sources simultaneously:

-- Query joins on-prem Iceberg table with cloud Snowflake table
-- No data movement required!

SELECT 
    o.order_id,
    o.amount,
    c.customer_name,
    c.segment
FROM onprem_lakehouse.gold.orders o
JOIN snowflake.analytics.customers c
    ON o.customer_id = c.customer_id
WHERE o.order_date >= '2024-01-01'

How it works:

Trino/Presto connectors:

Real-world example:

Business question: "What's the revenue by customer segment for Q1 2024?"

Data needed:
- Orders (on-prem Iceberg lakehouse)
- Customers (Snowflake)
- Product catalog (Databricks)
- Revenue adjustments (legacy EDW)

Traditional approach:
1. Copy orders to Snowflake (2 hours)
2. Copy product catalog to Snowflake (1 hour)
3. Copy revenue adjustments to Snowflake (30 min)
4. Run query in Snowflake
Total: 3.5 hours + storage costs

Federated query approach:
1. Run Trino query joining all 4 sources
Total: 5 minutes, no data movement

When to use federated query:

✅ Use federated query when:

• Ad-hoc analytics across multiple sources
• Exploratory data analysis
• One-time reports
• Data validation (comparing on-prem vs cloud)
• Low-latency requirements (< 1 minute)

❌ Don’t use federated query when:

• High-frequency queries (> 100/sec)
• Complex aggregations on large datasets
• Production dashboards with strict SLAs
• → Instead: Replicate data to a single location and optimize there

Deployment:

On-Prem Trino Cluster:
- 1 coordinator node (query planning)
- 5-10 worker nodes (query execution)
- Connects to: on-prem Iceberg, Snowflake (via VPN), Databricks (via VPN), EDW

Cost: ~$50K-100K/year (hardware + maintenance)
Benefit: Query across all systems without data movement

Layer D: Metadata, Catalog, and Discovery

Goal: One place to discover, understand, and trust data

Think of it like Google for your data. Instead of asking “Does anyone know where the customer table is?”, you search the catalog and find it in 10 seconds.

Your catalog experience should hold:

Example catalog entry:

Table: customer_360
Location: s3://lakehouse/gold/customer/customer_360/
Format: Iceberg
Owner: Customer Analytics Team
Domain: Customer
Tags: PII, Confidential
Quality Score: 98%

Schema:
  customer_id (int) - Primary key
  email (string) - PII, masked for non-finance
  lifetime_value (decimal) - Calculated monthly
  last_purchase_date (date)

Lineage:
  ← raw_customers (bronze)
  ← raw_orders (bronze)
  → revenue_dashboard (Tableau)
  → churn_model (Databricks ML)

Access:
  Finance team: Full access
  Marketing team: Email masked
  External contractors: No access

This is the “unified layer” people feel daily.

Layer E: Governance and Security (Policy Everywhere)

Goal: Consistent access control across all stores/engines

The problem: You set up masking in Snowflake, but forget to do it in Databricks. Now PII is leaking.

The solution: Central policy engine that enforces rules everywhere.

Design around:

• Identity: SSO + groups/roles (use existing AD/Okta)
• Authorization: RBAC + ABAC (tags like PII, PCI, Confidential)
• Masking / row filtering: Enforced centrally where possible
• Key management: Encryption at rest + in transit (CMK if needed)

Example policy:

Policy: mask_customer_email
Applies to: Tables tagged with "PII"
Rule:
  - IF user.role IN ["Finance", "Legal"]
    THEN show full email
  - ELSE mask email (show first 3 chars + "***@***.com")
  
Enforcement:
  - Snowflake: Column masking policy
  - Databricks: Dynamic view with CASE statement
  - EDW: Row-level security view

Best practice: Use a Policy Decision Point (PDP) with engine-specific Policy Enforcement Points (PEPs).

That’s how you stop “Snowflake rules” and “Databricks rules” from drifting.

Layer F: Data Product and Consumption

Goal: Stop everyone querying raw tables forever

Think of data products like APIs. You don’t give everyone direct database access. You publish well-defined products with contracts.

Publish Data Products (domain-owned) with:

• Contract + schema guarantees
• Documentation + examples
• Versioning + deprecation
• Access modes (SQL/BI, APIs, streaming topics)

Example data product:

Product: customer_360
Owner: Customer Analytics Team
Version: 2.1.0
SLA: Updated daily by 6 AM EST

Contract:
  - Schema: {customer_id, email, lifetime_value, segment, ...}
  - Freshness: < 24 hours
  - Quality: > 95% completeness
  - Breaking changes: 30-day deprecation notice

Access modes:
  - SQL: SELECT * FROM gold.customer_360
  - API: GET /api/v2/customers/{id}
  - Streaming: kafka://customer-updates

Documentation:
  - README: How to use this product
  - Examples: Common queries
  - Changelog: What changed in v2.1.0

Add a semantic layer for consistent metrics:

Metric: Revenue
Definition: SUM(order_amount) WHERE order_status = 'completed'
Owner: Finance Team
Used in: 47 dashboards

This ensures "Revenue" means the same thing everywhere.

Consumption modes:

• BI/SQL: Snowflake or SQL over lakehouse (Tableau, Looker, Power BI)
• ML/features: Databricks pipelines (feature engineering, model training)
• Real-time services: Streams + materialized views (fraud detection, recommendations)

Layer G: Observability and Operations

Goal: Treat data pipelines like production software

If your website goes down, you get paged. If your data pipeline fails, you should also get paged.

Platform-wide monitoring:

Example alert:

🚨 ALERT: Data Quality Violation

Pipeline: customer_360_daily
Issue: NULL values in customer_id column
Impact: 1,247 rows affected (2.3% of total)
SLA: Quality threshold is 98%, currently at 97.7%

Action required:
1. Check upstream raw_customers table
2. Review data quality rules
3. Quarantine bad records

Runbook: https://wiki.company.com/data/runbooks/customer-360
On-call: @data-platform-oncall

Concrete Enterprise Deployment Shape

Control Plane (Unified Data Infra Layer)

This is what you build once and everyone uses:

• Catalog + lineage + glossary (e.g., DataHub, Atlan, Collibra)
• Policy engine + RBAC/ABAC + masking (e.g., Immuta, Privacera, Open Policy Agent)
• Data contracts + schema registry (e.g., Confluent Schema Registry, custom)
• Quality framework + scorecards (e.g., Great Expectations, Monte Carlo, Soda)
• Orchestration standards (e.g., Airflow, Prefect, Dagster)
• Self-serve portal (e.g., Backstage, custom)

Data Plane (Where Data Lives)

This is where the actual data and compute happen:

On-Prem:

• Object storage (MinIO/Ceph/Dell ECS with Iceberg tables)
• Spark (transformations on-prem, can use CDP Spark)
• CDP/Hadoop (legacy EDL, gradual modernization)
• EDW (legacy, gradual modernization)

Cloud:

• Databricks (engineering + ML workloads)
• Snowflake (analytics marts + BI workloads)

Implementation Strategy (Avoid a 2-Year Project)

Start small, deliver value quickly, expand incrementally.

Phase 1: Unify Discovery + Governance First (Months 1-3)

Goal: Give everyone one place to find and understand data

What to build:

• Deploy catalog tool (DataHub, Atlan, etc.)
• Register existing datasets (EDW, Snowflake, Databricks)
• Capture lineage for top 20 critical pipelines
• Define RBAC/ABAC tags (PII, Confidential, etc.)
• Set up data product registration process

Success metrics:

• 80% of datasets discoverable in catalog
• Lineage visible for critical pipelines
• Teams can request access via portal

Effort: 2-3 engineers, 3 months

Phase 2: Unify the Curated Layer (Months 4-9)

Goal: One source of truth for curated data

What to build:

• Adopt Iceberg as platform standard
• Migrate top 10 curated datasets to Iceberg on S3
• Connect Snowflake to read Iceberg tables (external tables)
• Connect Databricks to read Iceberg tables (native)
• Deprecate duplicate curated tables

Success metrics:

• Top 10 datasets available in open format
• Both Snowflake and Databricks reading from same tables
• 50% reduction in duplicate datasets

Effort: 3-4 engineers, 6 months

Phase 3: Modernize EDW Usage (Months 10-18)

Goal: Reduce EDW dependency, move workloads to lakehouse

What to build:

• Identify EDW workloads that can move to lakehouse
• Migrate non-critical reports to Snowflake/Databricks
• Keep critical finance/compliance workloads on EDW
• Set up bi-directional sync (EDW ↔ Lakehouse)

Success metrics:

• 30% of EDW workloads migrated
• EDW costs reduced by 20%
• Critical workloads still running on EDW

Effort: 4-5 engineers, 9 months

EDW Migration Strategies: How to Retire Your Legacy Platform

The goal is not to replace the EDW overnight. The goal is to gradually reduce dependency until you can retire it safely.

Critical Principle: Invert the Dependency

The biggest mistake: Building your new modern system (DataHub, Snowflake, Databricks) that depends on the old EDW for data.

❌ WRONG: New system depends on old system

Sources → EDW (old) → Lakehouse (new) → Snowflake/Databricks
                ↓
         DataHub catalogs EDW tables

Problem: You can NEVER retire the EDW because everything depends on it.

The right approach: Invert the dependency. Make the lakehouse the source of truth, and let the EDW sync FROM the lakehouse if needed.

✅ RIGHT: Old system depends on new system (or coexists independently)

Sources → Lakehouse (new) → Snowflake/Databricks
            ↓                     ↓
         DataHub            (Optional: EDW syncs FROM lakehouse
         catalogs           for legacy reports that can't move yet)
         lakehouse
         
Result: You CAN retire the EDW when legacy reports are migrated.

How to implement this:

1. Replicate sources directly to lakehouse (not through EDW)
   • Set up CDC from source databases → Lakehouse Bronze
   • Bypass the EDW entirely for new data flows
2. Build curated layers in lakehouse (not in EDW)
   • Transform Bronze → Silver → Gold in Databricks/Spark
   • Store as Iceberg tables in object storage
3. Catalog the lakehouse (not the EDW)
   • DataHub/Atlan catalogs lakehouse tables
   • EDW tables are NOT in the catalog (they’re legacy)
4. Optional: Sync lakehouse → EDW (temporary, for legacy reports)
   • If you have critical reports that can’t move yet
   • Reverse ETL: Push curated data FROM lakehouse TO EDW
   • This keeps EDW alive temporarily, but it’s now a downstream consumer
5. Retire EDW when ready
   • Once all reports migrate off EDW
   • Stop the reverse ETL sync
   • Decommission the EDW

Key insight: The new system must be the source of truth from day one. The old system can consume from the new system temporarily, but never the other way around.

The Three Migration Strategies

Strategy 1: Coexist (Recommended for Most)

When to use: You have critical workloads that can’t move, but want to stop growing EDW footprint.

Approach:

• Keep EDW for critical finance/compliance workloads
• Set up bi-directional sync (EDW ↔ Lakehouse)
• Route all new workloads to cloud (Snowflake/Databricks)
• Gradually migrate non-critical workloads when safe

Timeline: Indefinite coexistence, EDW becomes smaller over time

Pros:

• ✅ Low risk (critical workloads stay put)
• ✅ Gradual cost reduction
• ✅ Teams learn cloud at their own pace

Cons:

• ❌ Maintain two systems indefinitely
• ❌ Sync complexity
• ❌ EDW costs never go to zero

Strategy 2: Gradual Sunset (Aggressive Modernization)

When to use: You have executive buy-in to retire EDW completely in 18-24 months.

Critical principle: Don’t modernize what you’re retiring

❌ WRONG: Modernize EDW pipelines before migrating

"Let's refactor this 10-year-old ETL job in the EDW first,
then migrate it to Databricks"

Problem: You spend 3 months modernizing code you'll throw away.
No business value. Wasted effort.

✅ RIGHT: Freeze EDW, rebuild in new platform

"Leave the old ETL job as-is. Build the new version directly
in Databricks with modern patterns. Run parallel. Cut over. 
Decommission old."

Result: You only invest in the new platform. Business gets
modern pipeline immediately.

The EDW is frozen, not modernized:

• No new features in EDW pipelines (they’re being retired)
• No refactoring of EDW code (it’s legacy, leave it alone)
• Only bug fixes if critical (minimal investment)
• All new work goes to lakehouse/Snowflake/Databricks

Approach:

• Categorize all EDW workloads (critical → nice-to-have)
• Rebuild (not refactor) in waves (easiest first, critical last)
• Set hard deadline to decommission EDW
• Allocate dedicated migration team
• Freeze EDW development (no new features, only critical fixes)

Timeline: 18-24 months to full retirement

Pros:

• ✅ Clear end date
• ✅ Full cost elimination
• ✅ Forces modernization

Cons:

• ❌ High risk if timeline slips
• ❌ Requires significant investment
• ❌ Business disruption

Strategy 3: Full Replacement (Not Recommended)

When to use: Rarely. Only if EDW is completely broken or vendor is EOL.

Approach:

• Rebuild everything in cloud from scratch
• Big-bang cutover
• High risk, high cost

We don’t recommend this. Use Strategy 1 or 2 instead.

Workload Migration Decision Framework

Use this framework to decide which workloads to migrate first:

Practical Migration Patterns

Pattern 1: Report Migration

EDW Report → Snowflake Report

Steps:
1. Identify source tables in EDW
2. Replicate tables to lakehouse (CDC or batch)
3. Rebuild report logic in Snowflake SQL
4. Run parallel (EDW + Snowflake) for 2-4 weeks
5. Validate numbers match
6. Cut over to Snowflake
7. Decommission EDW report

Timeline: 2-4 weeks per report
Risk: Low

Pattern 2: ETL Pipeline Migration

EDW ETL → Databricks Pipeline

Steps:
1. Document current ETL logic
2. Identify source systems
3. Set up CDC/streaming ingestion to lakehouse
4. Rebuild transformation logic in Spark
5. Run parallel pipelines
6. Validate data quality
7. Cut over to Databricks
8. Decommission EDW ETL

Timeline: 4-8 weeks per pipeline
Risk: Medium

Pattern 3: Data Mart Migration

EDW Mart → Lakehouse Gold Zone

Steps:
1. Understand mart schema and business logic
2. Migrate source data to lakehouse Silver zone
3. Rebuild mart as Iceberg tables in Gold zone
4. Create external tables in Snowflake (for BI access)
5. Migrate downstream reports one by one
6. Validate with business users
7. Decommission EDW mart

Timeline: 2-3 months per mart
Risk: Medium-High

Cost Comparison: EDW vs Cloud

Typical savings: 40-60% after full migration

Common Traps to Avoid

Trap 1: Building Two Curated Layers

❌ Bad:
Databricks: silver.customers (version A)
Snowflake: silver.customers (version B)
Result: They drift, teams confused

✅ Good:
Lakehouse: silver.customers (Iceberg)
Databricks: reads from lakehouse
Snowflake: reads from lakehouse
Result: One source of truth

Trap 2: No Data Product Ownership

❌ Bad:
Platform team owns all data products
Result: Bottleneck, slow delivery

✅ Good:
Domain teams own their products
Platform team provides tools + standards
Result: Scalable, fast delivery

Trap 3: Governance as Documentation Only

❌ Bad:
Policy: "Don't share PII"
Enforcement: Trust developers to follow it
Result: PII leaks happen

✅ Good:
Policy: "Don't share PII"
Enforcement: Automated masking in catalog
Result: Impossible to leak PII

Trap 4: Ad-Hoc Ingestion

❌ Bad:
Every team builds custom ingestion scripts
Result: No standards, maintenance nightmare

✅ Good:
Standard ingestion framework with contracts
Result: Reusable, maintainable, governed

Summary: The North Star

Create one logical platform, not one physical system.

You can keep multiple engines (on-prem EDW, Snowflake, Databricks). What makes it feel like “one platform” is that you standardize:

• How data lands (ingestion contracts)
• How it’s stored (one open table format + consistent zones)
• How it’s governed (one catalog + one policy model)
• How it’s observed (lineage + data quality)
• How it’s consumed (semantic layer + APIs)

If you do just that, you can modernize without a big-bang migration.

Two Questions to Tailor This to Your Environment

1. Are you aiming for Iceberg as the common table format, or are you already standardized on Delta somewhere?

   If you’re already deep into Delta Lake (Databricks-native), you might stick with Delta. But if you want true multi-engine interoperability (Snowflake + Databricks + Trino), Iceberg is the better choice.

2. Is the “CDP” (EDL) mainly a customer-event platform (streaming + identity), or a data lake landing zone?

   If it’s event-focused (Segment, mParticle style), treat it as a streaming source. If it’s a landing zone, treat it as Bronze storage.