Avro Deep Dive — Schema Evolution for Streaming Data
How Apache Avro enables schema evolution, efficient serialization, and seamless streaming data with embedded schemas.
Data
Data engineering, modeling, and storage notes.
Getting Started:
- Unified Data Infrastructure Layer (EDW + Snowflake + Databricks)
- Kubeflow Spark Operator - Getting Started - Run Spark on Kubernetes with simple YAML files
- Kubeflow Spark Operator - Enterprise & Multi-Tenant - Multi-team Spark with centralized logging and self-service portals
Apache Iceberg:
- Apache Iceberg — A Simple Guide
- Iceberg Query Engine Internals
- LTAP, the IRC Catalog, and the Hidden Layers of Apache Iceberg - A plain-English guide to LTAP, the REST catalog, and how a query travels through the Iceberg stack
- Lakehouse Serving Architecture: Bronze, Silver, Gold, and Serving Projections - Why Serving is a publishing architecture, not another Iceberg tier — Trino, StarRocks/ClickHouse, and Redis/Scylla projections from a single Gold source of truth, plus a semantic layer + MCP for governed AI agent access, with Mermaid diagrams throughout
Data File Formats — The Complete Guide
A beginner-friendly guide to every data file format from CSV to Delta Lake with simple explanations and visual examples.
Database Types — A Complete Guide
Understand different database types (RDBMS, Columnar, NoSQL, Vector) with simple explanations, popular options, and when to use each.
Delta Lake Deep Dive — ACID Transactions on Data Lakes
How Delta Lake brings ACID transactions, time travel, and schema evolution to Parquet files on data lakes.
Dimensional Modeling — A Beginner's Guide
Learn the fundamentals of Kimball dimensional modeling with simple analogies, diagrams, and practical examples.
Building a Modern Containerized Open Data Stack
A practical architecture guide for hybrid data platforms in the AI era — separating storage and compute, embracing open table formats, and building portable governed ecosystems.
ORC Deep Dive — Optimized Row Columnar Format Explained
A deep dive into how ORC (Optimized Row Columnar) stores, compresses, and indexes data for Hadoop and Hive workloads.
Parquet Deep Dive — How It Actually Stores Data
Understanding how Parquet stores, encodes, and compresses data under the hood — from row groups to encoding schemes.
Unified Data Infrastructure Layer (EDW + Snowflake + Databricks)
A very simple, practical reference architecture for a unified data platform across on-prem EDW + Snowflake + Databricks — implemented incrementally.
How Apache Iceberg Actually Works (And Why It's Perfect for AI)
A user-friendly technical guide to Apache Iceberg's architecture - how it works under the hood, why it's different from traditional tables, and why it's the go-to choice for AI/ML workloads.
How Iceberg Query Engines Work Internally — Trino, Spark & the REST Catalog
A deep dive into how query engines like Trino and Spark execute queries against Iceberg tables — from REST Catalog API calls to reading Parquet files from S3.
LTAP, the IRC Catalog, and the Hidden Layers of Apache Iceberg
A plain-English guide to what LTAP (Lake Transactional/Analytical Processing) actually means — including the important nuance that it does not replace the transactional database, but instead unifies OLTP and OLAP at the storage layer. Also covers how the Iceberg REST Catalog (IRC) works, and how a SQL query travels through the Iceberg stack from catalog lookup to Parquet rows.
Lakehouse Serving Architecture: Bronze, Silver, Gold, and Serving Projections
A practical architecture guide explaining why Bronze/Silver/Gold describe data quality while Serving describes consumption. Covers how to keep Gold as the authoritative Iceberg source of truth and publish workload-specific serving projections to Trino, StarRocks/ClickHouse, and Redis/Scylla, plus how to expose governed metrics to AI agents through a semantic layer and MCP server — with Mermaid diagrams throughout.
Kubeflow Spark Operator - Getting Started
Learn how to run Apache Spark on Kubernetes using the Spark Operator. This guide covers the basics: what it is, how it works, and how to submit your first Spark job without manual spark-submit commands.
Kubeflow Spark Operator - Enterprise & Multi-Tenant
Enterprise-grade Spark on Kubernetes: multi-tenant architecture, centralized logging and monitoring, self-service job submission portals, cost tracking, and security best practices for running Spark at scale.