Handbook

What is Avro? A file format designed for data that flows continuously (like sensor data or website clicks). The clever part? The file remembers its own structure, so even if you add new fields later, old and new systems can still understand each other. Think of it like a form that evolves over time but can still be read by old software.

Should you care about Avro? Only if you’re working with:

• Real-time data streaming (like Kafka)
• Systems where data structure changes frequently
• High-speed data transfer between systems

If you’re just analyzing data in Excel or Python, Parquet is simpler and more widely supported.

Why Avro Exists

Imagine you’re building a data pipeline:

• Day 1: User events have 5 fields
• Day 30: You add 3 new fields
• Day 60: You rename a field

Think of it like this: You design a customer feedback form with 5 questions. Later, you add 3 more questions. You still have old forms with only 5 answers — how do you process both old and new forms together? Avro solves this!

The Problem:

How do old systems read new data?
How do new systems read old data?

Avro’s Solution: Embed the schema IN the file, enabling perfect schema evolution.

Architecture: Schema + Data

Key difference from Parquet: Parquet stores columns together (all names, then all ages). Avro stores rows together (name+age, name+age, name+age). Think of Parquet as organizing a filing cabinet by topic, and Avro as organizing it chronologically. Each has its strengths!

Avro’s trick: The “form template” (schema) is saved right inside the file, so the file is self-describing!

Unlike Parquet/ORC (columnar), Avro is row-based with embedded schema:

File Structure

my_data.avro
│
├── Header
│   ├── Magic bytes: "Obj1" (4 bytes)
│   ├── Schema (JSON, embedded)
│   │   {
│   │     "type": "record",
│   │     "name": "User",
│   │     "fields": [
│   │       {"name": "id", "type": "long"},
│   │       {"name": "name", "type": "string"}
│   │     ]
│   │   }
│   └── Compression codec: "snappy", "deflate", "null"
│
├── Data Block 1
│   ├── Number of rows: 500
│   ├── Byte size: 10,000
│   ├── Serialized rows (binary):
│   │   Row 1: [123, "Alice"]
│   │   Row 2: [124, "Bob"]
│   │   ...
│   └── Sync marker (16 bytes)
│
├── Data Block 2
│   └── ... (same structure)
│
└── Data Block N

Key Insight: Schema is stored ONCE at the beginning, not repeated per row!

Type System

Avro has a rich, self-describing type system:

Primitive Types

Complex Types

Record (Struct) Example

{
  "type": "record",
  "name": "User",
  "namespace": "com.example",
  "fields": [
    {
      "name": "id",
      "type": "long",
      "doc": "User ID"
    },
    {
      "name": "username",
      "type": "string"
    },
    {
      "name": "email",
      "type": ["null", "string"],
      "default": null
    },
    {
      "name": "age",
      "type": "int"
    },
    {
      "name": "tags",
      "type": {"type": "array", "items": "string"},
      "default": []
    }
  ]
}

Key Features:

• Union types: ["null", "string"] = nullable string
• Default values: Enable backward compatibility
• Documentation: Inline docs for each field

Encoding: Compact Binary Format

How it works: Avro uses a clever trick — small numbers take up less space than big numbers. The number “5” takes 1 byte, but “1,000,000” takes more bytes. Most real-world numbers are small, so this saves tons of space! It’s like using abbreviations for common words in text messages.

Avro uses variable-length encoding for efficiency:

Integer Encoding (Zig-Zag + Variable Length)

# Small integers take fewer bytes
Value:   0  →  1 byte:  [0x00]
Value:   1  →  1 byte:  [0x02]
Value:  127 →  1 byte:  [0xFE]
Value:  128 →  2 bytes: [0x80, 0x02]
Value: 1000 →  2 bytes: [0xD0, 0x0F]

# Negative numbers also efficient (zig-zag encoding)
Value:  -1  →  1 byte:  [0x01]
Value: -64  →  1 byte:  [0x7F]

Why Variable Length?

• Most integers are small → save space
• Avro can store 1 in 1 byte instead of 4 bytes (like INT32)

String Encoding

# String: "hello"
Length:  5        → [0x0A]       # Variable-length encoded
Content: "hello"  → [0x68, 0x65, 0x6C, 0x6C, 0x6F]

# Total: 6 bytes (1 for length + 5 for content)

Array Encoding

# Array: [1, 2, 3]
Block count: 3    → [0x06]  # Positive = 3 items follow
Items:            → [0x02, 0x04, 0x06]  # Variable-length ints
End marker:       → [0x00]  # Zero = end of array

Record Encoding

# Schema: {name: string, age: int}
# Data: {name: "Alice", age: 30}

Serialized:
  [0x0A]                    # Length of "Alice" (5)
  [0x41, 0x6C, 0x69, 0x63, 0x65]  # "Alice"
  [0x3C]                    # Age: 30 (variable-length)

No field names in data! Schema defines field order, data is pure values.

Schema Evolution: The Killer Feature

What’s a schema? Think of it as the “template” or “blueprint” for your data. Like a form has specific fields (name, email, age), a schema defines what fields your data has and what type they are (text, number, date).

Schema evolution means: The form can change over time without breaking old submissions!

Avro shines when schemas change over time:

1. Adding Fields (Backward Compatible)

// Version 1 (Writer Schema)
{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "long"},
    {"name": "name", "type": "string"}
  ]
}

// Version 2 (Reader Schema)
{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "long"},
    {"name": "name", "type": "string"},
    {"name": "email", "type": ["null", "string"], "default": null}  // NEW!
  ]
}

Result: Reader with v2 schema reads v1 data → uses default null for missing email.

2. Removing Fields (Forward Compatible)

// Writer Schema (v2)
{
  "fields": [
    {"name": "id", "type": "long"},
    {"name": "name", "type": "string"},
    {"name": "email", "type": "string"}
  ]
}

// Reader Schema (v1)
{
  "fields": [
    {"name": "id", "type": "long"},
    {"name": "name", "type": "string"}
    // email field removed
  ]
}

Result: Reader ignores unknown email field.

3. Renaming Fields (Aliases)

// Writer Schema (old)
{"name": "name", "type": "string"}

// Reader Schema (new)
{
  "name": "username",
  "type": "string",
  "aliases": ["name"]  // Can read old "name" field as "username"
}

4. Changing Field Types (with caution)

Safe Changes:

• int → long ✅
• float → double ✅
• string → bytes ✅

Unsafe Changes:

• long → int ❌ (data loss!)
• string → int ❌ (incompatible)

Schema Registry: The Missing Piece

The problem it solves: Imagine if every email you sent included a 10-page document explaining what “From,” “To,” and “Subject” mean. That’s wasteful! Instead, email programs just know those fields exist. Schema Registry is like that — it stores the “form template” once, and messages just reference it. This saves tons of space!

Note: This is specific to Kafka and real-time streaming systems.

In streaming systems (Kafka), storing the full schema in every message is wasteful:

How It Works:

# Producer
1. Register schema with Schema Registry → Get schema_id = 42
2. Serialize data with Avro
3. Prepend schema_id to message: [42][binary_data]
4. Send to Kafka

# Consumer  
1. Read message: [42][binary_data]
2. Extract schema_id = 42
3. Fetch schema from registry (cached)
4. Deserialize data with schema

Message Format:

[Magic byte: 0x00] [Schema ID: 4 bytes] [Avro binary data]

Benefits:

• Messages are tiny (no repeated schema)
• Centralized schema management
• Schema evolution enforcement

Popular Implementations:

• Confluent Schema Registry
• AWS Glue Schema Registry
• Azure Schema Registry

Compression

Avro supports block-level compression:

Block-Level Compression:

Block 1: [500 rows] → Compress as one unit → Write
Block 2: [500 rows] → Compress as one unit → Write

Benefits:

• Splittable (can decompress blocks independently)
• Good compression (rows have similar structure)

Avro vs Parquet vs JSON

Real-World Use Case: Kafka + Avro

from confluent_kafka import avro
from confluent_kafka.avro import AvroProducer

# Define schema
user_schema = {
    "type": "record",
    "name": "User",
    "fields": [
        {"name": "user_id", "type": "long"},
        {"name": "username", "type": "string"},
        {"name": "email", "type": ["null", "string"], "default": null},
        {"name": "signup_date", "type": "string"}
    ]
}

# Configure producer with Schema Registry
producer = AvroProducer({
    'bootstrap.servers': 'localhost:9092',
    'schema.registry.url': 'http://localhost:8081'
}, default_value_schema=user_schema)

# Produce message
user_data = {
    "user_id": 12345,
    "username": "alice",
    "email": "alice@example.com",
    "signup_date": "2024-12-16"
}

producer.produce(topic='users', value=user_data)
producer.flush()

# Later: Add new field (backward compatible)
updated_schema = {
    "type": "record",
    "name": "User",
    "fields": [
        {"name": "user_id", "type": "long"},
        {"name": "username", "type": "string"},
        {"name": "email", "type": ["null", "string"], "default": null},
        {"name": "signup_date", "type": "string"},
        {"name": "country", "type": ["null", "string"], "default": null}  # NEW!
    ]
}

# Old consumers can still read new messages!

Why This Works:

1. New field has a default value
2. Schema Registry tracks both versions
3. Old consumers ignore unknown country field
4. New consumers get null for old messages

Performance Comparison

import time
import pandas as pd
import json

# Create test data (1M rows)
df = pd.DataFrame({
    'id': range(1_000_000),
    'name': ['User_' + str(i) for i in range(1_000_000)],
    'email': ['user{}@example.com'.format(i) for i in range(1_000_000)],
    'age': np.random.randint(18, 80, 1_000_000)
})

# Write as Avro
start = time.time()
df.to_avro('data.avro', compression='snappy')
avro_write_time = time.time() - start
avro_size = os.path.getsize('data.avro') / 1024 / 1024

# Write as Parquet
start = time.time()
df.to_parquet('data.parquet', compression='snappy')
parquet_write_time = time.time() - start
parquet_size = os.path.getsize('data.parquet') / 1024 / 1024

# Write as JSON
start = time.time()
df.to_json('data.json', orient='records')
json_write_time = time.time() - start
json_size = os.path.getsize('data.json') / 1024 / 1024

print(f"Avro:    {avro_size:.1f} MB in {avro_write_time:.2f}s")
print(f"Parquet: {parquet_size:.1f} MB in {parquet_write_time:.2f}s")
print(f"JSON:    {json_size:.1f} MB in {json_write_time:.2f}s")

# Read entire file
start = time.time()
df_avro = pd.read_avro('data.avro')
avro_read_time = time.time() - start

start = time.time()
df_parquet = pd.read_parquet('data.parquet')
parquet_read_time = time.time() - start

start = time.time()
df_json = pd.read_json('data.json', orient='records')
json_read_time = time.time() - start

print(f"\nFull read:")
print(f"Avro:    {avro_read_time:.2f}s")
print(f"Parquet: {parquet_read_time:.2f}s")
print(f"JSON:    {json_read_time:.2f}s")

Typical Results:

Size:
Avro:    125.0 MB in 2.3s  (fast writes)
Parquet:  85.0 MB in 3.5s  (slower writes, better compression)
JSON:    450.0 MB in 8.2s  (huge, slow)

Full read:
Avro:    1.8s  (decent for row-based)
Parquet: 0.5s  (fastest, columnar)
JSON:    15.2s (very slow)

Column read (age only):
Avro:    1.7s  (must read all columns)
Parquet: 0.1s  (only reads one column) 🏆

Conclusion:

• Avro: Fast writes, good for streaming
• Parquet: Best compression, fast analytics
• JSON: Human-readable, but slow & large

Best Practices

1. Always Use Default Values

// ❌ Bad: No default
{"name": "email", "type": "string"}

// ✅ Good: Has default (enables backward compatibility)
{"name": "email", "type": ["null", "string"], "default": null}

2. Use Unions for Optional Fields

// ❌ Bad: Field is required
{"name": "phone", "type": "string"}

// ✅ Good: Field is optional
{"name": "phone", "type": ["null", "string"], "default": null}

3. Version Your Schemas

{
  "type": "record",
  "name": "User",
  "namespace": "com.example.v2",  // Version in namespace
  "fields": [...]
}

4. Document Your Fields

{
  "name": "status",
  "type": "string",
  "doc": "User account status. Valid values: active, suspended, deleted"
}

5. Use Enums for Fixed Values

// ❌ Bad: Free-form string
{"name": "status", "type": "string"}

// ✅ Good: Enforced values
{
  "name": "status",
  "type": {
    "type": "enum",
    "name": "Status",
    "symbols": ["ACTIVE", "SUSPENDED", "DELETED"]
  }
}

6. Choose Snappy Compression (Default)

# Fast compression/decompression, good ratio
df.to_avro('data.avro', compression='snappy')

Common Pitfalls

1. Breaking Schema Compatibility

// ❌ Breaking change: Removed required field
// Old schema
{"name": "email", "type": "string"}

// New schema (missing email)
// Old readers will fail!

Solution: Never remove required fields without defaults.

2. Using Avro for Analytics

# ❌ Slow: Read 1 column from Avro
df = pd.read_avro('data.avro', columns=['age'])
# Must read ALL columns (row-based)

# ✅ Fast: Read 1 column from Parquet
df = pd.read_parquet('data.parquet', columns=['age'])
# Only reads the column you need

3. Not Using Schema Registry

# ❌ Bad: Embed full schema in every Kafka message
message = [schema_json, data]  # Huge overhead!

# ✅ Good: Use Schema Registry
message = [schema_id, data]  # Tiny overhead

4. Large Block Sizes

# ❌ Bad: 100 MB blocks (not splittable)
# ✅ Good: 1-10 MB blocks (parallelizable)

5. Changing Field Types Unsafely

// ❌ Breaking: int → string
{"name": "age", "type": "int"}     → {"name": "age", "type": "string"}

// ✅ Safe: int → long
{"name": "age", "type": "int"}     → {"name": "age", "type": "long"}

Should You Use Avro?

Quick Decision Guide:

• ✅ Use Avro: You’re building real-time data systems (like tracking website clicks as they happen)
• ❌ Use Parquet: You’re analyzing existing data (like last month’s sales report)
• ✅ Use Avro: Your data structure changes often and you need backward compatibility
• ❌ Use Parquet: Your data structure is stable

Honest take: Unless you’re specifically working with Kafka or real-time streaming, start with Parquet. It’s simpler, more widely supported, and easier to work with. Avro is for specialized use cases!

When to Use Avro

✅ Use Avro For:

1. Kafka / Event Streaming

   Producer → Kafka (Avro) → Consumer
   

2. Schema Evolution
   • Fields added/removed frequently
   • Need backward/forward compatibility
3. Write-Heavy Workloads
   • Fast serialization
   • Streaming ingestion
4. RPC / Data Exchange
   • Between services
   • Compact binary format

❌ Use Parquet Instead For:

1. Analytics / Data Warehouses
   • Aggregations, GROUP BY
   • Reading specific columns
2. Large-Scale Storage
   • Better compression (10-20% smaller)
3. Read-Heavy Workloads
   • Parquet’s columnar layout is faster

Summary

Quick Takeaways:

• What is Avro? A file format designed for real-time data streaming (like website clicks, sensor data)
• Special power: The file remembers its own structure, so old and new versions can work together
• Why use it? When data structure changes frequently and you need old/new systems to coexist
• When to use it? Kafka streaming, real-time data pipelines, rapidly changing data structures
• When NOT to use it? Analyzing historical data, working in Python/Excel, when your data structure is stable
• Bottom line: Unless you’re specifically working with Kafka or real-time streaming, use Parquet instead. Avro is specialized and less widely supported.

Avro is a schema-evolution-first format for streaming data:

Key Strengths:

1. ✅ Schema embedded in file (self-describing — knows its own structure)
2. ✅ Schema evolution (old and new versions work together)
3. ✅ Fast writes (optimized for streaming data)
4. ✅ Compact (efficient encoding saves space)
5. ✅ Kafka-native (industry standard for Kafka)

Key Weaknesses:

1. ❌ Row-based (slow for analytics that need specific columns)
2. ❌ No column pruning (must read all columns, even if you only need one)
3. ❌ Worse compression than Parquet (typically 10-20% larger)
4. ❌ Binary (can’t open in text editor or Excel)
5. ❌ Limited tool support (works best with Java/Kafka ecosystem)

Decision Matrix:

The Bottom Line:

• Streaming + schema changes? → Avro is the answer
• Analytics + big data? → Use Parquet

Further Reading

• Apache Avro Documentation
• Avro Specification
• Confluent Schema Registry
• Schema Evolution in Avro