Handbook

CEO: "How many bookings did we have last month?"

Product Manager: "500,000 bookings"
Marketing Team:   "520,000 bookings"
Finance Team:     "480,000 bookings"
Data Science:     "515,000 bookings"
Engineering:      "505,000 bookings"

CEO: "...which number is correct?"

Everyone: "Mine is!"

Question: "What is a 'booking'?"

Product team's definition:
✅ User clicks "Book" button
✅ Includes canceled bookings
Result: 520,000 bookings

Finance team's definition:
✅ Payment successfully processed
❌ Excludes canceled bookings
Result: 480,000 bookings

Marketing team's definition:
✅ User completes booking flow
✅ Includes pending payments
Result: 500,000 bookings

All three are measuring "bookings"
But getting different numbers!

Marketing meeting:
"Our campaign generated 50,000 bookings!" (using their definition)

Finance meeting later that day:
"Marketing spent $5M but only got 35,000 bookings" (using their definition)

CEO's conclusion:
"Marketing is terrible! Fire the CMO!"

Reality:
Both numbers were "correct" based on different definitions
But the decision to fire CMO was based on inconsistent metrics

Monday:
"Why does our dashboard show different numbers than theirs?"
- Investigate: 4 hours
- Root cause: Different SQL query
- Fix: 2 hours

Tuesday:
"Our A/B test results don't match the experiment platform"
- Debug: 6 hours
- Root cause: Different date filters
- Fix: 2 hours

Wednesday:
"Customer support says our metrics are wrong"
- Investigate: 4 hours
- Root cause: Different timezone handling
- Fix: 2 hours

Thursday:
"CEO wants explanation for discrepancy in board deck"
- Urgent investigation: 8 hours
- Root cause: Different definition of "active user"
- Write explanation: 2 hours

Friday:
"Fix production bug"
- Actually build features: 4 hours

Total: 36 hours on metrics issues, 4 hours on features!
That's 90% of time wasted!

Week 1:
Manager: "This dashboard shows 500K bookings"
Analyst: "Actually, this other dashboard shows 480K"
Manager: "Hmm, which is right?"

Week 5:
Manager: "These numbers don't match again"
Analyst: "Let me investigate..."
Manager: "This is getting annoying"

Week 10:
Manager: "I don't trust any of these dashboards anymore"
Analyst: "But this one is correct!"
Manager: "That's what you said about the last three..."

Week 15:
Manager: "I'll just use my gut feeling instead"
Company: *Makes decisions based on intuition, not data*

Result: $1B+ company making decisions with zero data!

Meeting to decide: "Should we expand to Japan?"

Hour 1: Discussing which metrics to use
- Product: "Based on our data..."
- Marketing: "No, our data shows..."
- Finance: "Actually, according to our numbers..."
- Data Science: "All your numbers are wrong..."

Hour 2: Still debating data quality
- "Why do your numbers differ from mine?"
- "Did you include canceled bookings?"
- "Which timezone are you using?"
- "Are you filtering test accounts?"

Hour 3: Finally getting somewhere
- Everyone recalculates using agreed definition
- Numbers start to align
- But meeting time is up!

Hour 4: Schedule follow-up meeting

Total meetings needed: 5-10
Time to decision: 2-3 months
Cost: Missed market opportunity

Marketing Team:
├─ Marketing Data Engineer
├─ Writes SQL: SELECT COUNT(*) FROM bookings WHERE source='ads'
├─ Creates Dashboard: "Marketing Bookings"
└─ Definition lives in their code

Product Team:
├─ Product Data Engineer
├─ Writes SQL: SELECT COUNT(*) FROM bookings WHERE status='confirmed'
├─ Creates Dashboard: "Product Bookings"
└─ Definition lives in their code (different!)

Finance Team:
├─ Finance Analyst
├─ Writes SQL: SELECT COUNT(*) FROM bookings WHERE payment_status='paid'
├─ Creates Report: "Financial Bookings"
└─ Definition lives in their spreadsheet (even more different!)

Result: 3 definitions of "bookings", none documented!

Year 1 (2008): 10 key metrics
- Bookings, Revenue, Users, etc.
- Everyone uses same definitions
- Life is simple ✅

Year 5 (2013): 500 metrics
- Different teams create variations
- "Bookings" becomes:
  - Total Bookings
  - New Bookings
  - Repeat Bookings
  - Canceled Bookings
  - Pending Bookings
  - 50 other variations...
- Some documented, most not

Year 10 (2018): 10,000+ metrics
- Nobody knows which is the "official" metric
- Same metric has 10+ different calculations
- Data teams spend all time reconciling
- Complete chaos ❌

Current state: Unsustainable

1. Currency?
   - USD, EUR, JPY, or local currency?
   - What exchange rate?
   - On what date?

2. Time period?
   - Booking date or check-in date?
   - UTC or local timezone?
   - Include today or not?

3. Booking status?
   - Include canceled?
   - Include pending?
   - Include refunded?
   - Include modified?

4. Booking type?
   - Include test bookings?
   - Include employee bookings?
   - Include promotional bookings?
   - Include instant book only?

5. Outliers?
   - Include $100,000 luxury mansions?
   - Include $1 promotional bookings?
   - What's the max/min?

Total combinations: 2^5 = 32+ different calculations!
All called "Average Booking Value"!

Day 1:
Data Engineer: "I'll improve the raw_bookings table"
- Changes booking_date from PST to UTC
- Seems like a good idea
- No warning systems

Day 2:
Dashboard A: Shows wrong numbers (but nobody notices yet)

Day 3:
Dashboard B: Shows wrong conversion rates
A/B test: Makes wrong decision based on bad data

Day 5:
CEO dashboard: Shows incorrect ROI
CEO: "Why did our performance tank suddenly?"
Everyone: Panic! 🔥

Day 10:
After investigation:
- 47 downstream metrics were broken
- 130 dashboards showed wrong data
- 12 A/B tests made wrong decisions
- $10M in bad business decisions

Root cause: One small change, no impact tracking

✅ Every change tracked in Git
✅ Can see who changed what and when
✅ Can roll back bad changes
✅ Can review changes before deploy
✅ Can see impact of changes

Example:
git log bookings.sql
- June 1: John changed filter logic
- May 15: Sarah updated timezone handling  
- May 1: Mike added new booking types

❌ No version control
❌ Metrics defined in various places:
   - SQL queries in dashboards
   - Python scripts on laptops
   - Spreadsheet formulas
   - Jupyter notebooks
   - Slack messages ("Just use this query")
   
❌ When metric changes:
   - No record of what changed
   - No record of who changed it
   - No record of why it changed
   - Can't roll back
   - Can't review

Result: Complete chaos

Product Team definition:
- User who logged in last 30 days
- Result: 50 million active users

Growth Team definition:
- User who searched for listings last 30 days
- Result: 35 million active users

Finance Team definition:
- User who made a booking last 90 days
- Result: 20 million active users

CEO to investors:
"We have 50 million active users!"

Investors do their own analysis:
"Actually, based on Finance data, it's 20 million"

Investor: "Are you lying to us?"
CEO: "No, we just have different definitions!"
Investor: "That's even worse!"

Result: Trust destroyed, stock price drops

Week 1: Launch A/B test
- Hypothesis: New flow will increase bookings by 10%
- Metric used: "Bookings" (Product team definition)
- Result after 1 week: +15% bookings ✅

Decision: "Ship it to 100% of users!"

Week 2: Rollout to everyone
- Marketing tracks using their "Bookings" metric
- Result: -5% bookings ❌
- Marketing: "Product team broke bookings!"

Week 3: Investigation
- Product: "Our metric shows +15%"
- Marketing: "Our metric shows -5%"
- Root cause: Different definitions
  - Product counted initiated bookings (up)
  - Marketing counted completed bookings (down)
  - New flow: More initiations, fewer completions
  - Net result: Feature made things worse!

Week 4: Rollback
- Feature removed
- 2 weeks of bad user experience
- Estimated cost: $50M in lost bookings
- All because of metric inconsistency!

Data Science Team:
- Builds ML model to forecast bookings
- Trains on 2 years of historical data
- Uses "bookings" from data warehouse
- Forecast: 10M bookings next quarter

Reality Check (next quarter):
- Product team: "We hit 11M bookings!"
- Marketing team: "We only got 9.5M bookings"
- Finance team: "Revenue suggests 9M bookings"

Post-mortem:
- Data Science used definition A
- Product used definition B
- Marketing used definition C
- Finance used definition D
- Model was accurate... for definition A
- But business used definition B/C/D
- Forecast appeared wrong (but wasn't!)

Result:
- CFO: "Fire the data science team!"
- Data Science: "Our forecast was correct!"
- CFO: "Then why are we missing targets?"
- Data Science: "Different definitions..."
- CFO: "This is unacceptable!"

Career impact: 3 people left the company

Metrics:
- 10,000+ defined metrics
- 50+ definitions of "bookings" alone
- 100+ definitions of "active user"
- 1,000+ definitions of "revenue"

Teams:
- 200+ data teams
- 2,000+ data engineers
- 5,000+ employees using data daily

Impact:
- 50-70% engineering time on reconciliation
- 100+ hours per week in "metrics meetings"
- $100M+ per year in wasted time
- $500M+ per year in bad decisions

Scalability:
- Problem grows O(N²) with company size
- More teams = exponentially more inconsistency
- Unsustainable!

Situation:
- Preparing for IPO
- Auditors reviewing financial metrics
- Need to certify accuracy

Auditor: "What's your 2017 revenue?"

Airbnb: "We have 5 different answers"
  - Finance: $2.6B
  - Product: $2.8B
  - Data warehouse: $2.7B
  - Tax team: $2.5B
  - CEO's dashboard: $2.9B

Auditor: "Which is correct?"

Airbnb: "Uh... all of them?"

Auditor: "I can't certify this. Fix it or no IPO."

CEO: "Drop everything. Fix this NOW."

Result: Emergency project launched

Small Startup (50 employees):
- 3-5 definitions of key metrics
- Weekly arguments about numbers
- Cost: 20% of engineering time

Mid-Size Company (500 employees):
- 50-100 definitions per metric
- Daily confusion
- Cost: 40% of engineering time

Large Enterprise (5,000+ employees):
- 1,000+ definitions per metric
- Complete chaos
- Cost: 60-70% of engineering time

Every company hits this eventually!

Phase 1: Startup (0-50 people)
├─ Everyone sits together
├─ Informal communication works
├─ One person knows all metrics
└─ No problem yet ✅

Phase 2: Growth (50-200 people)
├─ Teams start to specialize
├─ Each team builds own dashboards
├─ Small differences emerge
└─ Minor annoyance 😐

Phase 3: Scale (200-1,000 people)
├─ Teams in different locations
├─ Can't talk to everyone
├─ Metrics diverge significantly
└─ Major problem 😰

Phase 4: Enterprise (1,000+ people)
├─ Multiple business units
├─ Acquisitions add more systems
├─ Complete metric chaos
└─ Crisis! 🔥

The bigger you get, the worse it gets!

Week 1: Create "Metrics Wiki"
- Document all 10,000 metrics
- Define each one carefully
- Get everyone to read it

Week 5: Wiki has 100 pages
- But metrics keep changing
- Documentation out of date

Week 10: Wiki has 500 pages
- Nobody reads it
- Too much text
- Still inconsistent

Result: $500K spent, zero impact ❌

Hire "Metrics Team" to enforce standards

Month 1: Team defines standards
- "All metrics must use UTC"
- "All metrics must document assumptions"
- Etc.

Month 3: Nobody follows standards
- Too slow to wait for approval
- Engineers bypass the process
- Metrics team becomes bottleneck

Month 6: Metrics team overwhelmed
- 100+ requests per week
- Can't review everything
- Standards ignored

Result: Created bureaucracy, didn't solve problem ❌

Build internal BI tool with "standardized metrics"

Year 1: Build tool
- $5M investment
- 20 engineers
- Looks great!

Year 2: Launch
- But... teams still use their own SQL
- Tool too rigid for custom analysis
- Adoption: 30%

Year 3: Failure
- 70% of users still use custom queries
- Tool becomes yet another source of inconsistency
- Problem actually worse (now one more tool!)

Result: $15M wasted ❌

"Everyone meet weekly to align on metrics"

Week 1: Meeting with 50 people
- Takes 2 hours
- Align on "bookings" definition
- Everyone agrees ✅

Week 2: Already diverging
- Product: "We need to add one small filter..."
- Marketing: "For our use case, we should..."
- Definitions drift immediately

Week 10: Back to chaos
- Nobody remembers the meeting
- Everyone has their own definition again
- Meetings become pointless

Result: 100 hours wasted per week ❌

Old thinking:
"We need to educate people to use metrics correctly"

New thinking:
"We need to make it impossible to use metrics incorrectly"

Old approach:
- Document metrics
- Train people
- Review code
- Hope for the best

New approach:
- Define metrics once, in code
- Make everyone use the same code
- Automate verification
- Enforce consistency

1. Single Source of Truth
   ✅ One place to define each metric
   ✅ Everyone uses the same definition
   ✅ No way to bypass it

2. Version Control
   ✅ Track every change
   ✅ See who, what, when, why
   ✅ Roll back if needed

3. Impact Analysis
   ✅ Know what breaks when you change something
   ✅ Alert affected teams
   ✅ Prevent accidents

4. Automatic Propagation
   ✅ Change metric once
   ✅ Updates everywhere automatically
   ✅ No manual work

5. Testing Environment
   ✅ Test changes before production
   ✅ Verify correctness
   ✅ Zero downtime

6. Developer Friendly
   ✅ Easy to use
   ✅ Fast iteration
   ✅ Doesn't slow teams down

Requirements:
✅ Must work at massive scale (10,000+ metrics)
✅ Must be developer-friendly (or nobody uses it)
✅ Must be performant (can't slow queries down)
✅ Must be flexible (support all use cases)
✅ Must be reliable (can't break production)

Constraints:
❌ Can't disrupt existing systems (too risky)
❌ Can't force everyone to migrate at once (impossible)
❌ Can't slow down teams (they'll bypass it)
❌ Can't be expensive to maintain (not sustainable)

Challenge: Build all this while company is rapidly growing!

Minerva Platform:
├─ Single source of truth for all metrics
├─ Git-like version control
├─ Automatic impact analysis
├─ Staging environment for testing
├─ Automatic data backfilling
└─ Zero downtime updates

Results:
✅ From 10,000 inconsistent metrics to 10,000 consistent metrics
✅ From 70% time on reconciliation to 10%
✅ From chaos to clarity
✅ From blocks to IPO

Engineering time saved: 60%
Bad decisions prevented: $500M+/year
Time to implement: 2 years

The Question: "How many bookings yesterday?"

Before:
├─ Product: 520,000
├─ Marketing: 500,000
├─ Finance: 480,000
├─ Data Science: 515,000
└─ CEO: "...which one?!"

After (with Minerva):
├─ Everyone: 500,000
├─ All dashboards: 500,000
├─ All reports: 500,000
└─ CEO: "Finally!" ✅