Building Modern Data Platforms: The Architecture Decisions That Matter

The Data Platform Paradox

More tools have been built to solve data problems in the past five years than in the previous twenty combined. Yet most data teams still spend the majority of their time on data quality, pipeline reliability, and answering the same questions repeatedly.

The problem isn’t tooling. It’s architecture.

Foundational Architectural Decisions

Before evaluating any specific tool, three architectural decisions define everything else:

1. Centralized vs. Federated Storage

Centralized lakehouse: Single logical data store (S3/GCS/ADLS) with open table formats (Delta, Iceberg, Hudi). Lower governance overhead. Better for smaller organizations.

Federated / Data Mesh: Distributed domain ownership of data products. Higher autonomy, higher coordination cost. Right for large multi-domain enterprises.

For most organizations, start centralized and introduce federation as pain demands it.

2. Batch vs. Streaming First

Real-time streaming (Kafka, Flink, Kinesis) is seductive but expensive to operate. Most business decisions don’t require sub-second latency. Design for batch-first, stream-later — unless your core product is real-time (trading, fraud, live recommendations).

3. Transformation Layer Philosophy

The shift to SQL-first transformation with dbt has been one of the most productivity-positive changes in data engineering. Version-controlled, testable, documented SQL transformations that business analysts can contribute to.

The key decision: where does business logic live? In the transformation layer (dbt), in the serving layer (metrics store / semantic layer), or in application code? Centralizing it in dbt + a semantic layer dramatically reduces inconsistency.

The Stack That Works in 2025

Based on patterns across high-performing data organizations:

• Ingestion: Fivetran (managed connectors) + custom Python pipelines for proprietary sources
• Storage: Cloud lakehouse (S3 + Delta Lake or Iceberg)
• Compute: Databricks (heavy ML/engineering) or Snowflake (SQL-centric analytics)
• Transformation: dbt Core (open source) with SQLMesh for larger teams needing state management
• Orchestration: Dagster (best developer experience) or Airflow (most ecosystem support)
• Semantic layer: MetricFlow / dbt Semantic Layer or Cube
• Observability: Monte Carlo or Elementary (open source)

What Most Teams Get Wrong

1. Skipping data contracts: Define expectations between producers and consumers explicitly. Schema changes without contracts create silent, expensive failures downstream.

2. Underinvesting in metadata: A data catalog isn’t a nice-to-have. Teams waste enormous hours finding, understanding, and trusting data. Invest in discoverability early.

3. Building everything custom: Every custom pipeline is technical debt. Buy commodity workflows. Build only what’s genuinely proprietary to your data or business logic.

4. Neglecting data quality: Data quality is not a data engineering problem — it’s an organizational problem. The engineering is straightforward. The hard part is establishing ownership, SLAs, and accountability upstream.

The Principles That Don’t Change

Tools evolve rapidly. These principles don’t:

• Make data self-service: The data team should enable analysts, not be a bottleneck for them
• Treat data as a product: Build data assets with consumers in mind, with documentation and SLAs
• Automate observability: You cannot manually monitor a modern data platform at scale
• Design for change: Assume your stack will look different in 18 months. Avoid tight coupling.

The organizations winning on data in 2025 are not the ones with the most sophisticated stack. They’re the ones where data quality is trusted, answers are consistent, and business teams can self-serve.