What Is a Data Lakehouse? From Data Warehouses to Open Table Formats

Data infrastructure has had a messy decade. First, data warehouses were king — structured, reliable, fast. Then data lakes showed up promising cheap, flexible storage for massive amounts of raw data. And for a while, most companies ran both in parallel, with ETL pipelines constantly shuffling data between them. The result? Latency, consistency nightmares, and engineering teams maintaining two completely different systems.

The Data Lakehouse is the architectural pattern designed to end that tug-of-war.

Here's the core idea: instead of keeping a data warehouse and a data lake side by side, you collapse them into one layer. Your data lives in open file formats — like Parquet — sitting on cheap object storage, think S3 or Google Cloud Storage. And then you put an open table format on top of that storage layer to give you all the reliability features you expect from a warehouse: ACID transactions, time travel, schema evolution. One storage layer doing the job of two systems.

Now, to understand why this matters, let's quickly recap what made the old approaches painful.

Data warehouses are reliable and fast, but they're expensive. Most cloud warehouses tightly couple compute and storage, so scaling one means scaling both. And their strict schema requirements — where you define the structure before writing data — make them hostile to the messy, evolving data that machine learning and AI workloads actually need.

Data lakes flipped this around. You can dump anything in there cheaply. But without ACID transaction guarantees, concurrent writes can silently corrupt your data. There's no schema validation, so data quality becomes anyone's guess. And small-file proliferation — where thousands of tiny files accumulate over time — kills query performance until you do manual maintenance. Flexible, yes. Reliable, no.

The Lakehouse addresses all of this. Your data stays in cheap object storage, so costs stay low. But an open table format sits on top and manages a transaction log that enforces atomicity — either a write fully succeeds or it doesn't happen at all. And because the data files themselves are in open formats, multiple query engines can read the same data simultaneously. Spark, Trino, Snowflake, DuckDB — they all read the same Parquet files without you making copies.

Let's get into how this actually works under the hood, using Apache Iceberg as the example.

Iceberg manages a three-tier metadata layer. At the top, metadata files describe the table schema, how it's partitioned, and a history of snapshots. Below that, each snapshot points to a manifest list — essentially a directory of which data files belong to that snapshot. And each manifest file records the path, row count, and statistics like minimum and maximum column values for every individual Parquet file.

When a query engine runs a query, it walks down this hierarchy: metadata, then manifest lists, then manifest files. At each step, it uses those statistics to skip entire files that can't possibly contain relevant data. This is called partition pruning and file pruning, and it's what makes Iceberg fast even on massive datasets. And because table versioning is managed through metadata pointers rather than a centralized lock manager, multiple engines can read different snapshots simultaneously without blocking each other.

Delta Lake, the other major player, takes a similar approach but with a more Spark-centric design. Instead of a tiered metadata hierarchy, it stores a transaction log as a series of JSON files in a directory at the table root. Every ten versions, it writes a Parquet checkpoint file to speed up log loading. Simpler to understand, and deeply integrated with the Databricks ecosystem.

So which one do you pick?

Apache Iceberg came out of Netflix and is now an Apache Software Foundation project. Its design philosophy is cross-engine interoperability — it supports multiple catalog systems and is natively supported by Snowflake, Dremio, BigQuery, Flink, and others. If your architecture spans multiple query engines, Iceberg is the safer bet.

Delta Lake came from Databricks and moved to the Linux Foundation. It's optimized for Spark performance and makes common data manipulation operations simpler. If you're deeply in the Databricks ecosystem, Delta Lake is the path of least resistance.

But here's the 2025 development that changes the calculus somewhat: Delta Lake's UniForm feature. UniForm lets a Delta table simultaneously expose Iceberg-compatible metadata. So any engine that speaks Iceberg — Snowflake, Trino, whatever — can read a Delta table as if it were native Iceberg. Write in Delta, read from anywhere. The two formats are converging rather than staying isolated.

Three things to take away from all of this.

First, the Lakehouse isn't a product you buy — it's an architectural pattern. The principle is simple: put reliability in the metadata layer, leave flexibility and low cost in object storage.

Second, by 2025 this is no longer a cutting-edge experiment. It's the default starting point for most data engineering teams building new systems. The question isn't whether to use a Lakehouse — it's which table format fits your engine mix.

And third, the format war is softening. Delta UniForm and Iceberg's multi-engine catalog support mean you don't have to pick one and lock yourself in forever. The underlying files are the same Parquet format either way. The metadata layer is increasingly interoperable. That's genuinely good news for teams who don't want to bet everything on one vendor's ecosystem.