What Is a Data Lakehouse? From Data Warehouses to Open Table Formats

Data infrastructure has taken a pretty winding road over the last decade. First, data warehouses ruled everything. Then data lakes showed up as a cheap way to dump massive amounts of raw data — and they solved the cost problem, sure, but they handed us years of headaches around governance and slow queries. The Data Lakehouse is the architecture that's trying to end that tug-of-war once and for all. The pitch is simple: one storage layer that gives you warehouse-grade reliability *and* data-lake flexibility, at the same time.

So let's set the stage with the three players.

A data warehouse is structured, fast, and reliable — but it's expensive, and it uses what's called schema-on-write, which basically means you lock in the structure up front. Great for consistency, painful the moment you need to change that schema.

A data lake flips that. It's cheap and flexible, you can throw anything into it — but it lacks ACID transactions, it's hard to govern, and queries tend to crawl.

The Data Lakehouse takes the best of both. You keep your data in cheap object storage, like Amazon S3 or Google Cloud Storage, and then you layer an *open table format* on top of it. And that table format is where the magic happens.

Now, here's a key point: a Data Lakehouse is not a product you buy. It's an architectural pattern. The core idea in one sentence — you put a transactional metadata format on top of open-format object storage.

Think about the old way of doing things. You ran a data lake and a data warehouse side by side, and you had ETL pipelines constantly shuffling data between them. That means latency, and it means consistency problems, because now you've got two copies that can drift apart. The Lakehouse collapses all of that into a single layer. Your data lives as Parquet or ORC files sitting in object storage, and an open table format — the two big names are Apache Iceberg and Delta Lake — gives you ACID guarantees, time travel, and schema evolution right on top of those files.

Picture the flow like this: your data sources feed into object storage. On top of that storage sits the open table format. That format maintains a metadata and transaction log, and *that's* what delivers the three headline features — ACID transactions, time travel so you can query previous versions of a table, and schema evolution so you can change structure without breaking everything. And because it's all open, a whole crowd of query engines — Spark, Trino, Snowflake, BigQuery — can read the exact same copy of the data.

So why does any of this actually matter? Let's talk about the pain points it fixes.

Data warehouses hurt on cost and flexibility. Most cloud warehouses tie compute and storage together, so scaling gets expensive fast. And their strict schema requirements make machine learning and AI workloads a real struggle to fit in.

Data lakes hurt on reliability. No ACID means concurrent writes can literally corrupt your data. No schema validation means data quality is anyone's guess. And you get this endless small-file problem, where files pile up and you periodically have to do manual cleanup just to keep queries fast.

The Lakehouse addresses all of that. On cost, your data stays in cheap object storage and you spin up compute only when you need it. On reliability, the table format's transaction log guarantees atomic writes. On flexibility, one copy of the data can be read by Spark, Trino, Snowflake, DuckDB — all at once. And it plays nicely with ML and AI, because unstructured and semi-structured data can live right alongside your clean structured tables.

Okay, so how does it actually work under the hood? Let's use Apache Iceberg as our example, because its metadata layer is a nice clean three-tier design.

At the top, you've got metadata files. These record the table's schema, how it's partitioned, and the full history of snapshots. Below that are manifest lists — every snapshot points to a manifest list that says which data files belong to that particular version of the table. And then at the bottom are the manifest files themselves, which track each individual Parquet data file: where it lives, how many rows it has, and crucially, statistics like the minimum and maximum values in that file.

Here's why those statistics are the secret sauce. When a query engine runs, it walks down that tree — metadata, then manifest list, then manifest files — *before* it touches any actual data. Using those min/max stats, it can skip entire partitions and entire files that couldn't possibly match your query. That's called partition pruning and file pruning, and it dramatically cuts down how much data actually gets scanned. This same design also lets you switch table versions atomically across different engines, without needing some central lock manager coordinating everyone.

Delta Lake works on a similar principle, but it's more Spark-centric. At the root of a Delta table there's a folder called `_delta_log`, and it stores transaction records as JSON. Then, every ten versions, it writes out a Parquet checkpoint so engines can load the state faster instead of replaying every single change from the beginning.

Now let's compare the two formats head to head, because this is the choice most teams actually agonize over.

Apache Iceberg came out of Netflix and is now an Apache Software Foundation project. Its whole design philosophy is cross-engine interoperability and handling really large-scale partitioning. It supports lots of catalogs — Hive, Nessie, REST — and it's read by a broad lineup of engines: Snowflake, Dremio, BigQuery, Flink.

Delta Lake came out of Databricks and now lives under the Linux Foundation. Its focus is Spark performance and making data manipulation — inserts, updates, deletes — really simple. Its catalog story centers on Unity Catalog, and its engine support is primarily Databricks and Spark.

And here's the really interesting development — the two formats are converging. In 2025, the big trend is Delta Lake's UniForm feature. What UniForm does is let a Delta table simultaneously expose Iceberg-compatible metadata. So any engine that speaks Iceberg can read that Delta table as if it were native Iceberg. You write your data in Delta, and you read it from Snowflake. The slogan is "write once, read anywhere." Iceberg's moving the same direction too — its version three can read Delta. The walls between these two are coming down.

So how do you actually choose? It comes down to your ecosystem. If you're mostly living in the Databricks world, go with Delta Lake. If you need real multi-engine interop — Snowflake plus Spark plus Trino all reading the same data — go with Apache Iceberg. And if you genuinely want both worlds, that's exactly what Delta UniForm, or Iceberg paired with a multi-engine catalog, is there to give you.

Let me leave you with the three things worth remembering.

First: the Data Lakehouse isn't a concept anymore, it's the default starting point. If a data team is designing a new system today, this is where they begin — no more running a lake and a warehouse in parallel and syncing between them.

Second: the whole architecture rests on one simple principle. Put reliability in the metadata layer, and leave flexibility and low cost down in the object storage. That separation is the entire trick.

And third: for the format choice, don't overthink it. Databricks shop, pick Delta. Multi-engine future, pick Iceberg. And thanks to UniForm and the convergence happening right now, that decision is getting less permanent — and less scary — by the year.