Brownfield Lakehouse: When You Cannot Turn Off the Nightly Batch

Second in a series on hybrid lakehouse architecture for manufacturing. The first essay framed the catalog decision and production tiering. This one describes the brownfield reality behind that diagram — the constraints a Chief IT Architect cannot wish away.

Scale and source domains

Enterprise manufacturing ERP landscapes are not small pilots. A typical production platform loads more than three hundred operational tables into analytics every night. In the source database, row counts reach tens of billions across the full schema. The domains that dominate volume and operational risk are predictable:

Architecture that ignores this scale produces elegant slides and painful production incidents.

Batch still dominates — and you cannot switch it off

The production ingest path today is overwhelmingly nightly JDBC batch: hundreds of tables, a multi-hour window, landing in Delta Lake under Unity Catalog for Databricks consumers. Change Data Capture is real — often piloted on shop orders and a handful of hot tables — but it is incubation, not replacement.

Why not cut over? Because SSRS report suites, Power BI datasets, dbt models, and years of business logic sit on the batch path. Turning it off to "simplify architecture" would stop finance close reports and production KPIs the business runs every morning. As Chief IT Architect, my mandate is parallel operation until consumer migration is proven, not parallel promises.

Two lakehouse formats and three consumption paths

Brownfield hybrid architecture is messier than a single-format diagram suggests:

PostgreSQL is not a duplicate lakehouse for fun. It is a relational serving tier for legacy reporting that expects sub-second point queries on a hot working set. Tables in the billions of rows do not belong on that tier — they stay in Delta for cloud BI. Architecture explicitly scopes PostgreSQL to hot CDC tables; ultra-large financial and warehouse facts remain cloud-served. That scoping decision is architectural, not a performance tweak.

Warm and cold report tiers may later route through Databricks SQL Warehouse with ODBC instead of PostgreSQL — another branch in the consumer map, governed by freshness requirements rather than technology fashion.

Incubation mechanics

Initial load for high-volume tables cannot always ride the streaming bus: Oracle archive log retention and snapshot volume force bootstrap via batch extract with SCN capture, then CDC takes over incrementally. Supplemental logging rollout requires DBA windows. VPN or private link capacity becomes a single point of failure for the legacy cloud batch path. Every heavy hitter table needs its own ingest strategy — hash partitioning, date-range extraction, or CDC — because a full-table JDBC scan on billions of rows is last resort, not policy.

Delivery views (DLV) must present a single logical surface so consumers do not need to know whether today's row arrived via CDC, OData polling, or last night's batch. That abstraction is expensive to maintain during incubation. It is still cheaper than asking two hundred report owners to re-platform simultaneously.

$ kubectl get cronworkflows -n workflows 2>/dev/null | grep -E 'dbt|batch|pg'
dbt-silver-refresh          */2 * * * *     Active
dbt-gold-refresh            */5 * * * *     Active
nightly-batch-extract       0 23 * * *      Active
dlv-pg-sync                 */15 * * * *    Suspended
 
$ trino --execute "SELECT COUNT(*) FROM iceberg.information_schema.tables WHERE table_schema='dlv'"
 42
[WARN] CDC plus dbt path growing — batch still feeds 300+ sibling tables
 
$ echo "Consumers: SSRS ODBC | Trino JDBC | Databricks SQL"
Consumers: SSRS ODBC | Trino JDBC | Databricks SQL
[OK] three native consumption paths — architecture must tier, not unify by decree

Architecture as stewardship

Greenfield projects choose platforms. Brownfield manufacturing IT inherits them — then must tier consumers, document decisions in ADRs, and phase migration without halting the plant. The catalog decision from the first essay is what eventually binds these parallel paths under one governance model. Getting there is a programme, not a sprint.

Continue the series

Next: why silver materialization is the expensive middle — and how dual-track SLAs plus Tor C consumption beat a second MERGE factory in Azure.

PART 3 · SILVER

Why Nobody Tells You the Hard Part: CDC Lakes, Silver Layers, and the Azure Bridge

Bronze is easy. Silver MERGE is the bill for current state — and Azure should consume it, not rebuild it.

Read essay