Iceberg Won. Your Catalog Choice Is Now the Real Architecture Decision

As Chief IT Architect, I design enterprise data platforms where factory operations cannot wait for the cloud. This essay is the first in a short series on hybrid lakehouse decisions in manufacturing — not greenfield theory, but architecture under real constraints.

The problem is not the table format

In manufacturing enterprises, planning and production supervision depend on the same operational backbone as warehouse logistics and financial close: an ERP that records shop orders, inventory movements, and accounting entries in one transactional core. Shift planners ask whether a line will finish its order. Warehouse supervisors ask whether stock covers the next forty-eight hours. Finance asks whether ledger entries reconcile before period close. None of these questions tolerate a twenty-four-hour batch delay — and increasingly, none tolerate a multi-second round trip to a distant cloud region either.

At the same time, four structural pressures reshape what IT must deliver:

These forces do not argue against cloud analytics. They argue against depending on a single cloud path for everything that keeps a factory running.

Production planning needs a local path

When I map architecture for a production-heavy ERP landscape, the operational tier is non-negotiable. Change capture from the transactional database must feed a stream that lands in object storage close to the plant. SQL serving on-prem — through a federated query engine — gives planners and shift supervisors dashboards and ad-hoc queries within operational SLAs.

Cloud analytics remains valuable for forecasting, multi-site consolidation, and machine-learning features. The design principle is tiering: latency-sensitive supervision stays local; scale-sensitive intelligence syncs selectively upward. The catalog is what makes both tiers governable under one policy model instead of two siloed stacks.

Why the catalog — not Iceberg — is the architecture decision

Industry conversation in 2026 has largely settled the open table format question. Apache Iceberg interoperability across engines is real; vendor platforms increasingly expose REST catalog APIs. The decision that still defines enterprise outcomes is who owns the catalog layer: credential vending, attribute-based access control, audit attribution, and cross-engine consistency.

For a hybrid manufacturing platform, I evaluate three catalog patterns:

The hybrid pattern does not eliminate complexity. It contains it: operations teams get predictable local latency; enterprise analytics keeps Databricks or equivalent scale; security officers get one place to prove who accessed production data and when.

The matrix above is how I frame the conversation with CIO and plant leadership: not feature checklists, but trade-offs that survive production.

What this looks like in operation

A healthy hybrid deployment exposes the catalog as an API — engines request scoped credentials rather than sharing long-lived secrets. Local queries against Iceberg-served DLV views should sit comfortably under half-second to low-second p95 for operational dashboards. Cloud tables may number in the hundreds for enterprise BI; that path can remain authoritative for finance consolidation while incubation proceeds on-stream.

$ curl -s https://catalog.local/api/2.1/unity-catalog/tables/ops.production_orders
[OK] format=iceberg  engine=trino  policy=shift_read_only
 
$ trino --execute "SELECT approx_percentile(query_latency_ms, 0.95) FROM system.runtime.queries WHERE catalog='iceberg'"
 p95_latency_ms
----------------
            380
[OK] within ops SLA target under 500ms
 
$ databricks tables list analytics.production --output json | jq 'length'
327
[WARN] cloud path remains authoritative for enterprise BI during incubation

Chief IT architecture is a continuity mandate

My role is not to deliver a diagram that wins a workshop. It is to keep production planning, warehouse supervision, and financial reporting continuous while the organisation moves from batch-first ingestion toward event-driven serving — without a big-bang cutover that factory leadership would rightly reject.

The next essay in this series covers what that move actually looks like in brownfield: hundreds of tables still on nightly batch, Delta consumers in the cloud, PostgreSQL feeding legacy SSRS, and CDC incubating in parallel. That is where architecture earns its salary.