Why Nobody Tells You the Hard Part: CDC Lakes, Silver Layers, and the Azure Bridge

We thought the hard part was getting CDC into the lake.

It wasn't.

Oracle changes were landing in Kafka. Bronze Iceberg on MinIO was catching up almost immediately. The pipeline was alive, the events were there, and the lake was doing exactly what we expected it to do.

And yet people were still waiting for the table.

That is where the interesting part started.

This is the third essay in a series on hybrid lakehouse architecture for manufacturing. The first was about the catalog decision. The second was about what happens when you have to build a lakehouse around a brownfield estate that cannot simply be switched off.

This one sits between those two: current state, silver materialization, different freshness requirements, and the slightly uncomfortable question of what Azure should actually consume.

The setup

The environment is not unusual for manufacturing, but it is awkward enough to expose the gaps between architecture diagrams and production.

We run an IFS Oracle CDC platform on-prem:

There are also three deliberately different tracks:

A naming trap before we go further. In our docs, Tor C means the Databricks consumer path. Track C is an older ADR about Spark streaming into PostgreSQL — not Databricks. Do not conflate them.

The distinction matters because these tracks do not pay the same price for freshness. And that, it turned out, was the whole point.

Bronze was never the problem

The public version of this architecture is wonderfully simple:

CDC into Kafka → land it in the lake → expose it to Databricks → build dashboards.

And, to be fair, a lot of that works.

Debezium + Kafka is a good change log. Lakehouse medallion language is useful. Auto Loader and AUTO CDC make state management easier if Databricks owns the state. Iceberg keeps getting better at MERGE and DELETE. Catalog federation and sharing mechanisms keep improving.

None of that changes the uncomfortable bit.

Bronze is a log. It tells you what changed. It does not tell you what the current Oracle row looks like.

That sounds obvious. It becomes considerably less obvious when someone looks at a bronze table that is effectively caught up and asks:

Why isn't the report fresh?

Because the report is not reading the log. It is reading state — and state has to be materialized somewhere.

Then we started trying to make Azure simple

The first idea was obvious:

Let's just read Iceberg from Databricks.

Protocol support exists. The storage credential story is where things become less exciting.

MinIO is not AWS S3. The data plane still has to work. We have a VPN between the environments. And once you start scanning uncompacted CDC micro-files from Azure, the fact that something is technically readable becomes much less interesting than how long it takes to read it.

So:

Fine. Let's rclone the lake to ADLS.

Now we have micro-files crossing a VPN and another copy of the data.

That solves a different problem.

Then:

Let's just read Kafka in Databricks and rebuild it with Auto Loader / AUTO CDC.

That works. It also creates a second silver fleet.

Kafka is transport. Making Azure another owner of current state is a product decision, not a shortcut.

Then we tried something much less ambitious:

What about Trino JDBC from a Databricks notebook?

That actually worked as an interim solution. A ~9.2M-row count took ~3.2 seconds: useful, and incomplete — there was no durable Unity Catalog table.

And somewhere along the way the Azure question became almost secondary:

OK, forget Azure for a moment. Why is silver still slow?

This is where engine tourism begins.

The expensive middle

Our pilot Spark/dbt silver MERGE fleet routinely takes around 29–31 minutes of wall-clock time per hourly run. That number sounds terrible until you look at what it actually means.

The delay between a change event reaching bronze and the corresponding current state appearing in silver is usually much smaller — around 0–12 minutes for many tables, with light tables often around 0–2 minutes. Those are different metrics. A 30-minute job does not mean every row is 30 minutes stale.

The important question is not:

How long did the silver job run?

It is:

How long after T0 did this particular change become visible in current state?

That distinction matters because a quiet table can look stale forever if you measure freshness by the timestamp of its last change. So we started measuring propagation rather than wall-clock age.

What the pilot showed

One table, inventory_part_char, was pulled out of the fat fleet and given a dedicated run. Its bronze-to-silver delay dropped to roughly 3–4 minutes.

That was a useful result — not because Spark suddenly became faster, but because we stopped treating every table as if it deserved the same SLA. The silver problem was starting to look less like a SQL problem and more like an operations problem: how do you keep a batch state factory honest when hundreds of tables are sharing it?

Why Track A is faster

There is a simple mechanical difference between the two paths.

Track A:  Kafka event ──► JDBC Sink ──► UPSERT row in PG
                                          (at ingest)

Track B:  Kafka event ──► Iceberg APPEND
                              (bronze)

                         …wait for cron…

                         ──► Spark/dbt MERGE
                              (silver)

Track A builds current state at ingest. Track B records the event at ingest and builds current state later.

People sometimes describe this as:

PostgreSQL is faster than Iceberg.

That is the wrong framing. The difference is not primarily the database. It is when we choose to pay for state.

With Track A, one change becomes one upsert by primary key. With Track B, a bronze window becomes a MERGE over many keys. Track A has a continuously running JDBC sink. Track B has a Spark job that has to start, scan, merge, commit — and then do it again.

The model is actually the same.

Track A tables are current-state heaps, roughly SCD1 plus soft-delete: INSERT/UPDATE upsert by PK; an Oracle DELETE leaves the row with __deleted = true; history remains in Kafka / Iceberg.

SCD1 is not why this is faster. SCD1 is why continuous upsert is a sane model here.

Could Track B do the same? Conceptually, yes. Our silver *_dedup tables are already current state. Mechanically, not with the Connect stack we have: state-at-write in the Iceberg sink was not available in the way we needed it, so downstream dedup stayed mandatory.

That leaves a few possible paths:

The last one is less exciting than introducing another engine. It is also much easier to operate.

Silver is a cost map

Once you look at the architecture this way, the layers stop being fashionable names and become a map of where state is paid for.

You cannot honestly claim to materialize useful gold views straight from an event stream without holding the necessary dimension state somewhere. You need the initial load. You need continuous upsert or MERGE. Or you accept wrong answers for quiet keys forever.

Streaming joins without hydrated state are a demo. ERP reporting is not a demo.

That was another useful realization: there is no engine that magically removes the bill for current state. It can move the bill. It can make the bill cheaper. It can make the bill easier to operate. But it cannot make the bill disappear.

Engine tourism

We did what most architecture teams eventually do: we looked at engines.

In 2026-04 we ran an OLAP/CDC sink bake-off. StarRocks won on paper. In 2026-05 we rejected it under a zero-new-platform principle. In 2026-08 we re-checked — and the verdict was unchanged for our constraints. The re-check still rejected StarRocks under zero-new-platform. A 2026-08-11 Azure port probe also showed Sharing/StarRocks-class endpoints unreachable while HMS/MinIO controls were fine: another reason not to bet Tor C on a new OLAP logo.

Asking which OLAP engine removes silver is a little like asking which database removes indexes.

You can move the index.

You cannot abolish the need for one if you need to query current state.

So engine tourism was useful. It showed us that the problem was not waiting for the right logo.

And then there is Tor C

Databricks is still a real requirement. But the simplest-looking path is not necessarily the simplest path to operate. The constraints are fairly concrete:

• MinIO, not AWS S3.
• UC credential limitations around that storage.
• Site-to-site VPN carrying the data plane.
• CDC micro-files that need compaction.
• Air-gapped Spark images materializing state on Kubernetes.
• Azure consuming the result.

Compaction is mandatory for Azure file I/O, but it is not free of risk. On 2026-08-11 a Trino optimize(50MB) on a silver *_dedup table rewrote the tip to a few large Parquet files with a corrupt primary-key column chunk (PageHeader / uncompressed_page_size). That broke downstream gold — and the writer was on-prem Trino OPTIMIZE, not Databricks.

Fair rule: full column probe after OPTIMIZE (count on PK + peer columns); auto-rollback on failure; do not leave a blind large-file tip in place for VPN benches.

Near-RT state lives on Tor A (Kafka → JDBC → PG) on-prem. From Azure Databricks, direct JDBC to Tor A PG timed out on 2026-08-11. The preferred bridge for near-RT and lake in one session is Trino catalogs — Iceberg joined to Tor A PostgreSQL — not a new firewall hole by default.

This leaves a few options.

The important part is not which option wins. The important part is what we do not want to do:

Do not make Azure rebuild silver just because it is easier to connect to Kafka than to the existing state.

That would turn one state owner into two. We would have escaped the problem by duplicating it.

For Tor C, our current R&D exits are therefore fairly simple:

• C0 — Trino from UC notebook: keep using (proven).
• C0-pg — Tor A as Trino catalog: next federation step (not Azure→PG JDBC).
• C0b — HMS+MinIO: lab/prerequisite only; prefer Trino for product.
• C-CRIT — OpenSharing: blocked until Sharing/FW or AIStor; do not pretend VPN scan is the only remaining gate.
• C1 — silver → ADLS when zero-copy stays blocked.
• C2 — one-topic Auto Loader experiment only.

The useful automation was not another engine

At some point we built a small lineage-lag probe — not because we needed another platform, but because we needed to see what was actually happening.

$ kc_lineage_lag_matrix --view=propagation --sample latest
 
table                    bronze_after_t0  silver_after_t0  track_a_after_t0  wall_fleet
orders_tab               0m               4m               1m                —
inventory_part_char      0m               3m               1m                (dedicated)
cft_cost_heavy           0m               118m             9m                (fat fleet)
 
$ echo "job wall != row staleness"
 
job wall != row staleness
 
[OK] report delay after T0 — not now-MAX(source_ts)
[WARN] hot tables in fat silver fleet skew p50 — split before buying another engine

This turned out to be more useful than another architecture slide. We could see which tables were actually hurting; when a table was delayed because of its own workload and when it was simply waiting behind everything else; and the difference between a slow state factory and a stale row.

Silver fails as an operations problem long before it fails as a SQL problem.

So what do we actually do?

We stop looking for one silver SLA. The estate has different requirements: some tables need a few minutes; some are perfectly fine with an hourly refresh; some are rarely queried; some consumers do not need current state at all. And Azure does not need to own the state just because it happens to be the next place where someone wants to query it.

The resulting matrix is deliberately boring:

Run A and B in parallel on purpose. Accept that the lake has a silver + cron SLA. Keep near-RT where it actually matters. Split hot tables out of the fat silver job. Measure wall-clock time and delay-after-T0 before and after. And advance Tor C independently: there is no reason to wait for silver to become perfect before proving whether Trino, OpenSharing, or an ADLS copy is the right bridge into Azure.

Build the SLA → track matrix for five or ten pilot tables. Assign each one to a lane. Change one operations decision. Measure the wall-clock delta.

That is progress. Engine tourism is not.

The questions I would ask before buying anything else

• Have we separated event landing from current-state materialization in stakeholder language?
• Do we report delay after T0 rather than the wall age of quiet tables?
• Do we know the silver job wall and per-table propagation delay as two different numbers?
• Is near-RT an explicit track, or are we hoping batch MERGE becomes magic?
• If Azure is the goal, are we asking Databricks to consume silver or rebuild it?
• Have we tested the data plane — VPN, micro-files and compaction — rather than only the catalog API?
• Did the “upsert” flag we flipped actually write equality deletes / deletion vectors?
• Is the proposed OLAP engine solving entity mirrors or event aggregates?
• Are we about to add a platform we cannot operate at 3 a.m.?
• Can we name the one table we will split from the fat silver job this month?
• If Azure must see near-RT PG state, are we federating via Trino — or opening a firewall we do not need?
• After OPTIMIZE/compaction, did we probe PK columns end-to-end — or only celebrate file-count reduction?

The part nobody tells you

The silver layer is not a failure of Iceberg, Spark, or Databricks. And it is not a failure of the team building the platform. It is the bill for current state.

Bronze is a log. Gold is a join over state. Azure is a geography problem wrapped around a governance problem. Near-real-time is an upsert rhythm, not a lake format.

Once you accept that, the architecture gets less exciting — and considerably more useful.

Partition the estate by SLA. Materialize state where the SLA demands it. Share silver — don't rebuild it. Measure delay after T0. Compact carefully before you cross the VPN — and probe the tip after OPTIMIZE.

That is the article we wished we had found before the frustration peaked. It still ends with work: matrices, crons, OpenSharing PoCs, Trino notebooks. Not a new logo on a slide.

Updated 2026-08-11 with Tor C network and compaction findings from the production pilot.

In this series

Hybrid lakehouse essays — catalog, brownfield reality, then the silver bill.

PART 1 · CATALOG

Iceberg Won. Your Catalog Choice Is Now the Real Architecture Decision

Open table formats made multi-engine lakehouses possible. In manufacturing IT, the catalog — not the format — defines governance, hybrid SLA, and whether production planning can survive without a single cloud path.

Read essay

PART 2 · BROWNFIELD

Brownfield Lakehouse: When You Cannot Turn Off the Nightly Batch

Three hundred tables still run on nightly batch while CDC incubates on-prem. Warehouse and finance data reach billions of rows. Chief IT Architecture in manufacturing means parallel paths — Iceberg, Delta, and PostgreSQL for legacy SSRS.

Read essay