Brickster Newsletter

Change data feed stops being a decision, mostly

Mostly, no. On September 1 the release note announced that Automatic change data feed (Auto CDF) is generally available, and it describes a feature that computes row-level changes at query time using row tracking, the row IDs and row commit versions Databricks keeps for every row once you turn it on, “without requiring you to manually enable change data feed on each Delta Lake or Apache Iceberg v3 table”. The decision you used to make per table, in advance, before you knew whether you would ever need the history, is largely gone. Except on the tables where you were most likely to make it.

Read the two modes side by side and the shift is clear. The change data feed docs page describes legacy CDF this way: it “Materializes changes during table writes. Only supports Delta Lake tables. Requires individual table configuration.” It describes the automatic mode this way: it “Computes changes during table reads using row lineage metadata. This doesn’t require individual table configuration and works on Delta Lake and Apache Iceberg v3 tables.” Write time becomes read time, and Iceberg v3 comes along for the ride.

Nothing changes in how you read the feed. The docs say automatic change data feed "uses the same table_changes() and readChangeFeed APIs as legacy change data feed". That means SELECT * FROM table_changes('<table_name>', 0) in SQL and .option("readChangeFeed", "true") in Python, exactly as before, and the release note says it works with batch queries, Structured Streaming, and Delta Lake Sharing. Existing consumers keep their shape.

On performance, the release note and the docs page do not say the same thing. The September release note says Auto CDF removes write-time overhead and makes MERGE and UPDATE operations “about 15% faster on tables that you query for changes”. The change data feed docs page makes the same argument with no number at all, saying only that automatic change data feed “improves write performance and reduces storage costs, compared to legacy change data feed”. Both claims are the vendor’s, we have measured neither, and 15% is the release note’s number rather than a benchmark you can quote back to your platform team.

The entry bar is a runtime plus a catalog. The docs page asks for “Databricks Runtime 19 LTS or above”, the table registered in Unity Catalog, and then either a managed table “in Delta Lake format with row tracking enabled or in Iceberg v3 format”, or an external table “in Delta Lake format with row tracking enabled”. The release note adds the same runtime floor and the same row tracking condition, so there is no disagreement there.

Row tracking is the part people underestimate. Turning it on for an existing large table is not a costless metadata flip: the docs say Databricks assigns row IDs and row commit versions to existing rows, which takes time and creates multiple table versions. On your biggest tables, plan it like a rewrite window, not like a checkbox in a settings pane. We have not measured how long it takes, so treat this as a caution rather than a number.

Now the exclusions. Auto CDF “isn’t supported on tables with row filters or column masks”. Row filters and column masks are the governance controls you apply when only some rows or some columns should reach some people, which in practice means the tables holding something worth protecting. On top of that, “external Iceberg clients can’t query automatic change data feed”, change data feed queries “can’t span table versions where a non-additive schema change occurred”, and inside a multi-statement transaction, “if the source table was modified during the transaction, automatic change data feed isn’t supported”. Those last three are ordinary edges. The first one is the story.

If this feels familiar, it should: we covered the neighbouring step in issue #14 on August 11, when materialized views learned to publish their own change feed in Beta. That was one table type gaining the ability to emit changes. This is wider and further along: no table has to be configured for change data feed at all, the changes are computed when you read instead of when you write, and this one is GA rather than Beta. The community signal is thin days after GA: one Databricks Community thread from August 13, “Lakebase synced table doesn’t recognize Auto CDF on a SDP materialized view”, marked answered, and we have the title only.

Runtime 19 LTS or above The docs page and the release note both set the floor at Databricks Runtime 19 LTS or above. If your job runs on anything older, stop here.

Unity Catalog, and the right table shape The table has to be registered in Unity Catalog, and be either a managed table in Delta Lake format with row tracking enabled or in Iceberg v3 format, or an external table in Delta Lake format with row tracking enabled.

No row filters, no column masks The docs are flat about it: automatic change data feed isn’t supported on tables with row filters or column masks. We did not confirm what your options are on those tables, so check before you plan around it.

Three narrower blocks: external Iceberg clients, non-additive schema changes, multi-statement transactions External Iceberg clients can’t query automatic change data feed. Queries can’t span table versions where a non-additive schema change occurred. And in a multi-statement transaction it isn’t supported if the source table was modified during the transaction.

Budget for the row tracking backfill Enabling row tracking on an existing large table makes Databricks assign row IDs and row commit versions to existing rows, which takes time and creates multiple table versions. The docs we read do not say whether row tracking is already on for your managed tables, so check a real table before you assume either way, and measure the backfill on a copy before you schedule it.

So: do you still have to turn change data feed on? On most tables, no, and that is a genuine simplification, because the worst CDF failure was always the one where nobody enabled it and the history you needed was simply never written. But the tables carrying row filters and column masks are the tables holding something worth protecting, and those are exactly the ones Auto CDF skips. The feature arrives on the ordinary tables and stops at the governed ones, and the docs do not say whether that changes. The honest answer to “is CDF on?” moved from yes or no to “depends on the table”, and the tables where it depends are the ones you would most want a clear answer about.

Databricks Runtime 19 stops having point releases

From the what’s-coming page: “Starting with version 19, Databricks Runtime will use a unified release model. Instead of multiple feature versions (for example, 19.0, 19.1, 19.2), each major version will have a single release notes page.” The cadence is new features and fixes approximately weekly, with updates differentiated by date, and after approximately six months a version transitions to long-term support with three years of support. DBR 19 went GA on July 23, and the September 1 entry is a dated update on that same single page, which is the model working as described. It also means a cluster picks up changes when it restarts. “Pin 19.2” stops being a sentence you can say, so your reproducibility story now rests on restart timing rather than a version string.

pipeline_events system table (Beta)

The August 31 release note describes a table that “records Lakeflow pipelines event log entries for pipelines in your account, capturing lifecycle transitions, flow progress, data quality metrics, errors, and other operational data across all pipelines and workspaces in a region”, intended for querying historical pipeline activity, building alerts on pipeline failures, and correlating pipeline behavior with other Lakeflow system tables. It is append-only, regional, and carries 13 months of retention at no extra cost. Access needs metastore admin and account admin, or USE and SELECT on system schemas. Anything you build on it has a rename ahead: the Beta path is system.lakeflow_pipeline_events_preview.pipeline_events, and at general availability the table moves to the lakeflow schema.

Four hosted models in one week, and one repeated sentence

Between August 29 and September 4, four models landed via Foundation Model APIs and Unity Gateway: OpenAI GPT-6 Astra on Unity Gateway on September 4 as pay-per-token, Google Gemini 3.8 Flash on September 2, Anthropic Claude Fable 5.1 on September 1, and Zhipu AI GLM 5.3 on August 29. Only the GLM 5.3 note carries specifics, which is why it is the one we can describe: supported US and Americas regions, a text-only mixture of experts model, context up to 1,048,576 tokens, and up to 65,536 output tokens. All four notes carry the line “Customers are responsible for ensuring their compliance with applicable terms.” We read that as the gateway making a model easy to switch on and leaving the vendor’s terms for you to read.

Quick links this week:

→ brickster.ai/exam-prep: Sets for the Data Engineer Associate and Professional, Data Analyst Associate, Machine Learning Associate and Spark Developer exams. 15 questions a run, unlimited runs, no signup.

→ brickster.ai/daily-brick: Today’s five. Or pick one topic at one difficulty over at brickster.ai/practice.

→ brickster.ai/digest/week: The full weekly digest behind section two, with every source we read.

🔍 Do you know which of your tables carry row filters or column masks?

The Brickster Assistant searches our full archive and answers with citations.

Enjoying the Brickster Newsletter?

Forward this to a colleague who works with Databricks.