I spent years in master data management before I worked on the lakehouse, and the same truth held in both worlds: data quality is rarely a tooling problem. The tools are fine. The problem is that someone has to sit down and write the rules, then keep writing them as the data changes, forever.
Databricks has quietly built one of the better engines for enforcing those rules. The trouble is the engine runs on rules a human still has to author. That is the actual bottleneck, and it is exactly the kind of work large language models are good at. Not replacing the engine. Filling the gap above it.
Here is where that plays out, walking the data quality lifecycle from rule-writing to dedup. I will be clear about which parts are shipping platform features and which are patterns you build yourself, because that line matters and most write-ups blur it.
First, what the platform already gives you
Two primitives do most of the heavy lifting, and both are real, GA Databricks features.
Pipeline expectations (the declarative checks formerly branded as Delta Live Tables expectations) let you attach constraints to a table and decide what happens on violation: keep the bad rows, drop them, quarantine them, or fail the pipeline. The results are stored, so you can watch quality trend over time instead of finding out at 2am.
Lakehouse Monitoring, powered by Unity Catalog, profiles your tables automatically. It tracks statistical drift, the percentage change in nulls, distribution shifts, and lets you define custom metrics in SQL. It runs time-series, snapshot, and inference analyses and writes the results to metric tables you can alert on.
Together they are a strong mechanism. Define a rule, enforce it, observe it. The catch is in the first two words: define a rule. Every expectation and every custom metric is a line a human wrote and a human maintains. On a 40-table Info Mart that is tedious. On 400 tables it is a full-time job nobody wants. That is the toil AI can take off your plate.
1. Drafting expectations from the schema and a sample
The blank-page problem is real. Faced with a new table, most teams write three obvious expectations (not null on the primary key, a range check on the amount column, a date sanity check) and move on, leaving the long tail of real constraints unwritten.
An LLM closes that gap fast. Feed it the table schema plus a few hundred sample rows and ask for candidate expectations. It will propose the obvious ones, but it will also notice that country_code is always two uppercase letters, that status only ever takes five values, that created_at never precedes account_opened_at. You review the list, keep what is right, and you have spent ten minutes instead of an afternoon.
The important word is review. The LLM drafts; you ratify. It is a first-draft generator for the rules, not the source of truth. But going from blank page to a reviewable draft is most of the work, and it is the part people skip.
2. Triaging failures, not just catching them
Catching a violation is easy. Understanding it is the expensive part. An expectation fires, a pipeline quarantines 12,000 rows, and now someone has to figure out whether a source system changed, an upstream join went wrong, or it is a genuine data problem in the world.
This is where the agentic side of Mosaic AI gets interesting, as a pattern you assemble rather than a button you press. Give an agent read access to the failed rows, the expectation that tripped, recent schema-change history, and the upstream lineage from Unity Catalog. Ask it to produce a first-pass diagnosis: likely root cause, severity, and which upstream table to look at first. It will not always be right. But a triage note that is right 70% of the time, attached to the alert before a human opens it, turns a 30-minute investigation into a five-minute confirmation.
The value is not the detection. The platform already detects. The value is compressing the human time between "something broke" and "here is probably why."
3. Documenting rules in language the business understands
Every data quality rule has two audiences. Engineers read the SQL. The business owner who actually knows whether the rule is correct cannot. So rules drift out of sync with the business meaning they were meant to encode, and nobody catches it because nobody who could read the rule understood the intent.
An LLM bridges that cheaply. Point it at an expectation or a custom metric and have it generate a plain-language explanation: what this rule checks, why it likely exists, what a violation would mean for a report downstream. Store that next to the rule. Now the finance lead reviewing the data contract can actually confirm "yes, orders really can't predate the customer record" instead of nodding at a CASE statement.
This is the least flashy item on the list and quietly the highest-leverage. Documentation is the first thing to rot, and the thing whose absence causes the most silent damage.
4. Anomaly classification on top of monitoring metrics
Lakehouse Monitoring will tell you a metric moved. It will not tell you whether the move matters. Month-end always spikes transaction volume. A new market launch always shifts the geographic distribution. A backfill always dents the freshness numbers. These are not incidents, but a naive threshold alert treats them as such, and alert fatigue sets in within a week.
Feed the monitoring metric tables and a short window of history to a model and ask a sharper question than "did this cross a threshold." Ask "is this change consistent with the patterns we have seen, or is it genuinely new." Seasonal, explainable, expected: suppress. Novel and unexplained: escalate. You are using the platform's metrics as the input and adding a layer of judgment on top, so the alerts that reach a human are the ones that deserve one.
5. Fuzzy duplicate detection with embeddings
This one comes straight from the MDM world, and it is the angle classic data quality tooling has always handled worst. Exact-match dedup is trivial. The hard cases are "Acme Corp" versus "ACME Corporation Inc" versus "acme corp.", the same customer entered three slightly different ways across three systems.
Embeddings handle this far better than the edit-distance and phonetic rules MDM tools have leaned on for decades. Embed the entity records, index the vectors, and near-duplicates cluster by semantic similarity rather than surface string distance. On Databricks you can run this directly against Delta tables and keep the whole workflow on the platform, vectors and all. It will not auto-merge anything (you do not want it to), but surfacing the candidate clusters for a steward to confirm is the 90% of the job that was previously manual.
The honest summary
None of this replaces the engineer or the steward. Every item above is human-in-the-loop by design: the AI drafts, proposes, triages, and clusters; a person ratifies. That is the right shape, because data quality is ultimately a judgment about whether data matches the world, and a model does not know your world.
What changes is where the human time goes. Today too much of it goes into authoring and maintaining rules, reading SQL nobody else can read, and chasing alerts that did not matter. Push that work onto the model and the same team covers far more tables with far less toil.
The platform side is solid and shipping: pipeline expectations and Lakehouse Monitoring give you the mechanism. The AI side is mostly patterns you assemble yourself today, not features you switch on. But the patterns are real, they work now, and the gap they fill, the rule-writing bottleneck, is the one that has slowed data quality work for as long as I have been doing it.
If you have built any of these on your own stack, I would genuinely like to compare notes on what held up in production and what did not.