Brickster Newsletter
|
| 01 |
Your next AI use case starts with an invoice
Parse the PDF, then check the fields against data you already trust.
|
|
| 02 |
This week in Databricks
AI spend in budgets, a new jobs default, upstream triggers, and a runtime deadline.
|
|
| 03 |
From brickster.ai
Brickster Hub: practice a Databricks interview out loud, on TestFlight.
|
|
|
|
|
01 / THE LEAD
Your next AI use case starts with an invoice
Databricks has made document processing easier to put into a pipeline. The useful test is how much checking remains after extraction.
|
|
|
Picture an invoice arriving as a PDF.
Someone opens it, finds the supplier, copies the invoice number, checks the currency and enters the total. Then they look for the purchase order. Something doesn't match, so another person gets involved.
That workflow gives an AI project a concrete starting point: a document, an expected result and a person who can tell you whether it helped.
|
| Shift |
→ |
Problem |
→ |
Pitch |
→ |
Catch |
→ |
Your move |
|
|
1. The shift
Document AI can sit in a job
|
|
On November 4, Databricks introduced AI Function tasks for jobs in beta. Teams can configure document parsing, information extraction and classification as job tasks without writing SQL for those steps. Results land in Unity Catalog tables for downstream processing.
A workspace admin must enable Lakeflow AI Function Task on the Previews page. The underlying AI Functions must also be available in your workspace region. For a data team, the tasks then fit a familiar operating model: scheduled runs, task dependencies and outputs that other pipelines can consume.
|
|
2. The problem
Reading an invoice is only part of processing it
|
|
A supplier's name on the PDF may differ from the name in your purchasing system. A number labelled "total" might exclude tax. Two uploaded files might represent the same invoice.
Extracting a plausible value doesn't resolve those questions.
This is where your existing data becomes useful. Supplier records, purchase orders and previously processed invoices provide the context needed to check what the model found.
|
|
3. The pitch
Start with one narrow workflow
|
Invoice PDF |
→ |
Parse |
→ |
Extract |
→ |
Validate |
→ |
Review |
|
|
Once invoices are available in a Unity Catalog volume, an ai_parse_document task can read their content. An ai_extract task can then pull out fields you define: supplier, invoice number, date, currency, purchase-order reference and total. Databricks documents this parsing-to-extraction pattern directly, including an invoice example.
Next, build the checks around those candidate values. Match the supplier against your master data. Look for duplicate submissions. Compare amounts with purchase orders and verify arithmetic according to your business rules.
Give exceptions a review queue, with the original document available alongside the extracted values.
Those validation and review steps are your implementation. The new AI tasks supply the document-processing stages.
|
|
4. The catch
Two details worth putting beside the demo
|
| Detail |
What it means |
Extracted fields live inside response |
The extraction task stores its full result as a VARIANT in the response column. It does not add a column per field. You still select the values you need for joins, validation and reporting. Optional citations can point a reviewer at the source text. Extraction docs |
| Prompt changes leave earlier results unchanged |
Each run processes new and changed inputs. Changed fields or instructions apply only to inputs processed after the change. To apply revised instructions to all current inputs, set a new output table name. The next run processes those inputs again, adding cost. Processing behaviour |
|
|
A better prompt doesn't automatically fix yesterday's invoices. A prompt change deserves a release decision: what changed, which records need another pass, and how will you compare the results?
|
|
5. Your move
Start with one invoice type
|
|
Choose one invoice type and build a small, manually checked sample. Measure the current manual process first, including checking and corrections, so you have a baseline for time and cost. Include awkward examples: a poor scan, a missing purchase-order reference, a discount and a duplicate submission. Keep some documents aside to test changes independently.
|
| 1 |
Field accuracy
Are the important values correct?
|
| 2 |
Review effort
How many minutes of checking and correction remain?
|
| 3 |
Total cost per accepted invoice
Include processing, retries and human work.
|
|
|
Keep existing approval controls in place during the pilot. Then give the results to the person who currently processes those invoices. Ask them to complete the same work using the extracted records.
Their time, corrections and unresolved exceptions will tell you how useful the pipeline really is.
|
|
|
|
02
|
📰 This week in Databricks
|
|
Cost · Catch-up
AI Functions spend counts in Unity Gateway budgets
Since November 1, budgets have included ai_parse_document, ai_extract and ai_classify, on batch and real-time paths. Release note. Set alert or blocking thresholds before the first scheduled run. These controls are approximate: active requests can continue and final charges can exceed the threshold. Budget limits.
|
|
Jobs
New jobs compute defaults to Auto access mode
Beginning November 9, new jobs compute created in the UI defaults to Auto instead of Dedicated. Auto uses Standard unless you pick a machine learning runtime, a GPU instance, or a Databricks Runtime below 14.3. Existing jobs stay as they are. A cluster policy that enforces another mode takes precedence. Workspaces on the compliance security profile get this change in December. Release note. Check any runbook that assumed Dedicated on a brand-new job cluster.
|
|
Orchestration · Catch-up
A job can start when upstream jobs finish
In beta since November 3, a Lakeflow Job can wait for each monitored upstream job to finish with a matching result, including across workspaces on the same Unity Catalog metastore. Configure dependencies on the downstream job and enable the preview in every participating workspace. Release note. Useful when invoice checks should wait for a purchasing load. Verify data readiness too: the default success filter also accepts SUCCESS_WITH_FAILURES. Trigger behaviour.
|
|
Dates
A model retirement and a runtime change
Checked November 9 against the release notes and upcoming-change notices.
| Nov. 5 |
DeepSeek V4 Flash (0731) is retired. Databricks points to DeepSeek V4.1 Flash (1017) as the replacement. |
| Jan. 2027 |
From January 2027, AI Functions whose current minimum runtime is 15.4 LTS will need 17.3 LTS or newer. This excludes ai_similarity; REST API use is unaffected. |
Find affected SQL queries and upgrade the compute they run on.
November 18 reminder: new tables containing VARIANT columns, created on Runtime 17.3 or newer, will get shredding automatically. Runtime 15.4 cannot access those tables; upgrade affected consumers to 16.4 or newer. Shredding notice.
November 30 reminder: integrations using the Genie One MCP Beta endpoint must move to the generally available MCP in Unity Gateway. Migration notice.
|
|
|
|
📱 Brickster Hub is in TestFlight
Practice a Databricks interview out loud on iPhone: 24 questions in six decks, a checklist you can correct, and exam prep from the same 171-question bank as the site. No Brickster account required. iOS 16.4 or later. Not on Google Play yet.
Ask for a TestFlight invite →
|
|
|
Follow Brickster apps and see what’s new on our apps page.
|
|
What does Databricks document about reprocessing AI Function tasks?
The Brickster Assistant searches our indexed archive and answers with source links.
|
|
|
Enjoying the Brickster Newsletter?
Forward this to a colleague who works with Databricks.
Forward this issue ↗
|
|