LakeFlow

Lakeflow connect CT pipeline keeps running multiple connections in source

[Discussion] Lakeflow Jobs: If your data catalog could trigger a job on any change what would it be?

In addition to triggering on time (cron) intervals you can trigger on data landing: files arriving on a share, table gets a new commit and (soon) jobs completing upstream. While we were building Model update triggers (which trigger when UC registered models change), we were discussing the possibility of triggering on any entity in UC’s metadata changing. Examples here could be: Someone tags a column PII and you want to trigger a scan or retention job Someone changes a grant and an access review job gets triggered Someone is given SELECT permission on a table More generally it could be: Schema change on an upstream table (column added, dropped, renamed, type changed): run compatibility tests, rebuild the downstream model, ping the owner before it breaks. PII or sensitivity tag added: kick off a scan, apply retention, open an access review. Grant or ownership change: access recertification, sync to the entitlement system, log for audit. Table created, renamed or dropped in a schema: auto-register or clean up downstream assets. Questions: Which of these would you use and what are just noise? What kind of change would you really really want that you currently do by hand or after the fact? Anyone doing this off audit logs, system tables or just polling running jobs? submitted by /u/saad-the-engineer [link] [comments]

Private Network Gateway

Private Network Gateway is one of the year's biggest network innovations. Serverless can now be part of your VNET! more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

Best practices for SLA monitoring and automated retries across hundreds of Lakeflow Jobs

igrating from Cron-based Airflow to Lakeflow's Data-Aware Triggers — Real-world experiences?

Lakeflow Connect SQL Server Connector

I recently enabled Lakeflow Connect (lfc) on the source database - the issue is, some of the tables in the source database (managed by another team) does NOT have a primary key (which means that in lfc, a __databricks_id is used to identify a unique record). Thus, the DBAs enabled CDC on the source database. However, when I ingested the data into DBX using the Lakeflow Connect Managed SQL Server Connector, one of the tables in the source database had duplicate records (two or more records with the same value across all columns). This caused my Lakeflow Connect pipeline to break. Any ideas on how to fix this? (Other than dropping duplicate records in the source DB and implementing a unique constraint on the source DB)? I was wondering if there is a specific setting in Lakeflow Connect that I can toggle that I'm missing. submitted by /u/RazzmatazzLiving1323 [link] [comments]

Tutorial: Transform your Lakeflow Connect ad data into visual and conversational analytics

Read this if you use Streaming Tables in Lakeflow Spark Declarative Pipelines

🚀 We’re excited to announce that Lakeflow Spark Declarative Pipelines (SDP) now supports creating “vanilla” (i.e., non STREAMING) MANAGED TABLES and writing to them via one or more append flows , using the new CREATE TABLE ... FLOW ( SQL ) and create_table() (Python) APIs . What is this Beta? This Beta allows creating a managed table that is populated by append flows: CREATE TABLE ... FLOW (SQL) / create_table() + @append_flow (Python) create a managed table written by one or more flows. Fan multiple sources into one table — declare several flows targeting the same managed table. Full table surface works: partitioning, liquid clustering, expectations, row filters, table properties, and private (pipeline-local) tables. import_checkpoint on append_flow , which migrates an existing Structured Streaming workload into a pipeline without reprocessing the source — the flow imports the query's existing checkpoint and resumes from the last committed offset with state intact. Example (Python): from pyspark import pipelines as dp dp.create_table("combined") dp.append_flow(target="combined") def from_a(): return spark.readStream.table("source_a") u/dp.append_flow(target="combined") def from_b(): return spark.readStream.table("source_b") Example (SQL): CREATE TABLE events PARTITIONED BY (bucket) FLOW INSERT BY NAME SELECT id, bucket FROM STREAM read_files('abfss://my_path', format => 'json'); Where do we need help? We are in Beta, so there might be some rough edges. Please take this for a spin and share your feedback here . What’s next? Managed Tables support for other flow types (AutoCDC, Replace Using, and Replace Where) is coming soon! Learn more CREATE TABLE ... FLOW (SQL reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-create-table-flow create_table (Python reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-create-table import_checkpoint on append_flow — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-append-flow Questions, feedback, or help: comment below or share feedback in the form: https://forms.gle/7bGP5FYN7P1Z4WP27 submitted by /u/SlightImagination250 [link] [comments]

UC secrets in Key Vault

Secrets in Unity Catalog is a great feature introduced a few weeks ago, but since then, everyone has been asking to use Azure Key Vault as a secrets backend. Thanks to rapid development, we can now link our schema to Azure Key Vault; UC will read secrets as UC secrets, and permission management will be through Unity Catalog. In that scenario, you insert/update secrets in Azure Key Vault, but read/reference and grants can go through UC. more news https://databrickster.medium.com/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

Google Drive connector in Lakeflow Connect is now generally available (GA)

The Lakeflow Connect connector for Google Drive is now generally available ! It’s now easier than ever to ingest structured and unstructured files from Google Drive into Delta tables for analytics and AI workloads. You can configure a managed ingestion pipeline through the UI or managed API. Managed pipelines automatically handle incremental processing, automatic retries with exponential backoff for source API rate limits, failure recovery, and provide rich Google Drive metadata. For direct control over ingestion logic, you can also just use the Spark + SQL APIs directly: spark.read , Auto Loader, read_files, or COPY INTO pointed at Google Drive URLs. https://preview.redd.it/j65yaaft3uoh1.png?width=2048&format=png&auto=webp&s=0f9f13cc63572b991232a8fa617aa1fe6697369c Link to public docs + references: Google Drive managed connector documentation Spark + SQL APIs and examples Community blog and video tutorial: From PDF to insights Data + AI Summit session: Intelligent Document Processing with Lakeflow Common workloads include: Loading Google Sheets, Excels, CSV, JSON, and other structured files into Delta tables. Ingesting PDFs, Google Docs, Google Slides, and images. Parsing documents with ai_parse_document to prepare content for extraction, search, and agents. Examples of using the Spark + SQL APIs: Read an Excel sheet from Google Drive with spark.read : ​ df = (spark.read .format("excel") .option("databricks.connection", "my_gdrive_conn") .load("https://docs.google.com/spreadsheets/d/9k8j7i6f...")) Ingest unstructured documents + PDFs from a Google Drive URL with read_files , then easily parse them using ai_parse_document : ​ CREATE OR REFRESH STREAMING TABLE gdrive_documents_table AS SELECT *, "_metadata" FROM STREAM read_files( "https://drive.google.com/drive/folders/1a2b3c4d...", format => "binaryFile", `databricks.connection` => "my_gdrive_conn", pathGlobFilter => "*.{pdf,docx}"); CREATE OR REFRESH STREAMING TABLE documents_parsed AS SELECT *, ai_parse_document(content, map('version', '2.0')) AS parsed_content FROM STREAM gdrive_documents_table; Coming soon: Ingest Google Drive’s per-file permissions and ACL metadata to power permission-aware AI agents, enterprise search, and more. If you try it, share what you are building and let us know if you hit any friction! submitted by /u/BricksterJ [link] [comments]

SharePoint connector in Lakeflow Connect is now generally available (GA)

The Lakeflow Connect connector for Microsoft SharePoint is now generally available! It’s now easier than ever to ingest structured and unstructured files from SharePoint into Delta tables for analytics and AI workloads. You can configure a managed ingestion pipeline through the UI or managed API. Managed pipelines automatically handle incremental processing, automatic retries with exponential backoff for source API rate limits, failure recovery, and provide rich SharePoint metadata. Soon, our managed connectors will also support ingesting SharePoint Lists and per-file permissions metadata. For direct control over ingestion logic, you can also just use the Spark + SQL APIs directly: spark.read , Auto Loader, read_files, or COPY INTO pointed at SharePoint URLs. Common workloads include: Loading Excel, CSV, JSON, and other structured files into Delta tables. Ingesting PDFs, Word documents, PowerPoint files, and images. Parsing documents with ai_parse_document to prepare content for extraction, search, and agents. https://preview.redd.it/89i379aattoh1.png?width=2180&format=png&auto=webp&s=292350dfddfc9bc1a4daf3fc4447394821206054 Link to public docs + references: SharePoint managed connector documentation Spark + SQL APIs and examples Community blog and video tutorial: From PDF to insights Data + AI Summit session: Intelligent Document Processing with Lakeflow Examples of using the Spark + SQL APIs (after first creating a UC connection ) : Read an Excel sheet from SharePoint with spark.read : excel_df = (spark.read .format("excel") .option("databricks.connection", "my_sharepoint_conn") .option("headerRows", 1) .option("dataAddress", "Sheet1!A1:M20") .load(" https://mytenant.sharepoint.com/sites/Finance/Shared%20Documents/Monthly/Report-Oct.xlsx") ) Ingest unstructured documents + PDFs from a SharePoint URL with read_files , then easily parse them using ai_parse_document CREATE OR REFRESH STREAMING TABLE sharepoint_documents_table AS SELECT , "_metadata" FROM STREAM read_files( " https://mytenant.sharepoint.com/sites/Marketing/Shared%20Documents ", format => "binaryFile", databricks.connection => "my_sharepoint_conn", pathGlobFilter => " .{pdf,docx}"); CREATE OR REFRESH STREAMING TABLE documents_parsed AS SELECT *, ai_parse_document(content, map('version', '2.0')) AS parsed_content FROM STREAM sharepoint_documents_table; Coming soon: Ingest SharePoint Lists into Delta tables (coming super super soon) Ingest SharePoint’s per-file permissions and ACL metadata to power permission-aware AI agents, enterprise search, and more. If you try it, share what you are ingesting and where you hit friction! Don't hesitate to ask questions! submitted by /u/BricksterJ [link] [comments]

[Discussion] Lakeflow Jobs: How do you use table update triggers?

(databricks product manager here) Curious how people are using table update triggers in production. https://docs.databricks.com/aws/en/jobs/trigger-table-update Do you rely on the available debouncing capabilities (protect against over and under triggering) or do the options feel confusing enough that you mostly work around them? When a trigger needs to represent more than “run when this table changes,” how do you express the business logic? For example: Do you use control or checkpoint tables to signal that an upstream workflow has finished? Do you wait for a specific status, batch ID, watermark or set of tables before starting downstream work? Do you put that logic in the trigger itself, or in a separate workflow/job? What has worked well and what has been difficult to reason about or debug? Any other suggestions or feature requests relating to Table Update Triggers? I’m especially interested in real-world patterns and whether the current debouncing behavior is intuitive enough for you, or whether a control-table pattern ends up being the clearer approach. Edit: how many folks still use control tables instead of data tables with these triggers? Thank you 🙏 https://preview.redd.it/munx0r2u4ooh1.png?width=660&format=png&auto=webp&s=80e6b80a62cf77870bf444e7634d1ee7412feee3 submitted by /u/saad-the-engineer [link] [comments]

No more UNION ALL-ing all of your SDP pipeline event log tables for monitoring

https://preview.redd.it/3w0j4ss3inoh1.jpg?width=2048&format=pjpg&auto=webp&s=50bc79280a546544e8f795947e5af6b211e9596a Hi folks, Databricks PM here - I wanted to share an exciting update that you no longer have to manually publish and combine your pipeline event logs for monitoring across pipelines and workspaces. We just launched the beta for the pipeline events system table (system.lakeflow_pipeline_events_preview.pipeline_events). Key features: All pipeline events (regardless of cluster start) are automatically captured without any manual enablement or maintenance. Events are aggregated in a central system table without requiring any custom aggregation logic. This data is available at close to real time latency (based on our internal testing we achieve a P99 latency of less than 1 minute). An admin can grant a single user access and they can query events for every pipeline in the table. Fine-grained access controls support which scopes the visibility to only the pipelines the user has access to is coming soon. The data remains in the system table even after pipeline deletion and is retained for 13 months. If you want longer retention this is also possible with the configurable retention feature for System Tables. Query ergonomics are better now with the use of VARIANT. Here are some sample queries in case you want to try them out: -- The latest error for each pipeline that has errored in the last 7 days, with the outermost exception. -- The exception chain is ordered with the root cause last, so read element -1 for the root cause. -- On many errors only the first element carries error_class and sql_state. SELECT workspace_id, pipeline_id, event_time, event_type, message, error.exceptions[0].error_class AS exception_error_class, error.exceptions[0].sql_state AS exception_sql_state FROM system.lakeflow_pipeline_events_preview.pipeline_events WHERE level = 'ERROR' AND event_time >= current_timestamp() - INTERVAL 7 DAYS QUALIFY ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id ORDER BY event_time DESC) = 1 ORDER BY event_time DESC -- Flow throughput for a specific pipeline SELECT origin.flow_name, date_trunc('HOUR', event_time) AS hour, SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) AS rows_written FROM system.lakeflow_pipeline_events_preview.pipeline_events WHERE pipeline_id = ' ' AND event_type = 'flow_progress' AND event_time >= current_timestamp() - INTERVAL 7 DAYS GROUP BY origin.flow_name, date_trunc('HOUR', event_time) ORDER BY hour DESC, rows_written DESC -- Data quality: failed expectations by dataset, per update, in the last 1 day SELECT pipeline_id, update_id, origin.dataset_name, expectation.name AS expectation_name, SUM(expectation.failed_records) AS failed_records FROM system.lakeflow_pipeline_events_preview.pipeline_events LATERAL VIEW explode(variant_get(details, '$.flow_progress.data_quality.expectations', 'ARRAY >')) AS expectation WHERE event_type = 'flow_progress' AND event_time >= current_timestamp() - INTERVAL 1 DAY GROUP BY pipeline_id, update_id, origin.dataset_name, expectation.name HAVING SUM(expectation.failed_records) > 0 ORDER BY failed_records DESC; Beyond single queries you can build alerting (using Databricks SQL alerts) and dashboards. We will share a new dashboard template soon - I will update this post once its available. Call outs: This is in beta right now, if you are not opted in we will not capture your event log data. Enablement: Toggle on the “ Lakeflow Pipeline Events System Table ” from the account level preview. Docs are linked here , would love to hear your thoughts on how you will use it or what else you want to see to improve observability! submitted by /u/brickster_123 [link] [comments]

How to organize your notebook tabs?

It was a real pain, but now, with a few tricks, you can manage them better. First, in Workspace files, next to the DABs folder or git repo, there is a small shortcut to show only tabs from that DABs folder or git repo. Alternatively, you can also use the switcher in Home next to Notebook. If you need to organize your tabs differently, there is new functionality: spaces, which let you group them however you like. more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

Cost-optimized way to reflect source DB changes in Silver in <1 minute?

Due to new business requirements, we need to reflect the state of a few source DB tables (5 to 40 million rows each) in the Databricks Silver layer in less than 1 minute. Currently, the flow looks like this: Source DB → AWS DMS in CDC mode (ingests new data every 30 seconds to S3) → S3 landing bucket → DLT pipeline running on serverless compute in continuous mode. The DLT pipeline ingests the append-only data into the Bronze layer using file notification mode and updates the Silver layer using an Auto CDC flow. This works great, and we achieved what we wanted with relatively low effort because we already had DMS in place. We just added an extra replication task to ingest data more frequently for the tables we need. However, in this setup, the DLT pipeline costs are quite high. Ingesting just 6 Bronze tables and 6 Silver (Auto CDC) tables costs around $50 per day, which is about $1,500 per month. For comparison, DMS, which replicates more than 800 tables to S3, costs us less than half of that. My question is : is there any other more cost-optimized option we could consider to achieve less than 1 minute latency when reflecting the source DB state in the Silver layer? Maybe Lakeflow Connect or some custom process? Extra notes: - I know that adding more tables to the DLT pipeline makes the cost per table lower because Databricks can optimize the clusters more efficiently. - I know that using a cron schedule could reduce costs, but for these particular tables, we can’t use a schedule like every 10 minutes or similar because we need the data to be updated in less than 1 minute. - I know that for the relatively small tables currently in scope, we could eliminate the Auto CDC flow and create a normal view on top of the Bronze table, with deduplication and deletion logic. This would slightly sacrifice query performance, but we expect more similar use cases in the future, so I’m looking for a solution that can scale. submitted by /u/CyberEnzo [link] [comments]

Learn Databricks Lakeflow | Ingest, Orchestrate, and Build pipelines on one platform.

Using Databricks Asset Bundles and Lakeflow Jobs in a Real Project

Community BrickTalk | One Platform, Any Source: Unifying Enterprise Data with Lakeflow Connect

Hey r/Databricks ! We’re hosting a free, community-sponsored BrickTalk on Thursday, September 17, 2026, focusing on how to simplify and scale data ingestion using Lakeflow Connect! BrickTalks is a community event series where Databricks experts share real-world use cases, live demos, and practical insights, giving you a direct line to the people building the products. Stop struggling with fragmented data across disparate sources. In this session, we'll demonstrate how Lakeflow Connect enables seamless data ingestion from SaaS apps, databases, and cloud storage directly into the Databricks Platform with zero infrastructure management. 🛠️ What We’ll Cover Native Data Ingestion: Learn how Lakeflow Connect provides fully managed ingestion directly into Unity Catalog as governed Delta tables. Simple Integration: See how to easily connect data sources using a simple UI or API. Accelerated AI & Analytics: Discover how unifying your data powers Customer 360, Operations, and downstream AI agent workloads. ⏱️ Global Times PT: 9:00 AM ET: 12:00 PM BST (London): 5:00 PM IST: 9:30 PM 👉 Register here to save your spot! submitted by /u/Subject_Ant1789 [link] [comments]

Community BrickTalk | One Platform, Any Source: Unifying Enterprise Data with Lakeflow Connect

Tech Companies & Lakeflow Connect

Lakeflow SDP Append Flow

Serverless Env v6

Version 6 of the serverless environment is available, which corresponds to runtime 19. more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

Lakeflow connect Ingestion pipeline notification for gateway pipeline

Lakeflow Connect: Message Bus Ingestion - Now shipping your logs directly! (Beta)

Lakeflow Genie Code Task

We now have Genie Code Task in Lakeflow jobs. Can not yet send output to if/else, but more options for orchestration are planned. more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

Dynamic Select is now in Lakeflow Designer

You can now dynamically select columns in Lakeflow Designer. This makes it easy to bulk-keep, or bulk-drop columns from a very wide table. And your data prep will keep working as your underlying schema evolves. submitted by /u/zaboca_v [link] [comments]

External price control

In the Unity AI gateway, it is also possible to register an external model for which we pay the provider directly (OpenAI, Anthropic, etc.). In that case, Databricks now knows the prices for those models and can calculate, monitor usage, and alert or block based on budgets. more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

Lakeflow Designer - Source and Output operators now support parameters

You can now use parameters in Source and Output operators for Lakeflow Designer. This is pretty useful for iterating on your Data Prep in dev/test before running it in prod. submitted by /u/zaboca_v [link] [comments]

All 18 Lakeflow AUTO CDC configurations went green. Five failed my ship check

SDP-Meta Deep-Dive Demo: Building Data Pipelines at Scale on Databricks (w/ Databricks Sr. Staff FDE)

This s a helpful resource for those building pipelines at scale on Databricks. From the docs: SDP-META is a metadata-driven framework for Lakeflow Spark Declarative Pipelines . Define your Bronze and Silver pipelines in a JSON or YAML onboarding file — a single generic Declarative Pipeline reads the resulting DataflowSpec at runtime and builds the full processing graph automatically. No pipeline code to write. Who it's for: platform and data engineering teams standardizing repeatable Bronze/Silver pipelines across many datasets — onboarding new feeds through metadata instead of new pipeline code, with consistent data quality, quarantine, CDC, clustering, and sink patterns available through Bundles, CLI, UI, MCP, and agent workflows. When it's not the best fit: one or two simple pipelines, Gold-layer business modeling, tables that each need unique application logic, a managed connector and downstream logic that already satisfy the complete Bronze/Silver requirement, or a need for a formal support SLA (SDP-META is a Databricks Labs project). See the Introduction for the full positioning. You can find the project at https://github.com/databrickslabs/sdp-meta submitted by /u/JosueBogran [link] [comments]

Skills in Unity Catalog

Skills are available in Unity Catalog. They use a similar concept to volumes and are integrated with the AI gateway. New REST endpoints for skills are coming, and a new tool to manage them, ucode, is already available. more news: https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

Best practices for data quality in lakeflow

Lakeflow connect

Bye Bye Fivetran

I went to the Ingestion section and saw that Fivetran is no longer there. It was always there for many years. Also, at the same time, a few new Lakeflow connectors were added. submitted by /u/hubert-dudek [link] [comments]

Build up a data history in Databricks based on Azure SQL data

I have the following need: there's an OLTP Azure SQL DB, which holds transactional data for a period of roughly 30 days only. Now that data should be replicated to Databricks delta tables with a maximum delay of 15 minutes, not as a 1:1 copy, but instead growing over time. Ideally the timeframe covered on Databricks side should be several years. From what I've read so far, either CDC or CT with Lakeflow should be the way to go. The only thing I'm worrying about are breaking schema changes: as this is an OLTP DB managed by a different team, we have no chance to prevent such as incompatible column type changes (e.g. from a string to a date type), column renames or even column drops. I thought about using Views managed by the other team instead, as some kind of an abstraction contract, but neither CDC nor CT are applicable on Views. How did you guys solve such a requirement? Would also appreciate to hear some best practices of Databricks consultants based on real customer solutions. submitted by /u/Sea_Basil_6501 [link] [comments]

Five ways to build a pipeline in Lakeflow Designer — and what each one is good for

Genie Code as a Lakeflow Jobs task - autonomous agents inside your workflows [BETA]

Hi, A great new feature appeared in Lakeflow - you can now run Genie Code as an autonomous task using a natural-language prompt. This can be really handy when you need to automate some complex analysis or data operations as a part of a scheduled process. The documentation lists following use cases, but of course the only limit is your imagination 😄 Summarize overnight job results and email a report. Analyze incoming data and flag anomalies. Investigate a Jira ticket and propose a fix. Generate a weekly compliance audit. What's nice is you can also use job parameters directly in prompts, for example: Summarize yesterday's sales for {{region}} and flag anomalies. Each run creates a Genie Code conversation that you can open afterward and continue interactively. One important detail: auto-approval is always enabled for Genie Code job tasks and cannot be disabled. The docs explicitly say this should not be treated as a security boundary, so access to production data/resources needs to be designed carefully. Genie Code task for jobs - Azure Databricks | Microsoft Learn PS: Since AI behavior is inherently non-deterministic, Genie Code tasks should be used with caution in production workflows and important outputs should be validated rather than treated as guaranteed https://preview.redd.it/vq7swwaue2mh1.png?width=1189&format=png&auto=webp&s=d4e1eba041f6613eb9588068dd552da33a133379 submitted by /u/szymon_dybczak [link] [comments]

Title: Oracle CDC pipeline (Lakeflow Connect) never terminates

Declarative Automation Bundle: Bind an Existing Lakeflow Job

Lakeflow Connect | Marketo Connector (Beta)

Lakeflow Connect's Marketo connector is now available in Beta! It provides a managed, secure, and native ingestion solution for core Marketo objects: leads, static lists, activities (one table per activity type, like email opens and clicks), and custom objects. Try it now: Enable the Marketo Beta: Workspace admins can enable the Beta via: Settings → Previews → "Lakeflow Connect for Marketo" Set up Marketo as a data source Create a Marketo connection in Catalog Explorer Create the ingestion pipeline via a Databricks notebook or the Databricks CLI submitted by /u/Brickster_S [link] [comments]

Declarative Until It Isn't: Four Sharp Edges of Lakeflow Declarative Pipelines

Disabling Change Tracking and enabling Change Data Capture in SQL Server Lakeflow

Triggered vs. Continuous Mode: A Deep Dive into Serverless Lakeflow Spark Declarative Pipelines

Feedback on Deploy Workloads with Lakeflow Jobs SPL

NetSuite connector (Lakeflow Connect) — transactionline refresh takes 3.5–4.8 hrs regardless

Lakeflow Connect - Community Custom Connector - How to troubleshoot source ingestion logic "live"

Lakeflow Connect SQL Server gateway – intermittent Entra ID token auth failures (18456, state 132)

From Experiment to Prod: LakeFlow Spark Declarative Pipelines Testing Blueprint

Announcing General Availability of the Lakeflow Spark Declarative Pipelines Kafka Sink

Issue: Lakeflow Connect Microsoft Teams Community Connector - No module named 'databricks.labs'

INSERTS AND DELETES in a massive way for Lakeflow Spark Declarative Pipelines

sdp-meta (dlt-meta) vs lakeflow_framework: when should we use which?