Skip to content
All topics

Databricks CLI

Recent items mentioning Databricks CLI across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.

60 recent items38 releases11 videos11 community threads
What's happening in Databricks CLIAI synthesis · updated Nov 16, 2026

Databricks CLI v1.23.0 introduces workspace.sync_mode to replace deprecated deployment presets, restores legacy-import uploads for bundle deploys, and adds claude and codex support via databricks ssh connect 56. Additionally, updates to the Go and Java SDKs resolve CLI authentication errors by ensuring launcher scripts and version-manager shims are no longer misidentified as the legacy Python CLI 79.

Generated daily from the 10 most recent items mentioning Databricks CLI. Click any [N] to jump to the source.

Stack Overflow

I want to use Databricks CLI to get Serverless usage policy tags from the app

I want to use Databricks CLI to get Serverless usage policy tags from the app, as I want to be sure that the app not only has Serverless usage policy but also Serverless usage policy tags. This info is visible on UI however with cli and api only Serverless usage policy ID is possible to get. What I understand some sps can bypass the policy and redeploy apps without policy tag.

databricksazure-databricksdatabricks-cli
-70Eddmik2d ago
Reddit

[Beta] OneDrive connector in Lakeflow Connect now available!

Hey r/databricks , The Lakeflow Connect connector for Microsoft OneDrive (for business) is now available in Beta . You can ingest structured and unstructured files from a user's "My files" drive into Delta tables for analytics and AI workloads. Managed pipeline authoring is available through the Lakeflow Pipelines API or Declarative Automation Bundles (DABs) . UI-based pipeline authoring is not supported at this time, but will be available soon! Managed pipelines support incremental ingestion, automated retries, Unity Catalog governance, schema evolution for supported formats, and OneDrive-specific and standard file metadata. For direct control over ingestion logic, you can use the Spark + SQL APIs directly: spark.read , Auto Loader, read_files , or COPY INTO pointed at OneDrive URLs. Common workloads Loading Excel, CSV, JSON, XML, Parquet, Avro, and ORC files into Delta tables. Ingesting PDFs, Word documents, PowerPoint files, and images. Parsing documents with ai_parse_document for extraction, search, and agent workloads. Capturing file metadata without loading file contents. Examples using the Spark + SQL APIs Read an Excel range from OneDrive with spark.read : excel_df = (spark.read .format("excel") .option("databricks.connection", "my_onedrive_conn") .option("headerRows", 1) .option("dataAddress", "Sheet1!A1:M20") .load("https://mytenant-my.sharepoint.com/personal/user_mytenant_onmicrosoft_com/Documents/Report.xlsx")) Incrementally ingest PDFs and Word documents, then parse them with ai_parse_document: CREATE OR REFRESH STREAMING TABLE onedrive_documents_parsed AS SELECT *, ai_parse_document(content, map('version', '2.0')) AS parsed_content FROM STREAM onedrive_documents; CREATE OR REFRESH STREAMING TABLE onedrive_documents AS SELECT * FROM STREAM read_files( "https://mytenant-my.sharepoint.com/personal/user_mytenant_onmicrosoft_com/Documents", `databricks.connection` => "my_onedrive_conn", format => "binaryFile", pathGlobFilter => "*.{pdf,docx}"); Try the managed connector Ask a workspace admin to open Previews and enable Lakeflow Connect for OneDrive For Business . Create a Unity Catalog connection to OneDrive . Create a managed pipeline using Declarative Automation Bundles or the Lakeflow Pipelines API . The setup guide includes bundle YAML and an API example notebook . Schedule ingestion with Lakeflow Jobs. For DABs, define the scheduled job in the bundle. Current Beta scope The connector reads from a user's OneDrive for Business "My files" drive. For SharePoint sites, team sites, and shared document libraries, use the SharePoint connector . The connection authenticates as the signed-in Microsoft account and can ingest files that account can access. Managed ingestion requires Unity Catalog, Serverless compute, Databricks Runtime 19 or above, and the PREVIEW pipeline channel. DABs require Databricks CLI version 1.23.0 or above . Resources OneDrive managed connector documentation Managed pipeline setup: DABs and API examples Spark + SQL APIs and examples OneDrive connector limitations Community blog and video tutorial: From PDF to insights Data + AI Summit session: Intelligent Document Processing with Lakeflow If you try the Beta, please provide us your feedback! submitted by /u/BricksterJ [link] [comments]

00BricksterJ1w ago
Reddit

Lakeflow Connect | NetSuite Connector (GA)

Lakeflow Connect's NetSuite connector is now GA! It provides a managed, secure, and native ingestion solution that accesses NetSuite data through the SuiteAnalytics Connect JDBC driver and lands it in Databricks. The connector supports incremental ingestion, SCD Type 2, and automated schema evolution! Try it now: Set up NetSuite as a data source Create a NetSuite connection in Catalog Explorer Create the ingestion pipeline via the UI, a Databricks notebook, or the Databricks CLI submitted by /u/Brickster_S [link] [comments]

00Brickster_S3w ago
Reddit

Lakeflow Connect | LinkedIn Ads Connector (Beta)

Lakeflow Connect's LinkedIn Ads connector is now available in Beta! It provides a managed, secure, and native ingestion solution for LinkedIn Ads resource tables (ad accounts, campaign groups, campaigns, and creatives), as well as seven pre-built reports covering daily campaign and creative performance plus monthly member demographics. Try it now: Enable the LinkedIn Ads Beta: Workspace admins can enable the Beta via: Settings → Previews → "Lakeflow Connect for LinkedIn Ads" Set up LinkedIn Ads as a data source Create a LinkedIn Ads connection in Catalog Explorer Create the ingestion pipeline via a Databricks notebook or the Databricks CLI submitted by /u/Brickster_S [link] [comments]

00Brickster_S3w ago
Reddit

Run jobs and pipelines as a group

You can now run pipelines as a group, avoiding dependency on a single principal (user or SPN). Databricks CLI 1.20.0 adds run_as.group_name at the bundle and target levels. The group needs its own workspace and data permissions; it does not inherit a member’s personal access. more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

00hubert-dudek1mo ago
Reddit

Use Genie Code outside the workspace

Hey! I’m a Databricks product manager working on third-party development experiences. We just shipped Genie Code CLI , a new coding agent specialised for data and AI work. We’d love for you to try it out and share what you want to see next! Why we built this: Many of you prefer working locally with agents, but these agents might not always know the best way to work with Databricks. We wanted to provide an agent preconfigured to understand your Databricks data, pipelines, models, and apps. Use this when you want: Minimal setup and maintenance to use an agent that works effectively with Databricks. Governance provided through Unity Gateway. Access to agents outside of the Databricks workspace without model subscriptions or contracts. Getting started: (1) Install the latest version of the Databricks CLI . (2) Install Genie Code CLI : macOS, Linux, WSL: curl -fsSL https://github.com/databricks/genie-code-cli/releases/latest/download/install.sh | bash Windows powershell: powershell -ExecutionPolicy Bypass -c "irm https://github.com/databricks/genie-code-cli/releases/latest/download/install.ps1 | iex" (3) Start an interactive session by running genie in your project directory. Note that usage is billed through Unity Gateway’s model serving capabilities . For full details, check out the Genie Code CLI docs . Share your feedback: We’re releasing this fast to get your input asap. Please leave feedback and questions in the comments or reach out directly with a Reddit message! submitted by /u/BricksTrixTwix [link] [comments]

00BricksTrixTwix1mo ago
Reddit

Databricks needs an actual validation step in the CI

We use Databricks CLI in our CI-step to first validate and then print plan of the steps being applied - reflected in a comment on the PR. It successfully prints if something is being deleted, updated or created - but it never checks if actually can be applied. We've had green checks because validate and plan prints successfully, but there's no real step to catch a spelling error in a DAB-item if a service-principal exists. Any plans to add something like this to the CLI? submitted by /u/DeepFryEverything [link] [comments]

00DeepFryEverything2mo ago
Reddit

How to automate downloading files from Databricks to a local machine without PATs or CLI?

Hey everyone, Looking for some advice on automating a workflow in a pretty locked-down corporate environment. Context: Large enterprise with strict IT security and governance. Databricks was recently rolled out as our cloud data hub. The entire pipeline (ingestion, processing, and generating the final CSV) is already automated inside Databricks. I need to automatically save a copy of this generated CSV to a local machine / internal network. Right now, the only way I can do this is manually opening the workspace UI and clicking "Download." Databricks CLI is blocked and Personal Access Tokens (PAT) are disabled How do you usually automate pulling files from the cloud down to on-prem / local machines under these restrictions? Thanks! submitted by /u/Firm_Yogurtcloset835 [link] [comments]

00Firm_Yogurtcloset8352mo ago
Reddit

Automatically create local Python environments compatible with Databricks Runtime

Hey! I’m a Databricks product manager focused on third-party development experiences. We just shipped a new feature that ensures code you run locally works with Databricks, and we’d love your feedback. The problem: Ensuring that your local development environment is compatible with Databricks Runtime (DBR) is a completely manual process. Code that works locally often breaks once you move to the workspace due to minor version mismatches and package incompatibilities. The solution: The CLI command databricks environments setup-local and a new feature in the IDE extension take your existing dependencies and automatically create a uv-managed virtual environment compatible with DBR. You can now run code locally or remotely using Databricks Connect and be certain that the same code will work with the DBR or serverless version you’ve selected. To get started, install the latest version of the IDE extension or Databricks CLI and check out these docs: Set up your Python environment and Databricks Connect using the IDE extension Databricks environments setup-local CLI command Where we need your feedback Are there other environment managers (e.g. pip, poetry, conda) that you would like to see support for beyond uv? We’ve also recently launched an SSH tunnel (see docs ), which allows you to remotely access your Databricks workspace and compute directly from the IDE and terminal. What are the reasons you might prefer to work locally vs. remotely? Do you face issues managing your environment and dependencies in the workspace? Most importantly, please try this out and leave feedback and questions in the comments! submitted by /u/BricksTrixTwix [link] [comments]

00BricksTrixTwix2mo ago
Reddit

Databricks Automation Bundles - automatic migration to direct engine post deployment is coming in CLI v1.14.0

Starting with Databricks CLI v1.14.0 , the default bundle.engine will change to direct . If you're not familiar with the advantages of the direct deployment engine over Terraform, you can read more here: Migrate to the direct deployment engine - Azure Databricks | Microsoft Learn What does this mean for us? Bundles will automatically migrate to the direct deployment engine during deployment. If the migration encounter an error, Databricks will keep the bundle on the Terraform engine instead. Important things: You can opt out temporarily by setting bundle.engine: terraform in databricks.yml - either globally or per target. Databricks recommends migrating early by explicitly setting the engine to direct now. Most importantly: support for the Terraform deployment engine in Databricks Automation Bundles is planned for deprection in September 2026 . A future CLI version is expected to stop supporting engine: terraform . submitted by /u/szymon_dybczak [link] [comments]

00szymon_dybczak2mo ago
Reddit

Lakeflow Connect | Marketo Connector (Beta)

Lakeflow Connect's Marketo connector is now available in Beta! It provides a managed, secure, and native ingestion solution for core Marketo objects: leads, static lists, activities (one table per activity type, like email opens and clicks), and custom objects. Try it now: Enable the Marketo Beta: Workspace admins can enable the Beta via: Settings → Previews → "Lakeflow Connect for Marketo" Set up Marketo as a data source Create a Marketo connection in Catalog Explorer Create the ingestion pipeline via a Databricks notebook or the Databricks CLI submitted by /u/Brickster_S [link] [comments]

00Brickster_S2mo ago

Get Tuesday's version of this

Tracking Databricks CLI? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.

Read past issues first