Asset Bundles

Built a Databricks medallion pipeline for NYC Taxi data

Been working on this as a way to get hands-on with Databricks Asset Bundles and Unity Catalog governance. It's a migration of an old on-prem NYC Taxi analytics stack (ClickHouse + Spark + Docker + Terraform) into a proper Bronze/Silver/Gold lakehouse. A few things I focused on: Auto Loader for incremental ingestion, triggered by file arrival Data quality handling that doesn't just drop bad rows — duplicates, zero-distance trips, and reversed fares go into dedicated quarantine tables instead of being silently discarded Databricks Asset Bundles for deploy/orchestration (dev + prod targets) 3 published AI/BI dashboards on top of the Gold layer (fleet ops, finance, compliance) It's intentionally small in scope — meant to demonstrate the lakehouse pattern, not be a production-scale platform. Currently only Green Taxi data; FHV comparison is planned next. Repo: https://github.com/Hamza-Bouali/NYC-DATABRICKS Would love feedback, especially on the Silver-layer data quality rules or the bundle structure open to critique. https://preview.redd.it/ej6hvzrbokph1.png?width=1667&format=png&auto=webp&s=b090b2c5165efcc83fb3f2b5c38ff6ef34c8ef48 https://preview.redd.it/rih9a5sbokph1.png?width=1879&format=png&auto=webp&s=1b85b90608e1b1273da4f8cbb4fa65d9f2c397a7 https://preview.redd.it/kjdor4sbokph1.png?width=1687&format=png&auto=webp&s=677c6194ede0cd01a8630ee98f85d11a5146b410 submitted by /u/No-Pollution-2274 [link] [comments]

Azure AI Foundry + Databricks Architecture | Deploy Genie Agent with DAB...

Azure AI Foundry Databricks architecture, Deploy Genie Agent with DABs, Databricks Genie Agent, Azure Databricks Genie Space, how to deploy genie agent with declarative automation bundles, azure ai foundry + databricks integration, fully operating genie architecture databricks, databricks unity catalog genie agent, azure databricks bronze silver gold architecture, agent to agent nlq databricks, databricks spark python sql delta lake unity catalog, production ready genie agent deployment, databricks vector search index genie, microsoft purview databricks governance submitted by /u/macxima [link] [comments]

Databricks Asset Bundles: How to manage dependencies between volumes/files and cluster creation?

How to organize your notebook tabs?

It was a real pain, but now, with a few tricks, you can manage them better. First, in Workspace files, next to the DABs folder or git repo, there is a small shortcut to show only tabs from that DABs folder or git repo. Alternatively, you can also use the switcher in Home next to Notebook. If you need to organize your tabs differently, there is new functionality: spaces, which let you group them however you like. more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

Using Databricks Asset Bundles and Lakeflow Jobs in a Real Project

Managing Databricks Genie Agents as Code with Databricks Declarative Automation Bundles (DAB)

UI to DABs sync

When working from the web experience in development mode with source-linked deployment set to true (the default for development mode), you can edit jobs and pipelines in the UI, and changes are automatically propagated to your YML files. So it is like the best of two worlds: ow-code and IaC. Please remember to review any changes in Git, especially non-UI elements such as variables or mutators. Whole talk with audio https://www.databricks.com/dataaisummit/session/dabs-do-pro-all-best-tips-and-tricks submitted by /u/hubert-dudek [link] [comments]

What's new and what's next in DABs

What's new and what's next in DABs

Direct mode is GA and is now the default for bundles created from the workspace (since 13th August 2026); it will become the general CLI default on 26th August 2026. Let's see what direct mode brings and what will come next. https://medium.com/databrickscommunity/dabs-go-direct-ga-faster-better-and-life-after-terraform-edc502d54378 https://www.sunnydata.ai/blog/databricks-dabs-direct-mode-ga submitted by /u/hubert-dudek [link] [comments]

Declarative Automation Bundle: Bind an Existing Lakeflow Job

Declarative Automation Bundle - How to handle Managed resources?

Unity Catalog Schemas in Declarative Automation Bundles Are a Footgun

Productionizing Databricks Pipelines with Declarative Automation Bundles and Azure DevOps

Unity Catalog Secrets - storage location for catalog.schema.secret

Read the announcement regarding Databricks Unity Catalog and how secrets can now be stored under a schema as opposed to using secret scopes backed by a secrets manager e.g. Azure Key Vault. Does anyone know if the secrets are stored in plaintext in bundle state if using the direct deployment engine mode for DABs? submitted by /u/RazzmatazzLiving1323 [link] [comments]

DABs Migration Guide: Terraform to the Direct Deployment Engine

Visual, step-by-step curriculum for mastering DABs and Infrastructure as Code

Monorepo vs Multi-repo for Databricks Asset Bundles: A Decision Framework

DABs: immutable_folder

Genie Spaces (Agents) in DABs

can i deploy a metric view using DABs

How to Deploy Lakehouse Federation Using DABs from the Dev Environment to the Stage Environment

Databricks News: CLI v1.0.0, AI-tools, Docker, DABs UI sync, mutators

Databricks News: CLI v1.0.0, AI-tools, Docker, DABs UI sync, mutators

Learning Series | Automated Deployment with Declarative Automation Bundles

Asset Bundles Setup

UI sync back to DABs

Best resources to learn Databricks

hi! what are the best resources to learn Databricks from scratch up to management of infra (cluster), optimization of billing, asset bundles, etc? Many thanks.

Create and edit DABs without writing YAML!

Hey everyone! I’m a product manager for Declarative Automation Bundles (DABs) and our IDE experience.  I wanted to share two Beta features that **make it simpler to work with DABs** from the UI: **1. Edit a job or pipeline settings in the UI and have YAML update automatically** ([docs](https://docs.databricks.com/aws/en/dev-tools/bundles/workspace-author#edit-bundle-resources))**.**   [The user edits job parameters and the Job's YAML updates automatically.](https://i.redd.it/h4lcw3wg1h3h1.gif) 2. Use **custom bundle templates in the workspace** to quickly create standardized projects with best practices enforced by admins ([docs](https://docs.databricks.com/aws/en/dev-tools/bundles/templates#configure-a-custom-template-folder-in-the-workspace)).  # 💻 Get started‼️ 1. **Admins must enable “Visual authoring” and “Custom bundle templates in the workspace”** in the Databricks Preview Portal. See [managing workspace-level previews](https://docs.databricks.com/aws/en/admin/workspace-settings/manage-previews#-manage-workspace-level-previews).  2. **Check out docs:** * [\[Beta\] edit bundle resources in the UI](https://docs.databricks.com/aws/en/dev-tools/bundles/workspace-author#edit-bundle-resources) * [\[Beta\] configuring a custom template in the workspace](https://docs.databricks.com/aws/en/dev-tools/bundles/templates#configure-a-custom-template-folder-in-the-workspace) # 🗺️ What’s next ⁉️ We are introducing UI flows that leverage Genie Code to add existing resources to DABs, improve bundle configuration, and help set up CI/CD processes (e.g. AzureDevOps Pipelines and GitHub Actions).  # 🗣️ Share your feedback ‼️ Feel free to leave comments, reach out directly via Reddit message or email us at [dabs-feedback@databricks.com](mailto:dabs-feedback@databricks.com). I’m also happy to chat over a call as your feedback directly influences our product direction! 

CLI generally available

Databricks CLI and DABs are generally available, and the first version has been released! #databricks [https://databrickster.medium.com/databricks-news-cli-v-1-0-0-ai-tools-last-updated-25th-may-767ef39abe8a](https://databrickster.medium.com/databricks-news-cli-v-1-0-0-ai-tools-last-updated-25th-may-767ef39abe8a)

Attempting Data Engineer Associate with no real Databricks experience — is it doable?

1 year DE here. Comfortable with Python, SQL, and PySpark. My actual work shifted more toward GenAI/data tooling, so on Databricks I've hardly used anything. I haven't worked with things like: * Lakeflow Jobs * Auto Loader * COPY INTO * Unity Catalog * Governance/permissions * CI/CD/DABs * Spark monitoring/tuning For those who've taken the latest version, how much real-world Databricks platform experience did you have beforehand? Is hands-on practice and focused study enough, or are there topics that are difficult to grasp without working on production projects? I have 2 months time.

copy-pasting of job parameters

Stop copy-pasting job parameters between your DABS. Especially ones for the catalog and schema. Just read about mutators! #databricks [https://databrickster.medium.com/global-job-parameters-thanks-to-dabs-mutators-2ad0d94bda1c](https://databrickster.medium.com/global-job-parameters-thanks-to-dabs-mutators-2ad0d94bda1c) [https://www.sunnydata.ai/blog/declarative-automation-bundles-mutators-job-parameters](https://www.sunnydata.ai/blog/declarative-automation-bundles-mutators-job-parameters)

Repository organization in Databricks (Lakeflow/DLT)

Hey everyone, Looking for some architectural advice on directory and file organization for a large-scale project. We are migrating to Databricks’ new Lakeflow Pipelines (`pyspark.pipelines` / `dp`) using a fully config-driven Medallion architecture, and we're trying to prevent our repository from becoming unmaintainable. # The Scale & Setup * **Data Size:** \~300 raw tables across 3 distinct financial data providers. * **Architecture:** Medallion (Bronze/Silver/Gold) deployed via Databricks Asset Bundles (DABs) into Unity Catalog. * **The Pattern:** We are using a config-driven approach (YAML files defining schemas/DQ rules) passed into a Python `for` loop that dynamically generates the `dp.table` and `dp.view` structures. We are splitting the ingestion into separate pipelines by provider to avoid driver bottlenecks. # The Complexity (Where it gets messy) Bronze is a clean 1:1 loop from raw. However, when we hit Silver and Gold: * **Many-to-Many Mappings:** A single raw/bronze table often feeds into multiple business entities (e.g., one raw table splits into parts of our business objects that are created in gold). * **Cross-Provider Joins:** Gold entities require joining across the different providers to build the final target application objects. # Our Current Proposed Structure Plaintext ├── config/ │ ├── provider_a.yaml # Metadata for 80 tables │ └── provider_b.yaml ├── src/ │ ├── pipelines/ │ │ ├── ingest_provider_a.py # Generic loop for Bronze -> Silver │ │ ├── ingest_provider_b.py │ │ └── build_gold_ledger.py # Bespoke cross-provider joins # The Questions for the Community 1. **File Granularity:** For the Silver layer where a single config-driven table needs to fork into multiple complex business entities, do you isolate those transformations into bespoke Python files per entity, or handle the routing directly inside the config loop logic? 2. **Repo Organization:** If you've managed 300+ tables in a declarative framework, what does your actual `src/` folder look like? How do you organize the custom SQL/PySpark transformation snippets so they don't get buried in a monolithic script? 3. **Pipeline Boundaries:** Databricks recommends splitting pipelines by domain to avoid high initialization times. How do you split your Python files to align cleanly with separate pipeline definitions in your DAB bundle config? Would love to see examples or hear lessons learned from anyone who has tackled this scale without losing their sanity. Thanks!

DABs - state resolution nuances

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/direct#resource-state-diff-calculation I was reading the above documentation to deepen my understanding of how DABs in direct mode manage state (recall that direct mode no longer uses Terraform under the hood and instead uses the Databricks Go SDK under the hood). This is a snippet from the documentation that's pertinent to this discussion: "The resources state diff calculation is done in two steps: The local bundle configuration is compared to the snapshot configuration used for the most recent deployment. The remote state plays no role. The remote state is compared to the snapshot configuration used for the most recent deployment. The result is that: databricks.yml resource changes are never ignored and will always trigger an update. Resource fields not handled by the implementation do not trigger an inconsistent result error. These resources are deployed successfully by the direct engine, but this can result in a drift. The deployed resources are updated during the next plan or deploy." The qn I have is, I want to manage grants on a UC Securable object, say a catalog by using two bundles. One bundle has Business Unit (BU) specific grants i.e. only Entra Groups belonging to the BU get granted permissions on the BU catalog. The other bundle contains cross-BU grants i.e. grants on that catalog belonging to a certain BU to Entra Groups for another BU. The reason for the above separation is because of a difference in Governance process where cross BU grants must be approved by a central governance committee whereas BU specific grants only need to be approved by BU specific data stewards. The qn is, does the above model of using two bundles (local BU grants vs cross-BU grants) work? Or does it not bc of the possibility of the bundles being deployed at different times and reconciling against the remote state of the catalog grants? TLDR: Are mutually exclusive BU grants applied independently in a non-destructive manner through reconciliation of bundle state only with its previous bundle state (as opposed to reconciling with the state of the remote object)?