コミュニティが今、聞いていること
r/databricks と Stack Overflow の [databricks] タグから、最近のスレッドを紹介。実務でのつまずき、連携の疑問、知っておきたいエッジケースを取り上げます。
This week
97 questionsUnexpected Lakebase costs after new feature
Dynamic Continuous Data type and new column arrival adopt in silver in DLT
Your dashboard knows who opened it. Not what they looked at.
Bringing Genie Visualizations into Copilot: From Natural-Language Question to Charts
Recording | BrickTalk: Multi-Cloud Data Sharing with OpenSharing SecureConnect & Global Distribution
Reading Unity Catalog Tables in Snowflake
General Databricks Community Ask
Postgres 19 is out, and it's available on Neon and Databricks Lakebase
--- top comments --- [jedberg] REPACK is my favorite new feature. Vacuum was the bane of my existence when I ran Postgres at reddit. Partly because I didn't know what vacuum was for the first two years. So I thought, "huh, every time vacuum runs it slows down, I should turn this off". Yeah it was dumb to not even google what it was for. About two years in, I hit the point where I ran out of row IDs, and that's when I finally learned what vacuum was for (and reddit was down for a bunch of hours). Having REPACK would have been a godsend. [throwitaway222] It's very odd that both Snowflake and Databricks (two competitors) have a postgres 19 release announcement in the same 4 hour window and both have been updated but both companies barely ever gets comments on hacker news. It's definitely being upvoted by employees. [mmh0000] I’ve been super excited for this release! There’s a lot of performance improvements in 19. Now I just have to wait for AWS to launch it. [heavensteeth] Unfortunate that the only post of this announcement on HN is to a third-party offering to sell you Postgres for a fee and not the official announcement itself. [rixed] Better link: https://www.phoronix.com/news/PostgreSQL-19-Released
Unwarranted Exam Suspension Due to Normal Eye Movement
Did Databricks kill or buyout HighTouch?
Can Databricks temporarily raise the GCP workspace creation limit?
What Is – a DBU? (The kWh on your Databricks bill)
What is one technology or skill you believe every data professional should learn in 2027, and why?
Do we have any datalake diagram an enterprise should follow from Databricks Community of Practice.
How is "Estimated runtime" calculated in Query Profile ??
How is "Estimated runtime" calculated in Query Profile, and is it available via API?
CUSTOMER STORY | VfL Wolfsburg builds real-time football apps with Databricks and Lakebase Postgres
Lakeflow Designer: Your First Visual Data Prep, With Code Underneath
Unity Gateway: One Place to Govern Models, MCP Servers and Coding Agents
🌟 Community Pulse: Your Community Roundup! October 12 – November 15, 2026
Lakeflow Connect – Oracle Connector vs Foreign Catalog Connector
LTAP on Databricks: Unifying OLTP and OLAP on a Single Copy of Data
Tips/Tools/Recommendations for a Snowflake to Databricks Migration?
Predictive Optimization keeps running on system.lakebase after disabling it at account level
Preserving Salesforce connector SCD2 history across Full Refreshes
Announcement | Introducing the Databricks AI Index: The state of enterprise AI tokenomics
Hourly streaming table refresh jobs keep running after the table was dropped (Free Edition)
Undo a bad write on a Delta table: history, time travel and RESTORE
Auto Loader: Schema Evolution and the Rescued Data Column
The Small Files Problem, and What Predictive Optimization Does About It
Announcement | Migrate Effortlessly to the Databricks Lakehouse with Genie Migrate (Now GA)
Model access
DEEP CLONE fails with DELTA_CLONE_INCOMPLETE_FILE_COPY
Auto Loader: addNewColumns can still stop your stream
URGENT – Exam on 23 November – Immediate Webassessor Legal Name Correction Required
Cert rotation killed our continuous jobs. The paused one stayed dead
Suggestions for migrating legacy SAS workloads to Databricks?
Databricks Data Engineer Certified Associate
Databricks Apps deployment failing with invalid_client during source code download
Solution Accelerator Series | Salesforce Bring Your Own Model
Databricks Community Champion - October 2026 - Anuj Lathi
suspension of myDatabricks: Certified Associate Developer for Apache Spark exam
Suggestions/Recommendations for a Teradata to Databricks migration?
Not able to use 'Serverless Starter Warehouse'
Views in SDP - intended use and how to change object types (View -> MV)?
Have some questions for the SDP PM homies - apologies in advance for the post length. Our teams have started deploying views through SDP alongside MVs and streaming tables. They want one deploy unit per "data product," and I'm seeing them use views a lot. Not MVs - just views. That kind of surprised me. I hadn't anticipated them using simple views as they are. I suspect they'll want to start with a view and promote it to an MV (or an MV to an ST) later. I tested how that behaves and would appreciate your guidance on a few things. Setup: serverless, CURRENT channel, DBR 18.3, SQL, all deployed through DABs. Questions: Should we support or discourage publishing public VIEWs through SDP? I was surprised to see people deploying public views in a pipeline. It seems like they're wanting to roll those into the overall data processing flow and treat them like MVs or streaming tables as "just another resource." In hindsight - "duh"? But I was surprised. I've seen no references to doing PUBLIC views like this, so I have to ask - what's your take on using SDP to deploy and manage public views? What is the suggested approach to changing object types? My testing shows object types can't be changed in place. Changing CREATE VIEW to CREATE OR REFRESH MATERIALIZED VIEW (or any other pair) fails with CANNOT_CHANGE_DATASET_TYPE. --full-refresh doesn't help, either for one object or the whole pipeline. The only fix is to drop the object manually, then run. That means an ops team member manually triaging with explicit DROP statements, so it becomes a matter of adding the human into the loop. What's the proposed deployment/management model for these kinds of changes? Manual drop of changed objects before running the pipeline? Related - changing the object type from a MV/streaming table to a view fails silently. The update reports COMPLETED, but the old MV/ST stays in place. The error appears only in the event log for that flow. In the other directions, the whole update fails. Are views expected to support schema evolution? I suspect related to "should we use SDP to deploy public views?" I noticed that views deployed through a pipeline ignore/remove the SCHEMA EVOLUTION option when run. That means backing object changes won't propagate up to the view and may result in view failures, missing columns, etc. I suspect this is because views are meant to be temporary within a pipeline - and in that case there's no need for schema evolution. If the intent is to allow pipelines to publish public views, then I would have concerns with not honoring SCHEMA EVOLUTION. How are view changes managed? How are view logic changes managed? Views seem to be re-published on every run, even when the query did not change. The UC audit log shows updateTables with the full view definition on every update. It's an in-place update (created doesn't change), but last_altered moves on every run. From what I can tell, CREATE VIEW and CREATE OR REPLACE VIEW behave identically. I saw no real change in behavior using/omitting the REPLACE. Is there a preferred construct if we use pipelines to publish public views? Sorry - that was a lot. :) submitted by /u/lofat [link] [comments]
Photon cut this Databricks job from 60 to 40 minutes, but DBUs rose 33%
A 60-minute Databricks job dropping to 40 minutes sounds like a win. In this Photon example, DBUs still rose about 33%. On classic clusters, Photon’s DBU rate is roughly 2×. To see whether it paid off, compare the same workload on the same instance with Photon on and off. Check the query profile to see whether the slow operators ran in Photon, then compare runtime, DBUs, and total cost. Python and Scala UDFs can fall back to standard Spark, so Photon may not speed up the bottleneck. Disclosure: I work at Hackmamba and am sharing this SELECT guide as part of my work. submitted by /u/Plenty-Apricot9222 [link] [comments]
The Power of Native ai_ Functions: Settle Cases Before the Agent at a Third of the Cost
Private connectivity from Databricks on AWS to Amazon S3
Is legacy data migration still a major pain point for enterprises? What if we could automate most of it?
submitted by /u/Same_Perspective_955 [link] [comments]
Read a Query Profile: Find the Slow Step in Five Minutes
Databricks Associate Exam: Mastering Delta Lake Concepts
Lakeflow Jobs: Retries, Timeouts and Alerts That Actually Help
Retries not triggered for a Run Job task
AI Agents & Precedent: Three Gates Before Reusing a Decision
submitted by /u/Berserk_l_ [link] [comments]
Please check out my New Book on Azure-databricks certification DP-750
Building a Self-Validating AI Summary Pipeline on Databricks
How to Read XML in Databricks: rowTag, Auto Loader, and Rescued Data
Drop the spark-xml jar: read XML in Databricks with read_files and Auto Loader, now GA, with schema evolution and rescued data submitted by /u/Lenkz [link] [comments]
Tips and Utilities for migrating Tableau workbooks & Prep flows to Databricks?
Claude Opus 5, Opus 4.8 and Fable 5.1 Access Disabled – Databricks on AWS
Exam suspended
Validate API-Based Promotion / deploy of a Databricks Genie Agent from Dev to QC
Anyone using Kinaxis Data Publish (Delta Sharing) from Databricks Marketplace?
Validation of API-Based Promotion of Databricks Genie Agent from Dev to QC
Metric views on Free Edition?
Im new to the platform, but have quite a bit of experience on competing platforms. As such, I want to learn more about the platform and build out a semantic layer. It sounds like metric views are where I'll want to build my gold layer in Databricks. Assuming that is correct, are these included in the free edition? submitted by /u/SomeNeighborhood7126 [link] [comments]
Best Lakeflow ingestion pattern for Full, Append, Replace, SCD Type 1, and SCD Type 2?
I previously worked mostly with custom ingestion frameworks. Now I am starting a new project using Databricks Lakeflow Declarative Pipelines. I need guidance on the recommended pattern for these five write strategies: Full load / overwrite I need truncate-and-load behavior. A direct TRUNCATE did not work. Currently, I use REPLACE WHERE 1 = 1 , but I am concerned about full-table scans and performance on large tables. What is the most efficient approach? Append New batches should only insert rows. No updates, merges, or deletes. Is an Append Flow the recommended solution? Replace by key group The key identifies a group of rows, not one unique row. Each new batch must delete the existing group and insert its new version. Is REPLACE WHERE the correct pattern? SCD Type 1 I am using AUTO CDC for both real CDC feeds and incremental feeds without CDC operation columns. For non-CDC feeds, I use the source file modification timestamp from storage as the sequence key. Is this a reliable and recommended pattern? SCD Type 2 I use the same approach for SCD Type 2: AUTO CDC for both CDC and non-CDC incremental feeds. For non-CDC feeds, the sequence key is the source file modification timestamp from storage. Are there any limitations or better alternatives? What Lakeflow features or APIs would you recommend for each strategy? submitted by /u/warleyco96 [link] [comments]
Genie vs DBT Wizard
Any thoughts coming from people that use both ? submitted by /u/Sea-Glass7015 [link] [comments]
Query Performance Insights in action
Zerobus Ingest availability in Azure South Africa North (southafricanorth)?
What’s something you believed about Databricks for way too long before finding out you were wrong?
I’ll go first with two. For an embarrassingly long time I ran scheduled jobs on an all-purpose cluster because it was already up and I figured compute is compute. Nobody told me job clusters are billed at a lower rate. Found out when someone asked why our bill looked the way it did. Second one: I assumed Auto Loader schema evolution just quietly picked up new columns. It does, but only after the stream fails first and you restart it. Learned that from a 2am alert, not from the docs. Neither is complicated. I just never had a reason to question them until something broke. What’s yours? Could be Spark, Delta, Unity Catalog, billing, whatever. Bonus points if you found out in production. submitted by /u/UnluckySugar610 [link] [comments]
50% OFF any Databricks Certification
Join us for a four-week event dedicated to learning, upskilling, and advancing your career in data engineering, analytics, machine learning, and generative AI. Whether you are new to the field or aiming to deepen your knowledge, this is a great opportunity to invest in your professional development alongside a global community. 🎟️ Finish all modules in at least one learning pathway during the event window to receive: 50% OFF any Databricks Certification Learn more/ https://community.databricks.com/t5/learning-events/databricks-advanced-learning-festival-november-10-december-08/ev-p/172328 submitted by /u/Youssef_Mrini [link] [comments]
Databricks AI Index
--- top comments --- [pwendell] A new dataset that shows AI adoption trends amongst enterprises.
I want to use Databricks CLI to get Serverless usage policy tags from the app
I want to use Databricks CLI to get Serverless usage policy tags from the app, as I want to be sure that the app not only has Serverless usage policy but also Serverless usage policy tags. This info is visible on UI however with cli and api only Serverless usage policy ID is possible to get. What I understand some sps can bypass the policy and redeploy apps without policy tag.
Do you cache DataFrames in Spark as often as you can, or only when necessary?
I've seen people use cache() and persist() quite frequently in Databricks notebooks, especially when working with large DataFrames. But I've also read that caching everything can sometimes consume memory unnecessarily and even slow things down. Curious about how people handle this in real projects. Do you cache intermediate DataFrames when they're reused multiple times, or do you prefer letting Spark optimize the execution plan? Have you ever had caching make a job slower instead of faster? submitted by /u/shdw_0x0 [link] [comments]
Databricks Streaming table to Fabric for Gold Lakehouse
Has anyone built an external reference from Lakeflow databricks table to Fabric? The Lakeflow databricks table is sourced from On-prem SQL server and uses change tracking to identify rows that are changed. Databricks writes this out to ADLS Gen 2, which is then shortcut (external referenced) to Fabric. We need the data in near real time. The streaming delta tables in ADLS Gen 2 created by Databricks seem to have additional rows that keep track of deleted rows etc, is there a way to get only the final data after deletes and updates, and secondly, is there a way to get table names without mirroring the databricks unity catalogue? The unity catalogues don't support streaming tables which is what change tracking SQL tables are. Appreciate any help on this or alternative approaches to getting streaming data from Databricks to Fabric. submitted by /u/ObsidianHermit [link] [comments]
Guidance on resolving Delta Lake concurrency exceptions during parallel writes
CUSTOMER STORY | KaBuM! Accelerates AI Innovation on Databricks
Clarification Needed on AI/GenAI Billing Origin Products
Parameterised metric views in Excel: what the Databricks Add-in can and can't do (Nov 2026)
Announcement | From enterprise context to action: Genie Agents analyze more data sources
Databricks Cost Monitoring: Sharing My SQL Query – Feedback and Suggestions Welcome!
Approaches and utilities for migrating SSIS packages (.dtsx) to Databricks?
What's Newly GA in Databricks: June to November 2026
Level Up Your Genie Agent with Unstructured Files
My databricks certified data engineer associate exam got suspended
Subsidized tokens came to Databricks
Now, in Unity Gateway, you can connect a ChatGPT or Claude subscription and use cheaper tokens from there. https://medium.com/databrickscommunity/databricks-news-genie-cli-app-builder-agent-memory-dabs-41820427fcc2 submitted by /u/hubert-dudek [link] [comments]
5 Common Challenges in Data Engineering and How Databricks Can Help
Delta Lake vs. Traditional Data Lakes: Which One Is Better for Modern Data Engineering?
Announcement | Genie One: The AI cowork home for everyone
External Access and iFrame Embedding of Dashboards with Private Link and Public Net Access Disabled
No enrollment options available
I did not received certificate credentails
Query regarding last name while creating account on Webaccessor during booking of certification exam
Git folder fails with "Input/output error" on .git/index and edited files, even after a fresh clone
Undoing a bad MERGE on a Delta table: find it, size it, and RESTORE without losing later writes
Overwrite Job Errors
Spark Declarative Pipelines - possible to write UC Managed Iceberg Tables instead of materialized views?
Hi All, Based on my experience using SDP, the tables are typically rendered as "Materialized Views". My company is making a decision to move towards Managed Iceberg for tables and I was wondering if creating Managed Iceberg tables is supported by Databricks's native SDP capabilities? submitted by /u/RazzmatazzLiving1323 [link] [comments]
Lakeflow Declarative Pipelines with external: managed tables, or publish step?
I'm working on an Azure Databricks project where we must expose Delta tables through stable ADLS paths so external engines like Snowflake, Trino, and delta-rs apps can read directly from storage. We'd like to use Lakeflow, but managed streaming tables and materialized views live in Unity Catalog managed storage. Can we safely expose their physical paths like regular external Delta tables, or is path-based access unsupported? The alternative seems to be external Delta tables as sinks, but sinks look limited: Python/streaming only, no expectations or Auto CDC directly, and full refresh doesn't clear prior data. For this requirement, what architecture would you recommend: external sinks, a publish step, or something else? submitted by /u/warleyco96 [link] [comments]
Last week
103 questionsGoverned Spreadsheets on Databricks: Row Zero Acquisition & What It Means for Business Users on Databricks (Demo + Chat)
Hey everyone! Had the wonderful opportunity of recording a demo + chatting with Breck, the founder & former CEO of Row Zero, a governed spreadsheets platform Databricks just acquired. For business users, this will pair up nicely alongside the Genie One / Genie Agents experience. Some quick answers to common questions the video answers (used AI + transcript for them): When did Databricks acquire Row Zero? =The acquisition was announced in September 24, 2026. Can Databricks customers use Row Zero today? =Yes, with a Row Zero account. A built-in Databricks experience doesn't have a date yet. How will it be priced? =Not final. Expected to follow the rest of Databricks (pay for what you use), not per seat. Does it respect Unity Catalog permissions? =Yes. Row Zero queries Databricks as the signed-in user (OAuth recommended, service principals supported), so row-level security carries through. Can you write back to Databricks? =Yes, to tables with a primary key. You review a diff before changes apply. Can you export back to Excel? =Yes, to Excel or CSV. Exports over about 1 million rows get truncated because of Excel's row limit, and admins can turn export off entirely. All roads in data begin and end with a spreadsheet after all! PS: Always love to hear your feedback on the video. submitted by /u/JosueBogran [link] [comments]
Deploy models pattern via MLflow deployment jobs + multiple workspaces
Hi, I'm quite new to MLOps and need some hints for setting up what Databricks calls "Deploy model" pattern ( https://docs.databricks.com/aws/en/machine-learning/mlops/deployment-patterns ). Our Data Science team currently trains ML models in dev environment, in our dev Databricks account/workspace. They have direct access to production data, and the trainingㅤ is quite expensive. We'd like to train the model once and then promote it to production, which lives in a separate Databricks account/workspace. My idea to handle the deployment was to use MLflow deployment jobs ( https://docs.databricks.com/aws/en/mlflow/deployment-job ) and just allow the Data Science team to register the model in the production Unity Catalog. After the registration, the deployment job would kick in, do whatever else is needed, and then stop at an approval step. However, it turned out they can register the model version in the Unity Catalog from the dev workspace, but the deployment job doesn't run automatically. That makes the whole process suboptimal, as they now have to ping me to run the deployment job manually each time they register a new model version. I did some testing and the sole reason of that behavior seems to be the fact that the model comes from a different workspace, which doesn't "see" the attached deployment job that lives in the production workspace. Am I missing something here, or is it designed to work like that? What are the alternatives? submitted by /u/AvaPL [link] [comments]
Why your materialized view keeps doing full recomputes, and how to find out
DevOps Fundamentals: Demo: Deploying the Databricks Project
Federated PostgreSQL timestamps in Unity Catalog: a view that gives everyone the same answer
Databricks new Amsterdam office
Last week, I was invited to the opening of Databricks’ new Amsterdam office as a representative of the MVP community. I took quite a few photos, and I thought some of you might be interested in seeing what it’s like inside. The new office spans 11 floors in "The Rock" building, which offers stunning views of Amsterdam. Inside, there’s a barista, a pool table, foosball, a piano, and a dedicated game room with a PS5. I suspect the latter will become especially popular once GTA VI is released. submitted by /u/Significant-Guest-14 [link] [comments]
Databricks new office in Amsterdam: a photo report 2026
Event-Driven Triggers
One of the most important parts of FinOps is to run jobs only when needed. With event-driven triggers like job completion or table triggers, we can run a job as soon as data is available. https://www.sunnydata.ai/blog/lakeflow-jobs-triggers-events-conditions https://medium.com/databrickscommunity/lakeflow-jobs-triggers-06af6feae80c submitted by /u/hubert-dudek [link] [comments]
Databricks Latest Features: Genie Code CLI, App Builder, Agent Memory and DABs Deployment History
The Databricks Architect in the Age of AI: Why Judgement Matters More Than Knowing Every Feature
Databricks AI Dev round — anyone taken this recently?
Has anyone here done the Databricks AI dev on CoderPad recently? Trying to figure out how to prep for it since there isn't much info out there yet. Are you debugging a medium-sized repo, building a feature on top of existing code, or refactoring? Also curious what interviewers seem to grade heaviest on. Any pointers on how to prepare would be super helpful. Thanks! submitted by /u/ghacker_9 [link] [comments]
Does throwing more resources at a slow Spark job actually solve the problem?
I've noticed that whenever a Spark job starts running slowly, the first thought is often to increase the cluster size or add more workers. But sometimes the actual problem is something else, like data skew, unnecessary shuffles, poor partitioning, or inefficient joins. I'm curious how experienced Databricks users approach this. Do you usually start with the Spark UI and execution plan, or do you try increasing resources first? Have you ever had a job that became significantly faster just by changing the code instead of upgrading the cluster? submitted by /u/shdw_0x0 [link] [comments]
How to make skarf file completed?
I'm completing few learning modules on academy but there's this 'SORM' kind file which shows 'in progress' only although I've completed the quiz. Any tips?? submitted by /u/Automatic-Market8165 [link] [comments]
Using the Apache DataFusion Comet accelerator to save cost
I wonder if anybody has utilized the comet accelerator to speed up things and therefore save cost? Given the photon tax this seems like obvious choice but I'm not sure if the project is already mature enough. submitted by /u/22Maxx [link] [comments]
Is Databricks Losing Its Charm in the Age of AI?
How do you manage table schema changes in Databricks across dev/test/prod?
We run Databricks with Unity Catalog, and I'm trying to understand how other teams handle the lifecycle of managed tables (not pipelines or jobs, just the tables themselves). What I keep seeing: CREATE TABLE IF NOT EXISTS at the top of Job A DDL script in the repo that someone runs by hand Terraform for some tables and a different approach for the rest A few genuine questions: Where does the table definition live in your setup: in the pipeline code, in a separate repo, or only in the catalog? How do you review a schema change before it reaches prod? How do you roll the same change through dev, test, and prod? submitted by /u/ninilich [link] [comments]
Genie ignored my Unity Catalog wiki, so I tested what it actually reads
Last week I tried the new Unity Catalog wikis (Beta). I wrote a business rule into a table's wiki ("revenue = total_price of orders with status F only") and asked a Genie agent for revenue. It summed every order. When I asked who owns the table, it suggested checking a "wiki page outside this environment". I asked about it on the Databricks Community, and someone from Databricks replied: Genie agents don't use wikis as context today (no timeline yet). They read table and column comments. For a rule like this, use a metric view or a SQL expression, and keep text instructions for clarifications. So I tested that. Setup, in my own Azure workspace with the TPC-H sample data: - `samples.tpch.orders` copied into a table, all years - I asked about 1995 because it has F, O and P orders. The right answer (F only) is 32.69B. All orders is 171.63B - 5 Genie agents, each with ONE kind of context - 3 benchmark questions per agent ("What was our revenue in 1995?", "How much revenue did we make in 1995?", "What was our revenue in Q2 1995?"), run once in Agent mode and once in Chat mode Results: Context Agent mode Chat mode Nothing 0/3 0/3 Table + column comments 3/3 3/3 One text instruction 3/3 3/3 SQL expression (measure) + one example query 3/3 3/3 Metric view (filter inside the measure) 3/3 3/3 Things I noticed: - With no context, Agent mode wrote "This includes orders of every status." It states the assumption, but it doesn't ask. - The comments show up as the table and column descriptions under Configure > Sources. Two ALTER statements and it was fixed. - The measure editor wants table-qualified columns. `SUM(CASE WHEN status = 'F' THEN total_price END)` fails with "Table name or alias is required for column 'status'". `orders.status` works. - In Chat mode, Genie reused my example query word for word, `:year` parameter included. In Agent mode it didn't use the measure and wrote `WHERE status = 'F'` itself (still right). - The metric view agent used `MEASURE(revenue)` every time. The measure: ```yaml - name: revenue expr: SUM(total_price) FILTER (WHERE status = 'F') ``` Bonus: I also published a Unity Catalog Page called "Revenue". Genie One found it through the ontology ("Read page: Revenue"), answered 32.69B, said the definition "isn't certified yet" and listed the orders it left out. Best answer of the lot. Caveats: one run per mode, a simple rule, one table. Genie One also looked at an older table and agent in my workspace, but its reasoning says the Page decided the answer. And for a rule this simple even plain text worked, so this doesn't settle "structured context beats text" either way. Questions for you: - Has anyone seen text instructions break where a metric view or SQL expression held up? What kind of rule was it? - Where do you keep business rules for Genie today? Links: - The Databricks answer to my question: https://community.databricks.com/t5/data-governance/do-genie-agents-use-unity-catalog-wikis-as-context/td-p/175921 - Genie benchmarks: https://docs.databricks.com/aws/en/genie/benchmarks - Unity Catalog Pages: https://docs.databricks.com/aws/en/uc-semantics/pages submitted by /u/Zealousideal_Cry_889 [link] [comments]
Free edition - serverless option stopped working
How do you handle dirty data and data quality failures in your pipelines? (Quarantine vs. Fail Fast)
Hey everyone, I wanted to start a discussion on how different teams handle data quality failures in production pipelines—specifically dealing with bad records (e.g., null foreign keys, negative prices, schema mismatches). The two extremes are: Fail Fast: The pipeline crashes the moment a single bad row appears (very safe, but brittle and blocks all good data). Silent Drop: The pipeline quietly ignores bad rows (keeps data flowing, but causes massive data discrepancies later when numbers don't match source systems). Lately, I’ve been leaning heavily into the Quarantine Pattern (tagging bad rows, splitting the stream into Valid vs. Quarantine tables, and building monitoring alerts on top of the quarantine logs without blocking valid data). Curious how you all approach this: Do you use quarantine/dead-letter tables in your architecture, or do you prefer failing hard on errors? How do you handle operational alerts when quarantine tables spike? What tools or frameworks are you using to enforce and monitor data quality rules? submitted by /u/SalamanderOpen859 [link] [comments]
Are you using Databricks Materialized Views for your Gold layers yet? Thoughts?
Hey everyone, I’ve been implementing Databricks Materialized Views for executive reporting (Gold layer aggregates like daily revenue summaries, customer 360 metrics, etc.) and wanted to see how others are using them in production. Historically, keeping summary tables up to date meant writing custom MERGE INTO scripts, managing orchestration dependencies, or dealing with sluggish standard views that re-scanned massive Delta tables on every query. Having the platform automatically handle incremental updates under the hood via serverless compute has been a massive time-saver. Curious to hear your experiences: Are you fully embracing Materialized Views for Gold layer reporting, or sticking to traditional Delta tables with custom upsert logic? Have you run into any gotchas regarding compute costs or refresh frequencies with larger datasets? How do they hold up when dealing with complex multi-table joins and apportioned aggregations? submitted by /u/SalamanderOpen859 [link] [comments]
How are you handling enterprise CDC and file ingestion in Databricks lately? (Lakeflow Connect vs. Custom Spark)
Hey everyone, I’ve been leaning heavily into Databricks Lakeflow Connect and Auto Loader (cloudFiles) recently for managing multi-source ingestion (pulling relational CDC streams like SQL Server alongside high-volume S3 file drops), and I wanted to see how the community is approaching this. Historically, setting up reliable ingestion meant writing custom PySpark streaming jobs, managing checkpoints manually, and dealing with schema evolution headaches when upstream source systems changed unexpectedly. Using Lakeflow Connect combined with Unity Catalog Volumes has noticeably streamlined how raw data hits the Bronze layer, especially for incremental file processing and maintaining clean lineage. I’m curious: Are you guys relying heavily on native Lakeflow/Auto Loader connectors for ingestion, or still building custom Spark/Delta streaming pipelines for specific edge cases? How are you handling schema drift when upstream source tables alter their structure? Any major pain points you've run into with CDC replication in Databricks so far? submitted by /u/SalamanderOpen859 [link] [comments]
Scale Down NOW
This is how AI destroys society – it uses up every megawatt, every compute cycle. It shuts off your email and grounds flights. The AI Apocalypse has arrived. But you can do something about it. We CAN curtail AI data warehouse usage and it’s clear that we MUST. Step One: Use Airbrx right now. It’s free for most Databricks and Snowflake use cases (we give you a cap, trust me, it’s going to be hard to hit that cap. Airbrx sits between your AI tools and the database, and if you haven’t seen the way AI pummels databases, the last couple days should give you some idea just how bad it is. Go to app.airbrx.ai/signup and then paste your databricks or snowflake URL into the setup tool. Then give your AI the airbrx.ai URL instead of the warehouse and you’re done. Even our default 5 minute caching can save you, and the data center, a ton of transactions, and therefore money. Step Two: Scale down We don’t need a full stack of AI, Servers, Databases and frameworks to deliver a picture of a cat. We need thoughtful architecture that does more with less so that we don’t end up here again, unable to communicate, unable to travel, all because the AI wanted to check that resource a few more times, a resource that is over built and over powered. We do that with routing again, use a smaller warehouse, use a different engine (did you know you can get to your Databricks data without Databricks? Ping me…) This is URGENT Our infrastructure is already breaking… Yes our startup happens to have the tech right now and yes, we do sell it… (and yes I'm a founder) but this tech is the exact opposite of what everyone else is doing, we don’t throw more resources at the problem, we make it so you use less. And then maybe you can email your family about that Thanksgiving trip you’re planning on taking and know they’ll get the message and you can catch your flight. submitted by /u/bissellator [link] [comments]
Transactional apps on Databricks: Lakebase owns the decision, Delta gets the copy
What do you use for Databricks cost optimization beyond system tables?
We’re spending roughly $900 a day in DBUs and a bit over $2k a day on the AWS side for the same workspaces. For a long time, I only had a dashboard for the first number, it’s the usual one on system.billing.usage joined to list_prices and it looked fine until finance put it next to the AWS invoice. Tags get me part of the way because cluster tags land on the EC2 instances and cost explorer can group by them. Most of our scheduled jobs moved onto instance pools last year to cut startup time though, and on pool clusters our custom tags only show up on the DBU side. Idle pool instances don’t appear in system tables at all because they don’t burn DBUs while they sit there but AWS still bills every hour of them. On top of that, about 40 scheduled jobs run on one shared all-purpose cluster and that’s a single line on the EC2 side that I have no fair way to split back to teams. Which tool ended up tying the Databricks and AWS sides together for you, ofcourse past the built-in system tables? submitted by /u/Impressive-Craft1926 [link] [comments]
Job SQL condition
SQL condition can check whether a trigger should fire. This way, we can avoid unnecessary job runs. https://medium.com/databrickscommunity/lakeflow-jobs-triggers-06af6feae80c https://www.sunnydata.ai/blog/lakeflow-jobs-triggers-events-conditions submitted by /u/hubert-dudek [link] [comments]
Protecting Your Company Data When Integrating Coding Agents through Unity AI Gateway
Pushing a notebook with outputs to Github
Inside Apache Iceberg V3: Deletion Vectors and Puffin Files
submitted by /u/Low_Brilliant_2597 [link] [comments]
Genie Agents now analyze more data sources to transform your work
Databricks browser extension
Databricks shipped a Chrome extension this week! It has 293 users. FinOpsWay has 346. We are enjoying this for the three days it will last. The honest scoreboard: FinOpsWay has been in the Chrome Web Store since September. Genie One for Chrome went to Beta on November 10. They are also not the same game. Genie One puts an AI assistant on your new tab page. FinOpsWay puts today's Databricks cost in the workspace toolbar, next to the cluster, warehouse or job that spent it. submitted by /u/Significant-Guest-14 [link] [comments]
Databricks best practices for ad-hoc analytics alongside CI/CD
All Lakeflow Jobs Triggers
It started with scheduled triggers, and now it is starting to get really busy with triggers. I tried to summarize all the triggers available to us today. https://www.sunnydata.ai/blog/lakeflow-jobs-triggers-events-conditions https://medium.com/databrickscommunity/lakeflow-jobs-triggers-06af6feae80c submitted by /u/hubert-dudek [link] [comments]
SparkSession.getActiveSession() is Suddenly Broken in Python Multi-threading
Issue with CDC ingestion pipeline
Chlorophyll
Hello! To the Solutions Architects at Databricks: what are some of your tips for new joiners? What’s your recommendation for the first 3 and 6 months? (Outside of the mandatory onboarding stuff)
Especially for people who’re making a transition to Field Engineering from DS/MLE backgrounds. Thanks so much! submitted by /u/notnannanono [link] [comments]
Keeping data fresh
Hey eveyne how are you handling data freshness for dashboards or internal apps? Scheduled syncs are straightforward but I'm trying to figure learn when streaming or CDC is worth the extra complexity submitted by /u/A_A_ronnotaaron [link] [comments]
How often does anyone revisit the size of a scheduled job cluster?
https://preview.redd.it/jg2zzus31e1i1.png?width=1200&format=png&auto=webp&s=4dbed286bd04b300ea99cff9706d3b82414a2228 hey, a question for platform teams that run a lot of scheduled jobs . most job clusters get sized once, on day one, with a safety margin. then nobody looks at them again. in the telemetry we see across customer job clusters, most run far below the capacity they pay for. what do you do about it? a quarterly review, a cluster policy, something automated? we built Auto Tune for this. it learns from each job's own runs and changes one setting at a time, inside bounds you set. if a run regresses, it restores the old config. across the jobs it tunes, the median job runs 33% cheaper. submitted by /u/grahamdietz [link] [comments]
The Cost of Running Databricks Jobs on All-Purpose Compute
Announcement | The new Genie: autonomous AI for complex enterprise work
Issues making requests to Databricks from AWS Lambda
CUSTOMER STORY | Mural answers data questions in minutes with Genie Agents
From Kyriba to Lakehouse: A Near-Real-Time Treasury Data Pipeline on Databricks
How to Query Unity Catalog Views from External Engines
Your data pipeline runs successfully every night. That doesn't mean your data is trustworthy.
A green pipeline usually tells you the job completed. It doesn't necessarily tell you whether the output makes sense. Imagine an ETL process that runs without errors. But one source system changes how it represents cancelled orders. The pipeline still runs. The dashboard still updates. And suddenly, the business is reporting revenue that shouldn't be counted. I'd want data quality checks alongside pipeline monitoring. Things like: Unexpected changes in record counts Missing or duplicate identifiers Invalid values Schema changes Reconciliation against trusted sources I'd also make sure someone owns the business meaning of important metrics. A technically successful pipeline can still produce misleading information. The goal isn't just moving data reliably. It's making sure people can trust the decisions they make from it. Data engineers, what's the most difficult silent data-quality issue you've encountered? submitted by /u/djgoel123 [link] [comments]
Private Connectivity from Databricks Serverless on AWS to Your On-Prem Data Center
Announcement | Genie Ontology is Now Generally Available
DQX in Production
Has anyone taken DQX to production? It has been on our backlog for a while and we want to hear people’s experience and lessons from it. Particularly interested if you have integrated it with lakeflow pipelines or implemented contracts with it. submitted by /u/Agitated-Western1788 [link] [comments]
Request for Guidance: Enabling PostgreSQL CDC in Azure Databricks Lakeflow Connect
URGENT: Assistance Needed on Cases #01071585 & #01072696 – Switch Exam to Online & Exp (Exam Nov 14)
Is Databricks Genie limited to Databricks-related knowledge?
I've been exploring Databricks Genie and was wondering how its knowledge works. Is Genie mainly designed to answer questions based on the data and metadata available in Databricks, or can it also answer general questions like ChatGPT? For example, if I ask it about Python, SQL concepts, or general programming topics that aren't related to my Databricks workspace, will it be able to answer those as well? Just curious about how much of a general-purpose AI assistant Genie actually is compared to ChatGPT. submitted by /u/shdw_0x0 [link] [comments]
Show HN: Bricksentry – see what a data change does before it ships (Databricks)
[Megathread] self promotion
Hey r/databricks , In order to keep the main feed clean, we are implementing a weekly megathread for self promotion for companies who do lots of work with databricks. Please direct all self promotion posts here and keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules! submitted by /u/AutoModerator [link] [comments]
[Megathread] Certifications and Training
Hey r/databricks , please direct all certification and training posts here. Good luck to everyone on your certification journey! submitted by /u/AutoModerator [link] [comments]
Is Lake Transactional/Analytical Processing (LTAP) Just CDC?
[Megathread] Hiring and Interviewing at Databricks - Advice, Prep, Questions
Hey r/databricks , we're noticing a lot of repeated interviewing and hiring posts that tend not to get much engagement. We're going to combine them into a monthly thread so that you're more likely to get answers, plus we can ask our recruiting team to keep an eye on them if there are any general questions. submitted by /u/AutoModerator [link] [comments]
Horrible experience with Databricks CERTIFICATI0N EX4MS
If you are able to attend an examination centre, please do so! In my case, this was out of the question, as the closest one is really far away. I spent weeks preparing for this exam, and I never even got the chance to start it. I got suspended the first time before I was even able to begin the exam. Kryterion support is completely useless. They link you to an empty fucking blog about what to do (even when you tell them it's empty, they keep referring you to it in subsequent queries). They eventually end up telling you to contact Databricks Support for help. Databricks Support takes forever to respond, randomly closes your tickets, and asks you to create new ones. Completely stupid, especially because I'm already providing all the details in the ticket description, but they seem to have automated the process with some kind of AI filter. When I finally get a response explaining why I was suspended, they claim it was due to RAT malware. They offer me one extra free reschedule, which I accept. I ask for details about what could be causing the issue because I am sure there is nothing suspicious on my system. I get completely ignored. I spend multiple hours running scans with two different antimalware providers to make absolutely sure I don't have any such malware. I also prepare a different device in case the first one fails. Weeks later, I attempt to start the exam again, and exactly the same thing happens. This time, it doesn't even let me retry using a different device. I then spend another 3+ weeks waiting for responses and opening support tickets, only for them to eventually tell me to wait 14 days and pay again. This has been the worst online exam experience by far, and I've taken other certifications and much more important exams throughout my life. So, basically, I've spent weeks preparing for this exam and dealing with their bullshit support, only to never even get the chance to start it. And all of this because they don't seem willing to help me understand the actual reason behind the suspension. I verified every single running process, background service, telemetry activity, and internet connection. Everything was clear. At least I didn't have to spend any money, as this was covered by a company partner voucher. I just had to let this out. I checked the Databricks forums, and there are other people experiencing exactly the same issue. Databricks is a great product. You can do better than this. UPDATE: In case anyone thinks this is bullshit, here are a few examples: https://community.databricks.com/t5/certifications/exam-was-suspended-databricks-support-tickets-aren-t-helping/td-p/176201 https://community.databricks.com/t5/certifications/exam-suspended-before-start-kryterion-ticket-01054624/td-p/175619 https://community.databricks.com/t5/certifications/data-engineer-professional-exam-assistance-with-exam-suspension/td-p/166504 submitted by /u/Tiny-Comfortable3536 [link] [comments]
Capture/fork checkpointLocation in Spark Structured Streaming (recovery)
Can anyone help with a spark structured streaming question? Can you tell me if there is a common mechanism to capture a backup copy of the "checkpointLocation" files? Lets say I run a job every five minutes with a trigger (availableNow=true). Lets say it reads from one deltalake table (source) and writes to another deltalake table (target). Lets say there is a logic bug that goes into production mid-day that starts corrupting the target table and I'm not aware of the impact until the end of the day. In some certain streaming scenarios, I should be able reprocess the source table from the beginning of the day to revise all of the badly written data in the target table . But my "checkpointLocation" is doing a lot of the heavy-lifting and, by the end of the day, those files (metadata and state) have evolved and they are no longer configured to present me with the old data anymore. And my watermark duration is only a few minutes long. I don't know the correct approach for reprocessing the source data from the beginning of the day. I would need a mulligan, and a way to repeat the streaming microbatches for the past 12 hours. I haven't figured out what is to be done in this situation. The only thing I can come up with is to "fork" a new of my files into a new checkpointLocation each day, so that it will give me an option to take my mulligan and reprocess the past 24 hours of source data. Is this an approach that people use? Else can someone share the solution? submitted by /u/SmallAd3697 [link] [comments]
Genie agents and Unity Catalog wikis: what I tested, and what to use instead
Unity Catalog secrets: what each grant does, and the schema-level READ SECRET that fails
multiple bundle deployments
Hi, how do you manage deployments when one repo has multiple Databricks bundles (DABs)? We deploy each bundle on its own, but the bundles share some code. So a change in the shared code touches every bundle, even when it is really about one of them. - Do you version the shared code on its own and pin it per bundle? - Do you deploy every bundle on every shared change? - Or one deployment for the whole repo, and bundles that did not change are a no-op? And how do you keep release notes clean per bundle in that setup? What works for you? submitted by /u/ptab0211 [link] [comments]
Databricks Genie Ontology is now GA, and new Genie plugins bring it into ChatGPT and Claude. 🧞
Read more on LinkedIn: https://www.linkedin.com/posts/cenh_databricks-genie-dataengineering-ugcPost-7526718327315087360-rXFe/?utm_source=share&utm_medium=member_desktop&rcm=ACoAABmJHrsBNAC3x3H1M58JRKoHv_l4D61n0-8 submitted by /u/Lenkz [link] [comments]
Reinforcement Learning on Databricks: Building a Contextual Bandit
Context Needs Its Iceberg Moment
submitted by /u/Berserk_l_ [link] [comments]
Which Databricks feature do you think is the most underrated right now?
For me, it’s system tables. I ignored them for a long time, but now I use them to see which jobs cost the most, who’s running what, and which clusters sit idle. It helped us find waste we didn’t even know about. It made me wonder what other useful features people are missing. -What feature do you think doesn’t get enough attention? -How do you use it in your work? -Any tips for someone trying it for the first time? Let’s share some hidden gems! submitted by /u/No_Ambition8323 [link] [comments]
How do i automate this task?
submitted by /u/missing_oldmonk [link] [comments]
Request for Assistance – Databricks Data Engineer Associate Exam Suspended – Ticket #01071693
Announcement | Genie with Genie Ontology delivers 1.6× the accuracy with less than half the tokens
Field Engineering
How is the culture in field engineering? It seems like it is growing/changing rapidly and I am curious if that is having any kind of impact on the teams. I am currently in the negotiation phase for a role and am really excited but want to set my expectations appropriately for the team structure and expectations. Thanks! Also considering growth, is the onboarding process being compacted and rushed to get folks out on projects rapidly? submitted by /u/kujira- [link] [comments]
Lakebase in production: what surprised you?
Dataverse (D365): Synapse Link or Fabric Link?
Hi, in your view, what’s the best option for retrieving data from Dataverse (D365 F&O)? “Azure Synapse Link” (without Synapse Workspaces) or “Fabric Link”? Microsoft seems to be focusing primarily on Fabric. That’s why I’m concerned about starting with Synapse Link only to find out next year that they’re discontinuing it. However, with Synapse Link you have a bit more control over the exported data, as it’s stored in ADSL Gen 2 storage rather than in Dataverse, as is the case with Fabric Link. We also use Power BI in our company and, as far as I know, we have a small Fabric licence. submitted by /u/youderkB [link] [comments]
Streamhouse and Lakehouse with Apache Iceberg
submitted by /u/codingdecently [link] [comments]
Seeking feedback on my Databricks data engineering portfolio project
Hi everyone, I’m preparing to apply for junior/entry-level Data Engineer roles focused on Databricks, and I’d appreciate honest feedback on a portfolio project I’ve been building. It is a governed lakehouse project for a synthetic table-tennis e-commerce dataset: - Deterministic input contract and synthetic fixture with intentional duplicates, malformed records and quality issues. - Bronze ingestion designed around Databricks Asset Bundles, Auto Loader, append-only raw tables, lineage metadata and retained checkpoints. - Silver transformation with explicit validation rules and a quarantine layer. - Gold publication with privacy controls: only aggregated, non-PII cells that meet a k-anonymity threshold are published. - Unity Catalog / Terraform / dedicated service principal / least-privilege model. - Manual serverless Databricks Job with materialization followed by a read-only verifier. - 287 local tests covering contracts, transformations, bundle configuration, verifier logic and Terraform privilege rules. The current DEMO-001 cloud milestone produced: - Silver: 15 orders, 10 customers, 7 products - Quarantine: 6 expected rejected rows - Gold: €640 revenue, 5 orders, 0 refunded lines - Pre-suppression controls: €812.20, 13 orders, 2 refunded lines - Status: COMPLETED_WITH_QUARANTINE I have deliberately kept the scope honest: this is a fixed deterministic snapshot, not CDC, streaming, SCD, generic incremental processing, SQL Warehouse, Power BI, or a production monitoring setup. Repository: tt-ecommerce I would especially value feedback on: Does this demonstrate relevant Databricks/Data Engineering capability? What is missing before using it in applications? Would you prioritise finishing governance/consumer access, adding SQL Warehouse/BI, or CI/CD and operational monitoring? Is the documentation clear, or does it feel over-engineered for a portfolio project? What would you expect to see in a strong junior Databricks project that is not here yet? Thanks — direct criticism is welcome. submitted by /u/Alone-Cow-1563 [link] [comments]
CUSTOMER STORY | Aurasell powers AI sales agents 5x faster, at 7x lower cost with Lakehouse//RT
Enagement Manager and Technical ProgramManager
Serverless egress blocked (DNS REFUSED) on a Premium AWS workspace created via Express – only pypi/g
Can you please give me my certificate???????????
Databricks Metric Views are coming back to Power BI! [Private Preview]
First I'd like to give a bit of background because if you feel like you've heard about this feature before, you're right. Previous attempt to bring Metrics Views to PBI was "blocked" when MS removed the required opiton from the connector. In February 2026 - the Power BI Databricks connector introduced Metric View BI Compatibility Mode (Beta). It allowed Power BI to query Unity Catalog Metric Views, with Databricks rewriting the generated SQL to handle measures correctly. In April , Microsoft removed that option from the connector. Reports relying on it stopped working. Now, in November , Databricks has announced a different approach. In new approach Databricks pushes metric view to Power BI semantic model, so ach metric view becomes a table, dimensions become columns, and measures are translated into DAX. You can publish the model from Catalog Explorer or automate it using a Lakeflow Job. There are a few limitations: - Synchronization is one-way, so changes made in Power BI don't flow back to Unity Catalog. - Import mode also changes the security model - imported data is governed by Power BI permissions. For me it's a long awaited feature. I've never been a fan of defining the same business metrics separately in different BI tools. In my opinion metric definitions should live as close to the data as possible, in a shared layer that different reporting tools can use. Connect metric views to Power BI - Azure Databricks | Microsoft Learn https://preview.redd.it/zysb9za0o51i1.png?width=1355&format=png&auto=webp&s=ff3ce75b6bd2c25735f1c368a0721f09cbed8627 submitted by /u/szymon_dybczak [link] [comments]
Announcement | Real-Time Mode for Spark Declarative Pipelines is Now GA on Databricks
Serverless declarative pipeline ingestion from ADLS is running x4 SLOWER randomly
Lakebase Data API
Exam Cancelation appeal
Cannot reschedule exam and support refused to do so - case 01069034
Certification Exam Suspended Again
Show HN: Igloo – Self-hosted data warehouse built on DuckDB
I made a database (based on DuckDB) because Snowflake and Databricks is insanely expensive. It's open source and in testing it's 3x faster and 25x lower cost.
Configure AWS Backend Private Link
How do you test data quality in a Delta Lake pipeline?
Exam was suspended, Databricks support tickets aren't helping
Any tips for optimizing complex CTE queries in Databricks with a small warehouse?
I've been working with some SQL queries in Databricks that involve multiple CTEs, joins, aggregations, and filters. The queries work fine, but execution time becomes quite high when running them on a warehouse with limited resources. Increasing the warehouse size is one option, but that's not always possible, especially when trying to keep costs under control. I'm curious about how others handle this. Is it better to combine multiple CTEs or keep them separate? Does using the same CTE multiple times affect performance? Are temporary tables or intermediate results worth considering? What optimizations would you recommend before increasing warehouse size? Are there any particular things in the query profile that you check first? Would appreciate some practical tips from people who have optimized complex SQL queries in Databricks, especially when working with smaller warehouse configurations. submitted by /u/shdw_0x0 [link] [comments]
How Do I Choose the Right AI Development Services for My Business?
Feature materialization in DABs
DABs can now manage Feature View materialization to offline and online stores through materialization_sets. This requires CLI 1.24.0 or later and the direct deployment engine. More news https://medium.com/databrickscommunity/databricks-news-genie-cli-app-builder-agent-memory-dabs-41820427fcc2 submitted by /u/hubert-dudek [link] [comments]
[Announcement] : Serverless Jobs log delivery is in Beta
Hi Folks, We’re excited to announce the Beta for Serverless Jobs log delivery https://preview.redd.it/edm97xea101i1.png?width=1384&format=png&auto=webp&s=454ad8fb152141e7518146c352642e87f3f63cbe Why does this matter? Unblocks Serverless jobs for those who need to: (1) triage issues in Datadog/Splunk/ELK/Grafana, (2) retain logs for compliance, (3) query logs across jobs without clicking through the UI Prior to this feature, you only had access to logs through the UI What’s included? Delivering logs from user code (stdout/stderr) to a UC volume You can query the logs directly from UC volumes ( see example ) and build dashboards on top What’s next: GA with Logs delivery every few minutes while a long job runs, not only after it ends Logs from Spark Declarative Pipelines Support all serverless environment versions (today: version 5 and later only) Learn more: Docs ( AWS | Azure | GCP ), Release note submitted by /u/SparkConnective [link] [comments]
Announcement | Lakebase branching for CI/CD: lessons from 100 PR databases
Passed the Data Analyst Associate certification - sharing my study guide
Open source entity resolution tool on Databricks marketplace
Hello All, Zingg lead dev here! Excited to have the app available as on Databricks marketplace. Took a lot of effort from our tiny team, and we are glad we finally made it! Please do check it out and let us know your feedback. https://marketplace.databricks.com/details/6e0e1411-65c8-44ae-bdfa-ab706770ac11/Zingg-Labs-Inc_Zingg-Entity-Resolution-and-MDM submitted by /u/sonalg [link] [comments]