What the community is asking.

Ho creato un filtro deterministico per record JSON su Apify: accetto feedback.

submitted by /u/LucaDataTools [link] [comments]

Just ask Genie

"Just ask Genie." You hear this phrase more and more often when questions about Databricks arise. It frequently appears in Academy materials, too. AI reduces routine work it’s convenient, and we get used to it. But do we stop thinking in the process? The problem arises when a person no longer understands why a given answer is correct. Does AI really free us from the need to know the details, or does it actually make fundamental knowledge even more important? submitted by /u/Significant-Guest-14 [link] [comments]

I’m a certified associate data engineer. What’s next?

I’ve been working as a “data engineer” in Azure Databricks for a while, but I work on a team where my scope is extremely limited to silver/gold work. What studies can do I do next to continue developing? I’m thinking of diving into the azure certs submitted by /u/WeirdAnswerAccount [link] [comments]

De role evolution - where are things going?

submitted by /u/Cultural-Reserve-259 [link] [comments]

Writing nice unit tests is impossible

First of all we have a lot of classes that use the DatabrickSession import, which makes unit testing impossible and whenever I have the "normal" spark import and test it, then it looks absolutely ugly. Would it help to have the schema as json? I use json schema to read for transformations. If I want to test without json its horrible, do you guys test with json schema files? For example this is just the output, then I would need 2x this because of the input and the test is unreadable (according to my senior BUT WHAT AM I SUPPOSED TO DO ): schema = """ id INT, items ARRAY fortnite: STRING, babies: INT, moreStuff: MAP >> """ expected = spark.createDataFrame( [ { "id": 1, "items": [ { "fortnite": "ABC", "babies": 2, "moreStuff": { "size": "L", "color": "red", }, } ], } ], schema=schema, ) submitted by /u/Similar-Bug-350 [link] [comments]

Best practices for SLA monitoring and automated retries across hundreds of Lakeflow Jobs

igrating from Cron-based Airflow to Lakeflow's Data-Aware Triggers — Real-world experiences?

Task Success is not Routing Health: a Databricks Smart Routing Case Study

Build with Replit Against Managed Postgres in Databricks Apps

DevOps Essentials for Data Engineering: Lab does not match Videos

Databricks Solutions Architect Vibe Coding Round

Has anyone recently taken the databricks Solution Architect interviews? I have a vibe coding round with databricks in next week. I'd appreciate any guidance! Thank you 😊 submitted by /u/wilysarah [link] [comments]

How do you parse an xml that's in string format?

I have xml data that's for God knows why it's in string format. And the fking thing is so messed up or at least I think it's messed up because it's not consistent. Sometimes there is something else in there. Xml is something like this: .... ... ... Regex is not an option. Substring is not an option. Because it's so messed up it's not consistent. I just need to somehow parse and get that Name, Age, Gender values using Sql/Databricks sql Please, help you mate. submitted by /u/pineapple_brownies [link] [comments]

Choosing the right format, explained.

submitted by /u/ConstantNo2668 [link] [comments]

Jobs and Runs UX is Frustrating

I'm not new to Spark, but I'm pretty new to the Databricks platform. I am finding that the UX for monitoring jobs and runs is very rigid, and doesn't present my workloads as I would expect. Here is one simple example. If I submit a run with the "jobs/runs/submit" API then I can provide a custom and ad-hoc "run_name" that appears in the management console called "Runs". This is good. But if I submit a run that references a pre-existing job (using the "jobs/run-now" API) then there is NOT a way to provide a custom "run_name" that will be displayed in the databricks console. The only name that can be shown is the job's name. There are other things that don't seem right either. If I enter custom "tags" on my jobs, then I will be able to use the tags to filter on the Jobs list. But when I click on the Runs list, I can't filter on those same "tags" anymore. IMO, those tags are just as useful on BOTH screens. Another example - the UX doesn't allow me to show more than 20 completed runs at a time. I have to click the Next/Previous button to find runs. Paging thru a long list of runs is a really painful experience. I'm also a user of Microsoft Fabric. I once thought that the "Monitor" console of Fabric was pretty unfriendly ... but now that I'm in Databricks I realize that I'd much rather use their endless scrolling UX design, than having to spam-click the Next/Previous buttons. Even the HDInsight-yarnui allowed me to navigate my workloads more easily than I can in databricks; and that UX is a decade old by now! Is there a different UX experience for Jobs and Runs that I'm missing? Maybe a VS code extension in the community or something like that? Any tips would be appreciated. submitted by /u/SmallAd3697 [link] [comments]

Why External Secrets in Unity Catalog Matter

Lakeflow Connect SQL Server Connector

I recently enabled Lakeflow Connect (lfc) on the source database - the issue is, some of the tables in the source database (managed by another team) does NOT have a primary key (which means that in lfc, a __databricks_id is used to identify a unique record). Thus, the DBAs enabled CDC on the source database. However, when I ingested the data into DBX using the Lakeflow Connect Managed SQL Server Connector, one of the tables in the source database had duplicate records (two or more records with the same value across all columns). This caused my Lakeflow Connect pipeline to break. Any ideas on how to fix this? (Other than dropping duplicate records in the source DB and implementing a unique constraint on the source DB)? I was wondering if there is a specific setting in Lakeflow Connect that I can toggle that I'm missing. submitted by /u/RazzmatazzLiving1323 [link] [comments]

Delta Executor: Things to Check Before Using a Roblox Tool

Tutorial: Transform your Lakeflow Connect ad data into visual and conversational analytics

Broken x-axis sort order in combo chart

Meet SDP Rewind: An undo button for your ETL pipelines

Announcing Databricks AppQuest: Hands-On Guidance and Cash Prizes for APJ Developers

Read this if you use Streaming Tables in Lakeflow Spark Declarative Pipelines

🚀 We’re excited to announce that Lakeflow Spark Declarative Pipelines (SDP) now supports creating “vanilla” (i.e., non STREAMING) MANAGED TABLES and writing to them via one or more append flows , using the new CREATE TABLE ... FLOW ( SQL ) and create_table() (Python) APIs . What is this Beta? This Beta allows creating a managed table that is populated by append flows: CREATE TABLE ... FLOW (SQL) / create_table() + @append_flow (Python) create a managed table written by one or more flows. Fan multiple sources into one table — declare several flows targeting the same managed table. Full table surface works: partitioning, liquid clustering, expectations, row filters, table properties, and private (pipeline-local) tables. import_checkpoint on append_flow , which migrates an existing Structured Streaming workload into a pipeline without reprocessing the source — the flow imports the query's existing checkpoint and resumes from the last committed offset with state intact. Example (Python): from pyspark import pipelines as dp dp.create_table("combined") dp.append_flow(target="combined") def from_a(): return spark.readStream.table("source_a") u/dp.append_flow(target="combined") def from_b(): return spark.readStream.table("source_b") Example (SQL): CREATE TABLE events PARTITIONED BY (bucket) FLOW INSERT BY NAME SELECT id, bucket FROM STREAM read_files('abfss://my_path', format => 'json'); Where do we need help? We are in Beta, so there might be some rough edges. Please take this for a spin and share your feedback here . What’s next? Managed Tables support for other flow types (AutoCDC, Replace Using, and Replace Where) is coming soon! Learn more CREATE TABLE ... FLOW (SQL reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-create-table-flow create_table (Python reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-create-table import_checkpoint on append_flow — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-append-flow Questions, feedback, or help: comment below or share feedback in the form: https://forms.gle/7bGP5FYN7P1Z4WP27 submitted by /u/SlightImagination250 [link] [comments]

CUSTOMER STORY | Siemens Healthineers modernizes MRI scanner data on Databricks

Best Practice for Handling Schema Evolution with Auto Loader in Production?

Announcement | Autoscaling Lakebase Postgres

Changing Root Metastore Location

CSV export from AI/BI Dashboard table widget does not preserve visual column order

Getting started with Delta Lake basics

Storage Credential creation fails with "Access Connector ... could not be found"

Free Trial – “Daily limit for workspace creation” error

External secrets in Unity Catalog is in Beta, and it replaces Key Vault-backed secret scopes

This is the Databricks release I have been waiting for. Unity Catalog schemas can now hold external secrets, such as Azure Key Vault, and that will change how we manage and utilize secrets in our Databricks projects. On most of our engagements the secrets of record already live in Azure Key Vault, so we wire up a Key Vault-backed secret scope and move on. It works, but it is a workspace-level object from the pre-Unity Catalog era: configured per workspace, permissions managed through a separate secret ACL API, a flat scope/key namespace, and invisible to the governance model everything else on the platform now runs on. Read more: https://www.linkedin.com/posts/cenh_databricks-azure-unitycatalog-ugcPost-7504125176993800192-W9on/?utm_source=share&utm_medium=member_desktop&rcm=ACoAABmJHrsBNAC3x3H1M58JRKoHv_l4D61n0-8 submitted by /u/Lenkz [link] [comments]

How do you test whether a retriever stops too early?

Databricks Unity Catalog Explained | Full Governance Guide (Access Contr...

submitted by /u/macxima [link] [comments]

UC secrets in Key Vault

Secrets in Unity Catalog is a great feature introduced a few weeks ago, but since then, everyone has been asking to use Azure Key Vault as a secrets backend. Thanks to rapid development, we can now link our schema to Azure Key Vault; UC will read secrets as UC secrets, and permission management will be through Unity Catalog. In that scenario, you insert/update secrets in Azure Key Vault, but read/reference and grants can go through UC. more news https://databrickster.medium.com/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

Building for Failure: Implementing Data Quality Firewalls in Petabyte-Scale Medallion Architectures

Azure AI Foundry + Databricks Architecture | Deploy Genie Agent with DAB...

Azure AI Foundry Databricks architecture, Deploy Genie Agent with DABs, Databricks Genie Agent, Azure Databricks Genie Space, how to deploy genie agent with declarative automation bundles, azure ai foundry + databricks integration, fully operating genie architecture databricks, databricks unity catalog genie agent, azure databricks bronze silver gold architecture, agent to agent nlq databricks, databricks spark python sql delta lake unity catalog, production ready genie agent deployment, databricks vector search index genie, microsoft purview databricks governance submitted by /u/macxima [link] [comments]

Designing an Effective "Quarantine" Pattern for Failed DLT Expectations

How to create monotonic function to incrementally add obj_id for datasource in pyspark

The Future of Iceberg Isn't One Engine. It's an open Control Plane with many engines.

submitted by /u/codingdecently [link] [comments]

Managing Service Principal Permissions at Scale: External Locations vs. Managed Volumes

Auto Loader stream fails on RocksDB checkpoint after enabling managed file events

Controlling allow/ask/deny lists for predelivered MCP connectors

Migrating Large-Scale Z-Ordered Tables to Liquid Clustering: Strategies for Production Pipelines

Where Omnigent Fits Alongside ChatGPT Work, Claude Cowork and Cursor Projects

Do we still need fact and dimension tables in the Gold layer?

Data engineers traditionally modeled Gold layers using fact and dimension tables, partly because storage and compute were expensive. But with modern cloud data platforms, storage and compute are much cheaper and more scalable. So I’m curious: what does your Gold layer actually look like today? Are you still using a traditional star schema (facts + dimensions), or have you moved toward wider, denormalized tables / business-oriented models? And more importantly, why? submitted by /u/almightysosa888 [link] [comments]

Migrated our reproting layer to databricks and access control turned into a project of its own

The actual data movement into databricks did go fine (unity catalogue made lineage way easier to see than the older setups). But what we didn't expect was the time that went into access control, once everything was centralized instead of scattered across separate warehouses with their own permissions, data that used to be siloed was suddenly way visible to more people by default, which was found out when a couple of teams noticed they could see data which they probably shouldn't. Spent almost as much time on catalog level permissions/row filtering as on the actual pipeline work. Is this normal for a databricks migration or did we happen to have an unusually messy access management? submitted by /u/BrownAnclourne [link] [comments]

What Data Engineers Need To Know About Delta Lake 4.3

replaceUsing and replaceOn give you a better overwrite primitive, and every catalog-managed table operation now runs through the catalog. submitted by /u/Lenkz [link] [comments]

Google Drive connector in Lakeflow Connect is now generally available (GA)

The Lakeflow Connect connector for Google Drive is now generally available ! It’s now easier than ever to ingest structured and unstructured files from Google Drive into Delta tables for analytics and AI workloads. You can configure a managed ingestion pipeline through the UI or managed API. Managed pipelines automatically handle incremental processing, automatic retries with exponential backoff for source API rate limits, failure recovery, and provide rich Google Drive metadata. For direct control over ingestion logic, you can also just use the Spark + SQL APIs directly: spark.read , Auto Loader, read_files, or COPY INTO pointed at Google Drive URLs. https://preview.redd.it/j65yaaft3uoh1.png?width=2048&format=png&auto=webp&s=0f9f13cc63572b991232a8fa617aa1fe6697369c Link to public docs + references: Google Drive managed connector documentation Spark + SQL APIs and examples Community blog and video tutorial: From PDF to insights Data + AI Summit session: Intelligent Document Processing with Lakeflow Common workloads include: Loading Google Sheets, Excels, CSV, JSON, and other structured files into Delta tables. Ingesting PDFs, Google Docs, Google Slides, and images. Parsing documents with ai_parse_document to prepare content for extraction, search, and agents. Examples of using the Spark + SQL APIs: Read an Excel sheet from Google Drive with spark.read : ​ df = (spark.read .format("excel") .option("databricks.connection", "my_gdrive_conn") .load("https://docs.google.com/spreadsheets/d/9k8j7i6f...")) Ingest unstructured documents + PDFs from a Google Drive URL with read_files , then easily parse them using ai_parse_document : ​ CREATE OR REFRESH STREAMING TABLE gdrive_documents_table AS SELECT *, "_metadata" FROM STREAM read_files( "https://drive.google.com/drive/folders/1a2b3c4d...", format => "binaryFile", `databricks.connection` => "my_gdrive_conn", pathGlobFilter => "*.{pdf,docx}"); CREATE OR REFRESH STREAMING TABLE documents_parsed AS SELECT *, ai_parse_document(content, map('version', '2.0')) AS parsed_content FROM STREAM gdrive_documents_table; Coming soon: Ingest Google Drive’s per-file permissions and ACL metadata to power permission-aware AI agents, enterprise search, and more. If you try it, share what you are building and let us know if you hit any friction! submitted by /u/BricksterJ [link] [comments]

SharePoint connector in Lakeflow Connect is now generally available (GA)

The Lakeflow Connect connector for Microsoft SharePoint is now generally available! It’s now easier than ever to ingest structured and unstructured files from SharePoint into Delta tables for analytics and AI workloads. You can configure a managed ingestion pipeline through the UI or managed API. Managed pipelines automatically handle incremental processing, automatic retries with exponential backoff for source API rate limits, failure recovery, and provide rich SharePoint metadata. Soon, our managed connectors will also support ingesting SharePoint Lists and per-file permissions metadata. For direct control over ingestion logic, you can also just use the Spark + SQL APIs directly: spark.read , Auto Loader, read_files, or COPY INTO pointed at SharePoint URLs. Common workloads include: Loading Excel, CSV, JSON, and other structured files into Delta tables. Ingesting PDFs, Word documents, PowerPoint files, and images. Parsing documents with ai_parse_document to prepare content for extraction, search, and agents. https://preview.redd.it/89i379aattoh1.png?width=2180&format=png&auto=webp&s=292350dfddfc9bc1a4daf3fc4447394821206054 Link to public docs + references: SharePoint managed connector documentation Spark + SQL APIs and examples Community blog and video tutorial: From PDF to insights Data + AI Summit session: Intelligent Document Processing with Lakeflow Examples of using the Spark + SQL APIs (after first creating a UC connection ) : Read an Excel sheet from SharePoint with spark.read : excel_df = (spark.read .format("excel") .option("databricks.connection", "my_sharepoint_conn") .option("headerRows", 1) .option("dataAddress", "Sheet1!A1:M20") .load(" https://mytenant.sharepoint.com/sites/Finance/Shared%20Documents/Monthly/Report-Oct.xlsx") ) Ingest unstructured documents + PDFs from a SharePoint URL with read_files , then easily parse them using ai_parse_document CREATE OR REFRESH STREAMING TABLE sharepoint_documents_table AS SELECT , "_metadata" FROM STREAM read_files( " https://mytenant.sharepoint.com/sites/Marketing/Shared%20Documents ", format => "binaryFile", databricks.connection => "my_sharepoint_conn", pathGlobFilter => " .{pdf,docx}"); CREATE OR REFRESH STREAMING TABLE documents_parsed AS SELECT *, ai_parse_document(content, map('version', '2.0')) AS parsed_content FROM STREAM sharepoint_documents_table; Coming soon: Ingest SharePoint Lists into Delta tables (coming super super soon) Ingest SharePoint’s per-file permissions and ACL metadata to power permission-aware AI agents, enterprise search, and more. If you try it, share what you are ingesting and where you hit friction! Don't hesitate to ask questions! submitted by /u/BricksterJ [link] [comments]

Unable to Start Serverless Compute in Databricks Free Edition

Oracle NUMBER → DecimalType(38,10) on ingestion, and silver layer best practices

Handling New Columns in a Databricks Data Pipeline

Incremental data processing explained!

submitted by /u/ConstantNo2668 [link] [comments]

OAuth M2M (client-credentials) - getting error with github run

Dark mode setting ignored when embedding BI dashboard inside Databricks App

The Replit | Databricks Integration is now GA for building governed apps

submitted by /u/tony-dang [link] [comments]

Databricks metric views to PowerBI?

submitted by /u/FiftyShadesOfBlack [link] [comments]

[Megathread] self promotion

Hey r/databricks , In order to keep the main feed clean, we are implementing a weekly megathread for self promotion for companies who do lots of work with databricks. Please direct all self promotion posts here and keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules! submitted by /u/AutoModerator [link] [comments]

[Megathread] Certifications and Training

Hey r/databricks , please direct all certification and training posts here. There's upcoming learning festival September 16 - October 14 2026. You can get 50% discount voucher on any certification. Databricks Advanced Learning Festival: September 1... - Databricks Community - 166157 Good luck to everyone on your certification journey! submitted by /u/AutoModerator [link] [comments]

[Megathread] Hiring and Interviewing at Databricks - Advice, Prep, Questions

Hey r/databricks , we're noticing a lot of repeated interviewing and hiring posts that tend not to get much engagement. We're going to combine them into a monthly thread so that you're more likely to get answers, plus we can ask our recruiting team to keep an eye on them if there are any general questions. submitted by /u/AutoModerator [link] [comments]

🚀 Quest 5 Submission: Intelligent RAG Knowledge-Base & Note-Taking Workspace

Allow Public Network Access - Disabled, I can still access the workspace from public internet

Nexa: Genie is a Minute Away

Smart Routing in Unity AI Gateway: 30%+ Cost Savings on Coding Tasks

Any plans to make externally backed secrets in Unity Catalog enter public preview/GA?

Hi Databricks Team, Seeking your advice on the above. submitted by /u/RazzmatazzLiving1323 [link] [comments]

Metric views materialization failure

Building Deterministic Databricks Genie Agents

How do I read the databricks spark ui? Couldnt find any tutorials specifically for it.I know spark ui a bit.

submitted by /u/Intelligent_Duck_854 [link] [comments]

Databricks Community Contest | Winners of the Genie-Powered App Challenge!

Databricks Asset Bundles: How to manage dependencies between volumes/files and cluster creation?

Databricks Quest 4-(Deploy Your First Databricks App) success story

Databricks App suddenly unable to load valid Jupyter notebook

Actually understanding Unity Catalog Managed Tables

Looking for Databricks Data Engineers in EU - Fully Remote

I'm working on one of the largest projects in Europe currently, looking to onboard at least 5 data engineers with serious Databricks experience. Would be a 6-month initial contract, would be open to a further extension if needed. If this is something you'd be interested in, then comment below. I will ping you. €600-650 per day submitted by /u/Reuben_UMATR [link] [comments]

Agent outside databricks communication with databricks delta table

Materialized Views vs Streaming Tables in Databricks: A Practical Guide for Data Engineers

databricks-langchain PyPI installation stuck for 30+ minutes on multiple Azure Databricks clusters

Announcement | How we eliminated $1 million a year of wasted AI agent spend in one hour

shutil.copy from /local_disk0 to Unity Catalog Volume hangs for hours — recommended pattern for log

How are you combining forecasting models for time-series work?

[Discussion] Lakeflow Jobs: How do you use table update triggers?

(databricks product manager here) Curious how people are using table update triggers in production. https://docs.databricks.com/aws/en/jobs/trigger-table-update Do you rely on the available debouncing capabilities (protect against over and under triggering) or do the options feel confusing enough that you mostly work around them? When a trigger needs to represent more than “run when this table changes,” how do you express the business logic? For example: Do you use control or checkpoint tables to signal that an upstream workflow has finished? Do you wait for a specific status, batch ID, watermark or set of tables before starting downstream work? Do you put that logic in the trigger itself, or in a separate workflow/job? What has worked well and what has been difficult to reason about or debug? Any other suggestions or feature requests relating to Table Update Triggers? I’m especially interested in real-world patterns and whether the current debouncing behavior is intuitive enough for you, or whether a control-table pattern ends up being the clearer approach. Edit: how many folks still use control tables instead of data tables with these triggers? Thank you 🙏 https://preview.redd.it/munx0r2u4ooh1.png?width=660&format=png&auto=webp&s=80e6b80a62cf77870bf444e7634d1ee7412feee3 submitted by /u/saad-the-engineer [link] [comments]

Auto Loader Strategy: Balancing Cloud Notification Costs vs. Directory Listing in Massive Migrations

Databricks Support -0100505- Exam Suspended because non compliance requirements set by test sponsor

Managing Schema Drift and Evolution in Spark Declarative Pipelines (SDP)

No more UNION ALL-ing all of your SDP pipeline event log tables for monitoring

https://preview.redd.it/3w0j4ss3inoh1.jpg?width=2048&format=pjpg&auto=webp&s=50bc79280a546544e8f795947e5af6b211e9596a Hi folks, Databricks PM here - I wanted to share an exciting update that you no longer have to manually publish and combine your pipeline event logs for monitoring across pipelines and workspaces. We just launched the beta for the pipeline events system table (system.lakeflow_pipeline_events_preview.pipeline_events). Key features: All pipeline events (regardless of cluster start) are automatically captured without any manual enablement or maintenance. Events are aggregated in a central system table without requiring any custom aggregation logic. This data is available at close to real time latency (based on our internal testing we achieve a P99 latency of less than 1 minute). An admin can grant a single user access and they can query events for every pipeline in the table. Fine-grained access controls support which scopes the visibility to only the pipelines the user has access to is coming soon. The data remains in the system table even after pipeline deletion and is retained for 13 months. If you want longer retention this is also possible with the configurable retention feature for System Tables. Query ergonomics are better now with the use of VARIANT. Here are some sample queries in case you want to try them out: -- The latest error for each pipeline that has errored in the last 7 days, with the outermost exception. -- The exception chain is ordered with the root cause last, so read element -1 for the root cause. -- On many errors only the first element carries error_class and sql_state. SELECT workspace_id, pipeline_id, event_time, event_type, message, error.exceptions[0].error_class AS exception_error_class, error.exceptions[0].sql_state AS exception_sql_state FROM system.lakeflow_pipeline_events_preview.pipeline_events WHERE level = 'ERROR' AND event_time >= current_timestamp() - INTERVAL 7 DAYS QUALIFY ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id ORDER BY event_time DESC) = 1 ORDER BY event_time DESC -- Flow throughput for a specific pipeline SELECT origin.flow_name, date_trunc('HOUR', event_time) AS hour, SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) AS rows_written FROM system.lakeflow_pipeline_events_preview.pipeline_events WHERE pipeline_id = ' ' AND event_type = 'flow_progress' AND event_time >= current_timestamp() - INTERVAL 7 DAYS GROUP BY origin.flow_name, date_trunc('HOUR', event_time) ORDER BY hour DESC, rows_written DESC -- Data quality: failed expectations by dataset, per update, in the last 1 day SELECT pipeline_id, update_id, origin.dataset_name, expectation.name AS expectation_name, SUM(expectation.failed_records) AS failed_records FROM system.lakeflow_pipeline_events_preview.pipeline_events LATERAL VIEW explode(variant_get(details, '$.flow_progress.data_quality.expectations', 'ARRAY >')) AS expectation WHERE event_type = 'flow_progress' AND event_time >= current_timestamp() - INTERVAL 1 DAY GROUP BY pipeline_id, update_id, origin.dataset_name, expectation.name HAVING SUM(expectation.failed_records) > 0 ORDER BY failed_records DESC; Beyond single queries you can build alerting (using Databricks SQL alerts) and dashboards. We will share a new dashboard template soon - I will update this post once its available. Call outs: This is in beta right now, if you are not opted in we will not capture your event log data. Enablement: Toggle on the “ Lakeflow Pipeline Events System Table ” from the account level preview. Docs are linked here , would love to hear your thoughts on how you will use it or what else you want to see to improve observability! submitted by /u/brickster_123 [link] [comments]

Before you make your pipeline “near real-time”, check where the latency actually is

Here is a common pattern: a job polls a queue every 20 minutes, fans the payload out to 60 to 80 Bronze tables using MERGE operations, and takes 16 minutes to complete. When leadership asks for “near real-time,” the default response is to drop the polling interval to one minute. That approach fails because of simple arithmetic. Worst-case latency can be roughly the polling interval plus the batch duration, which puts total time at 36 minutes. Setting the trigger to 1 minute while the batch takes 16 minutes won’t give you 1-minute latency. It can instead create queued runs and additional contention between jobs. You need to optimize the batch duration first. In wide fan-out architectures, two bottlenecks can cause significant delays: MERGE operations running on empty targets. If a batch only updates 6 out of 82 tables, the other 76 MERGE operations are unnecessary work. Partition the payload first, check which targets actually received rows, and skip empty writes. Sequential writes. The 82 tables are independent, so processing them one by one in a driver loop can make the batch duration approach the sum of the individual write latencies instead of being closer to the slowest write. Where appropriate, independent writes can be processed concurrently. Address these two points first to reduce batch duration before shortening the trigger interval. Once that is done, re-evaluate whether you actually need a streaming architecture. For anyone being pushed to deliver “real-time” processing: what latency requirement did the business actually need once it was clearly defined? In practice, teams sometimes ask for seconds when minutes would actually meet the requirement. submitted by /u/AbilyticsEng [link] [comments]

Databricks is too expensive for small teams" is usually a workload problem, not a platform problem

Every few weeks someone posts a version of “our bill is going from $1k to $5k a month, is Databricks even worth it at our size?” The answer isn’t really about company size. It comes down to how you’re using the platform. All-purpose compute being used for scheduled jobs. Interactive clusters are convenient, but they can increase costs quickly. If a notebook runs on a schedule, moving it to jobs compute can make more sense. SQL warehouses sized for peak usage and left running. Using auto-stop and choosing a warehouse that can scale when needed can help avoid paying for idle capacity. Continuous triggers on jobs that don’t need them. This one gets misdiagnosed a lot. The fix usually isn’t “rewrite it as batch,” which costs you checkpointing and exactly-once. It’s Trigger.AvailableNow , which processes what’s available and shuts the cluster down. Databricks recommends it for incremental batch processing. If the table needs a 15-minute refresh, that’s a scheduled job with an AvailableNow trigger, not a cluster running at 3 AM. Once these three areas are addressed, the bill for a small team can often come down to a much more reasonable baseline. Then the more interesting question is: are you actually getting value from Unity Catalog, Delta, and the broader BI, ETL, and ML capabilities, or are you mainly paying for Spark compute that you don’t really need? If your data fits comfortably in Postgres or ADX, you have one main consumer, and you don’t need much governance or lineage, Databricks may not be necessary. No amount of cost tuning changes that. For teams running Databricks on relatively small workloads, what actually made it worthwhile for you? Was it a specific technical requirement, governance, or simply the convenience of having everything in one platform? submitted by /u/AbilyticsEng [link] [comments]

Looking for buddy

Hey guys Im from Hyderabad, India . A databricks dataengineer here. As we have event on 7th Oct 2026 in Mumbai, im planning to visit it. Who else are joining. Lets have some good connections 😌 submitted by /u/PrinceShahil6 [link] [comments]

Resolution of Missing Certificates in Accredible

The deployment decalogue

I am an ML engineer, but I come from a software engineering background: years of full-stack work, with heavy DevOps and Terraform experience. I come from teams that deploy to production five times a day with real continuous deployment. And honestly? Pressing the button still feels weird sometimes. Every engineer knows that feeling, no matter how good the safety net is. So I wrote down the list that settles it. Ten commandments, one flow, written with data scientists and ML teams in mind, but it works for batch jobs, realtime inference, and LLMs alike. Answer honestly, and if all ten are true, you can ship to production anytime, in any form or way. submitted by /u/SuspiciousPavement [link] [comments]

Enterprise workspace blocked by Databricks-set rate limit of 0 on all models (AWS Marketplace)

BOOTSTRAP_TIMEOUT on cluster start (Southeast Asia)

Databricks AppQuest - Dev kit !

Are Unscored Questions Really Unscored?

Unity Catalog Open Source in Name Only (UCOSINO)

Consider a callstack where something bad is happening in Spark or Unity Catalog (image above). Any software engineer will google for the message, and then for the Exception class, and then for the call frames shown on the stack (starting at the top or bottom). For any commonly encountered Exceptions from UC (something like com.databricks.sql.managedcatalog.acl.UnauthorizedAccessException), we will find dozens of results from a search engine. Others on the internet have already shared their experiences, and the search results are normally actionable. The users tell us what they had done to avoid or fix the error. But software engineers have heard for two years that "unity catalog is open source". So a software engineer will proceed to look for the source repo where they might find the full definition of "UnauthorizedAccessException", along with all the related references. No such thing exists. (Admittedly there is a public-facing github, called "unitycatalog", but it is virtually worthless and there is no overlap with the real-world UC in databricks, as we experience it.) It only takes one or two repeats of this, before a software engineer will realize that none of this stuff is actually open source. UC doesn't compare to a REAL open source software like Apach Spark. If we search for spark references in the call stack (eg. "org.apache.spark.sql.DataFrameReader"), then we are immediately taken to the source repo at github! I do give Databricks a lot of credit for open-sourcing spark. But nowadays they take too much liberty with the word "open source", to the point where it lost all of its meaning. UC is not opensource in any substantial way. Maybe there is an API spec that is open, but that is the extent of it. Another example is lakebase which the CEO claimed to be open source at the recent summit. There has never been any software as proprietary as neon/lakebase. It doesn't actually bother me if a CEO forgets how to use the term "open souce" correctly in English. What makes me more upset is when I expect to be able to use google to find the source code for "UnauthorizedAccessException", and come up with absolutely bupkis. Can anyone tell me a definition of "open source" which would potentially include either Unity Catalog or Lakebase? I'm assuming that when these words are used by the CEO, he does NOT intend to imply that the actual source is open to the public. submitted by /u/SmallAd3697 [link] [comments]

datatf: Automate importing Databricks workspace into Terraform

DataTf , bring an existing Databricks Workspace into Terraform (IaC). generates dynamic terraform.tfvars + import code built on the Databricks Go SDK compatible with Terraform or OpenTofu compatible with Databricks Omnigent, Claude Code, opencode, OpenAI Codex and others native support for Azure, with GCP and AWS support upcoming Disclosure: I am the Author/Founder, 536 Technologies. submitted by /u/536tech [link] [comments]

Has anyone tried the new Databricks AI/BI feature?

I recently came across Databricks AI/BI and was curious to know how people are finding it. It looks like Databricks is trying to bring BI and analytics more directly into the Databricks platform, with dashboards and Genie for asking questions in natural language. Has anyone actually tried AI/BI in a real project? How is it compared to Power BI or Tableau from your experience? Is it good enough for regular BI use cases, or is it still better to use a separate BI tool? Would like to know your experience, especially if you have used both. submitted by /u/Bhanuprakash_1947 [link] [comments]

How to organize your notebook tabs?

It was a real pain, but now, with a few tricks, you can manage them better. First, in Workspace files, next to the DABs folder or git repo, there is a small shortcut to show only tabs from that DABs folder or git repo. Alternatively, you can also use the switcher in Home next to Notebook. If you need to organize your tabs differently, there is new functionality: spaces, which let you group them however you like. more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

A single Databricks Architect Champion can potentially affect a company’s Partner Tier

There are 1,600+ Databricks Partner Champions worldwide and around 1,795 partners listed in the public directory. But Champions are distributed very unevenly. Some large partners have dozens. Others may have only enough to meet their current tier requirements. So if one Architect Champion leaves, the company could potentially fall below one of the requirements for its Partner Tier. That makes Champion status more than just another badge — it can have real value for the employer. For Databricks architects working at partners, I’d definitely put Partner Champion on the career roadmap. Do your companies maintain a buffer of Champions, or just the minimum required? submitted by /u/Significant-Guest-14 [link] [comments]

Reissue Voucher

Cost-optimized way to reflect source DB changes in Silver in <1 minute?

Due to new business requirements, we need to reflect the state of a few source DB tables (5 to 40 million rows each) in the Databricks Silver layer in less than 1 minute. Currently, the flow looks like this: Source DB → AWS DMS in CDC mode (ingests new data every 30 seconds to S3) → S3 landing bucket → DLT pipeline running on serverless compute in continuous mode. The DLT pipeline ingests the append-only data into the Bronze layer using file notification mode and updates the Silver layer using an Auto CDC flow. This works great, and we achieved what we wanted with relatively low effort because we already had DMS in place. We just added an extra replication task to ingest data more frequently for the tables we need. However, in this setup, the DLT pipeline costs are quite high. Ingesting just 6 Bronze tables and 6 Silver (Auto CDC) tables costs around $50 per day, which is about $1,500 per month. For comparison, DMS, which replicates more than 800 tables to S3, costs us less than half of that. My question is : is there any other more cost-optimized option we could consider to achieve less than 1 minute latency when reflecting the source DB state in the Silver layer? Maybe Lakeflow Connect or some custom process? Extra notes: - I know that adding more tables to the DLT pipeline makes the cost per table lower because Databricks can optimize the clusters more efficiently. - I know that using a cron schedule could reduce costs, but for these particular tables, we can’t use a schedule like every 10 minutes or similar because we need the data to be updated in less than 1 minute. - I know that for the relatively small tables currently in scope, we could eliminate the Auto CDC flow and create a normal view on top of the Bronze table, with deduplication and deletion logic. This would slightly sacrifice query performance, but we expect more similar use cases in the future, so I’m looking for a solution that can scale. submitted by /u/CyberEnzo [link] [comments]

Want to become a Forward Deployed Engineer?

We’re running a hands-on Forward Deployed Engineering (FDE) workshop focused on that gap: turning ambiguous customer problems into secure, governed, production-ready AI deployments. You’ll work through a realistic 90-day AI-agent deployment for a regulated customer, covering: evaluation and reliability security and governance rollout and deployment risks stakeholder alignment a CISO hot seat The workshop is led by Keith Bourne, Forward Deployed AI Engineer at Tribe AI, and Tanya Dixit, Forward Deployed Engineer at Google. There’s also a practical FDE career-path assessment to help you identify your strengths and gaps, explore relevant employers, and build a personalized 90-day career plan. Here is the link for futher details: https://luma.com/fde-workshop?utm_source=career submitted by /u/kunal_packtpub [link] [comments]

Genie Agent's idea of "the Midwest" includes Kentucky. Ours doesn't. Notes from [8 months] of Genie Spaces in prod.

Genie Space has been live for our sales folks for 8 months, maybe 500+ regular users. Short version of what I've learned, since everything I read before setting one up was either a demo or an argument about whether analysts are getting replaced. The failure mode isn't an error. It's a number that's slightly wrong and totally believable. Someone asked how the Midwest was doing, the number looked fine, sat in a deck for weeks. Genie's Midwest includes Kentucky. Our territory map doesn't. You can't catch that by looking at the output - you catch it when finance does. Four things you're configuring, roughly in order of how much they've mattered: Column comments. Free, and the biggest lever by far. Genie reads COMMENT metadata before writing SQL. No comment and segment is just a word - it has no idea whether your values are Enterprise/Mid-Market/SMB or something else, so it guesses. COMMENT ON COLUMN vw_sales_summary.segment IS 'Customer tier: Enterprise (>$1M ARR), Mid-Market ($100K-$1M ARR), SMB ( One pre-joined view, not raw tables. I did raw tables first. Every join it has to figure out is a coin flip. Also, put your test-data filter in the view - then every question anyone ever asks inherits it and you're not trusting the model to remember. SQL expressions. Register a named metric with your SQL and it uses yours instead of inventing one. Ask ten people what an "active customer" is and you'll get eleven answers; this is the box where you settle it. Name them how people talk - "Active Customers" matches, cnt_dist_cust_qtd never will. Example Q&A pairs. Nothing gets retrained, they just sit in context when something similar comes in. The shape travels further than I expected — registered revenue-by-category with a cancelled-order exclusion, and a Q2 question a month later inherited the exclusion in a query I never wrote. Two things from the instructions box worth stealing. One, tell it to ask instead of guessing when the time period is unclear - people trust it more when it occasionally asks. Two, ours has a rule about test customers with a TST_ prefix, whose orders carry real statuses so the status filter misses them entirely. Everyone on the team knew that. Nobody had ever written it down. Curious what other people have ended up putting in their instructions box. Assume everybody hits their own Kentucky eventually. (Here is the longer version with more SQL is on SQLServerCentral, it's mine https://www.sqlservercentral.com/articles/databricks-genie-spaces-for-sql-analysts-natural-language-querying-without-leaving-your-data-platform but the above is the useful part) submitted by /u/mehulbhuva [link] [comments]

Databricks Dashboard Embedded Ask Genie

Why we didn't build one big Genie Room

Can't access abfss data in azure databricks when providing shared key (fighting UC?)

Do UC table tags propagate to billing for Predictive Optimization and Data Quality Monitoring?

Introducing Stream-Stream Join Support in Apache Spark Real-Time Mode

CUSTOMER STORY | Rippling powers AI-driven GTM with Genie Agents on Databricks

Multiple gateway pipeline for same database

How are you separating dev, staging and prod in Unity Catalog without duplicating everything?

Data migration from teradata to databricks

I joined a new company recently and got a migration project here, they are migrating from teradata on prem to databricks, I have never done any migration in the past can anyone suggest some helpful yt videos or any other knowledge source? submitted by /u/WarPowerful740 [link] [comments]

Databricks Secrets

Databricks Account locked Issue

Learn Databricks Lakeflow | Ingest, Orchestrate, and Build pipelines on one platform.

Need some advice on Snowflake vs Databricks

submitted by /u/Ok_Independent_343 [link] [comments]

How to automate downloading files from Databricks to a local machine without PATs or CLI?

Hey everyone, Looking for some advice on automating a workflow in a pretty locked-down corporate environment. Context: Large enterprise with strict IT security and governance. Databricks was recently rolled out as our cloud data hub. The entire pipeline (ingestion, processing, and generating the final CSV) is already automated inside Databricks. I need to automatically save a copy of this generated CSV to a local machine / internal network. Right now, the only way I can do this is manually opening the workspace UI and clicking "Download." Databricks CLI is blocked and Personal Access Tokens (PAT) are disabled How do you usually automate pulling files from the cloud down to on-prem / local machines under these restrictions? Thanks! submitted by /u/Firm_Yogurtcloset835 [link] [comments]

Can I run jobs continuously without interruption on a permanent free (Community Edition) account?

What to learn? AWS databrics or Azure Databrics

I have experience in AWS and I want to learn Databricks now From future perspective what I need to learn Databrics with AWS or Databricks with Azure? I can see there are lot of openings related to Azure with Databricks Can anyone plz help me submitted by /u/mali_sagar [link] [comments]

What's new in Genie One - August 2026

submitted by /u/Youssef_Mrini [link] [comments]

Issue Regarding Missing Badge on Profile: DAIS24 Attendee

Databricks Dashboard Pivot Table: default collapsed state?

Actor for exporting database data into Datasets

submitted by /u/Hayder_Germany [link] [comments]

DELETE Removed the Customer Record—but Did It Remove the Data?

CosmosGenie — Your Universe, Answered (Genie-Powered App Challenge )

Issue in "Build a Declarative Pipeline with Spark Declarative Pipelines"

Harmonizing Informatica Governance Policies Across Databricks and Power BI Import Mode

Could Databricks app support the integration of Genei Room and Power BI?

Expanding Genie Agents: Deep analysis, file reasoning, and more

How are you attributing serverless costs back to individual jobs and teams?

Now that Unity Catalog manages Iceberg natively, is anyone actually switching?

AI/BI Dashboards

🌟 Community Pulse: Your Weekly Roundup! August 31 – September 06, 2026

Apache Iceberg Table Cleanup: A Production Guide

A guide to Iceberg table cleanup — snapshot expiration, orphan file removal, manifest rewriting, delete file resolution, streaming challenges, compliance, and cost. Why sequencing matters, where teams break tables, and how to automate the full lifecycle. submitted by /u/codingdecently [link] [comments]

Exploring the Databricks Application Development Ecosystem

Databrick Oauth federation

Using Databricks Asset Bundles and Lakeflow Jobs in a Real Project

Could Databricks app support the integration of Genei Room and Power BI?

Databricks 5 Minute Features: Governance Hub

Check out the Databricks Governance Hub in my latest 5 Minute Features. A one-stop-shop for all your governance needs; Cost Control, Access, Data Quality, AI Usage all in one place! submitted by /u/Remarkable_Rock5474 [link] [comments]

Solution Accelerator Series | Simplifying Product Onboarding With Generative AI

Genie Agent response Export to PDF via API

Incremental Load Issue with SDP

Secrets in Unity Catalog

Secrets in Unity Catalog store credentials like API keys as governed objects named catalog.schema.secret: created, granted, rotated and audited with standard Unity Catalog privileges and redacted from notebook and job logs. This is a full end-to-end demo: 1)Create a secret with the REST API and in the Catalog UI 2)Read it in a notebook with dbutils.secrets.get 3)Use it to call the OpenAI API 4)Grant read/reference/write access to a user or group 5)Rotate it programmatically 6)Audit every access from a system table submitted by /u/Youssef_Mrini [link] [comments]

Using Delta Sharing to hold ISV entitlement data outside the customer's metastore — reasonable fit?

Databricks Lakehouse Industry Data Models: What Data Engineers Can Learn from the GitHub Repository

Serverless compute resolves some public domains but not others

Managing AI spend? Participate in Databricks user study!

👋 Hi r/databricks , I'm Connie from the Databricks UX! The Databricks Platform team is conducting a user study to understand how admins and organizations control their AI spend/usage and to gather feedback on early redesigns of our AI budgets feature . Your input will help improve admin experiences on the platform! Please fill out the screener survey if you’re open to participating in a 60-min call in the upcoming week, or can connect me with a member of your team who may be the right fit. Study details are as follows: Topic: AI Budgets & Cost Management Duration: 60-minute remote interview over Google Meet Study date(s): Sep 9-18, 2026 (rolling) Compensation: $150 " thank you " gift card for the interview session, if your company policy permits. Completing this screener survey does not guarantee eligibility into the study. Note that completing this screener survey DOES NOT guarantee eligibility into the study. All responses are reviewed for consistency and authenticity, and any duplicate, inconsistent, or fraudulent entries will be disqualified. Thank you for taking the time to help us build a better platform! submitted by /u/Logical-Novel4483 [link] [comments]

Stop Asking an LLM Judge Questions Your Code Can Answer

WLB Databricks GTM

Considering a GTM role with Databricks. Compelling role, comp etc., but cannot get a proper read on the WLB and culture. Have a little one at home, can’t afford a job that requires major travel or 12+ hrs work. Love to hear from people in the company on what the reality on the ground is like. submitted by /u/Illustrious_Guest857 [link] [comments]

Do small companies actually use Databricks?

Sometimes I feel like Databricks is way too expensive. It feels like using a huge truck to move a single grain of sand. My company needs real-time data, but our data volume simply does not justify the use of Spark Structured Streaming. Despite this, they are insisting we move to Databricks. I'm worried our data infrastructure costs will jump from $1,000/month to $5,000/month or more due to the running costs of Databricks SQL Warehouses. Currently, I use Azure Container Apps with KEDA and Python, which helps me manage scaling and keep costs low. We ingest into Event Hubs, use ADX (Azure Data Explorer) as our OLAP warehouse, and archive cold data in a data lake. With this setup, I manage to process all our data with very low latency. When I tested this on Databricks Structured Streaming, I actually got higher latency and much higher costs. Would love to know what you guys think. submitted by /u/Puzzled-Mail-9092 [link] [comments]

Kedro Meets the Lakehouse: Rebuilding an Real World Evidence Pipeline on Databricks

Tuning with Optuna and MlflowSparkStudy

🚀 My Lakebase App is Working!

CUSTOMER STORY | Scottish Water: Capital Investment Insights via Databricks Genie

The hardest part of Genie rollout isn't building it, it's changing what people reach for at 9am. 🧠

Databricks Production Planning: How to Actually Use the Deployment Guide

A practical read of the 10-phase Databricks deployment guide: what to decide first, and how to use it on a platform you already run. Most Databricks platforms get designed one of two ways. On the fly, project by project, as teams onboard and workspaces appear. Or properly, once, right at the start, and then never looked at again. Neither ages well. One leaves you with a platform nobody chose. The other leaves you with a platform that was right three years ago. submitted by /u/Lenkz [link] [comments]

I merged two databases (Postgres and Elasticsearch) into Lakebase, then threw 200 AI agents at it.

submitted by /u/Limp-Park7849 [link] [comments]

Specialized GPU Kernel Generation

Lakebase Postgres branch stuck "disabled" after auto-archive → unarchive (Public Preview)

Community BrickTalk | One Platform, Any Source: Unifying Enterprise Data with Lakeflow Connect

Hey r/Databricks ! We’re hosting a free, community-sponsored BrickTalk on Thursday, September 17, 2026, focusing on how to simplify and scale data ingestion using Lakeflow Connect! BrickTalks is a community event series where Databricks experts share real-world use cases, live demos, and practical insights, giving you a direct line to the people building the products. Stop struggling with fragmented data across disparate sources. In this session, we'll demonstrate how Lakeflow Connect enables seamless data ingestion from SaaS apps, databases, and cloud storage directly into the Databricks Platform with zero infrastructure management. 🛠️ What We’ll Cover Native Data Ingestion: Learn how Lakeflow Connect provides fully managed ingestion directly into Unity Catalog as governed Delta tables. Simple Integration: See how to easily connect data sources using a simple UI or API. Accelerated AI & Analytics: Discover how unifying your data powers Customer 360, Operations, and downstream AI agent workloads. ⏱️ Global Times PT: 9:00 AM ET: 12:00 PM BST (London): 5:00 PM IST: 9:30 PM 👉 Register here to save your spot! submitted by /u/Subject_Ant1789 [link] [comments]

Community BrickTalk | One Platform, Any Source: Unifying Enterprise Data with Lakeflow Connect

Building Stateful Agents on Lakebase

What's new in Databricks Genie Agents in August 2026 ?

submitted by /u/Youssef_Mrini [link] [comments]

What’s new in Databricks - August 2026

Databricks shipped many major Generally Available features in August 2026. Here is the breakdown of what just landed: 🚀 Unity AI Gateway Enterprise AI governance layer covering model access, Model Context Protocol (MCP) management, and cost observability. 🔒 Role-Based Access Control (RBAC) Switch to scoped, temporary role assumptions instead of dealing with permission bloat. 🔑 Secrets in Unity Catalog Unified security secrets are now governed, 3-level namespace securable objects. ⚙️ Serverless Compute Access Control Granular admin controls over who can trigger serverless workloads across your organization. ⚡ Lakebase Postgres APIs & LTAP Direct Writes Accelerated synced-table loads and improved transactional data integration. 🤖 Genie Agent Upgrades Official GA releases for both the Agent mode API and Full-page Genie Code view. submitted by /u/Youssef_Mrini [link] [comments]

What’s new in Databricks - August 2026

Automatic change data feed is now generally available!

With automatic CDF, Databricks computes row-level changes at read time using row tracking, rather than materializing those changes during every write. Use change data feed on Databricks | Databricks on AWS Why does that mattre? - Better write performance for MERGE INTO and UPDATE workloads - No need to enable CDF individually on every eligible table - Lower storage overhead compared with legacy CDF - The same familiar APIs still work: table_changes() and readChangeFeed - Works with batch processing, Structured Streaming, and Databricks-to-Databricks Delta Sharing For Delta Lake, the main requirements include: • Databricks Runtime 19 LTS+ • A managed table or external table in Delta Lake format with row tracking enabled And if you’re already using legacy CDF, migration is really simple.Once the table meets the requirements, disable legacy CDF: https://preview.redd.it/0dddizd442oh1.png?width=710&format=png&auto=webp&s=fddc536d6aeca423b2e418f4856ae06843393749 submitted by /u/szymon_dybczak [link] [comments]

Tech Companies & Lakeflow Connect

How to extract table-level execution time and resource allocation within a multi-table Job?

Understanding Parquet File Storage for Large Datasets

We reduced a Databricks pipeline from 126 hours to under 30 minutes by removing driver-side orchestration

We inherited a weekly production pipeline that processed 24 months of historical data. The implementation used: Python for-loop ThreadPoolExecutor replaceWhere Spark was barely distributing work across the cluster. After analyzing the Spark UI we found that almost all orchestration was happening on the driver. We redesigned the execution model using: Dynamic partition overwrite Adaptive Query Execution Native Spark distributed execution Runtime dropped from ~126 hours to under 30 minutes while preserving business correctness (validated with the business team). Have you encountered similar driver-side bottlenecks in production? What was the root cause in your case? I documented the full investigation, Spark UI analysis, and redesign here for anyone interested: Full Case submitted by /u/al_coper [link] [comments]

Databricks SA/S.SA

I am looking to connect with people at Databricks and learn new things and also want to evaluate my skillset for some FDE roles. Is there someone who can help me with? submitted by /u/on_rampage [link] [comments]

Acquiring/Processing from a MQ to a Delta

Has anyone tried acquiring data from a MQ at scale using apache spark on databricks cluster? I was trying to solve this problem at work but so far havn't seen an native lib or efficient ways to do this. The legacy system seems to be pulling data using a java based utility and wanted to if there are any imporvements or new patterns of access for spark based workflows. Any documentation or nudge is the right direction will be greatly appreciated. submitted by /u/raja3194 [link] [comments]

Best Place to Buy Instagram Likes?

Bulk Cold Email Service Provider in USA by Real Experience

MetaData Framework for Multi-Level Silver Layer PK/FK Creation

Apache Iceberg Compaction Best Practices

submitted by /u/codingdecently [link] [comments]

Automating Apache Iceberg Table Maintenance

submitted by /u/codingdecently [link] [comments]

What is a Checkpoint in Structured Streaming?

Omnigent Local Coding Model Rec

After watching Matei's webinar and the post on controlling spend , been trying to use the other harnesses and models folks are suggest and trying out Qwen 2.5 coding and 3.6 with Polly in local Omnigent (not connected to a workspace). I have Codex and Claude but ideally thinking best to use paid higher model to plan and then have the local Ollama based Qwen model on my Mac build but so far I haven't seen Polly use it much. What are folks experience, is there a good local model I should use, should I be giving Polly and the sub-agents more direction? (This is on a MacBook M5 btw) submitted by /u/ecp5 [link] [comments]

Just Cleared the Databricks Certified Context Engineer Associate Exam — My Key Takeaways

Lakeflow SDP Append Flow

Snowflake’s AI generated slop blog

submitted by /u/Useful-Swordfish4946 [link] [comments]

Serverless Env v6

Serverless Env v6

Version 6 of the serverless environment is available, which corresponds to runtime 19. more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

Recommendation for Data Reconciliation Frameworks (Legacy vs. Migrated Validation)

I Stopped Sending Every Data Engineering Task to an LLM - A Cost Aware Routing Pattern on Databricks

Databricks-Native AI Agent for Job Incident Detection, RCA & Safe Remediation

Building Custom Agents on Databricks:LangGraph, Atlan-Grounded Routing, and Per-Run Cost with MLflow

Databricks Puts an Autonomous Agent Inside Your Job Scheduler — Genie Code

submitted by /u/macxima [link] [comments]

Evaluating Unity AI Gateway for Enterprise Cost Control

Databricks vs Snowflake comparison

Are there any unbiased comparisons between these two popular platforms? Seen a lot but most of them are biased views, based on experience and commercial motives. submitted by /u/medici2022 [link] [comments]

Databricks vs Snowflake Pyspark Performance

How data engineer do effective testing in Databricks?

I have been writing SQL scripts to ensure data sanity.What are the other ways ? Is pytest useful? Let's say , i populated my bronze table from the source. I want to check if the correct mapping is done. I wrote SQL scripts. What are better ways submitted by /u/Otherwise_Address241 [link] [comments]

What is a Data Skipping in Delta Lake?

Genie - Cost monitoring and usage

Databricks Apps - Deployment process - Need help !!

Genie Agents Toggle between Agent and Chat Modes

Do you guys have this issue where in genie agents, after you ask your first question, the toggle between Agent mode and Chat mode just disappears? And you have to exit that chat to get it back? Before: https://preview.redd.it/cl6wec5nuknh1.png?width=739&format=png&auto=webp&s=d2a0bb2c78b63a962ecdf28aa17dc6b47d806f04 After: https://preview.redd.it/bhsleq3ouknh1.png?width=733&format=png&auto=webp&s=ce877c577b07410124b3bd6f97648dfcb46eca43 Idk if this is because I'm on the free edition/free trial of Databricks. However, I'm confused because in the second video demo in this link, the toggle still exists after sending a question to the AI: https://docs.databricks.com/aws/en/genie-agents/concepts submitted by /u/kcxl [link] [comments]

How to do cross-cloud sharing with OpenSharing with added security (demo)

Hey folks! In this demo, Akram from Databricks' product team shares how you can leverage SecureConnect to better your security posture when doing cross-cloud sharing on OpenSharing! If you have no idea what OpenSharing is, how it applies to you, or how we got from Delta Sharing to OpenSharing, also encourage you to watch this video: https://youtu.be/0mfuNybtmdE Hope you find this helpful! submitted by /u/JosueBogran [link] [comments]

Who is The Best Cold Email Service Provider in 2026?

Does enabling Photon improve performance while lowering compute costs?