Skip to content
All topics

Cost Optimization

Recent items mentioning Cost Optimization across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.

23 recent items4 news13 videos6 community threads
What's happening in Cost OptimizationAI synthesis · updated Nov 15, 2026

Spark declarative pipelines (SDP) improve pipeline cost efficiency by replacing manual ETL scripting with automated orchestration, incremental updates, and engine-level execution 3. Meanwhile, data practitioners looking to control spend are actively seeking tooling beyond native Databricks system tables 1, pointing toward dedicated multi-provider cloud FinOps solutions 2.

Generated daily from the 3 most recent items mentioning Cost Optimization. Click any [N] to jump to the source.

Reddit

What do you use for Databricks cost optimization beyond system tables?

We’re spending roughly $900 a day in DBUs and a bit over $2k a day on the AWS side for the same workspaces. For a long time, I only had a dashboard for the first number, it’s the usual one on system.billing.usage joined to list_prices and it looked fine until finance put it next to the AWS invoice. Tags get me part of the way because cluster tags land on the EC2 instances and cost explorer can group by them. Most of our scheduled jobs moved onto instance pools last year to cut startup time though, and on pool clusters our custom tags only show up on the DBU side. Idle pool instances don’t appear in system tables at all because they don’t burn DBUs while they sit there but AWS still bills every hour of them. On top of that, about 40 scheduled jobs run on one shared all-purpose cluster and that’s a single line on the EC2 side that I have no fair way to split back to teams. Which tool ended up tying the Databricks and AWS sides together for you, ofcourse past the built-in system tables? submitted by /u/Impressive-Craft1926 [link] [comments]

00Impressive-Craft19264d ago
Reddit

Cloud FinOps Directory with 40 Providers

Hi everyone! I've recently created a Cloud FinOps Directory featuring 40+ providers, covering FinOps platforms, cloud cost optimization tools, Kubernetes cost management, and more. I’m trying to make the directory the most helpful and useful for the others. I’m looking to expand the providers list, so I'd really appreciate your help. Are there any providers or tools you think should be included? If you are a provider and you’d be interested to be in the list DM me. (It’s free) You can check it out here: https://finops.monetizacreator.com/ submitted by /u/anazt_yt1 [link] [comments]

00anazt_yt11w ago
HackerNews

Databricks cost optimization: monitoring all-purpose clusters with the API

20protmaks1mo ago
Reddit

Databricks Cost Optimisation 3rd Party Tools

Does anyone have experience with third party optimisation tools like espresso.ai or Zipher? Espresso has an enticing pricing model but I wonder if dumping a whole heap of logs and system stats into Claude would get me 80% of the way there. submitted by /u/Agitated-Western1788 [link] [comments]

00Agitated-Western17881mo ago
Databricks CommunityGenerative AI

The Open-Weight Revolution: A Game Changer for Our LLM Cost Optimization Odyssey

003mo ago
RedditDiscussion

Snowflake to Databricks Migration in 12 weeks and cut cost per run by ~77%. AMA.

Lovelytics wrapped up a Snowflake-to-Databricks migration; 847 DBT models, 35 Info Mart tables, \~77% lower cost per run on a 2XL warehouse. **TL;DR What helped:** * Treated the migration as engineering, not translation. Each dbt model was tested in isolation, not just row counts vs Snowflake. * Routing macro to resolve cross-layer references at runtime, so the same codebase could read from Snowflake, federated Snowflake, and Unity Catalog without forking logic. * Dual model trees in one repo, which let the migration stay in lockstep with live Snowflake changes. * Script-generated wave selectors enabled parallel builds while preserving dependency order. * Used reference-slice validation subsets vs. waiting on full mart refreshes. **TL;DR Cost reduction:** * Reworked joins to use narrow staging dimensions instead of wide marts where possible. * Added incremental predicates to reduce MERGE target scans. * Split wide models into parallel sub-models where the dependency graph allowed it. * Copied static reference data into Delta instead of repeatedly reading it through federation. * Loaded static copies into Delta rather than reading via federation (predicate pushdown is poor). Happy to go into the gotchas: HASH() not being portable, Snowflake MERGE tolerating duplicate keys that Delta doesn't, NULL ordering, and timestamp handling. AMA [Full Blog Post](https://community.databricks.com/t5/technical-blog/partner-blog-847-models-12-weeks-77-less-inside-r1-s-snowflake/ba-p/157284)

5713m_goo5mo ago

Get Tuesday's version of this

Tracking Cost Optimization? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.

Read past issues first