Skip to content
All topics

Snowflake

Recent items mentioning Snowflake across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.

50 recent items1 release17 news32 community threads
What's happening in SnowflakeAI synthesis · updated Nov 18, 2026

Databricks launched Genie Migrate for Lakehouse, deploying AI to automate legacy warehouse migrations by converting SQL into open ANSI SQL and PySpark 1. Meanwhile, cross-platform interoperability is advancing as Catalog Linked Databases now enable Snowflake to write UC-managed Iceberg tables directly into Unity Catalog 5, alongside practical patterns for querying live Snowflake data from Databricks 6 and executing Snowflake-to-Iceberg migrations 8.

Generated daily from the 10 most recent items mentioning Snowflake. Click any [N] to jump to the source.

Reddit

Lakeflow Declarative Pipelines with external: managed tables, or publish step?

I'm working on an Azure Databricks project where we must expose Delta tables through stable ADLS paths so external engines like Snowflake, Trino, and delta-rs apps can read directly from storage. We'd like to use Lakeflow, but managed streaming tables and materialized views live in Unity Catalog managed storage. Can we safely expose their physical paths like regular external Delta tables, or is path-based access unsupported? The alternative seems to be external Delta tables as sinks, but sinks look limited: Python/streaming only, no expectations or Auto CDC directly, and full refresh doesn't clear prior data. For this requirement, what architecture would you recommend: external sinks, a publish step, or something else? submitted by /u/warleyco96 [link] [comments]

00warleyco963d ago
Reddit

Scale Down NOW

This is how AI destroys society – it uses up every megawatt, every compute cycle. It shuts off your email and grounds flights. The AI Apocalypse has arrived. But you can do something about it. We CAN curtail AI data warehouse usage and it’s clear that we MUST. Step One: Use Airbrx right now. It’s free for most Databricks and Snowflake use cases (we give you a cap, trust me, it’s going to be hard to hit that cap. Airbrx sits between your AI tools and the database, and if you haven’t seen the way AI pummels databases, the last couple days should give you some idea just how bad it is. Go to app.airbrx.ai/signup and then paste your databricks or snowflake URL into the setup tool. Then give your AI the airbrx.ai URL instead of the warehouse and you’re done. Even our default 5 minute caching can save you, and the data center, a ton of transactions, and therefore money. Step Two: Scale down We don’t need a full stack of AI, Servers, Databases and frameworks to deliver a picture of a cat. We need thoughtful architecture that does more with less so that we don’t end up here again, unable to communicate, unable to travel, all because the AI wanted to check that resource a few more times, a resource that is over built and over powered. We do that with routing again, use a smaller warehouse, use a different engine (did you know you can get to your Databricks data without Databricks? Ping me…) This is URGENT Our infrastructure is already breaking… Yes our startup happens to have the tech right now and yes, we do sell it… (and yes I'm a founder) but this tech is the exact opposite of what everyone else is doing, we don’t throw more resources at the problem, we make it so you use less. And then maybe you can email your family about that Thanksgiving trip you’re planning on taking and know they’ll get the message and you can catch your flight. submitted by /u/bissellator [link] [comments]

00bissellator4d ago
HackerNews

Show HN: Igloo – Self-hosted data warehouse built on DuckDB

I made a database (based on DuckDB) because Snowflake and Databricks is insanely expensive. It's open source and in testing it's 3x faster and 25x lower cost.

20AbrahamParangi1w ago
Reddit

Catalog Linked Database - Snowflake writes UC Managed Iceberg tables to Unity Catalog

Is Unity Catalog REST API only accessible via public internet from Snowflake when attempting to set-up a private link connection (catalog integration) to a Unity Catalog (Catalog Linked Database)? I need a private link connection between Databricks and Snowflake. The use case is Snowflake needs to be able to write Managed Iceberg tables to Unity Catalog. submitted by /u/RazzmatazzLiving1323 [link] [comments]

00RazzmatazzLiving13231w ago
Databricks CommunityData Engineering

How to Connect Databricks to Snowflake & Query Live Data?

002w ago
Reddit

ACTUALLY USEFUL Big Data Project Ideas

Heyy everyone! We have a Big Data class project proposal coming up, and I want to make an actually useful project (something challenging, or something that could sell, or something that solves an actual problem people have). Not yet another fancy showpiece class project. Requirements: The dataset must be at least 10GB, be genuinely large-scale and require distributed processing The project must include some or all (or even more): data warehouse (snowflake/redshift/any other) airflow elt/dbt spark, sparkml mlflow kafka/spark structured streaming/flink nosql, vectordb llmops Practitioners, help me with some ideas? (ChatGPT and Claude didn't help much). submitted by /u/mulberry-cream [link] [comments]

00mulberry-cream2w ago
Reddit

Snowflake to Iceberg Lakehouse Migration Guide

submitted by /u/codingdecently [link] [comments]

00codingdecently2w ago
HackerNews

Show HN: Dbprint – Database Context for Agents

CLI that produces an offline snapshot of a database (DDL, column- and relation-level statistics, values with frequencies), combines it with annotations written by you or an agent, and exposes it as MCP and as a web UI. Works with PG, MySQL, Snowflake, ClickHouse, Databricks, BigQuery. How to try it: uvx dbprint context \ arboretum.seedbank.germination_trial \ --purpose query \ --project https://github.com/jakubro/dbprint/blob/main/docs/format/v1/examples/production/.dbprint.yaml Features: - Philosophy: snapshots are meant to sit next to your code and versioned so that you have a history of changes and agents can hit them when searching codebase. - Offline: no need to expose live credentials to agents. - CI: "dbprint check" can verify schema/data drift. - Annotations: complement statistics by documenting the meaning behind data. An agent can prepare them for you ("dbprint annotate"). - Redaction: values can be masked/excluded/hashed either explicitly with FQNs or automatically based on inferred sensitivity and value shape. - Format: Snapshots are YAML and markdown, so greppable and usable by other tools. Why I built this: SQL queries produced by agents were inaccurate, and the overall time to result was slow and costly because of finding the answers in codebase or in live database.

10jakubro2w ago
Reddit

Agentic code converter is GA for SQL

Migration · GA Agentic code converter is GA for SQL Genie Code converts T-SQL, Snowflake, Redshift, Oracle, BigQuery, and Teradata SQL to Databricks SQL. It validates each file and fixes errors before writing to a workspace folder. Work in a migration project, or run /migrate in a Genie Code chat for one file or folder. Afterward, keep chatting to clear action items. The Notebooks target is not part of this GA. https://preview.redd.it/u5m0quj71fyh1.png?width=1808&format=png&auto=webp&s=1a5f16a1d66fa88e9bc171e10efb16c33dadfa2b Looking to migrate workloads but don't have the labor to do so. This is going to continue to expand its SQL dialect but taking an agentic approach to migrations really speeds up migrations and cuts down on time/money from labor submitted by /u/Wise_Ear_4064 [link] [comments]

00Wise_Ear_40642w ago
Reddit

Open source benchmark for Snowflake, Databricks, BigQuery, Redshift, Fabric, ClickHouse and MotherDuck: 244 queries, 1 GB to 1 TB, up to 100 concurrent users, every compute size

submitted by /u/dan_the_lion [link] [comments]

00dan_the_lion3w ago
HackerNews

Show HN: 10x faster, 3x cheaper ETL

For the past few months our team has been working on an agentic framework that takes large, expensive SQL workloads and compiles them to Rust, backed by a formal proof of correctness. This is the sort of work I used to do by hand as a systems performance engineer, but now it happens in hours instead of months. We first redid our internal pipelines with this technique and they're now 10x faster and 3x cheaper. Next, we went live with our Snowflake and Databricks SQL customers. They're seeing similar results with critical production workloads that originally cost 6-7 figures per year. We'll be publishing benchmarks and code samples over the coming weeks; for now, you can read a bit more (and watch our anime launch trailer) on the site. Happy to answer questions about how it all works. --- top comments --- [gtirloni] Why was SQL slow? Where did processing shift to?

11karamazov3w ago
HackerNews

MiniLake Is a Local Databricks API Emulator with SQL, Delta Lake and Spark

--- top comments --- [gavinray] If this actually works, this is such a huge boon. Let me share why: I work on connectors to various datasources, mostly SQL DB's The most useful tests are the integration tests, sending actual queries to the source For most DB's we have a giant Docker Compose which runs a local instance But things like Snowflake, Databricks, Redshift, Athena suck to test because we need an external hosted instance I would kill to be able to run test versions of all of these locally [akbarian] Getting tired of the standard AI design, all websites look the same now [ngruhn] You should call it "pond". [lysecret] Want to hijack this someone made experience recently with https://ducklake.select/ ? We need to setup a warehouse soon and this looks enticing. [viccis] I'm tempted to use this but the rub with Databricks is that there are always a bunch of different restrictions on what features work in what products (for example RT//Lakehouse looks great but runs a more limited regex parser than other warehouses), and so I'm basically forced to rely heavily on integration tests to catch weird behavior. This would need to copy it closely enough to catch those. The other problem is that Databricks renames everything like every 6 months so keeping up with them is non-trivial. Their DLT product alone has gone on a jazz odyssey of renames.

10125kermatt3w ago
Reddit

Title: Data Engineering Roadmap: Azure + Databricks vs AWS + Snowflake + Airflow

Hi everyone! I’ve learned Python, SQL, and PySpark, and I’m looking for guidance on what to learn next to become job-ready for a Data Engineering role. I’m currently considering two learning paths: Option 1: Azure + Databricks Option 2: AWS + Snowflake + Airflow Which path would you recommend for someone starting out in Data Engineering, considering job opportunities, industry demand, and long-term career growth? I’d also appreciate recommendations for: Free learning resources and YouTube tutorials Hands-on, end-to-end Data Engineering projects A practical roadmap to become job-ready If you’re already working as a Data Engineer or have experience with these technologies, I’d love to hear your advice if you know any free learning resources, YouTube tutorials, or end-to-end project tutorials please share the links 🙏 Thanks in advance for your suggestions! submitted by /u/harishvangara [link] [comments]

00harishvangara3w ago
Reddit

Databricks vs. Snowflake: Are we comparing features instead of foundations?

submitted by /u/Zealousideal_Ant5362 [link] [comments]

00Zealousideal_Ant53621mo ago
Reddit

Databricks Article Summary:

Just recently publish my whole thesis on Databricks on this subreddit. Got really great feedback so figured I would write a summary for those too busy for a 20 minute read. The Core Thesis: Databricks is one of the largest and most influential companies mostly lying behind the scenes. Databricks is a platform that allows organizations to store, process and analyze massive amounts of data. Built on their open-source platform Apache Spark, Databricks created software that they released publicly. This software eventually gained extremely widespread adoption and built much of Databricks’ moat. Led by Ali Ghodsi, the 47-year-old Iranian-Swedish-American computer scientist, Databricks has leapfrogged competition in the last decade and is now ahead of its main competitor Snowflake in valuation, revenue, and revenue growth rate. The Numbers to Know: $190 billion valuation. $5.2 billion in 2025 revenue. 71% - 2025 revenue growth. 27x - valuation to revenue multiple using Databricks’ estimated July 2026 ARR. 140% net revenue retention in 2024–2025. 74% - last disclosed gross margin cited, down from 80%. $5 billion raised in the last funding round despite $15–20 billion in investor interest. 80% of Fortune 500 companies using Apache Spark. The Bear Thesis: Databricks trades at 27x annualized revenue which assumes that their fast growth will continue to be sustainable. In addition, while AI is certainly beneficial for Databricks in accelerating revenue, their gross margins are starting to compress, down to a (still healthy) 74% from a previously elite 80%. The company has already acknowledged further compression as compute costs rise. If gross margins compress significantly, even if Databricks’ revenue continues growing at it’s current pace, the company may not be profitable enough to justify its valuation. The Verdict: Databricks was one of the most impressive companies I have researched to date. A few years ago, Databricks was behind its main competitor Snowflake in revenue and valuation. Today, that script has flipped on its head with Databricks being ahead of Snowflake in valuation and revenue. This is largely due to Ali Ghodsi and the incredible team at Databricks and it is partly for this reason that Databricks trades at a slight premium in comparison to Snowflake. Their sticky moat, strong leadership and impressive revenue growth are all strong reasons to invest in Databricks. However, I think due to the premium valuation perhaps the right call with Databricks at this moment is to have patience. I will particularly be watching the revenue growth to see if the company is able to maintain the revenue growth they currently have and watch to see if the gross margins continue to compress. This is certainly a company I want to add to my portfolio, at the right time and right price. submitted by /u/Roadtochessmaster [link] [comments]

00Roadtochessmaster1mo ago
Reddit

Stop migrating data to Databricks: Are you using Lakehouse Federation yet?

I’ve been diving deep into Unity Catalog lately, and the "Lakehouse Federation" feature is genuinely a game-changer for reducing ETL bloat. For a long time, the default answer to "how do I analyze this external data?" was "copy it into the lake." We’re finally moving away from that. Here is the breakdown for anyone trying to cut down on unnecessary data movement: The two ways to federate: Catalog Federation (The "Data-Centric" approach): How it works: Connects Databricks to external metadata (AWS Glue, Hive, Snowflake Iceberg). The Magic: Databricks compute does the heavy lifting (reading raw files in S3/ADLS). It’s fast, low-cost, and creates a "live" bridge to your storage. Best for: Incremental migrations and long-term hybrid storage. Query Federation (The "Database-Centric" approach): How it works: Pushes SQL via JDBC to remote databases (Postgres, MySQL, SQL Server, Synapse, etc.). The Magic: The remote DB does the compute. You get the result set back without touching the raw underlying tables. Best for: Ad-hoc reporting against operational/production databases where you don't want to break their performance with heavy ETL loads. Why it’s actually useful: Centralized Security: Unity Catalog acts as the single security guard. You don't need to manage separate credentials for your users on the remote DB. Automatic Lineage: This is the biggest win for me—Databricks automatically maps the lineage from the remote system to your report. No more "mystery data." Unified Syntax: You query everything using catalog.schema.table regardless of whether it’s sitting in Databricks, Snowflake, or an on-prem SQL Server. submitted by /u/SalamanderOpen859 [link] [comments]

00SalamanderOpen8591mo ago
HackerNews

Launch: Assury Cross platform agent lifecycle mangement

Hey HN, I am David Girvi, co-founder of Assury.ai. We are practitioners who became vendors last year. I wanted to share that we have launched agent lifecycle management for cross-platform agents. One centralized spot for all your agents. AWS, Databricks, Snowflake, Salesforce, etc. Lifecycle does auto-discovery in every environment, plus auto-tagging. We bring everything into a tamper-proof, searchable audit trail. Easily see who or what teams own the agents etc. We also have Kill/resume controls based on tagging so you can granularly shut down certain groups during an incident. We built this also to handle swarms, as one of our design partners has 30k agent swarms that spin up and down for research. Our thesis is that we want everyone to use the agents that will help them with their work. We want to be the observability, audit, governance, and incident response for all those different platforms. We started with one of the first MCP gateways that did execution level governance, then the endpoint with hooks, then our agent deception technology, and now lifecycle management. We are a small team of practitioners building product for our customers, not a marketing team. Cheers.

20girvin1mo ago
Databricks CommunityData Engineering

Connecting to Snowflake via Workload Identity Federation

001mo ago
Reddit

Which Came First For You... Spark or Databricks?

Saw a post this morning claiming you shouldn't touch Databricks until you've already mastered Apache Spark. That advice is backwards. It treats learning like a 90s waterfall project and traps engineers in tutorial hell. The fundamentals are non-negotiable: - You need SQL and Python. - You need to understand distributed shuffles, partitions, and memory spills. - You need to understand Parquet structures, metadata footers, and row groups. - You need to understand Delta Lake transaction logs and ACID guarantees. If you don't know what happens under the hood, you will write brittle pipelines that burn cloud budgets. The false assumption is that you must learn these concepts in an abstract vacuum before touching a modern platform. You don't learn networking by memorizing RFC packet headers on a whiteboard for six months before opening Wireshark. You fire up the tool, capture packets, and inspect the traffic. Databricks, Fabric, or Snowflake is your network analyzer. It is the laboratory. Nobody in enterprise production is setting up bare-metal Spark clusters on a home lab just to learn how shuffles work. And running local[*] on a laptop masks real-world distributed realities: - Local filesystems use POSIX atomic renames. Cloud object stores (S3, ADLS) do not. You cannot truly grasp why Delta's _delta_log exists until you see it write commits against cloud storage. - A local JVM masks network serialization, executor skew, and cross-node shuffle overhead. The caveat: don't treat the platform as a black box. Don't just click "Run All" on serverless compute and assume you know Spark. Use the platform to inspect the engine: - Run .explain(True) and read the physical plan. - Open the Spark UI to watch tasks, stages, and memory spill. - Inspect the JSON commits inside _delta_log after a merge operation. - Compact small files and see how file pruning changes scan times. You don't master the theory first and then graduate to a platform. You master the fundamentals by using the platform as your workbench. Learn the platform. Learn the engine. Learn the storage layer. Do it concurrently, in the environment where production actually lives. submitted by /u/OkImprovement7010 [link] [comments]

00OkImprovement70101mo ago
HackerNews

Show HN: Like LeetCode, but multi-file and multi-step

On LeetCode you write one function, it passes, and you're done. Most AI labs such as Anthropic and OpenAI, as well as some big tech companies such as Meta, Snowflake and Databricks, don't stop in one function and instead ask multi-step questions that are closer to real-world problems. Say you're building a key-value store. Step 1 is get, set, delete, count. Once your tests pass, Step 2 unlocks: expiring keys. Then nested transactions. Then atomic batches. I built this because I couldn't find a good place to practice for multi-step, multi-file questions. The questions are based on ones people have reported from those loops. Check it out at https://loopprep.dev. There are 24 questions, 3 free, and no signup to try them. Happy to answer questions. --- top comments --- [MiroslavPokorny] Writing code in a browser like this or leet code is wrong on so many levels its not funny.

13mrtie2mo ago
Reddit

Need some advice on Snowflake vs Databricks

submitted by /u/Ok_Independent_343 [link] [comments]

00Ok_Independent_3432mo ago
Reddit

Snowflake’s AI generated slop blog

submitted by /u/Useful-Swordfish4946 [link] [comments]

00Useful-Swordfish49462mo ago
Reddit

Databricks vs Snowflake comparison

Are there any unbiased comparisons between these two popular platforms? Seen a lot but most of them are biased views, based on experience and commercial motives. submitted by /u/medici2022 [link] [comments]

00medici20222mo ago
Databricks CommunityData Engineering

Databricks vs Snowflake Pyspark Performance

002mo ago
Reddit

Data Engineers, what does your actual day-to-day work look like? And what should I learn next?

I’m currently trying to transition deeper into Data Engineering and would really appreciate some perspective from people who are already working in the field. I have 1.3 yrs experience as a Junior Python Developer. What I want to do is slowly transform into a Data Engineer. How would you suggest my choice? Basically what I do is make web scraping scripts to get the data from web and give the data in excels. Our company is currently not using git or CI/CD or anything like that. The problem I’m running into is that when I look at Data Engineering jobs on Naukri, LinkedIn, etc., the requirements seem endless. One job asks for Python, SQL, Airflow and AWS; another wants Spark, Kafka and Databricks; another wants Snowflake, dbt, Terraform, Kubernetes, CI/CD, etc. It becomes difficult to understand what I should actually prioritize. So I’d like to hear from people who are actually working as Data Engineers . What does your day-to-day work look like? What kind of problems do you solve, what technologies do you use regularly, and which skills have turned out to be genuinely important in your job? More importantly, based on my current experience, what would you suggest I improve or learn next to become a stronger candidate for Data Engineering roles? Are there any gaps that you think I should focus on, or technologies/concepts that are worth learning through projects rather than just studying theoretically? I’m not really looking for a generic “learn SQL → Python → Spark → AWS” roadmap. I’m more interested in understanding the reality of the job and getting advice from people who have actually gone through the transition. If you’re a Data Engineer with 1–5+ years of experience , I’d especially appreciate your perspective. Even a short description of what you work on and what you wish you had learned earlier would be extremely helpful. Thanks in advance! submitted by /u/Stunning-Space8032 [link] [comments]

00Stunning-Space80322mo ago
Databricks CommunityData Engineering

Lakehouse Federation (Snowflake) — large query results fail to download from internal stage

002mo ago
Databricks CommunityMachine Learninganswered

Snowflake OR Databricks

004mo ago
HackerNews

Show HN: Ingestr CDC – open-source CDC replication in Go

Hi all, this is Burak, one of the founders at Bruin. I created ingestr to make data ingestion easy. ingestr is a CLI tool that can ingest data from 130+ sources. I have shared it on HN after our Go rewrite as well, which made it the fastest ingestion tool in the space. However, ingestr had always been a batch tool. I am personally a big fan of batch workloads due to their simplicity and have built ingestr around that assumption as well; however, over time, the cracks started to show when we started working with larger orgs. Turns out there are some scenarios where CDC proves beneficial: - For legacy systems where it is not possible to introduce cursor columns due to technical, but mostly organizational, concerns, it becomes impractical to deploy batch pipelines. - For systems that do not have a way to reliably know the update timestamp, also due to legacy reasons. Think usecases where the columns are updated without the timestamp being updated. - For hard deletes. Even though I do believe there are ways to solve each of these, it ended up putting us in a disadvantage, and we decided to build the CDC connectors instead. ingestr CDC works in two modes now: batch (bad name, I know) and stream. The batch mode is reading the changelog entries from the last load until the starting timestamp, and the streaming mode keeps reading and landing them to the destination databases. ingestr has a few advantages compared to a more traditional debezium + kafka setup: - it's a standalone Go binary and does not require any additional infra. - it has very low resource consumption, ~100MB baseline. - it can run on your own computer during development, and can be converted into a streaming prod deployment when it is ready. - supports 20+ destinations already, primarily analytical platforms like snowflake, databricks, generic iceberg destinations, etc. i would love to hear any feedback on what we could do to make it easier for cdc workloads! https://github.com/bruin-data/ingestr --- top comments --- [mercutio93] Man I have had a lot of gripes with debezium... the setup alone in a kubernetes cluster with strimzi kafka connect... And dealing with the kafka connect errors and fixing replication slots... ehhh was painful. Does this tool make it any well easier. One problem I expirienced in production systems with staging doubles is that sometimes we do database restores from production onto staging and that makes the sinks and sources go haywire... does ingestr handle this well? Is support for kubernetes/helm anywhere in the pipeline

22karakanb4mo ago
HackerNews

Show HN: Connect DuckDB to any database that has an ADBC driver

Hi everyone, I'm a PhD student in databases at CMU. Over the past few months, I've been interning at Columnar and building a community extension for DuckDB that lets you query Snowflake, Databricks, BigQuery, PostgreSQL, MySQL, and any other system with an ADBC (Arrow Database Connectivity) driver. The extension supports querying ADBC databases directly through a read_adbc table function. It also supports using ATTACH to connect to an ADBC database and then running SELECT, INSERT, COPY, and CTAS statements as if the database were local to DuckDB. You can install it from DuckDB by running: INSTALL adbc FROM community; LOAD adbc; The ADBC extension is open source and available under the Apache-2.0 License. We plan to contribute this extension to the Arrow project so it can become an official ADBC library. Give it a try, and leave feedback if you have ideas about how to make it better. Leave an issue on the GitHub repo if you hit any rough edges! --- top comments --- [eveerett] Super cool project! Curious your thoughts on LakeSail. It can run single node on your laptop similar to DuckDB all the way to a cluster like Spark (it adopts the Spark interface, but built entirely in Rust without the JVM). Might be an interesting look for ya. [StarlaAtNight] Haven’t tried it yet, but awesome work! This seems huge. I try to use DuckDB whenever feasible, and I also tend to work with Snowflake, Databricks, and BigQuery (as well as Fivetran which uses DuckDB for local dev/debug mode). Excited to have ADBC-enabled connections w/ DuckDB, and have a feeling this is going to start coming up as a use case a lot more

72sarch4mo ago
HackerNews

Show HN: Dex – Cost-aware analytics engineering skills for agents

Hi I’m Marco, co-founder of Exmergo. Me and my team created Dex to help Analytics Engineers do real work with Claude Code (and any other coding agent). We’ve found that Data and Analytics teams are stuck between a rock and a hard place cost-wise: - On one side, they are using some of the most expensive consumption-billed software on the planet (Snowflake, Databricks etc.). - On the other side, Anthropic and Open AI want you to tokenmax (and with data analytics it’s very easy to burn your context window). So we created Dex, our open source skills plugin (Apache-2.0), to solve both of these problems: - Dex forces the agent to use a cost guard when performing exploration queries and transformations. - Dex uses a set of tooling that makes it hard to get burned when transforming data. It achieves these things with tight control scripts that the SKILL.md files are pointed towards when using /dex:explore, /dex:transform and /dex:maintain. Bonus: Dex makes agents really good data exploration, building sql and dbt models and detecting drift. 76% performance on ade-bench with Claude Sonnet 5 (and, per our measures, 2.5x cheaper than Fable 5). Install on any agent with this command in your terminal: npx skills add exmergo/dex Install on Claude Code running these commands (separately): /plugin marketplace add exmergo/exmergo-agent-plugins /plugin install dex@exmergo If you want to see more visual examples you can go through the README or browse here: https://www.exmergo.com/dex Let me know if this helps your analytics workflow and makes your agents more cost-aware.

50marcociavarella4mo ago
Databricks CommunityData Engineering

Ingest data from snowflake to databricks

005mo ago
RedditDiscussion

Snowflake to Databricks Migration in 12 weeks and cut cost per run by ~77%. AMA.

Lovelytics wrapped up a Snowflake-to-Databricks migration; 847 DBT models, 35 Info Mart tables, \~77% lower cost per run on a 2XL warehouse. **TL;DR What helped:** * Treated the migration as engineering, not translation. Each dbt model was tested in isolation, not just row counts vs Snowflake. * Routing macro to resolve cross-layer references at runtime, so the same codebase could read from Snowflake, federated Snowflake, and Unity Catalog without forking logic. * Dual model trees in one repo, which let the migration stay in lockstep with live Snowflake changes. * Script-generated wave selectors enabled parallel builds while preserving dependency order. * Used reference-slice validation subsets vs. waiting on full mart refreshes. **TL;DR Cost reduction:** * Reworked joins to use narrow staging dimensions instead of wide marts where possible. * Added incremental predicates to reduce MERGE target scans. * Split wide models into parallel sub-models where the dependency graph allowed it. * Copied static reference data into Delta instead of repeatedly reading it through federation. * Loaded static copies into Delta rather than reading via federation (predicate pushdown is poor). Happy to go into the gotchas: HASH() not being portable, Snowflake MERGE tolerating duplicate keys that Delta doesn't, NULL ordering, and timestamp handling. AMA [Full Blog Post](https://community.databricks.com/t5/technical-blog/partner-blog-847-models-12-weeks-77-less-inside-r1-s-snowflake/ba-p/157284)

5713m_goo5mo ago

Get Tuesday's version of this

Tracking Snowflake? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.

Read past issues first