Skip to content
All topics
Data EngineeringSee on /pulse →

ETL

Recent items mentioning ETL across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.

47 recent items9 news35 videos3 community threads
What's happening in ETLAI synthesis · updated Oct 17, 2026

Databricks is actively targeting the elimination of dedicated ETL pipelines through HTAP architectures 2 and built-in search in Lakebase Postgres, which runs hybrid search directly on operational data 4. For teams maintaining existing transformation stacks, Databricks outlined a cutover strategy to repoint dbt pipelines to the platform using Lakeflow Jobs for production orchestration 3.

Written October 2026 from the 4 most recent items mentioning ETL at that time. It refreshes when this topic next has enough new material. Click any [N] to jump to the source.

Reddit

What's the most common reason your ETL pipelines fail in production?

I've seen ETL pipelines fail for different reasons, from unexpected schema changes to missing data and transformation errors. I'm curious to know what causes the most issues in real-world projects. Is it usually source system changes, bad data, SQL errors, connection issues, or something else? Also, how do you usually identify the root cause and prevent the same issue from happening again? Would love to hear about your experiences and the problems you deal with most often. submitted by /u/ETL_Pipeline_Dev [link] [comments]

00ETL_Pipeline_Dev1mo ago
Databricks CommunityTechnical Blog

Meet SDP Rewind: An undo button for your ETL pipelines

002mo ago
HackerNews

Postgres data stored in Parquet on S3: LTAP architecture explained

--- top comments --- [dsauerbrun] Maybe I'm too stupid to understand the article... How does this achieve performant querying for olap and oltp purposes? Based on my understanding, olap queries will go to the parquet files which are stored in a columnar fashion and oltp style queries will go to a caching layer that sits on top of those parquet files? What's the special sauce here? Seems like they're just caching the data which, for all intents and purposes, seems like the same solution of storing another copy of the data which is what they say they're avoiding. [ianberdin] Surprisingly, I'm already encountering a second solution that involves storing data chunks on S3 — and this is all within the same week. This is becoming popular. At Playcode, we built what we believe is a revolutionary file system for our Playcode Cloud (https://playcode.io/cloud), which enables the creation of full-stack web software. The FS built completely from scratch using Rust. We thought we were the smartest ones around and that nobody else had figured this out. But it turns out Databricks, Neon, and several others have as well. The idea behind a *Bottomless File System* is really cool, and it works very well for us. Essentially, as described here: - There is a *page server* - A *Linux file system* split into chunks (let's call them chunks instead of pages) - A *cache on NVMe* - And of course, *object storage*, where everything is asynchronously synchronized It works quite well, though it has its downsides. One clear advantage is that NVMe drives have become expensive lately, while object storage remains cheap — so the benefits are undeniable. That said, latency is also a factor. On top of that, uplink costs are rising. To run an object storage-backed file system, you need a very strong uplink with consistent speed — 1 Gbps is simply not enough. Ideally, you want *5 to 10 Gbps*, depending on the load. We spend a lot of time optimizing and experimenting with different hosting providers — specifically bare metal hardware. The main challenges are: - *Slow disks* - *Slow uplink* - And as it turns out, *object storage can be unreliable* — unless you're using S3 But AWS hardware is expensive, so nothing in life is ever that simple. [Avalaxy] Super cool stuff. Being able to combine your analytical platform and transactional database into one storage layer without having to set up ETL pipelines in between is really a game changer. Especially since it's just postgres, instead of some proprietary database. [saisrirampur] But why? I’m skeptical of the idea of unifying storage just because it sounds “elegant” or “cool”. It’s not obvious to me how a single storage engine can compete with purpose-built OLTP and OLAP systems like Postgres and ClickHouse, without significant tradeoffs. You also mention removing CDC pipelines. I’m curious if the materialization (conversion across formats) can catchup to an OLTP workload that is heavy (50K+ tps), which is pretty common these days. Also CDC if done right and with care can be magical for users and stays native to the OLTP/OLAP data-store. Third, data Lakes and open formats are suitable for Data Warehousing / Data analyst use-cases than real-time customer facing apps. Sure, you might work on changing that, which is what you are upto, but you’ll always run into tradeoffs, which will make it hard to unleash the best performance, much needed for the latter category. [andrenotgiant] Here's what I don't understand: Part of the value of doing an ETL pipeline via streaming replication is you get the full history of data in a table. An SCD type 2 table where each row also has a valid_from and valid_to timestamp column. How would someone do the same thing with this architecture?

18257andrenotgiant4mo ago

Get Tuesday's version of this

Tracking ETL? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.

Read past issues first