Snowflake

Snowflake OR Databricks

Show HN: Ingestr CDC – open-source CDC replication in Go

Hi all, this is Burak, one of the founders at Bruin. I created ingestr to make data ingestion easy. ingestr is a CLI tool that can ingest data from 130+ sources. I have shared it on HN after our Go rewrite as well, which made it the fastest ingestion tool in the space. However, ingestr had always been a batch tool. I am personally a big fan of batch workloads due to their simplicity and have built ingestr around that assumption as well; however, over time, the cracks started to show when we started working with larger orgs. Turns out there are some scenarios where CDC proves beneficial: - For legacy systems where it is not possible to introduce cursor columns due to technical, but mostly organizational, concerns, it becomes impractical to deploy batch pipelines. - For systems that do not have a way to reliably know the update timestamp, also due to legacy reasons. Think usecases where the columns are updated without the timestamp being updated. - For hard deletes. Even though I do believe there are ways to solve each of these, it ended up putting us in a disadvantage, and we decided to build the CDC connectors instead. ingestr CDC works in two modes now: batch (bad name, I know) and stream. The batch mode is reading the changelog entries from the last load until the starting timestamp, and the streaming mode keeps reading and landing them to the destination databases. ingestr has a few advantages compared to a more traditional debezium + kafka setup: - it's a standalone Go binary and does not require any additional infra. - it has very low resource consumption, ~100MB baseline. - it can run on your own computer during development, and can be converted into a streaming prod deployment when it is ready. - supports 20+ destinations already, primarily analytical platforms like snowflake, databricks, generic iceberg destinations, etc. i would love to hear any feedback on what we could do to make it easier for cdc workloads! https://github.com/bruin-data/ingestr --- top comments --- [mercutio93] Man I have had a lot of gripes with debezium... the setup alone in a kubernetes cluster with strimzi kafka connect... And dealing with the kafka connect errors and fixing replication slots... ehhh was painful. Does this tool make it any well easier. One problem I expirienced in production systems with staging doubles is that sometimes we do database restores from production onto staging and that makes the sinks and sources go haywire... does ingestr handle this well? Is support for kubernetes/helm anywhere in the pipeline

Show HN: Connect DuckDB to any database that has an ADBC driver

Hi everyone, I'm a PhD student in databases at CMU. Over the past few months, I've been interning at Columnar and building a community extension for DuckDB that lets you query Snowflake, Databricks, BigQuery, PostgreSQL, MySQL, and any other system with an ADBC (Arrow Database Connectivity) driver. The extension supports querying ADBC databases directly through a read_adbc table function. It also supports using ATTACH to connect to an ADBC database and then running SELECT, INSERT, COPY, and CTAS statements as if the database were local to DuckDB. You can install it from DuckDB by running: INSTALL adbc FROM community; LOAD adbc; The ADBC extension is open source and available under the Apache-2.0 License. We plan to contribute this extension to the Arrow project so it can become an official ADBC library. Give it a try, and leave feedback if you have ideas about how to make it better. Leave an issue on the GitHub repo if you hit any rough edges! --- top comments --- [eveerett] Super cool project! Curious your thoughts on LakeSail. It can run single node on your laptop similar to DuckDB all the way to a cluster like Spark (it adopts the Spark interface, but built entirely in Rust without the JVM). Might be an interesting look for ya. [StarlaAtNight] Haven’t tried it yet, but awesome work! This seems huge. I try to use DuckDB whenever feasible, and I also tend to work with Snowflake, Databricks, and BigQuery (as well as Fivetran which uses DuckDB for local dev/debug mode). Excited to have ADBC-enabled connections w/ DuckDB, and have a feeling this is going to start coming up as a use case a lot more

Show HN: Dex – Cost-aware analytics engineering skills for agents

Hi I’m Marco, co-founder of Exmergo. Me and my team created Dex to help Analytics Engineers do real work with Claude Code (and any other coding agent). We’ve found that Data and Analytics teams are stuck between a rock and a hard place cost-wise: - On one side, they are using some of the most expensive consumption-billed software on the planet (Snowflake, Databricks etc.). - On the other side, Anthropic and Open AI want you to tokenmax (and with data analytics it’s very easy to burn your context window). So we created Dex, our open source skills plugin (Apache-2.0), to solve both of these problems: - Dex forces the agent to use a cost guard when performing exploration queries and transformations. - Dex uses a set of tooling that makes it hard to get burned when transforming data. It achieves these things with tight control scripts that the SKILL.md files are pointed towards when using /dex:explore, /dex:transform and /dex:maintain. Bonus: Dex makes agents really good data exploration, building sql and dbt models and detecting drift. 76% performance on ade-bench with Claude Sonnet 5 (and, per our measures, 2.5x cheaper than Fable 5). Install on any agent with this command in your terminal: npx skills add exmergo/dex Install on Claude Code running these commands (separately): /plugin marketplace add exmergo/exmergo-agent-plugins /plugin install dex@exmergo If you want to see more visual examples you can go through the README or browse here: https://www.exmergo.com/dex Let me know if this helps your analytics workflow and makes your agents more cost-aware.