How WMCHealth cut a 72-minute dbt pipeline down to 5 minutes, with one environment variable
Summary
WMCHealth cut a dbt pipeline from a 72-minute refresh to five minutes by switching one model between two execution modes with a single `ENV_NRT` environment variable: active records only as a table every five minutes, or the full dataset as an incremental overnight. The pattern runs from one codebase and cuts median runtime 93.6% (4,340 to 278 seconds) with over 99% less data processed per near real-time run, and it relies on inner-join propagation to carry the narrower scope through the DAG for free.
Summary generated by brickster.ai. For the full article, follow the source link above.
More from dbt Labs Blog
Healthcare AI starts with data you can trust
Trustworthy healthcare AI depends on defining metrics once, giving agents context, and keeping data pipelines fresh. Learn how healthcare teams put these foundations to work to build AI systems their care teams can trust.
Making the economic case for warehouse-first AI context
A warehouse-first approach to AI context dramatically reduces token costs, establishing a clear economic case for AI agents. This strategy tackles immediate monthly token bills while long-term agent investments take years to pay off.
A letter from our founders
The age of intelligence will be driven by agents, open data infrastructure, and the humans building the agentic future. Databricks founders share their perspective on these core themes and their vision for navigating this next era.
How organizations progress from trusted data to trusted AI and agents
Autonomy is an outcome of AI maturity that requires establishing trusted data, governance, and context. Learn how organizations can progress from foundational data trust to deploying AI and agents that act with control.
How to assess your organization’s AI readiness
Assess six dimensions of your data foundation to pinpoint where it limits AI, since adoption is moving faster than readiness. The post lays out a framework for judging your organization's AI readiness so you know which gaps to close first.
How to put generative AI directly into a dbt pipeline with plain SQL
Google Cloud shows how to call BigQuery's AI SQL functions from inside dbt models, so generative AI steps like sentiment scoring run as ordinary SQL transformations in the pipeline. It extends the pattern to LLM-generated product briefings, all written in plain SQL without separate orchestration code.