Metadata-Driven Streaming Ingestion Using DLT, Azure Event Hubs and a Schema Registry
Description
At Plexure, we ingest hundreds of millions of customer activities and transactions into our data platform every day, fuelling our personalisation engine and providing insights into the effectiveness of marketing campaigns. We're on a journey to transition from infrequent batch ingestion to near real-time streaming using Azure Event Hubs and DLT. This transformation will allow us to react to customer behaviour as it happens, rather than hours or even days later. It also enables us to move faster in other ways. By leveraging a Schema Registry, we've created a metadata-driven framework that allows data producers to: Evolve schemas with confidence, ensuring downstream processes continue running smoothly. Seamlessly publish new datasets into the data platform without requiring Data Engineering assistance. Join us to learn more about our journey and see how we're implementing this with DLT meta-programming - including a live demo of the end-to-end process! Talk By: Vicky Avison, Principal Data Engineer, Plexure Here’s more to explore: Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering The Big Book of Data Engineering: https:…
Description from YouTube. Full content on the video page.
More from Databricks
NewsWhat Is a Meta-Harness for AI Agents? Omnigent explained
A harness is the software wrapper around an LLM that turns it into an agent, handling connections, security, and tools, but each harness is proprietary. Databricks' Omnigent is an open-source meta-harness layer that allows multiple agents and models to communicate and be composed into a single workflow.
ReleasesDatabricks launches across the Data + AI stack in 90 seconds
Databricks announced LTAP to unify lakebased and lakehouse data, eliminating ETL and enabling a single copy of data for analytical and operational needs. They also introduced Unity AI Gateway for governance, Genie Ontology for enterprise knowledge graphs, and open-sourced Omniant for managing multiple coding agents.
ReleasesIntroducing Omnigent: The Ultimate Meta-Harness for AI Agents
Omnigent is a new open-source meta-harness for AI agents that provides a unified interface for composition, control, and collaboration across multiple models and agent workflows. It enables stateful, data-centric policies for guardrails and allows real-time sharing and steering of live agent sessions with teammates.
NewsHow DEFRA and Natural England Accelerate Peatland Restoration with AI and Databricks
DEFRA and Natural England utilize AI and Databricks to accelerate peatland restoration by automating the mapping of peatland features and peat dams across England. This technology significantly reduces the time required for mapping, enabling faster identification and restoration of these crucial carbon-storing habitats.
NewsAI Stack Explained in 3 Layers (LLM, Agent Harness, Omnigent)
The AI stack now includes a third layer, the meta harness, which sits above individual agent harnesses. This meta harness, exemplified by Databricks' open-sourced Omnigent, allows for routing queries to appropriate agents and orchestrating tasks across multiple agents, enabling seamless interaction and context sharing between them.