Building the Petcare Data Platform using Delta Lake and 'Kyte': Our Spark ETL Pipeline
Description
At Mars Petcare (in a division known as Kinship Data & Analytics) we are building out the Petcare Data Platform – a cloud based Data Lake solution. Leveraging Microsoft Azure, we were faced with important decisions around tools and design. We chose Delta Lake as a storage layer to build out our platform and bring insight to the science community across Mars Petcare. Migrating away from Azure Data Factory completely, we leveraged Spark and Databricks to build ‘Kyte’, a bespoke pipeline tool which has massively accelerated our ability to ingest, cleanse and process new data sources from across our large and complicated organisation. Building on this we have started to use Delta Lake for our ETL configurations and have built a bespoke UI for monitoring and scheduling our Spark pipelines. Find out more about why we chose a Spark-heavy ETL design and a Delta Lake driven platform, the advantages (and difficulties) of migrating away from Azure Data Factory, and why we are committing to Spark and Delta Lake as the core of our Platform to support our mission: Making a Better World for Pets! Key Takeaways: -Leveraging Delta Lake as Engineers for exposing data to Data Scientists -Advantages …
Description from YouTube. Full content on the video page.
More from Databricks
NewsDatabricks for Good x MapAid: Creating a Searchable Database for Groundwater Discovery
MapAid is using Databricks to build an AI groundwater mapping system called Well Mapper that processes water documents to identify accurate well-drilling locations in Ethiopia, improving on the current 30% success rate. The system analyzes 400 well logs in half a second instead of four weeks, with potential to double Ethiopia's food supply through improved irrigation.
NewsHow Databricks Genie Automates Data Workflows with Genie Ontology and Scheduled Tasks
Databricks Genie enables ontology by default for business context and adds document/PDF uploads, direct Unity Catalog queries, and team collaboration features in chat. Scheduled tasks automate recurring workflows with embedded visualizations and PDF outputs accessible across web, desktop, and mobile platforms.
NewsGenie One Beginner's Guide: Explore Data & Automate Tasks
Genie One helps subject matter experts avoid repetitive questions by using business ontology to understand your data and run real-time queries that generate instant answers and shareable reports. The tool can also automate monitoring with conditional alerts that only notify you when specific business thresholds are met.
TutorialsHow to Schedule Automated Meeting Prep in Genie One
Genie One allows users to connect multiple data sources like Databricks tables and Google Calendar, then create analyses through natural language prompts to prepare for customer meetings. The platform can schedule these analyses to run automatically on a recurring basis and deliver results via email or mobile app.
TutorialsHow to Build Custom Skills in Genie One in Minutes
Genie 1 can populate business report templates by analyzing their format and running SQL queries against organizational data. Users can save these report workflows as reusable skills that execute with a single slash command, enabling automation of recurring reports.
TutorialsHow to Create & Share Polished Documents in Genie One
Genie One documents enable creation of polished reports with AI-generated narratives and interactive charts sourced from Databricks data. The feature supports real-time collaborative editing between users and Genie, sharing with team members for comments, and scheduling automated report generation.