Description
Processing large amounts of data for analytical or business cases is a daily occurrence for Apache Spark users. Cost, Latency and Accuracy are 3 sides of a triangle a product owner has to trade off. When dealing with TBs of data a day and PBs of data overall, even small efficiencies have a major impact on the bottom line. This talk is going to talk about practical application of the following 4 data-structures that will help design an efficient large scale data pipeline while keeping costs at check. 1. Bloom Filters 2. Hyper Log Log 3. Count-Min Sketches 4. T-digests (Bonus) We will take the fictional example of an eCommerce company Rainforest Inc and try to answer the business questions with our PDT and Apache Spark and and not do any SQL for this. 1. Has User John seen an Ad for this product yet? 2. How many unique users bought Items A , B and C 3. Who are the top Sellers today? 4. Whats the 90th percentile of the cart Prices? (Bonus) We will dive into how each of these data structures are calculated for Rainforest Inc and see what operations and libraries will help us achieve our results. The session will simulate a TB of data in a notebook (streaming) and will have code sam…
Description from YouTube. Full content on the video page.
More from Databricks
NewsHow adidas Uses Databricks to Build Better Products
Adidas uses Databricks' lakehouse platform to centralize all its data—from product to football-related insights—enabling faster analytics across the organization. The company's Genie analytics tool helps analysts spend less time processing data and more time on strategic questions, ultimately supporting better product development.
NewsDatabricks for Good x MapAid: Creating a Searchable Database for Groundwater Discovery
MapAid is using Databricks to build an AI groundwater mapping system called Well Mapper that processes water documents to identify accurate well-drilling locations in Ethiopia, improving on the current 30% success rate. The system analyzes 400 well logs in half a second instead of four weeks, with potential to double Ethiopia's food supply through improved irrigation.
NewsHow Databricks Genie Automates Data Workflows with Genie Ontology and Scheduled Tasks
Databricks Genie enables ontology by default for business context and adds document/PDF uploads, direct Unity Catalog queries, and team collaboration features in chat. Scheduled tasks automate recurring workflows with embedded visualizations and PDF outputs accessible across web, desktop, and mobile platforms.
NewsGenie One Beginner's Guide: Explore Data & Automate Tasks
Genie One helps subject matter experts avoid repetitive questions by using business ontology to understand your data and run real-time queries that generate instant answers and shareable reports. The tool can also automate monitoring with conditional alerts that only notify you when specific business thresholds are met.
TutorialsHow to Schedule Automated Meeting Prep in Genie One
Genie One allows users to connect multiple data sources like Databricks tables and Google Calendar, then create analyses through natural language prompts to prepare for customer meetings. The platform can schedule these analyses to run automatically on a recurring basis and deliver results via email or mobile app.
TutorialsHow to Build Custom Skills in Genie One in Minutes
Genie 1 can populate business report templates by analyzing their format and running SQL queries against organizational data. Users can save these report workflows as reusable skills that execute with a single slash command, enabling automation of recurring reports.