Description
Spark Performance Tuning Master the art of Spark Performance Tuning and Data Engineering in this comprehensive Apache Spark tutorial! Data skew is a common issue in big data processing, leading to performance bottlenecks by overloading some nodes while underutilizing others. This video dives deep into a practical example of data skew and demonstrates how to optimize Spark performance by using a technique called 'Salting'. Salting involves adding some randomness to the values before computing the hash for partitioning, thus distributing the data more evenly across partitions and reducing skew. With clear step-by-step explanations, you'll learn how to apply salting in practice, understand the concept behind it, and ultimately improve your data engineering skills. 📄 Complete Code on GitHub: https://github.com/afaqueahmad7117/spark-experiments/blob/main/spark/1_data_skew/4_salting.ipynb 🎥 Full Spark Performance Tuning Playlist: https://www.youtube.com/playlist?list=PLWAuYt0wgRcLCtWzUxNg4BjnYlCZNEVth 🔗 LinkedIn: https://www.linkedin.com/in/afaque-ahmad-5a5847129/ Chapters: 00:00 Salting Concept 07:06 Applying Salting In Joins 12:53 Code Examples For Salting In Joins 16:56 Appl…
Description from YouTube. Full content on the video page.
More from Afaque Ahmad
CommunityHow I Mastered System Design Interviews
This video teaches a six-step framework for mastering data engineering system design interviews, covering requirements gathering, pipeline design, data modeling, storage and file formats, data quality and observability, and pipeline resilience. It demonstrates how to apply this framework with practical examples and back-of-the-envelope calculations to justify design choices.
TutorialsDatabricks End-To-End Project | Zero-To-Expert | Streaming, AI, Lakeflow, Unity Catalog, AI/BI
This video demonstrates building an end-to-end restaurant analytics platform on Databricks, covering streaming and batch data ingestion, AI-powered sentiment analysis, and dashboard creation. It teaches how to use Unity Catalog, Lake Flow Connect for CDC, Spark declarative pipelines for real-time data from Event Hub, and how to construct a medallion architecture with fact and dimension tables.
CommunityHow Much DSA Do You Need To Crack Data Engineering Interviews?
Data engineers need to understand DSA concepts at an easy to medium level, focusing on practical applications like Big O intuition, arrays, hashmaps, and basic trees/graphs, rather than advanced algorithms. The video provides a practical DSA roadmap, differentiating between "must-knows," "good-to-knows" for stronger product/infra roles, and "overkill" topics for most classic data engineering interviews.
CommunityWill AI REPLACE Data Engineers?
AI will not replace data engineers, but it will shift their role from typing code to designing solutions, guiding AI tools, and verifying outputs. Data engineers should focus on core coding fundamentals, system and product thinking, and effectively using AI and other tools.
CommunityApache Spark Was Hard Until I Learned These 30 Concepts!
The video explains 30 key Apache Spark concepts, starting with a comparison to MapReduce to highlight Spark's in-memory processing and DAG-based execution model. It then details Spark's cluster architecture, job execution flow (driver, executors, tasks), and memory management within executor containers.
TutorialsDelta Lake Masterclass | Azure Databricks | PySpark | From Zero-To-Expert
This video provides a comprehensive masterclass on Delta Lake using Azure Databricks and PySpark, covering its core concepts, internal workings, and practical applications. It demonstrates how Delta Lake solves data lake problems like lack of ACID support, DML operations, and schema enforcement, and teaches features like time travel, concurrency control, and optimization techniques.