Data Engineering is Dead (Or How We Can Use AI to Avoid It...)
Description
Last week Simon gave a talk at SQLBits, a Data Conference in the UK, outlining some of the recent product announcements and how we as Data Engineers can get ahead of the game as AI swamps the industry. Now, whilst the people in the room enjoyed the talk, it sparked a fair bit of outrage in the LinkedIn influencer camps - some of that's fair, some of it just needs context, so in this video Simon lays out the core messages of his talk for all to see. If you still disagree, that's fine, but lets all have a bit of context! Maybe this vision of a future, full of vibe-coded spaghetti pipelines won't come to pass. Maybe we'll achieve a techno-utopia where we all get paid to fiddle with pipeline code to our hearts content. We'd certainly rather be prepared for the worst, then be surprised by the best.
Description from YouTube. Full content on the video page.
More from Advancing Analytics
NewsEnhancing your Skills with Databricks Genie Code
Databricks Genie Code is an agentic coding system that allows users to build custom "skills" using markdown files, enabling it to generate code and perform tasks according to specific in-house standards and conventions. These skills provide context-on-demand, ensuring repeatable and consistent output for various engineering tasks like schema documentation or metric view creation.
NewsDatabricks Genie, Unity AI Gateway, Project Glasswing, and Model Mania | AI Newsround - April 2026
Databricks Genie is now the business user home screen for Databricks, offering a unified chat interface, external knowledge store connections, and a mobile app. The Unity AI Gateway, integrated with Unity Catalog, provides comprehensive governance for agentic AI, including permissions, auditing, and policy controls for models and tools.
ReleasesDatabricks Genie Code, Carl, Bull**** Bench & more! | AI Newsround - March '26 | Advancing Analytics
The video discusses Databricks' new AI tools, Genie Code for autonomous data work and Carl for faster, cost-efficient enterprise knowledge agents using custom reinforcement learning. It also covers the Bench V2 for evaluating AI models' ability to detect and push back on nonsense, along with updates to various models like Qwen 3.5, Gemini 3.1 Flashlight, and OpenAI's GPT-5.3 Instant, 5.4, Mini, and Nano, highlighting their focus on agent capabilities and cost-efficiency.
ReleasesIntroducing Pantheon - Agentic Engineering At Scale
Pantheon is a Databricks application that uses a multi-agent system to generate Lake Flow pipelines for data engineering, allowing users to define data ingestion and transformation rules through a conversational interface. It automates the design, validation, and code generation for lakehouse pipelines, enabling citizen engineers to build robust data solutions without deep PySpark knowledge.
NewsDatabricks Lakebase - Instant OLTP for Apps & Agents
Databricks Lakebase provides an OLTP-style database within the Databricks Lakehouse ecosystem, enabling rapid, scalable transactional processing for applications and AI agents. It allows users to quickly provision autoscaling databases that can spin up and down in milliseconds, offering a cost-effective solution for operational data storage.
NewsOpenClaw, Databricks Agentic Data Monitoring & more! | AI Newsround - February 2026 | Advancing AI
The video discusses OpenClaw, an open-source framework for AI agents, and Databricks' new agentic data quality monitoring solution. It also introduces Advancing Analytics' Lake Forge and Pantheon, a framework and AI layer for developing scalable Lake Flow pipelines, and highlights new model releases from Anthropic, Google, and OpenAI.