Ultra-Fast Anomaly Detection using Apache Spark Real-Time Mode
Summary
Databricks practitioners can now implement a reusable pattern for ultra-fast, real-time fraud and anomaly detection using Apache Spark Real-Time Mode. This operational workload pattern enables data engineers to process and detect anomalies at extremely low latencies for critical business use cases.
Summary generated by brickster.ai. For the full article, follow the source link above.
More from Databricks Blog
Sizing Lakebase Search: lessons from a real app and 4 million passages
Sizing the Local File Cache to match the vector index keeps up to 98% of pages in memory and drops warm vector latency to single-digit milliseconds directly inside Lakebase Postgres. Reducing the search candidate pool to 200 cuts page reads by two-thirds while preserving 90.5% of top-10 results, with an always-on 0.5 CU floor providing an optimal production baseline for just $31 to $40 per month.
Unlock powerful insights and forecasts with Predictive AI Functions
Databricks has introduced five Predictive AI Functions, featuring an upgraded ai_forecast alongside the new ai_top_drivers, ai_detect_anomalies, ai_predict_value, and ai_predict_class. These tools allow SQL analysts and business users to generate predictions, detect anomalies, and uncover metric drivers directly within Genie and existing workflows without building or maintaining models.
How FactSet cut model training and deployment from days to hours with Databricks AI Runtime
FactSet cut the time required to set up, train, and deploy its Text-to-Formula models from days to hours by adopting Databricks AI Runtime. By leveraging serverless GPU compute directly alongside existing Lakehouse workflows, the team can now continuously retrain fine-tuned open models every two to three weeks to convert natural-language queries into FactSet Query Language formulas.
Power real-time applications and agents with Document Intelligence
Document Intelligence real-time REST APIs enable applications to parse, classify, and extract information from documents as they arrive rather than relying on delayed batch pipelines. You can embed these APIs directly into Databricks Apps or custom agents to power interactive workflows and drive real-time business decisions.
Introducing Private Network Gateway
Databricks has introduced Private Network Gateway (PNG), a managed service that connects serverless workloads to your private networks and resources through a single network-level configuration. Utilizing your existing routes, DNS, and firewall rules, PNG is now available in Public Preview on Azure and Private Preview on AWS.
Reducing Qwen training time by 6x on AI Runtime
Qwen fine-tuning can be accelerated by 6x on AI Runtime by installing the correct kernels and improving the data pipeline, even when baseline GPU utilization already reaches 98%. AI Runtime enables you to diagnose these performance issues using a standard torch trace of an LLM.