Reducing the noise: How Atlassian built an OCSF security lakehouse on Databricks
Summary
Atlassian normalized 44 billion daily events into an OCSF security lakehouse on Databricks, cutting ingestion costs by 80% and accelerating queries by up to 95%. The unified architecture now powers over 500 production threat-detection signals while driving a 60% increase in security analyst productivity.
Summary generated by brickster.ai. For the full article, follow the source link above.
More from Databricks Blog
How our AI Research team uses AI Runtime
Databricks AI Research uses AI Runtime to train and fine-tune models across hundreds of GPUs, supporting an 8x surge in training jobs fueled by agent-driven research. Built-in GPU reliability, fast job launches, and native integrations with MLflow and Unity Catalog accelerate these automated experimentation workflows at scale.
Geospatial queries are up to 38X faster with Lakehouse//RT
Lakehouse//RT now supports spatial SQL and spatial joins directly on governed lakehouse data, delivering up to 38X faster query performance than traditional data warehouses under 50 concurrent users. This eliminates the need for a separate geospatial serving layer while bringing real-time latency, Unity Catalog governance, and more than 100 spatial functions to your architecture.
Sizing Lakebase Search: lessons from a real app and 4 million passages
Sizing the Local File Cache to match the vector index keeps up to 98% of pages in memory and drops warm vector latency to single-digit milliseconds directly inside Lakebase Postgres. Reducing the search candidate pool to 200 cuts page reads by two-thirds while preserving 90.5% of top-10 results, with an always-on 0.5 CU floor providing an optimal production baseline for just $31 to $40 per month.
Unlock powerful insights and forecasts with Predictive AI Functions
Databricks has introduced five Predictive AI Functions, featuring an upgraded ai_forecast alongside the new ai_top_drivers, ai_detect_anomalies, ai_predict_value, and ai_predict_class. These tools allow SQL analysts and business users to generate predictions, detect anomalies, and uncover metric drivers directly within Genie and existing workflows without building or maintaining models.
How FactSet cut model training and deployment from days to hours with Databricks AI Runtime
FactSet cut the time required to set up, train, and deploy its Text-to-Formula models from days to hours by adopting Databricks AI Runtime. By leveraging serverless GPU compute directly alongside existing Lakehouse workflows, the team can now continuously retrain fine-tuned open models every two to three weeks to convert natural-language queries into FactSet Query Language formulas.
Power real-time applications and agents with Document Intelligence
Document Intelligence real-time REST APIs enable applications to parse, classify, and extract information from documents as they arrive rather than relying on delayed batch pipelines. You can embed these APIs directly into Databricks Apps or custom agents to power interactive workflows and drive real-time business decisions.