Photon
Recent items mentioning Photon across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Photon is now the execution engine behind two new Databricks Runtime features. NEAREST BY uses a fused Photon operator with a custom blocked GEMM kernel to run exact or approximate vector search as a SQL join on liquid-clustered Delta tables, with no external vector store to sync 2. The native IP functions, GA on DBR 18.3+, are also optimized in Photon and complete IP CIDR joins up to 3.1x faster and 6.4x cheaper than an alternative, without UDFs or regex 3.
Written October 2026 from the 3 most recent items mentioning Photon at that time. It refreshes when this topic next has enough new material. Click any [N] to jump to the source.
Photon cut this Databricks job from 60 to 40 minutes, but DBUs rose 33%
A 60-minute Databricks job dropping to 40 minutes sounds like a win. In this Photon example, DBUs still rose about 33%. On classic clusters, Photon’s DBU rate is roughly 2×. To see whether it paid off, compare the same workload on the same instance with Photon on and off. Check the query profile to see whether the slow operators ran in Photon, then compare runtime, DBUs, and total cost. Python and Scala UDFs can fall back to standard Spark, so Photon may not speed up the bottleneck. Disclosure: I work at Hackmamba and am sharing this SELECT guide as part of my work. submitted by /u/Plenty-Apricot9222 [link] [comments]
Using the Apache DataFusion Comet accelerator to save cost
I wonder if anybody has utilized the comet accelerator to speed up things and therefore save cost? Given the photon tax this seems like obvious choice but I'm not sure if the project is already mature enough. submitted by /u/22Maxx [link] [comments]
Optimize serverless job performance with query metrics & performance insights
Hi folks, Databricks PM here - I wanted to share an update about serverless jobs observability since this is a topic I get asked about all the time. One of the hardest adjustments for users adopting serverless is changing your mindset from how you monitored classic jobs and investigating very low level details (think deep diving into the Spark UI) to understanding how you can monitor serverless jobs and have confidence in their performance. Query Profile already simplifies performance investigations for serverless jobs by providing in depth metrics on query execution. The addition of query performance insights (generally available since September) have made this even easier: Databricks flags queries that have optimization potential directly in the UI and you can take action via Genie Code by using the “Optimize” button which will propose optimizations directly on the asset (e.g. in your notebook for code optimization). Here is the list of insights that are currently covered: Query optimization insights: COVERAGE_FILTER_KEYS_CLUSTERING, COVERAGE_FILTER_KEYS_PARTITIONING, COVERAGE_PHOTON, EXPLODING_JOIN, FLOW_FULL_RECOMPUTE, REDUNDANT_AGGREGATION, REDUNDANT_JOIN, SELECTIVE_JOIN, WIDE_PROJECTION Data layout insights: CONCURRENT_WRITE, COVERAGE_STATS_DELTA, COVERAGE_STATS_OPTIMIZER, DATA_FILE_SIZE, DATA_SKEW, MANUAL_DATA_LAYOUT Compute and resource insights: DATA_SPILL, EXCESSIVE_QUEUE_TIME, IO_THROTTLING Applied accelerations: AUTO_LIQUID_CLUSTERING, HISTORY_BASED_JOIN_STRATEGY, SHORT_QUERY_PRIORITIZATION, METRIC_VIEW_MATERIALIZATION_MATCH Having this available only at the query level can make it cumbersome to figure out which queries are worth optimizing for your job, so we have brought these insights to the Jobs UI. So how can you try this out for your jobs? Open any serverless job that has a recent run ( You can either drill down and pick which query you want to optimize or you can use the Optimize button directly from the job run sidebar and let Databricks prioritize insights which are estimated to help achieve the most performance improvements for a given job. Timeline view Side note: we also roll up query metrics on data volume and number of queries on the job and task level now to help you quickly identify if a change in job performance can simply be attributed to a change in data volume. Limitation: the roll up of metrics and insights is currently capped at 100 queries You can check out our documentation to learn more. If you have tried it out already, I would love to hear any feedback! Are there any other insights or query performance metrics that you are missing? What other observability improvements would you like to see for serverless jobs? submitted by /u/brickster_123 [link] [comments]
NEAREST BY Join: Scaling Vector Search in Databricks Runtime
Databricks Runtime now features NEAREST BY, a new SQL join that executes exact or approximate batch vector searches directly against your Lakehouse data. Powered by a fused Photon operator with a custom blocked GEMM kernel, it turns standard liquid-clustered Delta tables into partition-pruned vector indexes with no external vector store to sync or manage.
IP Functions are Generally Available, bringing high-performance network analytics to the Lakehouse
Native IP functions are now generally available on Databricks Runtime 18.3+, providing built-in support to parse, validate, canonicalize, and join IPv4 and IPv6 addresses and CIDR blocks without UDFs or regex. Optimized in Photon across SQL, PySpark, and Scala, these native functions run demanding network workloads in seconds and complete IP CIDR joins up to 3.1x faster and 6.4x cheaper than another leading cloud data warehouse.
Photon enabled but a large share of the plan is falling back, cost up and runtime flat
Photon enabled but a large share of the plan is falling back, cost up and runtime flat
Does enabling Photon improve performance while lowering compute costs?
Does Photon support CSV?
Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator. https://preview.redd.it/1amu93oqqslh1.png?width=260&format=png&auto=webp&s=4ca1c1ffec6cb6f6cd484b4645b0b5d7c5f6be86 I'm not running anything complex: df = ( spark.read .format("csv") .option("header", "true") .schema(schema) .load(csv_path) ) df.write.mode("overwrite").saveAsTable("...") I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator. https://preview.redd.it/g1tlotk8rslh1.png?width=940&format=png&auto=webp&s=f3d2b22ebab4f716c5b6831d96305c082e1126a8 DBR 19 + Serverless SQL Warehouse (current) Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything. submitted by /u/Common_Jaguar474 [link] [comments]
Detecting Photon fallback in-cluster + safe right-sizing from system tables
Get Tuesday's version of this
Tracking Photon? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.