Skip to content
All topics

Photon

Recent items mentioning Photon across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.

10 recent items2 news8 community threads
What's happening in PhotonAI synthesis · updated Oct 21, 2026

Photon is now the execution engine behind two new Databricks Runtime features. NEAREST BY uses a fused Photon operator with a custom blocked GEMM kernel to run exact or approximate vector search as a SQL join on liquid-clustered Delta tables, with no external vector store to sync 2. The native IP functions, GA on DBR 18.3+, are also optimized in Photon and complete IP CIDR joins up to 3.1x faster and 6.4x cheaper than an alternative, without UDFs or regex 3.

Written October 2026 from the 3 most recent items mentioning Photon at that time. It refreshes when this topic next has enough new material. Click any [N] to jump to the source.

Reddit

Photon cut this Databricks job from 60 to 40 minutes, but DBUs rose 33%

A 60-minute Databricks job dropping to 40 minutes sounds like a win. In this Photon example, DBUs still rose about 33%. On classic clusters, Photon’s DBU rate is roughly 2×. To see whether it paid off, compare the same workload on the same instance with Photon on and off. Check the query profile to see whether the slow operators ran in Photon, then compare runtime, DBUs, and total cost. Python and Scala UDFs can fall back to standard Spark, so Photon may not speed up the bottleneck. Disclosure: I work at Hackmamba and am sharing this SELECT guide as part of my work. submitted by /u/Plenty-Apricot9222 [link] [comments]

00Plenty-Apricot9222yesterday
Reddit

Using the Apache DataFusion Comet accelerator to save cost

I wonder if anybody has utilized the comet accelerator to speed up things and therefore save cost? Given the photon tax this seems like obvious choice but I'm not sure if the project is already mature enough. submitted by /u/22Maxx [link] [comments]

0022Maxx4d ago
Reddit

Optimize serverless job performance with query metrics & performance insights

Hi folks, Databricks PM here - I wanted to share an update about serverless jobs observability since this is a topic I get asked about all the time. One of the hardest adjustments for users adopting serverless is changing your mindset from how you monitored classic jobs and investigating very low level details (think deep diving into the Spark UI) to understanding how you can monitor serverless jobs and have confidence in their performance. Query Profile already simplifies performance investigations for serverless jobs by providing in depth metrics on query execution. The addition of query performance insights (generally available since September) have made this even easier: Databricks flags queries that have optimization potential directly in the UI and you can take action via Genie Code by using the “Optimize” button which will propose optimizations directly on the asset (e.g. in your notebook for code optimization). Here is the list of insights that are currently covered: Query optimization insights: COVERAGE_FILTER_KEYS_CLUSTERING, COVERAGE_FILTER_KEYS_PARTITIONING, COVERAGE_PHOTON, EXPLODING_JOIN, FLOW_FULL_RECOMPUTE, REDUNDANT_AGGREGATION, REDUNDANT_JOIN, SELECTIVE_JOIN, WIDE_PROJECTION Data layout insights: CONCURRENT_WRITE, COVERAGE_STATS_DELTA, COVERAGE_STATS_OPTIMIZER, DATA_FILE_SIZE, DATA_SKEW, MANUAL_DATA_LAYOUT Compute and resource insights: DATA_SPILL, EXCESSIVE_QUEUE_TIME, IO_THROTTLING Applied accelerations: AUTO_LIQUID_CLUSTERING, HISTORY_BASED_JOIN_STRATEGY, SHORT_QUERY_PRIORITIZATION, METRIC_VIEW_MATERIALIZATION_MATCH Having this available only at the query level can make it cumbersome to figure out which queries are worth optimizing for your job, so we have brought these insights to the Jobs UI. So how can you try this out for your jobs? Open any serverless job that has a recent run ( You can either drill down and pick which query you want to optimize or you can use the Optimize button directly from the job run sidebar and let Databricks prioritize insights which are estimated to help achieve the most performance improvements for a given job. Timeline view Side note: we also roll up query metrics on data volume and number of queries on the job and task level now to help you quickly identify if a change in job performance can simply be attributed to a change in data volume. Limitation: the roll up of metrics and insights is currently capped at 100 queries You can check out our documentation to learn more. If you have tried it out already, I would love to hear any feedback! Are there any other insights or query performance metrics that you are missing? What other observability improvements would you like to see for serverless jobs? submitted by /u/brickster_123 [link] [comments]

00brickster_1234w ago
Databricks CommunityData Engineeringanswered

Photon enabled but a large share of the plan is falling back, cost up and runtime flat

001mo ago
Databricks CommunityData Engineeringanswered

Photon enabled but a large share of the plan is falling back, cost up and runtime flat

002mo ago
Databricks CommunityAdministration & Architecture

Does enabling Photon improve performance while lowering compute costs?

002mo ago
Reddit

Does Photon support CSV?

Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator. https://preview.redd.it/1amu93oqqslh1.png?width=260&format=png&auto=webp&s=4ca1c1ffec6cb6f6cd484b4645b0b5d7c5f6be86 I'm not running anything complex: df = ( spark.read .format("csv") .option("header", "true") .schema(schema) .load(csv_path) ) df.write.mode("overwrite").saveAsTable("...") I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator. https://preview.redd.it/g1tlotk8rslh1.png?width=940&format=png&auto=webp&s=f3d2b22ebab4f716c5b6831d96305c082e1126a8 DBR 19 + Serverless SQL Warehouse (current) Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything. submitted by /u/Common_Jaguar474 [link] [comments]

00Common_Jaguar4742mo ago
Databricks CommunityData Engineeringanswered

Detecting Photon fallback in-cluster + safe right-sizing from system tables

005mo ago

Get Tuesday's version of this

Tracking Photon? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.

Read past issues first