Native IP functions became generally available for SQL, PySpark and Scala on Databricks Runtime 18.3+

The main shipping item for Spark users in October 2026 was general availability of native IP functions on Databricks Runtime 18.3 and later [23]. They parse, validate, canonicalize and join IPv4 and IPv6 addresses and CIDR blocks without UDFs or regex. They are optimized in Photon and available from SQL, PySpark and Scala.

Related open source releases were smaller. Delta Lake 4.4.1 added safeguards for catalog-managed tables, including blocking unsupported cleanups and skipping automatic maintenance so writes can succeed [21]. It also fixed snapshot construction errors caused by unratified staged commits, which allows S3A fast listing to be re-enabled safely [21]. Unity Catalog 0.7.0 turned on native Apache Iceberg REST catalog support by default. It also made dropping managed tables, volumes or models purge the underlying cloud storage files after a default seven-day delay [14]. A separate open-source library, Funke, parsed HL7v2 healthcare messages into native Spark types while keeping the message hierarchy [17].

Spark Real-Time Mode appeared in two videos, one on a manufacturing digital twin and one on energy trading and risk, both pairing it with Lakebase [2][10]. Databricks also described using Free Edition, which includes Apache Spark, to teach data and AI courses [4].

Community threads centered on everyday PySpark work. Practitioners asked what to check first when a job runs slowly [16], whether supported memory settings come down to trial and error [11], and how to run PySpark code locally [13]. Others covered unit testing PySpark [18], converting Informatica PowerCenter XML to PySpark [9], and rewriting an 800-line PySpark pipeline as 200 lines of SQL [19]. One thread asked whether anyone had contributed to Apache Spark itself [6].