Accelerate GPU training on Unity Catalog tables with UCDataset
Summary
Now in beta, UCDataset is a native data loader that streams data directly from Unity Catalog tables to GPUs, accelerating training times by 10x and reducing compute costs. It plugs directly into standard PyTorch training loops to handle multi-GPU data sharding without custom code while ensuring automatic governance.
Summary generated by brickster.ai. For the full article, follow the source link above.
More from Databricks Blog
Databricks' Commitment to European Sovereignty
Over a Decade of Investment in EuropeDatabricks has been building in Europe since...
Turn buying signals into pipeline with Sendoso gifts in Genie One
Sendoso's MCP server is now available on Databricks Marketplace, allowing teams to convert intent signals into previewed, approved gifts directly within Genie One conversations. Governed through Unity Catalog and Unity Gateway for access control and call logging, the integration keeps gift histories in the same workspace as pipeline data to track deal influence.
Read, don't copy: replacing file feeds with OpenSharing - Part 2 (How to Implement)
OpenSharing lets you expose a Delta table through an open protocol so recipients read it live, replacing per-recipient file and API exports along with the file tracking and reconciliation they require. This implementation guide shows recipients consuming the shared table as a checkpointed incremental stream, receiving row-level inserts, updates, and deletes through Change Data Feed.
Read, don't copy: replacing file and API feeds with OpenSharing - Part 1 (The Why)
Instead of copying customer data via SFTP drops or API pulls, you can read the customer's shared table in place with OpenSharing and follow its changes through streaming or change data feed, with no need to track what's been processed yourself. This removes the per-feed pipelines, fragile incremental loads, backfills and duplicate storage, and it makes sharing easier for customers, since adding you as a recipient is just a grant on a table they already maintain, with audit logs recording every access.
How public sector organizations can govern AI at scale
Unity Gateway lets public sector teams govern AI the way they already govern data in Unity Catalog, with controls for model access, spend caps, and request-level tracking. Databricks also manages data residency for its hosted models by geography, giving compliance teams a documented answer on where AI requests are processed.
Real-time fraud detection with Feature Views
Feature Views let you define fraud features once and reuse them for both point-in-time training and online scoring, which reduces data leakage and training-serving skew. The example pairs rolling windows for payment bursts with a 30-day Sawtooth baseline and an on-demand feature that scores each payment against it, and a single materialize_features() call sets up managed streaming pipelines into the Online Feature Store on Lakebase with sub-200ms feature freshness.