How Flo Health built safer, more useful open models with Databricks AI Runtime
Flo Health improved open model medical safety from 34% to approximately 61% and usefulness from 42% to 76% by running distributed fine-tuning on 32 NVIDIA H100 GPUs with Databricks AI Runtime. The team achieved these results without human-annotated data by combining privacy-first synthetic datasets, supervised fine-tuning, and Direct Preference Optimization with an evaluation framework calibrated against clinician reviews.
* Flo Health built a specialized evaluation framework to measure AI responses across both medical safety and usefulness, calibrated against clinician review. * Using privacy-first synthetic data and AI judges, Flo created training data without human-annotated training pairs, then used supervised fine-tuning and Direct Preference Optimization (DPO) to improve an open model. * Flo ran distributed fine-tuning on 32 NVIDIA H100 GPUs with Databricks AI Runtime, improving medical safety from 34% to approximately 61%, usefulness from 42% to 76%, and the combined score from 16% to 47%.