Evaluating AI Agents Live at the Grounded Reasoning Cup
Stanford's team won the Grounded Reasoning Cup with 63.3% accuracy on OfficeQA Pro V2, a new 120,000-page U.S. Treasury document benchmark, using an end-to-end agent optimization approach combining reusable skills, document-representation fallbacks, and adaptive verification. Across all 11 academic teams, out-of-the-box frontier agents averaged under 30% accuracy, showing that agent performance tuned on one benchmark doesn't reliably generalize to a new corpus.
* The Grounded Reasoning Cup challenged 11 academic teams to apply agents developed on OfficeQA Pro to OfficeQA Pro V2, a newly released benchmark built from approximately 120,000 pages of U.S. Treasury documents. * Results showed that generalization cannot be assumed. Approaches developed on a familiar benchmark did not always transfer reliably to a new corpus, and out-of-the-box frontier agents averaged less than 30% accuracy. * Stanford’s winning team achieved 63.3% accuracy through an end-to-end agent optimization strategy that combined a library of reusable skills, targeted document-representation fallbacks, and adaptive verification.