Vals AI is an independent AI evaluation platform that builds private, domain-specific benchmarks to measure how well large language models perform on real work in law, finance, healthcare, and coding. It has raised $40 million in Series A funding at a $400 million valuation, with its results cited in model cards from OpenAI, Anthropic, Google, Meta, and xAI .
At-a-Glance: Key Takeaways
- Vals AI builds private, expert-written benchmarks that test AI models on real professional tasks, not public academic exams that can leak into training data .
- The Vals Index ranks 30+ benchmarks across legal, finance, healthcare, and coding domains, showing that no single model leads everywhere .
- It raised $40M in Series A funding at a $400M valuation in August 2026, led by Andreessen Horowitz (a16z) .
- Vals Smith lets anyone build custom coding benchmarks from GitHub repositories, with 120 free credits to start .
- Medical billing reveals a 32-point gap: leading models score 85-88% on clinical notes but only 49-53% on ICD-10 coding, exposing where AI truly struggles .
- The company operates with Zero Data Retention policies, ensuring model providers cannot retain or train on its evaluation prompts .
What Is Vals AI?
Vals AI is an independent AI evaluation and model auditing platform that measures how well large language models (LLMs) and AI agents perform on real-world professional tasks. The San Francisco-based company was founded in 2024 by Rayan Krishnan and Langston Nashold, who began asking how to measure the new generation of AI while at Stanford .
The core premise is blunt: the AI industry has invested trillions in generating intelligence but comparatively little in measuring it. Public benchmarks saturate within months, and tests leak into training data, making it impossible to know whether a model is genuinely capable or just memorized the answer key .
Vals AI addresses this by building private test sets that models cannot study for. The company partners with reference institutions in each field—law firms, financial analysts, tax professionals—to create a taxonomy of representative tasks, then uses automated scoring plus expert review to evaluate model outputs .
The approach has gained rapid traction. Vals AI reports revenue grew 8x compared to all of 2025, its customer base doubled, and its team tripled in six months. Its results have been cited in model cards from OpenAI, Anthropic, Google, Meta, and xAI, and it has supported the Department of Commerce and members of Congress working on AI policy .
How Vals AI Works
The Private Test Set Architecture
Vals AI’s methodology centers on preventing benchmark contamination. The company divides its proprietary exams into three tiers: public validation material, a larger private validation set it can license for internal work, and a final private test set used for published scores .
That last set stays hidden. Once an answer key is on the web, models may have read it during training. By keeping the final questions private, Vals ensures that published scores reflect genuine capability rather than memorization .
The trade-off is that buyers cannot independently inspect every final question. They can examine sample tasks and methodology notes, but must trust Vals’ expertise in constructing and grading the benchmarks .
The Vals Index
The Vals Index is the company’s flagship evaluation, built to measure the impact of AI on real-world agentic tasks. It is agentic and long-horizon: many tasks are long, with the median Terminal-Bench 4.0 task estimated at four hours of expert work, and Excel Modeling tasks taking five hours or more .
The Index comprises eight components spanning finance, coding, legal, and tax work: Finance Agent v2, Excel Modeling Benchmark, Terminal-Bench 4.0, Vibe Code Bench, Code Migration, Legal Research Bench, Harvey’s Legal Agent Benchmark (HLAB), and Tax Agent Bench .
Six of the eight benchmarks are private, with tasks, answers, and grading rubrics written by professionals who do this work: financial analysts for finance benchmarks, practicing lawyers for Legal Research Bench, and tax professionals for Tax Agent Bench .
Scoring and Reliability
Vals AI uses a rubric-based scoring system that evaluates correctness (50%), authoritativeness (40%), and appropriateness (10%). The platform combines LLM-as-a-judge, string matching, hallucination detection, and custom Python functions in its “Checks” mechanism. When confidence is low, the system automatically routes to human expert review .
Error bars are reported alongside accuracy, cost, and latency. Vals’ methodology states that these error bars cover uncertainty in the benchmark but do not capture every variation in prompts, settings, or model randomness .
Zero Data Retention
Evaluations are run with Zero Data Retention (ZDR) policies in place, meaning model providers cannot retain or train on Vals’ prompts. This is a critical distinction for enterprises concerned about data leakage into third-party models .
Key Features and Products
Vals Smith: Custom Coding Benchmarks
Launched in August 2026, Vals Smith lets anyone create a custom coding benchmark from any GitHub repository. Users provide a repo, and Vals generates a benchmark tailored to that codebase, with 120 free credits to get started .
This addresses a common problem: generic coding benchmarks like SWE-bench test models on tasks that may not reflect a specific company’s tech stack or coding standards. Vals Smith lets organizations test models against their own code .
Frontier Risk Benchmarks
Alongside the Series A, Vals announced new benchmarks for frontier risks:
- RSI Index (in collaboration with CoreWeave): measures recursive self-improvement
- CyberBench: a cyber risk benchmark built with leading academics
- Mental health benchmarks: initial work announced
These benchmarks address concerns that go beyond productivity: whether AI systems could improve themselves, whether they introduce security vulnerabilities, and whether they handle sensitive domains safely.
Live Evals (Beta)
Vals offers a Python SDK that allows developers to run evaluations during live deployments, not just during development. The evaluate() function takes model output and an array of checks, returning results for each check .
Example checks include grammar verification, length constraints, and safety filters. This enables real-time monitoring of production AI systems, not just pre-deployment testing .
Valkyrie: Open-Source Evaluation Infrastructure
Vals open-sourced Valkyrie, a microservices-based evaluation platform that runs agent evaluations in parallel sandboxes. It is licensed under AGPL-3.0 and was presented at ACM CAIS 2026 .
Pricing and Access
Vals AI does not publish public pricing for its enterprise platform. According to Japanese startup database First CVC, the revenue model is a hybrid: subscription base fee plus usage-based billing tied to evaluation volume, sold through consultative sales .
The public leaderboards at vals.ai/benchmarks provide free access to model rankings across legal, finance, healthcare, coding, and academic categories. Developers can use the valsai Python package (CLI and SDK) with an API key from platform.vals.ai .
Privacy and Security Considerations
Vals AI operates with Zero Data Retention (ZDR) policies for its evaluations, ensuring that model providers cannot retain or train on the prompts used in benchmarking .
For enterprises, this addresses a critical concern: when evaluating a model through a third-party platform, does the evaluation data leak into the model provider’s training pipeline? Vals’ ZDR commitment means it does not.
The company’s public benchmarks are accessible to anyone. Private benchmark results and custom evaluation data are restricted to enterprise customers .
Important distinction: SOC 2 certification or similar compliance frameworks are not referenced in available documentation. Enterprises with strict compliance requirements should conduct independent due diligence on Vals’ data handling practices.
Real-World Findings from Vals AI
Vals AI’s benchmarks have produced several findings that challenge common assumptions about AI capabilities.
The Medical Billing Gap
A February 2026 study found leading models scoring 85-88% on MedScribe, a clinical documentation benchmark. On MedCode, a medical billing test with 2,755 samples, most leading models clustered around 49-53%, with the top result at 55.92% .
The roughly 32-point gap is significant. A fluent answer and a correct operational decision are different accomplishments. Models that appear competent on clinical notes may fail on the coding decisions that determine reimbursement and audit exposure .
Multi-Agent Teams Cost More Without Guaranteed Gains
A Vals AI study on Vibe Code Bench pitted GPT 6 Sol and Claude Opus 5.5 against themselves, running each as a solo coder and as a multi-agent team .
Only one of four team configurations produced a statistically significant gain. Teams cost 1.8x to 5.1x more than single agents. Opus’s maximum-effort team achieved the highest score at 93.2% but cost $122 and took 174 minutes per app, compared to $4.08 and 20.3 minutes for its medium-effort single agent .
For Sol, raising reasoning effort beat adding a team: +11.4 points solo versus +7.3 from a team .
The Oracle Router Opportunity
Vals calculated the performance of an “oracle router”—one that picks the perfect model for every task in hindsight. On the Vals Index, that oracle would beat the best single model by 14 points while costing about 40% less .
This underscores a key finding: no single model leads every component. The model that tops Harvey’s Legal Agent Benchmark ranks 40th on Terminal-Bench 4.0. Different models lead different components, and the second-ranked model overall costs about 36% more per task than the first .
Who Should Use Vals AI
Enterprises deploying AI in regulated industries—law, finance, healthcare—need objective measurement of model performance before committing to production. Vals provides the infrastructure for model selection, ROI measurement, and procurement auditing .
AI research labs use Vals scores in model cards and as external validation of capability claims. The results are cited by OpenAI, Anthropic, Google, Meta, and xAI .
Developers building AI applications can use Vals Smith to create custom coding benchmarks from their own repositories, and the SDK/CLI for CI/CD integration and regression testing .
Governments and policymakers working on AI safety and competitiveness can use Vals’ independent evaluations to measure frontier capabilities and cyber risk .
Enterprises with strict data governance benefit from Vals’ Zero Data Retention policies, which prevent evaluation data from leaking into model training pipelines .
Frequently Asked Questions
What is Vals AI?
Vals AI is an independent AI evaluation platform that builds private, domain-specific benchmarks to measure how well large language models perform on real professional tasks in law, finance, healthcare, and coding. It was founded in 2024 and has raised $40 million in Series A funding .
How does Vals AI make money?
Vals AI uses a hybrid revenue model: subscription base fee plus usage-based billing tied to evaluation volume. Public benchmarks are free to access, while the enterprise platform with private benchmarks and custom evaluations is priced through consultative sales .
What is Vals Smith?
Vals Smith is a tool that lets anyone create a custom coding benchmark from any GitHub repository. It launched in August 2026 with 120 free credits to get started .
Why do Vals AI benchmarks matter?
Public benchmarks saturate and leak into training data. Vals AI uses private test sets that models cannot study for, providing more reliable measurement of genuine capability. Its results are cited in model cards from major AI labs and used by enterprises for model selection .
What is Zero Data Retention in Vals AI?
Vals AI runs evaluations with Zero Data Retention policies, meaning model providers cannot retain or train on the prompts used in benchmarking. This addresses enterprise concerns about evaluation data leaking into model training pipelines .
Conclusion
Vals AI has built a position that the AI industry urgently needs: an independent measurement institution for a trillion-dollar market. The company’s private test sets and expert-written benchmarks address the fundamental problem that public tests saturate and leak, making it impossible to know whether a model is genuinely capable or just memorized the answers.
What matters most is the evidence Vals produces. The 32-point gap between clinical note performance and medical billing accuracy reveals that fluency and operational correctness are different things. The finding that multi-agent teams cost 1.8x to 5.1x more without guaranteed gains challenges assumptions about orchestration. And the oracle router calculation—14 points better while costing 40% less—shows that model selection matters more than picking the single best model.
Vals AI may suit enterprises deploying AI in regulated industries, AI research labs seeking external validation, developers building custom benchmarks, and governments measuring frontier capabilities. It may not suit users seeking simple, free model comparisons without the need for domain-specific rigor.
Before relying on any benchmark, including Vals, understand its methodology and limitations. Vals’ error bars cover benchmark uncertainty, not every variation in prompts, settings, or model randomness. A company using the leaderboard still needs to test its actual workflow . Vals provides a rigorous starting point, not a final answer.