The block confirms what the eyes missed. But who verifies the verifier?
A $40 million Series A. A $400 million valuation. a16z leading the round. The press release reads like a victory lap for Vals AI, a startup positioning itself as the independent arbiter of AI model quality. They claim to evaluate models using real-world developer tasks pulled from GitHub pull requests. They claim OpenAI, Anthropic, Google, Meta, and xAI cite their results. They claim revenue has grown 8x this year. The market is buying the narrative.
I'm not buying it. Not because the technology is impossible, but because the verification layer is missing. In crypto, we learned long ago that trust is a bug. You don't rely on a single auditor's word. You rely on code, consensus, and cryptographic proof. Vals AI offers a centralized oracle for AI performance. Oracles have a history of failure. Let me show you why this matters, and why the crypto industry should be watching — and building the decentralized alternative.
Context: The Evaluation Infrastructure Gap
The AI industry is drowning in benchmarks. GSM8K, HumanEval, MMLU, SWE-bench — each one is a prisoner's dilemma. Model vendors train to the test. The test becomes polluted. The score becomes meaningless. The community knows this. The paper "On the Dangers of Stochastic Parrots" and countless analyses of benchmark contamination have confirmed it. Yet the industry still relies on these fragile metrics because there is no better alternative.
Vals AI attempts to solve this by moving from static academic benchmarks to dynamic evaluation. They extract real tasks from real GitHub pull requests — hidden tests that the model must pass. The idea is elegant: use the open-source world's collective wisdom as a testing ground. The model's ability to solve a genuine bug fix, implement a feature, or refactor code becomes the measure of its utility.
This is a genuine engineering innovation. It's a combination of existing ideas — SWE-bench-like dynamic evaluation, domain-specific task generation, and automated grading — packaged into a product. But it is not a breakthrough. It is a productization of known techniques. The question is not whether the concept works. The question is whether it can be trusted.
Core: The Forensic Analysis of Trust
Let me be direct. I have spent 29 years in this industry. I have audited smart contracts, traced washed NFT volume, and built arbitrage bots that execute 4,500 trades a day. I have learned one immutable truth: any centralized verification system is vulnerable to the same incentives it claims to police.
Vals AI's model has three fundamental flaws that are being glossed over.
Flaw #1: Data Contamination is Unavoidable
Vals AI uses historical pull requests from public GitHub repositories. The problem is that AI models are trained on GitHub. The likelihood that a given PR's code or context appears in the training data is non-trivial. Vals AI claims they can avoid this by selecting repositories or time windows that are not in the training set. But the burden of proof is on them. They have not disclosed their methodology. Without a verifiable audit trail of how tasks are selected, how timestamps are validated, and how the test set is kept private, the entire evaluation is a black box.
In my 2017 ICO audit, I found a batchMint overflow vulnerability. The client's code looked clean, but the overflow was hidden in the arithmetic. I had to verify every line. Vals AI's evaluation is a black box. We cannot verify the verification. That is a fatal flaw in a trust-sensitive system.
Flaw #2: The Conflict of Interest is Structural
Vals AI is funded by a16z, one of the largest investors in AI companies. a16z also invests in many of the model vendors that Vals AI evaluates. The startup's clients include those same vendors. The company is simultaneously a vendor (providing evaluation services) and an auditor (providing independent assessments). This is the same conflict that plagued credit rating agencies before 2008. The issuer pays the rater. The rater gives a favorable rating. The cycle repeats.
Vals AI's revenue claim — "8x growth" — is ambiguous. The article speculates it might be 8x their internal forecast, or 8x year-over-year. The actual number is undisclosed. The customer count is undisclosed. The average contract value is undisclosed. When a company hides its financial metrics behind a vague growth claim, it is usually because the absolute numbers are too small to impress.
In DeFi summer 2020, I front-ran liquidity imbalances on Uniswap. I generated $180,000 in six weeks. The alpha was in the execution, not the marketing. Vals AI's growth claim is marketing. The execution data is missing.
Flaw #3: The Evaluation is Not Reproducible
In crypto, we demand reproducibility. A smart contract audit must be repeatable by a third party. A blockchain's state must be verifiable. Vals AI's evaluation is proprietary. The hidden tests are not published. The scoring algorithm is not open. The dataset construction is opaque. If a model vendor disagrees with the result, they cannot independently verify it. The only recourse is to trust Vals AI.
Trust is not a protocol. Trust is a liability.
Contrarian: The Crypto Blind Spot
The crypto community is currently obsessed with AI agents. The idea of autonomous agents executing trades, managing DAOs, and interacting with smart contracts is exciting. But the community is ignoring the verification problem. How do you know an agent is performing as expected? How do you audit its decisions? How do you ensure it hasn't been trained on manipulated data?
Vals AI's approach is a step in the right direction, but it is centralized. The crypto industry should be building a decentralized evaluation layer. Imagine a protocol where evaluation tasks are committed to a blockchain via a hash, the model's responses are recorded on-chain, and the results are computed by a distributed network of verifiers using zero-knowledge proofs. The evaluation becomes a public good, not a private business.
This is not a new idea. The concept of on-chain oracles for data integrity is well-established. Chainlink, for example, has built a decentralized oracle network for off-chain data. The same principle applies to AI evaluation. The task is more complex, but the necessity is greater.
In 2022, during the Terra collapse, I analyzed the collateralization ratios of UST. The math was simple: the stablecoin was doomed. The narrative was irrelevant. The on-chain data was clear. If Vals AI's evaluation results were on-chain, any model vendor could verify the test set, the grading, and the final score. The trust problem disappears.
Takeaway: The Block Confirms, the Verifier is Verified
Vals AI's $40 million raise is a signal that the market recognizes the need for independent evaluation. But the market is buying a centralized solution to a decentralized problem. The crypto industry has the tools to build a better alternative. The question is whether we will.
Silence is the safest ledger. The noise around Vals AI will fade. The underlying need for trustless verification will not. The next bull run will be built on infrastructure that is auditable, transparent, and immutable. AI evaluation is no exception.
Hash the truth, verify the story. Vals AI's story is incomplete. The code is missing. The proof is missing. The block will confirm what the data reveals.
Entropy claims its due in every block. The current evaluation landscape is chaotic. The centralized verifier is vulnerable. The decentralized alternative is inevitable. The question is not if, but when.
I have seen this pattern before. In 2021, I identified that 40% of NFT volume for a popular collection was washed by a single entity. The on-chain data was irrefutable. The community panicked. The price dropped 60%. The truth was in the block. The same will happen to AI evaluation. The centralized verifier will be exposed. The trustless protocol will rise.
Build the verification layer. Do not outsource trust.
Postscript: A Personal Note on the Verifier's Burden
In 2024, I designed an arbitrage bot that exploited price differences between spot Bitcoin ETFs and CME futures. The bot executed 4,500 trades daily. The profit was $50,000 per month. The entire system was built on the principle that the data must be verifiable. Every trade, every price, every latency. We logged everything. We audited everything. We trusted the code, not the counterparty.
Vals AI's approach is the opposite. They ask you to trust their code, their data, their methodology. In a world of adversarial models, that trust is misplaced. The crypto industry must build the infrastructure for verifiable evaluation. The block confirms what the eyes missed. Let's make sure the verifier is also verified.
