OpenAI has introduced a new system called EVMbench, designed to measure how well artificial intelligence agents can find and fix security flaws in crypto smart contracts.
Summary
- OpenAI has introduced EVMbench, a new framework designed to measure how well AI agents can detect, fix, and exploit smart contract vulnerabilities.
- Developed with Paradigm, the benchmark is built on real audit data and focuses on practical, high-risk security scenarios.
- Early results show strong progress in exploit tasks, while detection and patching are still challenging.
The company announced on Feb. 18 that it has developed EVMbench in partnership with Paradigm. The benchmark focuses on contracts built for the Ethereum Virtual Machine and is meant to test how AI systems perform in real financial settings.
OpenAI said smart contracts currently secure more than $100 billion in open-source crypto assets, making security testing increasingly important as AI tools become more capable.
Testing how AI handles real security risks
EVMbench evaluates AI agents across three main tasks: detecting vulnerabilities, fixing flawed code, and carrying out simulated attacks. The system is built using 120 high-risk issues drawn from 40 past security audits, many of them from public auditing competitions.
Additional scenarios were taken from reviews of the Tempo blockchain, a payments-focused network designed for stablecoin use. These cases were added to reflect how smart contracts are used in financial applications.
To build the test environment, OpenAI adapted existing exploit scripts and created new ones where needed. All exploit tests run in isolated systems rather than on live networks, and only previously disclosed vulnerabilities are included.
In detection mode, agents review contract code and try to identify known security flaws. In patch mode, they must fix those flaws without breaking the software. In exploit mode, agents attempt to drain funds from vulnerable contracts in a controlled setting.
Early results and industry impact
OpenAI said a custom testing framework was developed to ensure results can be reproduced and verified.
The company tested several advanced models using EVMbench. In exploit mode, GPT-5.3-Codex achieved a score of 72.2%, compared with 31.9% for GPT-5, released six months earlier. Detection and patching scores were lower, showing that many vulnerabilities are still difficult for AI systems to handle.
Researchers observed that agents performed best when goals were clear, such as draining funds. Performance dropped when tasks required deeper analysis, such as reviewing large codebases or fixing subtle bugs.
OpenAI acknowledged that EVMbench does not fully reflect real-world conditions. Many major crypto projects undergo more extensive reviews than those included in the dataset. Some timing-based and multi-chain attacks are also outside the system’s scope.
The company said the benchmark is intended to support defensive use of AI in cybersecurity. As AI tools become more powerful, they could be used by both attackers and auditors. Measuring their capabilities is seen as a way to reduce risk and encourage responsible deployment.
Alongside the release, OpenAI said it is expanding security programs and investing $10 million in API credits to support open-source and infrastructure protection. All EVMbench tools and datasets have been made public to support further research.