AI Proves Code Safety at Scale, Reshaping Software Verification

The automation of software safety has reached a significant milestone as artificial intelligence begins to handle the rigorous task of proving code correctness. This development addresses a longstanding challenge in the tech industry, where verifying that software is error-free has traditionally been a resource-intensive process reserved for high-stakes environments like aviation.

Anthropic and its partner Theorem utilized Claude to generate formal proofs for existing software. The team focused on 123 open-source libraries from Google's OSS-Fuzz security project. Over a period of just a few weeks, the model produced approximately 180 million lines of mathematical proof code. This approach differs from standard AI coding tasks because Claude did not write new software. Instead, it translated crash tests into mathematical problems and analyzed compiled machine code to create extensive proofs. An independent tool named Rocq automatically checks these proofs, a process the researchers describe as fully autonomous, although the underlying formal foundations were built by human experts at Theorem.

Historically, formal verification was prohibitively expensive. For instance, a team of specialists required 20 person-years to verify the seL4 operating system kernel. Consequently, the method was largely limited to critical systems such as Airbus avionics. By leveraging AI, Anthropic has dramatically reduced the time required for this work, making it feasible for a broader range of software applications.

The initiative yielded immediate practical benefits in the form of bug detection. While generating proofs, Claude identified hundreds of bugs that conventional testing had overlooked. Many of these issues only manifested under specific conditions, such as unusually large input files or particular access sequences. In one notable example, the team found as many security bugs in a widely used PDF reader in a single week as are typically reported for that software in an entire year. In another instance, Claude uncovered a logical error that had remained undetected for over a decade. Anthropic notes that even when a complete proof fails, the process still surfaces genuine defects.

Despite these successes, the technology has clear limitations. Anthropic clarifies that formal verification does not guarantee software is entirely error-free. The proofs are valid only under strict conditions, including specific processor architectures, single-threaded execution, and inputs under 64 MB. The compute costs are also substantial, with individual proofs consuming hundreds of millions of tokens. However, efficiency is improving. Newer models, such as Opus 5.5, solve half the proofs with about 40 percent fewer tokens than the previous version.

The project suggests a shift in how developers approach AI-generated code. If AI can write code and prove its safety under defined conditions, the focus moves from basic programming capability to verified correctness. This could reduce the time developers spend on manual reviews and lower overall costs. Anthropic expects formal verification with AI to become as routine as standard testing in the near future. Theorem’s roadmap already targets critical infrastructure libraries such as glibc, OpenSSL, and cURL, indicating a push toward broader application in essential software systems.