As AI models increasingly take on tasks traditionally handled by humans, including writing code and identifying vulnerabilities, a disturbing trend is emerging: even when these systems are able to generate patches for security flaws, they often fail to effectively fix the problem. In fact, research suggests that only about half of all AI-generated patches successfully eliminate the underlying vulnerability.
A recent study conducted by identity management firm 1Password’s Off-By-1 research team reveals just how flawed this approach can be. The researchers generated over 6,000 patches using two large language models (LLMs): OpenAI’s ChatGPT-5.5 and Anthropic’s Opus 4.8. While the AI systems were able to produce effective patches for about half of the vulnerabilities tested, many of these “working” patches introduced new vulnerabilities or were easily bypassed by attackers.
The issue is not just with the quality of the patches themselves, but also with the fact that AI systems are often better at exploiting flaws than patching them. This asymmetry in capabilities has serious implications for defenders trying to keep up with the ever-evolving threat landscape. As Keith Hoodlet, director of security research for 1Password, points out, “Attackers are expanding their ability to attack more companies, more individuals, more codebases at scale in a way that I don’t think we’ve ever seen before.”
The study also highlights the limitations of relying on AI-generated patches. For six major open source flaws, the researchers found that only about 26% of patches successfully fixed the original vulnerability without introducing new security risks. The majority of patches (49%) failed to address the issue at all, while a significant number introduced new vulnerabilities.
This research has implications for efforts to patch vulnerabilities using AI before attackers can exploit them. As Hoodlet notes, “Because of AI and agent capabilities, defenders are under pressure to keep up with an ever-increasing number of attacks.” To stay ahead, he advises companies to focus on “burning down vulnerability backlogs” by investing time and energy into addressing existing security flaws.
The study’s findings are not isolated – application security firm Veracode has also found similar results in its own research. Across over 100 models and 80 coding tasks, the company found that AI-generated code had an average security pass rate of just 56%. This is concerning, given that nearly half of the generated code (44%) introduced detectable OWASP Top 10 vulnerabilities.
The take-away from this research is clear: while AI has the potential to revolutionize the way we approach cybersecurity, it’s not a silver bullet. Instead of relying solely on AI-generated patches, companies should focus on developing more robust security strategies that prioritize human oversight and verification. By doing so, they can minimize the risk of introducing new vulnerabilities and ensure that their systems remain secure in an increasingly complex threat landscape.
Source: Dark Reading — 2026-08-07