As security teams grapple with the increasing number of vulnerabilities in their applications, they’re turning to large language models (LLMs) for help. But a recent study has found that these AI-powered tools are not the silver bullet they seem to be.
The problem lies in the high false-positive rates associated with LLMs. In tests conducted by Arshan Dabirsiaghi, chief technology officer and co-founder at Pixee, an AI-powered application-security startup, more than 60% of flagged vulnerabilities were found to be either false positives, unreachable code, or low-severity issues. This means that security teams are wasting time and resources on non-essential patches, rather than focusing on the real threats.
The issue is not just with the LLMs themselves, but also with the context in which they’re being used. Dabirsiaghi argues that these models are “dumb” because they lack a deep understanding of the specific application and its deployment environment. As a result, they’re unable to provide accurate prioritization of vulnerabilities.
This is not just a minor issue; it has significant implications for security teams. With the number of valid vulnerabilities surging – estimates suggest that issues assigned a Common Vulnerabilities and Exposures (CVE) identifier will jump 50% this year – companies are struggling to keep up. The situation is further complicated by the fact that LLMs often require human checking and oversight, which can be time-consuming and costly.
The use of LLMs in vulnerability scanning has become a hot topic in recent years, with some vendors promoting their AI-powered tools as a game-changer for application security. However, Dabirsiaghi’s research suggests that these models are not yet ready for prime time. “Companies don’t remotely have the manpower to scale” human-led vulnerability triage, he says.
So what can be done? According to Dabirsiaghi, good triage needs three sorts of context: organizational context, technical context, and wider code context. This means that security teams need to provide more information about their applications and deployment environments in order for LLMs to be effective. It’s also essential to use task-specific scanning tools rather than relying solely on generalist AI models.
Finally, Dabirsiaghi highlights the importance of reachability in vulnerability triage. By measuring which code is actually used at runtime, security teams can significantly reduce the amount of code that needs to be scanned and patched. For example, one study found that 62% of open source libraries are never used at runtime, while another found that only 12% of Java code is used.
In conclusion, while LLMs hold promise for application security, they’re not yet ready for widespread adoption. Security teams need to be aware of the limitations and challenges associated with these tools and focus on providing the necessary context and using task-specific scanning tools to get the most out of them. By doing so, they can improve their vulnerability triage processes and reduce the risk of false positives and over-fixing.
Source: Dark Reading — 2026-07-21