OpenSSL Servers Left Vulnerable to Memory-Consuming Attack
A newly discovered vulnerability in OpenSSL, dubbed HollowByte, allows unauthenticated attackers to trigger a denial-of-service (DoS) condition on vulnerable servers with just an 11-byte malicious payload. The issue has been silently patched by the OpenSSL team and backported to older releases.
The problem lies in the way vulnerable versions of OpenSSL handle TLS handshakes. When a message is declared, the server allocates memory for it before receiving the actual data. An attacker can exploit this by sending an 11-byte input with a header claiming that a much larger message body will follow. By repeating this process across multiple connections, the server’s memory becomes bloated, causing it to become unresponsive.
The issue affects all software projects that embed OpenSSL, including popular web servers like NGINX and Apache, language runtimes such as Node.js and Python, and databases like MySQL and PostgreSQL. In a test on NGINX, Okta researchers found that low-capacity environments can be easily depleted of memory using HollowByte, while higher-spec servers may lose up to 25% of their memory.
Although DoS flaws are generally considered less severe than vulnerabilities that enable data theft or code execution, they can still cause significant operational disruptions and reputational damage. The OpenSSL team has fixed the issue in version 4.0.1 and backported it to older releases, but organizations should prioritize upgrading to a fixed version of the library.
The root cause of HollowByte is due to how memory is allocated by the GNU C Library (glibc). When buffers are freed, glibc does not immediately return small-to-medium allocations to the operating system; instead, it keeps them for potential reuse. By launching waves of connections with randomized claimed sizes, an attacker prevents the allocator from reusing those freed chunks, causing the heap to fragment and the server’s Resident Set Size (RSS) to climb continuously.
To fully reclaim the space, a restart of the process is necessary. The OpenSSL team has categorized this issue as a “hardening fix” rather than a security vulnerability, but Okta recommends that organizations upgrade their distribution’s OpenSSL packages immediately.
In today’s threat landscape, it’s essential for security teams to stay vigilant and proactive in testing every layer of their defenses before attackers do. According to a recent study, 54% of successful attacks go undetected until they have already caused damage. Regular breach and attack simulation tests can help identify weaknesses in SIEM and EDR rules, ensuring that threats are caught before they cause harm.
In the wake of HollowByte, it’s clear that organizations must prioritize upgrading to a fixed version of OpenSSL and regularly testing their defenses against similar attacks. By taking these steps, businesses can minimize the risk of DoS disruptions and reputational damage caused by vulnerabilities like HollowByte.
Source: Bleeping Computer — 2026-07-17