A newly discovered vulnerability in OpenSSL, known as “HollowByte,” has left a significant number of servers and applications at risk of being crippled by a denial-of-service (DoS) attack. The bug, identified by Okta’s red team, allows an attacker to exhaust a server’s memory before any security handshake can take place.
The vulnerability arises from the way older versions of OpenSSL pre-allocate receive buffer sizes based on the incoming message body length declared in the handshake message’s 4-byte header. An attacker can send a malicious payload of just 11 bytes that claims a larger incoming message body, triggering an unvalidated buffer allocation of up to 131 KB. This not only consumes valuable memory but also blocks the worker thread indefinitely, waiting for data that will never arrive.
The problem is exacerbated by the GNU C Library’s (glibc) behavior in retaining small-to-medium memory allocations for potential reuse and not immediately returning them to the OS when a connection drops. An attacker can launch successive waves of connections with randomized claimed sizes, preventing the allocator from reusing freed chunks and causing the server’s memory to become permanently bloated.
In real-world testing, Okta demonstrated that even on a 1 GB RAM system, a successful attack could freeze nearly half of the available memory after just 547 MB was fragmented. On a more robust 16 GB RAM system, the attack successfully locked up 25% of the system’s total memory while staying safely under the connection ceiling, rendering standard connection-limiting defenses ineffective.
The vulnerability affects various types of applications, servers, runtimes, and databases that rely on OpenSSL unless they have upgraded to a patched version. Thankfully, patches for HollowByte were silently included in OpenSSL version 4.0.1 and backported to versions 3.6.3, 3.5.7, 3.4.6, and 3.0.21. The latest versions of the library now increase buffer sizes as bytes actually land rather than relying on the handshake header for buffer growth.
It’s essential for system administrators and developers to be aware of this vulnerability and take immediate action by updating their OpenSSL installations to the latest patched version. Regularly monitoring and maintaining software dependencies is crucial in preventing such attacks, which can have devastating consequences for online services and applications.
Source: SecurityWeek — 2026-07-20