Monday, September 21, 2015 At 2:12PM
Introduction
GDS recently discovered a client-side NULL pointer dereference in all versions of mbed TLS using afl-fuzz. The issue was reported to ARM who quickly released fixes for all affected versions. See below for further details.
Fuzzing is a testing technique that feeds a program under inspection with random inputs and observes its behavior. The goal of fuzzing is to discover bugs in the inspected program. Michal Zalewski’s fuzzer american fuzzy lop (afl) is currently one of the most sophisticated open source fuzzers due to its unique approach to fuzzing. Afl utilizes compile-time instrumentation and genetic algorithms to discover test inputs that crash the program under inspection. The fuzzer has gained popularity due to finding plenty of bugs in open source projects. The findings include the recent vulnerability in the BIND9 DNS server which enables attackers to crash the DNS server via a crafted TKEY query. Hanno Boeck reported that the Heartbleed vulnerability in OpenSSL which allows attackers to read parts of the server memory could have been found using afl. A popular alternative to OpenSSL is mbed TLS. The library was formerly called PolarSSL and is owned by ARM Limited. Inspired by Hanno’s OpenSSL fuzzing efforts, this blog post describes how afl can be used to fuzz the mbed TLS 2.0.0 and mbed TLS 1.3.12 libraries.
Creating a Fuzzable Instance
The Transport Layer Security (TLS) protocol is a commonly used protocol which allows a TLS client and a TLS server to establish an encrypted and integrity-protected communication channel over a computer network such as the Internet. TLS is the successor of the Secure Sockets Layer (SSL) protocol which was developed by Netscape. SSL has been considered insecure since the advent of the POODLE attack in 2014. TLS/SSL serves as the basis for many other network protocols such as HTTPS which allows web browsers to securely communicate with web servers. RFC 5246 defines TLS 1.2 which is the latest released version of TLS. More high-level descriptions of TLS/SSL are available from Microsoft, a blog post by Alvaro Castro-Castilla, and the tls.openmirage.org test page.
As TLS is a network protocol, the network packets which the client and server applications exchange constitute interesting candidates for fuzzing. A network packet that crashes the client or server application exhibits a severe vulnerability, as attackers who can communicate with the vulnerable application only need to replay the corresponding packet (and potentially previous packets) in order to mount a Denial-of-Service (DoS) attack. Depending on the nature of the vulnerability, it might also be possible to gain control over the application and the underlying operating system.
To be able to fuzz the network packets, a program that runs the TLS protocol has to be implemented. The program needs to support reading program input from files and use it as the content of a specific network packet. Implementing such a program poses several challenges. A straightforward approach to implement a run of the TLS protocol would be to launch a client thread and a server thread. While it is possible to fuzz a multi-threaded binary, it is highly recommendable to stick to a single thread to obtain predictable results. Therefore, our approach is to implement a single-threaded TLS client and server combination. Hanno Boeck also adopted this approach to fuzz OpenSSL for the Heartbleed vulnerability.
The source code under programs/ssl/ in the mbed TLS source folder served as a starting point for our implementation of a fuzzable program. In particular, the files ssl_server.c and ssl_client1.c provide suitable TLS server and client implementations. The server and client code has to be intertwined accordingly such that the program completes a TLS handshake and exchanges data over the TLS channel. One obvious challenge here is to handle the communication in a way that prevents the client and server from infinitely waiting for a response. In the handshake phase, for example, the client and server may send and receive several packets in a row. Simply letting the client and server carry out one handshake step (which involves merely sending or receiving a packet) in a loop is not sufficient, as the client or server may block waiting to receive a packet that the other side has not sent. Instead, it is necessary to let the client and server execute a specific number of handshake steps in each iteration of the loop until both finish the handshake. As the client and server run in the same thread, it is desirable to let the client and server communicate over a buffer in memory rather than network sockets for performance reasons. Therefore, our program utilizes memory buffers by default, but users can instruct the program to use network communication instead. An additional challenge is to render the program deterministic in order to allow for thorough testing with predictable results. To make the implementation deterministic, we replaced the random number generator function with a function that returns a nonzero constant (using zero is not possible due to requirements for key generation). Moreover, the client and server send a timestamp as part of their Hello messages. Therefore, we modified mbed TLS to use a constant timestamp to achieve deterministic behavior. In addition to reading the content of a network packet from a file, our program allows for saving all network packets that are exchanged in the deterministic program run as separate files.
Fuzzing Stage
Afl requires the target binary to be compiled with instrumentation capabilities. Moreover, we have to instrument the mbed TLS library. To do this, we wrote a compilation script that builds mbed TLS and our test program using afl’s best optimizing compiler afl-clang-fast. The llvm_mode/README.llvm of afl states that using afl-clang-fast “some slow, CPU-bound programs will run up to around 2x faster.”
In addition to the compilation script, we wrote a script to automate the fuzzing process. The script expects the number of the packet to fuzz and the fuzzer number as its arguments. Our script is able to leverage the parallel fuzzing capability of afl. To run the afl fuzzing tasks in parallel on different processor cores, it is necessary to set up a master instance by setting the fuzzer number argument of the script to 1. The script then sets up the folder structure including the test input and output folders, generates the initial network packet to fuzz, and launches the master instance. All other fuzzing instances are slaves. Our script provides for a convenient way to launch slaves.
It is important to note that memory access errors do not necessarily lead to a program crash. If that is the case, afl will not be able to detect the bug. However, it is possible to compile the targeted code with the powerful LLVM/Clang tools AddressSanitizer (add AFL_USE_ASAN=1 to the environment) or MemorySanitizer (add AFL_USE_MSAN=1 to the environment) activated and then run afl. Memory access error will now crash the target program. Compiling the binary with the environment variable AFL_HARDEN=1 set, also facilitates detecting simple memory access errors. While the tools enable afl to detect more bugs, the runtime performance of the compiled binary decreases. On 64-bit systems, AddressSanitizer has to be used with caution, since it reserves a lot of virtual memory which can render the system unstable. Under Linux, it is possible to use the script experimental/asan_cgroups/limit_memory.sh that ships with afl to leverage cgroups to restrict the resource usage. A tutorial of the Fuzzing Project by Hanno Boeck and the afl documentation located at docs/notes_for_asan.txt provide further details.
The following screenshot shows the master instance of afl fuzzing the fourth network packet.
Our fuzzing scans discovered client-side NULL pointer dereference vulnerabilities in mbed TLS which affect all maintained versions of mbed TLS. ARM fixed the issues in versions 2.1.1, 1.3.13 and PolarSSL version 1.2.16. See the release notes for details.
Conclusions
In this blog post, we explored an approach to fuzz the mbed TLS library using afl. The tool was able to discover previously unknown vulnerabilities in mbed TLS. Going forward, we aim that our tool remains useful for testing mbed TLS. Moreover, we hope that the description of our efforts aids others in using afl to find bugs in (network protocol) libraries. Since June 2015, afl supports the so called persistent mode which avoids the overhead of constantly forking a new process for every fuzzing input. The idea of persistent mode is to let afl iteratively feed test input into a long-lived process. In each iteration, the process resets the state, reads the test input from afl, and executes the code under inspection. While persistent mode is still considered experimental, it potentially facilitates the process of making programs fuzzable and offers promising performance benefits.
Our test program and scripts are available on GitHub: https://github.com/GDSSecurity/mbedtls-fuzz/
Acknowledgements
We thank Michal Zalewski for creating the wonderful fuzzer afl, Hanno Boeck for starting the Fuzzing Project and his Heartbleed rediscovery efforts using afl, and the mbed TLS team at ARM for their rapid response to this issue.
Author: Fabian Foerg
©Aon plc 2023
