Fuzzing rust code: cargo-fuzz and honggfuzz

This post explains how to test Rust code using fuzzers.

Parsers are good target for fuzzers, especially because they usually are functions that only takes bytes as input.

Preamble: why fuzz Rust code ?

Since Rust provides some kind of memory safety, it may first appear strange to fuzz Rust code. However, there are some kind of bugs that a fuzzer will help you find, including:

debug or unfinished code, like unimplemented! and panic! calls
out of range accesses, like array[i]
integers overflows/underflows, like base + offset (see end of article)
stack overflows, unbound recursions
crashes in unsafe code
direct calls to std::process::exit
timeouts and functions that take too long

The last kind of bugs is very similar to the previous list, but is quite annoying:

functions from other crates, even std, which can panic (like Add, Sub and other ops for Duration)

Note that fuzzing has other advantages, like acting as regression tests when you change/update code, and can even be used as functional testing if you add assertions.

The good news is that some tools are well integrated with cargo, making it very easy to use. We’ll show how to use 2 tools: cargo-fuzz and honggfuzz.

cargo-fuzz

cargo-fuzz is a nice command-line wrapper around libFuzzer, an LLVM library for coverage-based fuzzing.

Install cargo-fuzz:

$ cargo install cargo-fuzz

Note that you will need the nightly version of the toolchain.

Note: I’m using stable as default, so all commands requiring nightly will explicitly set +nightly.

Initialize directories:

$ cargo +nightly fuzz init

This will create the fuzz/ directory, where fuzzers code and data will live.

We’ll then add a new fuzzer. I’ll take the example of the parse_der function of der-parser (DER is always a good target for fuzzing).

$ cargo +nightly fuzz add fuzzer_parse_der

It will create fuzz/fuzz_targets/fuzzer_parse_der.rs with template code, and add stuff to fuzz/Cargo.toml so it builds. Edit the fuzzer_parse_der.rs, and add a call to the targeted function:

#![no_main]
extern crate libfuzzer_sys;
extern crate der_parser;
#[export_name="rust_fuzzer_test_input"]
pub extern fn go(data: &[u8]) {
    let _ = der_parser::parse_der(data);
}

That’s all you need to start. Run the fuzzer:

$ cargo +nightly fuzz run fuzzer_parse_der

It will show you the progress of execution. To understand output, see libFuzzer Output. Many items are interesting:

NEW items show that a new code path has been discovered. With time, NEW events will appear less often
cov and ft give an idea of the current coverage (edges and comparisons)
exec/s shows how many times per second the function has executed. This should remain high for the fuzzer to be efficient.

Next, a few tips that will make fuzzing much more efficient.

Use a corpus

libFuzzer is a mutation-based fuzzer. If you give it enough time, it should be able to find all execution paths and branches. However, discovery of new paths can be slow.

Providing examples (as much as possible, good and bad) in the corpus makes this process much faster. This is especially true if the input data should be structured.

Simply copy files to the fuzz/corpus/fuzzer_parse_der/ directory.

Neat point: if your fuzzer is already running, do nothing more! Files copied to the corpus are automatically detected, and the fuzzer will issue a RELOAD.

Use parallelism

cargo-fuzz can start many processes (instead of 1 by default) using the --jobs n option.

$ cargo +nightly fuzz run --jobs 24 fuzzer_parse_der

All processes share the same corpus, and paths discovered by one process are automatically used by others (since a new item is added to corpus, others will reload it).

Handling crashes

When a crash is detected, the input will be saved in the fuzz/artifacts/<fuzzer_name>/ directory.

This is useful, especially for testing a candidate fix. Giving the name of a file as input argument of cargo-fuzz will run the fuzzer only for this file.

$ cargo +nightly fuzz run fuzzer_parse_der fuzz/artifacts/<fuzzer_name>/<input_file>

Minimize corpus

libFuzzer adds new samples to the corpus for every new path. With time, the corpus can grow to become very large, which will slow down fuzzing and become hard to manage (please do not commit a corpus of several gigs to github when your source code is only a few kilobytes!).

The cmin subcommand can be used to minimize corpus examples, while preserving coverage.

$ cargo +nightly fuzz cmin fuzzer_parse_der

Also fuzz in release mode

By default, cargo-fuzz uses the debug mode (which is good, because operations on integers are only instrumented in debug mode by default). Fuzzing in release mode has several advantages: it is much faster, and it provides a target closer to the code that will be executed in the end.

Just add --release to command-line arguments:

$ cargo +nightly fuzz run --jobs 24 --release fuzzer_parse_der

Note that fuzzing in release mode is complementary to debug mode, but does not replace it.

Visualizing code coverage

After torturing the parser, I want to look at the coverage of source code by the current corpus. For this I used kcov, passing the entire corpus as argument of the fuzzer:

$ cd fuzz
$ mkdir cov
$ kcov ./cov ./target/debug/fuzzer_parse_der corpus/fuzzer_parse_der/*

with not much success:

WARNING: Failed to find function "__sanitizer_acquire_crash_state".
WARNING: Failed to find function "__sanitizer_print_stack_trace".
WARNING: Failed to find function "__sanitizer_set_death_callback".
INFO: Seed: 631975293
kcov: Process exited with signal 11 (SIGSEGV) at 0x5555555a8f75

After fighting a bit with search results, I found this kcov issue which helped solving the problem. After adding some arguments (to include source path for the fuzzer and the der-parser lib), kcov worked:

kcov --include-path .,.. ./cov ./target/debug/fuzzer_parse_der corpus/fuzzer_parse_der/*

Overview

honggfuzz

honggfuzz is another fuzzer, maintained by Google, easy to use in Rust using honggfuzz-rs.

I have not a complete comparison of of honggfuzz versus cargo-fuzz. Here are a few points that looks important to me:

cargo-fuzz makes it easier to create multiple fuzzers in the same project. It is more efficient to write multiple, small fuzzers if your project can parse several formats. It is faster, and makes managing corpus easier.
honggfuzz seems faster in terms of execution per seconds (not sure why, just looking at the numbers)
honggfuzz does not require nightly

Similarly to the fuzz directory of cargo-fuzz, I used a hfuzz directory to store the project.

Create a new crate:

$ cargo new --bin hfuzz

Enter this directory, and edit the Cargo.toml file:

# ...
[dependencies]
honggfuzz = "0.5"

[dependencies.der-parser]
path = ".."

Edit src/main.rs file, and call the targeted function:

#[macro_use]
extern crate honggfuzz;
extern crate der_parser;

fn main() {
    println!("Starting fuzzer");
    loop {
        // The fuzz macro gives an arbitrary object (see `arbitrary crate`)
        // to a closure-like block of code.
        // For performance reasons, it is recommended that you use the native type
        // `&[u8]` when possible.
        // Here, this slice will contain a "random" quantity of "random" data.
        fuzz!(|data: &[u8]| {
          let _ = der_parser::parse_der(data);
        });
    }
}

Build and run fuzzer:

$ cargo hfuzz build
$ cargo hfuzz run hfuzz

This will display the fuzzing console, with some log output and current progress display.

Parallelism

By default, honggfuzz uses half the logical number of CPUs. To set the number of workers, use the -n argument:

$ HFUZZ_RUN_ARGS="-n 12" cargo hfuzz run hfuzz

Corpus

honggfuzz stores its corpus (by default) in hfuzz_workspace/<fuzzer_name>/input/.

Note that the corpus is similar to the one used in libFuzzer or cargo-fuzz, so files can just be copied to populate the initial corpus.

Exit upon crash

Unlike cargo-fuzz, honggfuzz will continue when the target function crashes. While this can be interesting in some cases, during the development phase it is easier to stop the fuzzer and fix the bug before continuing.

To do that, add --exit_upon_crash to HFUZZ_RUN_ARGS.

Release mode

By default, honggfuzz uses instrumented release mode. To build (run) without instrumentation, or in debug mode, use the following targets: build-no-instr, build-debug (run-no-instr, run-debug) instead of build (run).

honggfuzz arguments

The list of possible arguments for HFUZZ_RUN_ARGS can be displayed using:

$ cargo honggfuzz --help

Digression on checked integer operations

In Rust, the overflowing_xxx and checked_xxx family of functions helps detecting overflows, returning the result of operations and a boolean set to true if an operation occurred.

This is great, because a parser should be careful with overflows (especially when adding numbers coming from untrusted sources like the network).

However, the checked_shl and overflowing_shl are misleading in the performed check: they do not check for overflow, but that the number of bits shifted is not greater than the representation of the left argument. While this is important to test, this is not sufficient to detect shift overflows.

See this rust playground example for a silent overflow.

Note that this is usually the case for the CPU instructions (sal for x86, lsl, etc.) where the shifted bit goes to the carry flag, so if you shift several bits and the last is a 0, the overflow cannot be detected using carry. LLVM does not provide intrinsics to test it.

As such, it is neither a bug nor a new thing, but something that you have to be careful with.

To test for an overflow when shifting bits, you have to test manually.

Other articles

UEFI SecureBoot on Debian
Date Thu 15 October 2015 Tags Security Debian UEFI x509 PKI SecureBoot

This post explains how to enable UEFI SecureBoot on Debian, using your own trust chain. The technical part itself is very light, most of the post is explanations and what and why.

What is SecureBoot ?

UEFI SecureBoot is a mechanism to verify a cryptographic signature of UEFI Images before loading them into the Firmware (the new name for the BIOS¹). This provides a way to control which images are allowed, and also drivers and option ROM used by the Firmware, and to fight bootkits and malwares based on that. For an example of such dangers, see my past presentations on malicious UEFI Option ROMs ([FR] at SSTIC, and [EN] at PacSec).

Roughly, SecureBoot will rely on cryptographic signatures (mainly using SHA-256 and RSA-2048) that are embedded into files using the Authenticode file format. The integrity of the executable is verified by checking the hash, and the authenticity and the trust by checking the signature, based on X.509 certificates, which has to be trusted by the platform.

At a high level, the Firmware has 4 different set of objects (see figures for details):
- the Platform Key (PK): this is the main key. This keys, usually belonging to …
read more
Debian SELinux Documentation
Date Tue 04 October 2011

When I installed this server, I have decided to enable SELinux, and run it in enforcing mode. And it works !

Finding relevant documentation for SELinux and Debian was more difficult than expected, and even when it exists, it is often outdated. Also, It does not give real examples, and one problem I encountered very often is a policy module with correct labels, but created for another distribution, and thus not labeling Debian packages correctly. Some other distros (Gentoo and Fedora) have made huge progress on RBAC security, it would be nice to see the same on Debian.

I have started a guide for SELinux on Debian. The goals are to:
- give a practical approach on using SELinux on Debian,
- fight some false ideas (like not being usable on a Desktop, or that you have to enable it globally etc.),
- describe how to mix confined and unconfined services,
- give some examples (Cyrus IMAP, git, redmine, PostgreSQL, etc) see the examples page
- explain how to use it with PaX/Grsec
- give some example on using the
- give some generic hardening tips.
Please notes that the examples and solutions given in the guide are only my own explanations and solutions, and that of …
read more
Project page for SIEM-live
Date Fri 19 November 2010 Tags Security IDS Debian

I’ve created a project in redmine for SIEM-live, so there is now a wiki, a tracker, and a repository. I’ll add some documentation and instructions on how to build the CD soon.

Contributors would be gladly accepted :)

I’ve also updated the Git repository for recent versions of live-build, where all variables have been renamed without keeping compatibility :/

The bug where booting with no network (no DHCP, for ex.) made many command fail with a weird error message has been fixed:
```
could not resolve 127.0.0.1: address family for hostname not supported
```
For the record, this was caused by .. IPv6 ! Disabling it during the configuration sequence fixes the problem.
- Project page: https://www.wzdftpd.net/redmine/projects/siem-live
read more
LE pare-feu OpenOffice
Date Wed 16 June 2010 Tags OpenOffice Firewall

Internet ça a du bon. Enfin, pas toujours .. Et en combinant une jolie boulette avec quelques outils, on peut réaliser des choses dont personne n’aurait pensé au début …

En bref, et notamment pour faire suite au challenge que certaines personnes m’ont lancé au SSTIC 2010 (très bonne année d’ailleurs), le voila: LE pare-feu OpenOffice ! Mais attention: pas un truc codé à l’arrache, nan, un vrai pare-feu avec un design toussa

Donc, comment ça marche:
- on veut filtrer des paquets avec OpenOffice, donc on a besoin d’OpenOffice
- n’ayant pas eu le temps de planifier le portage d’OpenOffice en mode noyau, j’ai donc fait l’inverse: amener les paquets en espace utilisateur avec nfqueue + python
- on manipule OO avec python-uno
Donc, je commence à coder: on crée une feuille calc avec les numéros de ports à filtrer, je récupère la valeur des cellules (et là je peux vous le dire: quand on a l’habitude de LaTeX, OO c’est pas la joie), et on s’en sert pour filtrer les paquets récupérés par nfqueue.

Premier problème: pour utiliser nfqueue, il faut être root. Et faire tourner OpenOffice en root (donc avec une interface …
read more
Creating a live cd for open source SIEM Prelude and Suricata
Date Fri 05 February 2010 Tags Debian Security IDS Prelude Suricata Snort

I have started to work on a Live CD for Open Source tools like Prelude SIEM, and software like Suricata, Snort, OpenVAS to send alerts. The goal is to easily test these tools, register new agents, get some alerts and be able to correlate them etc. I also want to add some visualization tools, so this CD could maybe become a reference for security alert detection and report.

First, a few points on applications used:
- Debian Live for building the CD. It’s very easy, it’s based on Debian, and it allows me to re-use some work I’ve done
- Suricata IDS, which is a very promising project
- Snort IDS, with the free signatures
- OpenVAS to be able to generate alerts
- Prelude SIEM is the key point: suricata, snort, syslog etc. will send alerts to Prelude, which has a database, a correlator, a web interface (Prewikka) etc.
- Standard useful tools: nmap, scapy, wireshark, p0f, etc.
This first version is based on Debian Lenny and arch x86. Everything is based on packages (.debs) to make it easier to maintain, upgrade versions or add patches: most of the time, I just have to rebuild packages from squeeze or sid.

The build …
read more
implementing the evil maid attack on linux with Luks
Date Wed 28 October 2009 Tags Security Linux Encryption

This month, Joanna Rutkowska implemented the “evil maid” attack against TrueCrypt.

This kind of attack can be done on any OS with disk encryption: when using whole-disk encryption, you have to infect to bootloader. Linux includes dm-crypt/LUKS, which has some nice features (including TKS1 and working encrypted suspend-to-disk). But how does it play with this attack ?

Sadly, the answer is: pretty bad. LUKS has no protection against this attack, and even requires a /boot partition in clear. Before looking at the possible solutions, we’ll play with the /boot partition to see how simple the attack is.

Linux boot sequence basics

The boot sequence (See http://www.ibm.com/developerworks/library/l-linuxboot/index.html) is the following:
- System startup: the BIOS is loaded, searches for a boot medium, loads the MBR, and yields control to it.
- Boot loader stage 1: the job of the primary boot loader is to find and load the secondary boot loader (stage 2)
- Boot loader stage 2: its jobs is to search and load the Linux kernel and initial RAM disk (initrd) images.
- Linux kernel: it starts by uncompressing itself, then mounts the initrd image. This image contains modules and scripts required to find …
read more
Playing with OpenDPI
Date Thu 17 September 2009 Tags Security DPI

So, Ipoque has published its deep inspection engine under a free license (LGPLv3): OpenDPI This is always good news when a company decides to release source code to the community, so first of all thanks to Ipoque for this.

After downloading the source code on OpenDPI google project’s page, I started to look at it.

Basically, the project looks quite unprepared for release (only a Makefile, no configure script - though no-one can be blamed for not using autotools -), but after looking at the code it seems not so bad:
- the code is reasonably clean
- it builds fine on x86 or x86_64 platforms
- the code is provided with a decent list of identified protocols
- the demo uses pcap files
There are a few minor annoyances:
- the provided lib is a static lib … building a shared library would be better !
- the build system is pretty awful, rebuilding everything each time, without using deps, no install system etc.
- no docs (looking at the demo file was sufficient to understand most of the function).
- no correct website, forums or whatever. I’m sure it will get better in the future
- pcap only
This last point was the most annoying to me, so I …
read more
LinkedIn group for Prelude

Date Mon 01 June 2009 Tags Work Prelude SIEM

I’ve just created a LinkedIn group for Prelude IDS.

All Prelude users are welcome to join the Prelude IDS group to stay in touch with other Prelude users, use the forums, get news etc.

read more
New Syslog RFCs
Date Wed 27 May 2009 Tags Logs

Several new RFCs for syslog have been issued in March:
So what are the improvements since the previous RFC (3614), especially in RFC5424 ¹ :
1. In section 5.1, “Minimum Required Transport Mapping”:
  
  All implementations of this specification MUST support a TLS-based transport as described in RFC5425.
  Yay ! So they discovered TLS, that’s great. Especially since RFC 5425 supports certificates authentication (section 4.2.1), certificate path validation, fingerprints, etc. 2. Improved timestamps (Section 6.2.3) with supports for milliseconds, time zones, UTC offsets 3. Section 6.3 describes structured data (name-value pairs) 4. Section 7: Structured Data IDs
  This allows using an enterprise ID (registered to the IANA) for the structured data elements
However, nothing really useful on reliability (resending events, making sure they were delivered, etc.) except the very poor (and useless) section 8.5, which only acknowledges the lack of support :/ Well, Prelude IDS can do that pretty good.

Also, nothing on taxonomy, though it may be improved with structured data. However, it would require a good definition of events …
read more