( Pollux's blog )

The Prelude Correlator is now packaged in Debian.

From the description of the package:

Prelude is a general-purpose hybrid intrusion detection system.
.
This package provides the Prelude Correlator, which is a powerful
correlation engine using Lua to write correlation rules.
.
The features currently include:
 * Rapid identification of important security events, enabling the analyst to
   assign task priorities
 * Alert correlation originally from heterogeneous sensors deployed on the
   whole infrastructure
 * Real-time analysis of events received by the Prelude Manager

You can contribute ! If you use the correlation engine, please share your correlation rules.

Related links:

• Technical documentation
• Download Prelude Correlator
• Lua official site

Since recent versions, entropy generation is very slooooooow on Linux (mainly because the entropy generation algorithm has been changed for security). This can be really painful when generating certificates, for ex. for Prelude. Reading /dev/random only gives a few bytes per second !

If you're lucky, you will have a hardware random generator (RNG). However this is generally not the case, so how to speed up random generation ?

One solution is to take random data from another source and then inject if to the kernel random pool. You have to be careful though, because if you read data from /dev/null it will be fast, but you will have problems !

Install rng-tools, which reads data from a configurable input source (hardware RNG, or, in our case, /dev/urandom). Before injecting data to the kernel, rng-tools test data with the FIPS 140-2 test, and then add it.

Warning: if you want to generate, for ex, certificates for Prelude, you must fully install and configure rng-tools before installing Prelude !

apt-get install rng-tools
vi /etc/default/rng-tools

Change the values, for ex:

HRNGDEVICE=/dev/urandom
RNGDOPTIONS="-W 80% -t 20"

The second line tells rng-tools to fill 80% of the kernel random pool (default is 50%) every 20 seconds maximum (default is 60). You may want to adjust these not to be too aggressive, or your "tainted" random data will completely replace real random data. See man rngd for more information.

/etc/init.d/rng-tools start

Caution: if you want 100%-sure random data, then stick to the default random generator, or use other libraries

The code for nfqueue-bindings is now almost ready, I have made some progress since last week:

* you can now modify packets in live, and send the new packet with the verdict
* new functions are wrapped, and the creation of the queue can be done in one function
* more examples

I have presented a special script for SSTIC, using the weather to decide if a packet should be accepted or dropped :) While the utility of the module still has to be proven, it is a good example of how easy it is to use the new bindings.

The slides can be found online here, and contains some code examples (with some funny things ;). They are in french, but they should be quite easy to understand.

Random ideas:

* The Netfilter workshop will be held in Paris from 30 September to 3 October 2008.
* Eric has presented nf3d, a nice tool to view netfilter logs (from ulogd2) in 3D.

Gamers will recognize a nice try to convert network logs into Guitar Hero tracks ;)

* Some people have weird habits at SSTIC !

I am currently working (amongst other projects ..) on nfqueue-bindings, set of high-level modules for several languages (Python and Perl, for the moment), for libnetfilter_queue.

The goal is to provide a library to gain access to packets queued by the kernel packet filter. For more details, see nfqueue-bindings project site.

Current state

Actually, you can

* access the module in Perl or Python
* create a queue connected to netfilter
* register a callback
* access the contents of the packet. As I do not want to do what was already done many times, I use some other libraries to decode the packet: 
 * NetPacket::IP for Perl
 * dpkg for Python.
 * If you know some other libraries, please let me know.
* set the verdict (decision) to ACCEPT or DROP for the packet

I have written some scripts to show what can be done in a few lines of code. The current examples are:

* example Perl script
* example Python script
* Packet dumper, in pcap format (use scapy)
* HTTP request checker
* A surprise I will present in a lightning talk at SSTIC :)

I will make a release as soon as the code is stable (and can be installed).

Examples

Create and bind the queue (Perl)

use nfqueue;

use NetPacket::IP qw(IP_PROTO_TCP);
use NetPacket::TCP;

my $q;

sub cb
{
...
}

$q = new nfqueue::queue();
$q->open();
$q->bind();
$q->set_callback(\&cb);

$q->try_run();

Create and bind the queue (Python)

import nfqueue
from dpkt import ip

q = None

def cb(dummy, payload):
  ...

q = nfqueue.queue()
q.open()
q.bind()
q.set_callback(cb)
q.create_queue(0)

q.try_run()

Decode packet (Perl)

Now all we have to do is to use the callback !

my $ip_obj = NetPacket::IP->decode($payload->get_data());
print("$ip_obj->{src_ip} => $ip_obj->{dest_ip} $ip_obj->{proto}\n");

if($ip_obj->{proto} == IP_PROTO_TCP) {
  my $tcp_obj = NetPacket::TCP->decode($ip_obj->{data});
  	print "TCP src_port: $tcp_obj->{src_port}\n";

Decode packet (Python)

data = payload.get_data()
pkt = ip.IP(data)
print "proto:", pkt.p
print "source: %s" % inet_ntoa(pkt.src)
print "dest: %s" % inet_ntoa(pkt.dst)
if pkt.p == ip.IP_PROTO_TCP:
  print "  sport: %s" % pkt.tcp.sport
  print "  dport: %s" % pkt.tcp.dport

Set verdict

Perl:

$payload->set_verdict($nfqueue::NF_DROP);

Python:

payload.set_verdict(nfqueue.NF_DROP)

Links

* libnetfilter_queue
* nfqueue-bindings project site
* nfqueue-bindings source (git browser)
* Get source: git clone http://git.inl.fr/git/nfqueue-bindings.git

(See part 1)

Remember: you must compile with -02 if you want the checks to be effective

DEB_BUILD_HARDENING_FORTIFY (gcc/g++ -D_FORTIFY_SOURCE=2)

The idea behind FORTIFY_SOURCE is relatively simple: there are cases where the compiler can know the size of a buffer (if it's a fixed sized buffer on the stack, as in the example, or if the buffer just came from a malloc() function call). With a known buffer size, functions that operate on the buffer can make sure the buffer will not overflow.

Example:

void foo(char *string)
{
	char buf[20];
	strcpy(buf, string);
}

Execution will fail:

[home ~/harden] ./bad $(perl -e 'print "A"x100')
zsh: segmentation fault  ./bad $(perl -e 'print "A"x100')

When compiling with -D_FORTIFY_SOURCE=2, gcc will add some checks to detect the overflow and terminate the program:

[home ~/harden] DEB_BUILD_HARDENING=1 make
[home ~/harden] ./bad $(perl -e 'print "A"x100')

*** buffer overflow detected ***: ./bad terminated
======= Backtrace: =========
/lib/libc.so.6(__fortify_fail+0x37)[0x2ba8d18fb787]
/lib/libc.so.6[0x2ba8d18f9e70]
./bad(main+0x26)[0x555555554856]
/lib/libc.so.6(__libc_start_main+0xf4)[0x2ba8d18411c4]
./bad[0x555555554789]
======= Memory map:  ========
2ba8d1607000-2ba8d1622000 r-xp 00000000 03:01 468316                     /lib/ld-2.7.so
2ba8d1622000-2ba8d1625000 rw-p 2ba8d1622000 00:00 0 
2ba8d1821000-2ba8d1823000 rw-p 0001a000 03:01 468316                     /lib/ld-2.7.so
2ba8d1823000-2ba8d1961000 r-xp 00000000 03:01 471074                     /lib/libc-2.7.so
2ba8d1961000-2ba8d1b61000 ---p 0013e000 03:01 471074                     /lib/libc-2.7.so
2ba8d1b61000-2ba8d1b64000 r--p 0013e000 03:01 471074                     /lib/libc-2.7.so
2ba8d1b64000-2ba8d1b66000 rw-p 00141000 03:01 471074                     /lib/libc-2.7.so
2ba8d1b66000-2ba8d1b6c000 rw-p 2ba8d1b66000 00:00 0 
2ba8d1b6c000-2ba8d1b82000 r-xp 00000000 03:01 1045795                    /lib/libgcc_s.so.1
2ba8d1b82000-2ba8d1d82000 ---p 00016000 03:01 1045795                    /lib/libgcc_s.so.1
2ba8d1d82000-2ba8d1d83000 rw-p 00016000 03:01 1045795                    /lib/libgcc_s.so.1
555555554000-555555555000 r-xp 00000000 03:07 1225                       /home/pollux/harden/bad
555555754000-555555755000 r--p 00000000 03:07 1225                       /home/pollux/harden/bad
555555755000-555555756000 rw-p 00001000 03:07 1225                       /home/pollux/harden/bad
555555756000-555555777000 rw-p 555555756000 00:00 0                      [heap]
7fffd948e000-7fffd94a3000 rw-p 7ffffffea000 00:00 0                      [stack]
7fffd95fe000-7fffd9600000 r-xp 7fffd95fe000 00:00 0                      [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
zsh: abort      ./bad $(perl -e 'print "A"x100')

The difference is that there is no segmentation fault, because the program was properly terminated, so no exploit possible. A backtrace is also printed to help.

It is possible to detect if a program is using such checks by using objdump:

[home ~/harden] objdump -d bad | grep call |grep _chk
 84b:   e8 c0 fe ff ff          callq  710 <__stack_chk_fail@plt>

If there is one line or more, FORTIFY_SOURCE is active. However if the output is empty, FORTIFY_SOURCE either might not be enabled, or the program code is such that FORTIFY_SOURCE is not applicable (for example: secure code that has no static buffers and always checks buffer sizes, and thus FORTIFY_SOURCE cannot identify any potentially dangerous operations in the program).

A firewall has to find the difference between good and bad packets, and for this, nothing is better than humans ! (french people could add this is the same difference as for good and bad hunters).

So the next generation firewall will be:

• better than stateful
• better than layer 7 analysis
• compliant with encrypted traffic
• able to detect malware, suspicious traffic, virus, etc.

Preview screenshot:

Source code should be released on monday, on the Netfilter mailing lists. Stay tuned !

Since recent versions (>= 4.0, maybe before), gcc (and ld) has some nice security features. Debian has created a wrapper for the toolchain, to make the use of these features easy.

To install the wrapper, run:

apt-get install hardening-wrapper

To enable the hardening features, you have to export the environment variable:

export DEB_BUILD_HARDENING=1

The features include additional checks for printf-like functions, stack protector, using address-space layout randomization (ASLR), marking ELF-sections as read-only after loading when possible, etc.

Please note that you must compile with -02 if you want the checks to be effective

DEB_BUILD_HARDENING_FORMAT (gcc/g++ -Wformat -Wformat-security)

Ask gcc to make additional checks on format strings, to prevent attacks.

The following code, for ex:

printf(buf);

will result in a warning:

[home ~/harden] DEB_BUILD_HARDENING=1 make
gcc     bad.c   -o bad
bad.c: In function ‘main’:
bad.c:10: warning: format not a string literal and no format arguments

Why is this code vulnerable ? Because the buffer (buf) could contain format characters like %s, and the printf function will interpret these characters to pop arguments from the stack, and can result in the execution of arbitrary code.

Solution:

• Replace previous code by

printf("%s",buf);

• Remember this is also true for other functions like syslog()
• Tell gcc to help ! You can mark your own printf-like functions using gcc attributes, for ex:

void out_log(int level,const char *fmt,...)
#ifdef __GNUC__
  __attribute__((__format__(printf,2,3)))
#endif
;

DEB_BUILD_HARDENING_RELRO (ld -z relro)

When loading a program (which is handled by ld-linux.so), many ELF sections are written and so marked as read-write. However, most of them could be turned read-only after. This options tells the linker to mark as much sections as possible read-only.

[home ~/harden] objdump -x bad | grep RELRO
[home ~/harden] DEB_BUILD_HARDENING=1 make
[home ~/harden] objdump -x bad | grep RELRO
  RELRO off    0x0000000000000df0 vaddr 0x0000000000200df0 paddr 0x0000000000200df0 align 2**

Links

• Format string attacks (Wikipedia)
• Debian wiki page on source hardening

This article explains how to build, install and configure ulogd 2 for use with netfilter/iptables. It explains how to use plugins to store logs in databases (MySQL and PostgreSQL), use plugins to filter data, and gives some iptables rules to log packets.

Get the sources

You can use the official repository:

svn co https://svn.netfilter.org/netfilter/branches/ulog/ulogd2 ulogd2

You can also use our work repository (Regit and I) which contains the same files, with some additional patches here:

git clone http://home.regit.org/~regit/git/ulogd2.git ulogd2

Prerequisites

• libnfnetlink (>= 0.30) http://www.netfilter.org/projects/libnfnetlink/index.html
• libnetfilter_log
• mysql-dev
• postgresql-dev

Build

Use the standard autotools method for configure, build and install:

./autogen.sh
./configure --prefix=/path/to/prefix
make
sudo make install

Configuration

Edit ulogd.conf

1. enable plugins

You will have to choose the input and output plugins according to your setup. NFLOG is present in recent kernels (and iptables installation), and should be preferred if possible.

• Input plugin: ULOG or NFLOG
• Output: MySQL or PostgreSQL

You have to enable the corresponding in the configuration before you can use them:

plugin="/path/to/prefix/lib/ulogd/ulogd_inppkt_ULOG.so"
plugin="/path/to/prefix/lib/ulogd/ulogd_output_MYSQL.so"

See "Stack configuration" later.

2. buid the stack

For MySQL, we will use a very simple plugin stack. As MySQL is quite inefficient in storing IP addresses (and has no standard type for it), we will log the IP addresses in binary format using the IP2BIN plugin. You won't be able to run SQL commands directly, but GUI tools (like Nulog can decode them.

stack=log1:ULOG,base1:BASE,ip2bin1:IP2BIN,mysql1:MYSQL

For PostgreSQL, the stack is similar, except that we choose to log IP addresses using the standard inet type.

stack=log1:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,pgsql1:PGSQL

3. configure input plugin

[log1]
# netlink multicast group (the same as the iptables --ulog-nlgroup param)
nlgroup=1

4. configure output plugin

[mysql1]
db="ulog2"
host="localhost"
user="ulog2"
table="ulog"
pass="password"
procedure="INSERT_PACKET_FULL"

The configuration is the same as usual and should be easy, except maybe for the last parameter: it is the name of a procedure which will be used to insert data.

For PostgreSQL, you can use the following:

[pgsql1]
db="ulog2"
host="localhost"
user="ulog2"
table="ulog"
pass="ulog2"
procedure="INSERT_PACKET_FULL"

The configuration is the same, assuming you are running on the default port and have configured PostgreSQL, and can connect.

Stack configuration

Each module is a plugin which is dynamically loaded during ulogd start. A module has:

• Input keys: It defined the entry which are needed (mandatory or not) by the module to be able to work (ie output other keys)
• Output keys: The module outputs key->value association with key in the output list.

Each key has a type and may have a default value.

To have a working stack, you will need to provide an input module and filter which will be able to modify the initial set of key->value provided by the input module in a set of key->value which will contain all the input keys which are mandatory for the output plugin.

Information about a given module can be obtained through the info command of ulogd:

# /opt/ulogd2/sbin/ulogd --info /opt/ulogd2/lib/ulogd/ulogd_filter_IP2STR.so
Name: IP2STR
Input keys:
       Key: oob.family (unsigned int 8)
       Key: oob.protocol (unsigned int 16)
       Key: ip.saddr (IP addr)
       Key: ip.daddr (IP addr)
       [...]
Output keys:
       Key: ip.saddr.str (string)
       Key: ip.daddr.str (string)
       [...]

From this output, we see that the IP2STR module output IP in string format and take in input IP in rw format and some additional information (oob.family and oob.protocol).

Input plugins

• ULOG: get packet select via the iptables target ULOG
• NFLOG: get packet from NFLOG target which is the successor of ULOG
• NFCT: get flow information from Netfilter connection tracking via libnetfilter_conntrack

Output plugins

• LOGEMU: log packet/flow into a file
• OPRINT: log packet/flow to a file in multiline format
• SYSLOG: log packet/flow to syslog system
• MYSQL: log packet/flow to a MySQL database
• PGSQL: log packet/flow to a PGSQL server
• SQLITE3: log packet to a SQLITE3 file
• PCAP: log packet to a Pcap file
• IPFIX: log flow via IP Flow Information Export
• NACCT: log flow to a nacct compatible format (accounting)

Create database schema

PostgreSQL

1. create user and database

# su - postgres
$ createuser -P ulog2
$ createdb -O ulog2 ulog2

2. add support for plpgsql language (for procedures)

$ createlang plpgsql ulog2

3. insert schema

$ psql -U ulog2 -h localhost ulog2 -f doc/pgsql-ulogd2.sql

MySQL

1. create user and database

# mysql -uroot mysql
> CREATE DATABASE ulog2;
> GRANT ALL PRIVILEGES TO 'ulog2'@'localhost' IDENTIFIED BY 'password';

2. insert schema

$ mysql -uulo2 -ppassword ulog2 < doc/mysql-ulogd2.sql

Start ulogd2

As root, start the process:

/path/to/prefix/sbin/ulogd

If an error happens, it will only print:

Fatal error, check logfile.

Edit /var/log/ulogd.log (be careful, you will have to scroll many lines, so be sure not to use tail).

Wed Mar 26 22:19:54 2008 <7> ulogd.c:698 cannot find key `timestamp' in stack
Wed Mar 26 22:19:54 2008 <1> ulogd.c:807 destroying stack

Add iptables rules

iptables -A OUTPUT -p tcp --dport 80 -m state --state NEW -j ULOG

Check the result

mysql> select count(*) from ulog2;
+--+
| count(*) |
+--+
|        5 |
+--+
1 row in set (0.00 sec)

So now, we are logging to the database. Next time, we will detail the SQL schema: how to get information, how to insert data, tables, view, procedures, etc.

References

• http://software.inl.fr/trac/wiki/ulogd2/user
• ulogd 2 articles, part 1

The Black Hat Europe 2008 Media Archives are now online. I wasn't there, but the archives contains some interesting materials:

• Spam-Evolution
• LDAP Injection & Blind LDAP Injection (see my post)
• New Viral Threats of PDF Language
• 0-Day Patch -Exposing Vendors (In)Security Performance
• Client-side Security
• Attacking Anti-Virus (there was a presentation on the same subject at CanSecWest and Hack.lu)
• Investigating Individuals and Organizations Using Open Source Intelligence
• DTRACE: The Reverse Engineer's Unexpected Swiss Army Knife
• ...

Related links:

• Hack.lu presentations
• USENIX Proceedings online
• Defcon Media Archives

Sébastien and I were giving a presentation on IDS Correlation: A Weapon of Mass Investigation at CanSecWest.

Most of the presentations were very interesting, including attacks against the anti-virus software (they are the most interesting targets, imho : run with system privileges, include parser for many protocols, are present on almost all machines, etc.), secure programming with gcc and glibc, snort 3 (our presentation was just after Marty's), fuzzing with Peach, and some others I do not remember at the moment.

We also gave two lightning talks, one on the Authenticating Firewall NuFW (slides here) and one on the Signatures.NU project (slides here).

We even won a beer for doing the presentation :)

After so much work (and eating so much sushis with a delicious wild salmon), we went to Whistler for skiing, that was great.

Many thanks to all the people from the conference and to all who helped us !

Some links:

• Our photos
• Photos from Hirosan
• Photos from Sid
• CanSecWest last events and materials

To install Prelude, the Hybrid IDS (or Meta IDS) on Debian, on less than ten minutes, just use the packages:

• install a database (PostgreSQL or MySQL)
• install the Prelude manager, all needed packages will be installed automatically

apt-get install prelude-manager

• during the installation, dbconfig will ask to configure the database. Say yes, and give the parameters. dbconfig will create a new user, set a password, create the SQL schema and configure prelude-manager to use it.

This should be enough for the manager. You will have to configure the listen address for the manager (the default is restricted to localhost) to listen on the network.

To add agents (sensors), you have to install the package and register a new profile for each sensor. For ex:

apt-get install prelude-lml
apt-get install snort

Create a new profile:

prelude-admin register prelude-lml "idmef:w" <manager address> --uid 0 --gid 0
...
prelude-admin register snort "idmef:w" <manager address> --uid 0 --gid 0
...

Follow the instructions for the registration.

Check the address of the manager in the config (global file is /etc/prelude/default/client.conf):

server-addr = 192.168.1.1

For a complete installation guide (with explanations) including the web interface Prewikka, look at the Prelude Etch installation guide and the Prelude Handbook.

We have given a presentation with Sébastien, for OSSIR.

Subject is: Visualisation appliquée à la détection d'intrusion (which can roughly be translated to "visualization applied to intrusion detection"). The slides of the conference can be found on the list of presentation materials.

The conference was on our work since one year on intrusion detection, with the proposed implementation of a correlator, some explanations on classifications and current limitations, and how some graphs can help, especially the parallel axes representation.

Ulogd (and also ulogd2) is a powerful and flexible logging system for Netfilter/Iptables, based on a plugin system. It allows, for example, to log packets in a SQL database, and have some interface to analyze it (see Nulog2) :

Architecture

Ulogd2 combines plugins to create a stack, where each plugin is chained to another. There are three types of plugins:

• Source
• Filter
• Output

A stack must have only one source, and one output (yet it can have several filters). It is possible to define several stacks in the configuration.

Each plugin has a type (for ex, PGSQL), and must be instanciated (using a name chosen by the user). Each instance is a particular version of the plugin, defining parameters. This way, we will be able to output data in several formats using different stacks.

For ex, the following stack :

stack=log1:NFLOG,base1:BASE,ifi1:IFINDEX,print1:PRINTPKT,pgsql1:PGSQL

defines a stack with the following properties:

• input plugin is NFLOG, this means we'll use the -j NFLOG target of iptables as source
• filters are BASE, IFINDEX, and PRINTPKT (we'll cover what they do later)
• output plugin is of type PGSQL, so data will be logged to a PostgreSQL database.

What's new in ulogd2 ?

Ulogd2 brings a new SQL schema (see Linux Weekly News) where data are split in several tables (instead of one monolithic table as for ulogd), allowing to add additional data easily without modifying the schema (just add a new table and use the packet id as foreign key).

To hide the complexity of the new SQL schema (to select all data for a packet, one now has to use a JOIN in the request. The insertion must be done in several tables instead of one), a system of views (for select) and stored procedures (for insertions) has been written, so that ulogd2 does not have to know the SQL schema : if the schema is changed, only the procedures must be changed, and not ulogd2.

Next time

What ? That's all ? Yes ! That was a (short) introduction to what ulogd is. Next time, we'll cover the installation and the use of ulogd2.

Fusil (http://fusil.hachoir.org) has been accepted into Debian.

The description:

Fusil is a fuzzing framework designed to expose bugs in software by
changing random bits of its input.
.
It helps to start process with a prepared environment (limit memory,
environment variables, redirect stdout, etc.), start network client or
server, and create mangled files. Fusil has many probes to detect
program crash: watch process exit code, watch process stdout and syslog
for text patterns (eg. "segmentation fault"), watch session duration,
watch cpu usage (process and system load), etc.
.
Fusil is based on a modular architecture. It computes a session score
used to guess fuzzing parameters like number of injected errors to
input files.