ulogd2: the new userspace logging daemon for netfilter/iptables (part 2)
This article explains how to build, install and configure ulogd 2 for use with netfilter/iptables. It explains how to use plugins to store logs in databases (MySQL and PostgreSQL), use plugins to filter data, and gives some iptables rules to log packets.
Get the sources
You can use the official repository:
git clone git://git.netfilter.org/ulogd2.git/
- libnfnetlink (>= 0.30) http://www.netfilter.org/projects/libnfnetlink/index.html
Use the standard autotools method for configure, build and install:
./autogen.sh ./configure --prefix=/path/to/prefix make sudo make install
1. enable plugins
You will have to choose the input and output plugins according to your setup. NFLOG is present in recent kernels (and iptables installation), and should be preferred if possible.
- Input plugin: ULOG or NFLOG
- Output: MySQL or PostgreSQL
You have to enable the corresponding in the configuration before you can use them:
See “Stack configuration” later.
2. buid the stack
For MySQL, we will use a very simple plugin stack. As MySQL is quite inefficient in storing IP addresses (and has no standard type for it), we will log the IP addresses in binary format using the IP2BIN plugin. You won’t be able to run SQL commands directly, but GUI tools (like Nulog can decode them.
For PostgreSQL, the stack is similar, except that we choose to log IP addresses using the standard inet type.
3. configure input plugin
[log1] # netlink multicast group (the same as the iptables --ulog-nlgroup param) nlgroup=1
4. configure output plugin
[mysql1] db="ulog2" host="localhost" user="ulog2" table="ulog" pass="password" procedure="INSERT_PACKET_FULL"
The configuration is the same as usual and should be easy, except maybe for the last parameter: it is the name of a procedure which will be used to insert data.
For PostgreSQL, you can use the following:
[pgsql1] db="ulog2" host="localhost" user="ulog2" table="ulog" pass="ulog2" procedure="INSERT_PACKET_FULL"
The configuration is the same, assuming you are running on the default port and have configured PostgreSQL, and can connect.
Each module is a plugin which is dynamically loaded during ulogd start. A module has:
- Input keys: It defined the entry which are needed (mandatory or not) by the module to be able to work (ie output other keys)
- Output keys: The module outputs key->value association with key in the output list.
Each key has a type and may have a default value.
To have a working stack, you will need to provide an input module and filter which will be able to modify the initial set of key->value provided by the input module in a set of key->value which will contain all the input keys which are mandatory for the output plugin.
Information about a given module can be obtained through the info command of ulogd:
# /opt/ulogd2/sbin/ulogd --info /opt/ulogd2/lib/ulogd/ulogd_filter_IP2STR.so Name: IP2STR Input keys: Key: oob.family (unsigned int 8) Key: oob.protocol (unsigned int 16) Key: ip.saddr (IP addr) Key: ip.daddr (IP addr) [...] Output keys: Key: ip.saddr.str (string) Key: ip.daddr.str (string) [...]
From this output, we see that the IP2STR module output IP in string format and take in input IP in rw format and some additional information (oob.family and oob.protocol).
- ULOG: get packet select via the iptables target ULOG
- NFLOG: get packet from NFLOG target which is the successor of ULOG
- NFCT: get flow information from Netfilter connection tracking via libnetfilter_conntrack
- LOGEMU: log packet/flow into a file
- OPRINT: log packet/flow to a file in multiline format
- SYSLOG: log packet/flow to syslog system
- MYSQL: log packet/flow to a MySQL database
- PGSQL: log packet/flow to a PGSQL server
- SQLITE3: log packet to a SQLITE3 file
- PCAP: log packet to a Pcap file
- IPFIX: log flow via IP Flow Information Export
- NACCT: log flow to a nacct compatible format (accounting)
Create database schema
1. create user and database
# su - postgres $ createuser -P ulog2 $ createdb -O ulog2 ulog2
2. add support for plpgsql language (for procedures)
$ createlang plpgsql ulog2
3. insert schema
$ psql -U ulog2 -h localhost ulog2 -f doc/pgsql-ulogd2.sql
1. create user and database
# mysql -uroot mysql > CREATE DATABASE ulog2; > GRANT ALL PRIVILEGES TO 'ulog2'@'localhost' IDENTIFIED BY 'password';
2. insert schema
$ mysql -uulo2 -ppassword ulog2 < doc/mysql-ulogd2.sql
As root, start the process:
If an error happens, it will only print:
Fatal error, check logfile.
Edit /var/log/ulogd.log (be careful, you will have to scroll many lines, so be sure not to use tail).
Wed Mar 26 22:19:54 2008 <7> ulogd.c:698 cannot find key `timestamp' in stack Wed Mar 26 22:19:54 2008 <1> ulogd.c:807 destroying stack
Add iptables rules
iptables -A OUTPUT -p tcp --dport 80 -m state --state NEW -j ULOG
Check the result
mysql> select count(*) from ulog2; +--+ | count(*) | +--+ | 5 | +--+ 1 row in set (0.00 sec)
So now, we are logging to the database. Next time, we will detail the SQL schema: how to get information, how to insert data, tables, view, procedures, etc.