Support This Project

  1. How it works
  2. Setting up the box
    1. Install Debian
    2. Install packages
    3. Fetch the code and install packages
    4. Setup chroot
  3. Give root to your buildd
  4. Setup sbuild
    1. Setup buildd
    2. Configure dupload
    3. Add cron jobs
  5. Setup wanna-build
    1. Building arch=all packages
    2. Useful commands
    3. List packages which needs to be build
    4. Give back a package for build (aka retry)
    5. Remove package state from database
    6. binNMU
  6. Mutt script for buildd auto-reply
    1. Folder-hook
    2. Mutt preferences and send-hook
    3. Extracting changes file from build log
  7. Related Links

This documentation is written to explain how to setup a build daemon to recompile the whole Debian unstable archive.

How it works

Buildd setup works with 3 packages:

  • wanna-build
  • sbuild
  • buildd

wanna-build is used to store informations about packages in a database. sbuild is used to actually build packages into a chroot. buildd is used to fetch packages to build via wanna-build and run sbuild to build them.

Setting up the box

Install Debian

Install debian. Version installed here is etch.

Install packages

The following packages are needed: 

build-essential devscripts bzip2 fakeroot patch subversion

Fetch the code and install packages

Currently, everything is not packaged. That's a shame.

sbuild is in the archive, but wanna-build and buildd are not. You fetch the code via Subversion from http://svn.cyberhqz.com/svn/wanna-build

Here is how I build sbuild and wanna-build: 

$ svn co http://svn.cyberhqz.com/svn/wanna-build
$ cd wanna-build
$ vi debian/control
--> Remove versionned dependency on sbuild, because we can use the version from unstable
$ debuild -uc -us

Then, install packages: 

# aptitude install sbuild debootstrap cron-apt libmldbm-perl dupload quinn-diff
# dpkg -i wanna-build_2006.01.08_sparc.deb buildd_2006.01.08_sparc.deb

You can use the sbuild version from sid or from the svn repository. Currently I use both.

Create system environment

You can use /usr/share/doc/buildd/examples/buildd-setup-system.gz directly (zcat | sh), but I prefer to do everything by myself. 

# as root
adduser --system --shell /bin/sh --uid 60000 --gecos 'Build Daemon' --group --disabled-password buildd
mkdir -p /var/lib/debbuild/srcdep-lock
chown -R buildd:buildd /var/lib/debbuild
chmod -R 2775 /var/lib/debbuild
cd ~buildd
zcat /usr/share/doc/buildd/examples/buildd.conf.gz > buildd.conf
zcat /usr/share/doc/sbuild/examples/sbuildrc.gz > .sbuildrc
chown buildd:buildd buildd.conf .sbuildrc
mkdir -p logs build old-logs upload bin stats/graphs mqueue
echo "|/usr/bin/buildd-mail-wrapper" > .forward
chown buildd:buildd logs build old-logs upload bin stats .forward mqueue

Setup chroot

Now it's time to create our chroot! The buildd-make-chroot command is not available in the sbuild package from sid, so use the one you grabbed from svn, be smart! 

# cd ~buildd
# ~/wanna-build/buildd-make-chroot buildd etch build/chroot-etch http://ftp2.fr.debian.org/debian

Note: actually, this command fails because of dependencies, with the following error: 

dpkg: dependency problems prevent removal of makedev:
 gnupg depends on makedev (>= 2.3.1-13) | devfsd | hurd; however:
  Package makedev is to be removed.
  Package devfsd is not installed.
  Package hurd is not installed.

The solution is to keep makedev (which is removed by buildd-make-chroot). Apply the following patch: 

--- buildd-make-chroot.orig     2007-05-11 12:31:44.000000000 +0000
+++ buildd-make-chroot  2007-05-11 12:31:57.000000000 +0000
@@ -74,7 +74,7 @@
 chmod -R 02775 $TARGET/var/debbuild
 echo include /etc/ld.so.conf.d/*.conf >> $TARGET/etc/ld.so.conf
 (cd $TARGET/dev ; ./MAKEDEV fd)
-sudo chroot $TARGET dpkg -P debconf-i18n debconf liblocale-gettext-perl libtext-charwidth-perl libtext-iconv-perl libtext-wrapi18n-perl procps makedev
+sudo chroot $TARGET dpkg -P debconf-i18n debconf liblocale-gettext-perl libtext-charwidth-perl libtext-iconv-perl libtext-wrapi18n-perl procps
 echo "Successfully setup chroot for a buildd"
 echo Possible commands to append to fstab:
 echo echo $SUITE-proc $TARGET/proc proc defaults 0 0 \>\> /etc/fstab

You can finish the install: 

# as root
mkdir -p ~buildd/build/chroot-etch/var/lib/sbuild/srcdep-lock
chown -R buildd.buildd ~buildd/build/chroot-etch/var/lib/sbuild/
mkdir ~buildd/build/chroot-etch/home/buildd
chown buildd.buildd ~buildd/build/chroot-etch/home/buildd
cd ~buildd/build && ln -s chroot-etch chroot-stable

After, edit the /etc/apt/sources.list in your chroot: 

# as root
vi ~buildd/build/chroot-etch/etc/apt/sources.list
chroot build/chroot-etch /bin/sh
apt-get update
exit

You should mount proc into your chroot otherwise some build will failed. Add the following line to your /etc/fstab: 

etch-proc /home/buildd/build/chroot-etch/proc proc defaults 0 0

Give root to your buildd

sbuild should be able to use schroot, but, in fact, that's terribly b0rked right now. So we will use the old sudo method. 

# visudo
Add:
Defaults:buildd env_keep+="APT_CONFIG DEBIAN_FRONTEND SHELL"

buildd  ALL= NOPASSWD: ALL

Setup sbuild

Change your user to buildd (sudo -H -s -u buildd), and configure sbuild. 

# vi .sbuildrc
# egrep -v '^(#|$)' .sbuildrc
$build_dir="$HOME/build";
$log_dir = "$HOME/logs";
$mailto = "pollux\@inl.fr";
$maintainer_name='Pierre Chifflier <chifflier@inl.fr>';
$purge_build_directory="always";
$stalled_pkg_timeout = 90;
1;

Setup buildd

Change your user to buildd and edit ~buildd/buildd.conf: 

$ egrep -v '^(#|$)' buildd.conf 
require "/etc/buildd.conf.global";
@take_from_dists = qw(stable testing unstable);
@no_auto_build = qw(boot-floppies);
@weak_no_auto_build = qw();
$autoclean_interval = 86400;
$admin_mail = "buildd admin <pollux\@inl.fr>";
$build_log_keep = 1;
$statistics_mail = 'pollux\@inl.fr';
$dupload_to = "anonymous-inl";
$dupload_to_security = "anonymous-inl";
$log_queued_messages = 0;
$wanna_build_user = "buildd_arch-hostname";
$wanna_build_dbbase = "build-db";
$sshcmd = "";
$sshsocket = "";
1;

Configure dupload

The buildd user should have a working dupload installation (.dupload.conf): 

package config;

$default_host = 'anonymous-inl';

$cfg{'anonymous-inl'} = {
        fqdn => "xxx.xxx.xxx.xxx",
        method => "scpb",
        login => "pollux",
        incoming => "/var/www/debian/mini-dinstall/incoming/",
        dinstall_runs => 1,
};

Add cron jobs

Some commands should be added to the crontab, to ensure that buildd is running, and to upload packages: 

$ crontab -l
# m h  dom mon dow   command
MAILTO=pollux@inl.fr
30 0,6,12,18 * * * /usr/bin/buildd-uploader
20 0,6,12,18 * * * /usr/bin/buildd-watcher

Setup wanna-build

Copy wanna-build local configuration: 

# zcat /usr/share/doc/wanna-build/examples/wanna-build.conf.local.gz > /etc/wanna-build.conf.local
# chown buildd /etc/wanna-build.conf.local

Then you can create an empty wanna-build database: 

# # as root:
# install -m 755 -d -obuildd -gbuildd /var/debbuild
# # as user buildd:
$ wanna-build -d stable --create-db --clean-db

Now you will need to fill up the database with all packages. That's easy to do with a little script (thanks to Julien Louis). 

#!/bin/sh

cd /tmp
wget -q http://ftp.fr.debian.org/debian/dists/unstable/main/source/Sources.gz -O Sources.main.gz
wget -q http://ftp.fr.debian.org/debian/dists/unstable/contrib/source/Sources.gz -O Sources.contrib.gz

if [ -e Sources.main.gz ] && [ -e Sources.contrib.gz ]; then
    zcat Sources.main.gz Sources.contrib.gz > Sources
fi

touch Packages
quinn-diff -A `dpkg-architecture -qDEB_HOST_ARCH` 2>/dev/null | wanna-build --merge-quinn -d stable
rm -f Sources Packages

You can run this script once a day to update the database.

Building arch=all packages

The default configuration of buildd is not to build architecture-independant packages. To force buildd to build arch=all packages, apply the following patch to '/usr/bin/buildd': 

--- buildd.OLD  2007-05-14 11:59:20.000000000 +0000
+++ buildd      2007-05-14 11:59:47.000000000 +0000
@@ -466,7 +466,8 @@
 
        my @sbuild_args = ( 'nice', '-n', "$conf::nice_level", 'sbuild',
                            '--batch', "--stats-dir=$main::HOME/stats",
-                           "--dist=$dist" );
+                           "--dist=$dist",
+                           "--arch-all" );
        my $sbuild_gb = '--auto-give-back';
        if ($conf::sshcmd) {
                $sbuild_gb .= "=";

Useful commands

List packages which needs to be build

$ /usr/bin/wb-short/list-needs-build -d stable
unknown/glpi_0.68.3-3 [unknown:uncompiled]
Total 1 package(s)

Give back a package for build (aka retry)

$ /usr/bin/wb-short/give-back-build -d stable -U buildd_arch-hostname glpi_0.68.3-3

Remove package state from database

$ /usr/bin/wb-short/forget-build -d stable -U buildd_arch ocsinventory-server_1.01-2

binNMU

$ cd ~buildd/build
$ sbuild --make-binNMU="Rebuild for erase unwelcomed dependency" --binNMU=1 -d unstable -v hello_2.1.1-4
  • "--make-binNMU" defines message to add changelog.
  • "--binNMU" defines minor revision. If you set 1, hello_2.1.1-4.0.1 will be built.

Mutt script for buildd auto-reply

When a build is complete (and successful), a mail is sent (see ~buildd/.sbuildrc). To complete the build and upload the package, the .changes file must be extracted from the build log, must be signed and sent back to the buildd. As this is quite boring, here is some scripts to make this part easier.

Folder-hook

First, we will apply the script only to mails stored in the =buildd folder. So we use a folder-hook to associate this folder with a specific script: 

folder-hook =buildd                     source ~/.mutt/defaults.buildd

Mutt preferences and send-hook

This is the big part: we set some preferences for how to display the folder, and we set a send-hook to make an almost automatic reply, passing the build log to a shell script to extract the .changes file, and then signing it. Practically, all what you have to do is to press 'r' (reply) on a mail, then 'y' to send it (you will have to enter the passphrase if not in memory). 

# -*- muttrc -*-

# Use the global defaults first.
source ~/.mutt/defaults.global

# This is the index format for mailing lists.
set index_format="%4C %Z %{%b %d} %-15.15n (%4l) %s"

# Sort on threads
set sort=threads

# Give mailing lists a more "slrn" like feel.
set pager_index_lines=9

# do not ask for 'to' and 'subject' on reply
set fast_reply=yes

# do not save a copy of the reply
set record=""

send-hook '~t buildd ~s success' "set signature='' pgp_autosign=yes indent_string='' edit_headers=yes editor='~/bin/buildd-script.sh' fast_reply=yes pgp_create_traditional=yes include=yes pgp_sign_as=chifflier@inl.fr"
send-hook '!(~t buildd ~s success)' "set pgp_autosign=no indent_string='> ' editor=vim fast_reply=no pgp_create_traditional=no include=yes"

'Note': please note that I am no mutt expert, so the line can be wrong, or there may be an easier way to do that. In both cases, I'm interested if you have any suggestion.

Extracting changes file from build log

We use a shell script buildd-script.sh, containing a bit of sed magic to suppress all lines before and after the .changes file: 

#!/bin/bash +e
cat $1 > ~/.buildd/orig
# 11 is the number of lines of your headers in default reply
egrep -v '^Fcc:' $1 | sed -e '11,/\.changes\:$/d' -e '/\/.*\.deb\:/,$d' > ~/.buildd/changes
cat ~/.buildd/changes > $1

Related Links