Terry Burton's Blog

Upgrading from PostgreSQL 7.4 to 8.1 in Debian Etch

2008-06-25T00:12:00.002+01:00

Debian allows both versions 7.4 and 8.1 of PostgreSQL to coexist on the same system which somewhat simplifies the upgrade procedure.

Assuming that the PostgreSQL 7.4 packages are currently installed with a fairly standard configuration, we start by installing PostgresSQL 8.1.

apt-get install postgresql-8.1 postgresql-client-8.1

The newly installed system will be started on a port 5433 whilst the old system continues to run on standard port 5432. For the time being it is best to leave both instances running on these ports which will allow you to migrate the data via a simple pipe.

Amend the configuration files for the new instance by hand, based on the existing configuration and restart the instance.

/etc/init.d/postgresql-8.1 restart

Populate the new instance with the existing databases:

sudo su - postgres
/usr/lib/postgresql/7.4/bin/pg_dumpall -p 5432 | /usr/lib/postgresql/8.1/bin/psql -p 5433

Stop the old database instance and use the new client.

/etc/init.d/postgresql-7.4 stop

Switch the new instance to using standard port 5432.

vi /etc/postgresql/8.1/main/postgresql.conf [set port = 5432]
/etc/init.d/postgresql-8.1 restart

Use the new psql client and your own applications to ensure that the new installation is operating correctly and that the data import was successful.

sudo -u postgres /usr/lib/postgresql/8.1/bin/psql -p 5432

If all is well then remove the old package, maybe backing up the old data.

tar cvzf ~/oldpgdata.tgz /var/lib/postgresql/7.4/data
apt-get purge postgresql-7.4 postgresql-client-7.4

Catching up with Barcode Writer in Pure PostScript

2008-04-13T01:44:00.009+01:00

It has been a while since I last wrote about Barcode Writer in Pure PostScript. The project has been far from dormant in recent months so here is a chance to catch up with what's been keeping me busy in my “spare” time. (And even very busy at other times!)

It's hard to recall from memory all of the improvements that have been made to the project over the last year but thankfully the commit logs do not forget! There have been the usual bug fixes, some code optimisations and new miscellaneous features, but the main highlight has to be the inclusion of support for 2D barcodes which went in to the mix as follows:

MaxiCode (June to July 2007)

MaxiCode is an irregular matrix symbologies whose symbols consist of a hexagonal grid of dots around a bullseye finder pattern. The sequencing of these dot positions does not follow any regular pattern and so unfortunately the mapping matrix must be hard-coded into the software. MaxiCode also has various different "modes" of operation, some of which impose a strict format on the initial part of the data which makes the input encoding quite complicated.

PDF417 (Boxing Day to New Year's Day)

Technically speaking, you might refer to PDF417 as a "stacked-linear" symbology, however BWIPP renders it using a grid of tall, rectangular cells. The worst thing about this symbology is that it requires a set of lookup tables that contain the "cluster sets" - three groups of 930 numbers used to convert from codewords to bar/space widths. The sequencing of the numbers within these sets appears to be quite random (if you know otherwise then please let me know) and so they must be hard-coded into the software which leads to a lot of uninteresting code – ouch!

Data Matrix (early- to mid-January)

The is a matrix symbology that can be rendered using a grid of squares through which the data zig-zags in eight-module, L-shaped clusters. Whilst the ordering of the modules within the grid is reasonably complicated, it can nevertheless be determined algorithmically for only a small amount of computational cost and requires only some minor tweaking to fix up the corner cases for matrices that do not contain some multiple of eight modules. So overall this symbology can be coded very nicely. We can generate both the standard square symbols types as well as the optional rectangular symbols.

Aztec Code (early- to mid-February)

This is a matrix symbology that can be rendered using a grid of squares with the data wrapping clockwise in two-module wide layers around a square finder pattern in the centre of the symbol. Whilst there are a few different types of symbols it is possible to fold the implementation for each of these into a single relatively sophisticated but direct algorithm that does containing excessive branching. So again, this symbology can be coded quite cleanly.

QR Code (February to late-March)

This is a matrix symbology that can be rendered using a grid of squares with the data vertically meandering in two-modules columns from right to left. With respect to implementation this symbology is quite hideous with its one saving grace being that is does not require the inclusion of hard-coded lookup tables for module placement. Firstly, in certain symbols the final data codeword is defined to be four bits wide rather than the usual eight which results in an awkward bit shift having to be applied to the trailing codewords in order to avoid propagating the exceptional processing required for these specials cases throughout the remainder of symbol generation process. Secondly, the "drunken walk" algorithm for placing the modules within the symbol (whilst avoiding the pre-defined static feature placeholders) has an unexplained inconsistency in the way that you perform the hop over the vertical timing pattern. Thirdly, the format and version information functions are unnecessarily complicated, however since their domain is very small it is possible to use a small set of pre-calculated lookup tables for these in order to avoiding using a significant amount of complex code. But finally, the worst aspect of this symbology is the optional, but recommended, process of apply eight distinct mask patterns to the candidate symbol in turn and then to evaluate these in order to select the one that would produce an output symbol with the fewest undesirable properties. To perform the evaluation algorithm as given by the specification turns out to be significantly more operationally expensive that the entire remainder of the symbol generation process! So for the time being we always select one particular mask.

So, we presently support all major 2D barcode formats, but with one major caveat - the user (or application developer) that is working with BWIPP has to do some preparative work to process the barcode data into the particular intermediate format required by each encoder for which they require the corresponding specification. This is a small task compared to the sometimes sophisticated numeric manipulation involved in the remainder of the symbol generation process. However it does involve extensive string manipulation which is a task for which PostScript is definitely not well suited whilst purpose-built application development languages (such as Perl and C++) have much better support for this task either natively or through libraries.

So the next major set of challenges on the BWIPP roadmap is to integrate the high-level encoding routine for each 2D symbology that convert from a user-supplied ASCII string to the intermediate format that is required by the encoders at present. The result will be that the novice user can simply enter the data that they require to place into a barcode, with only the minimal restrictions as necessarily imposed by each symbology, and our code will create the most optimal encoding that produces the best symbol for the given data, thereby making the system much easier to use for the uninitiated user.

Lastly, but by no means least, an extremely useful component in the implementation of support for 2D barcode generation has been the extensive testing performed by Jean-François Barbeau. He has helped detect and fix a number of bugs, some obvious, and some much more subtle so that we can place much greater confidence in the correctness of the output – so a big thank you on behalf of the PostScript barcoding community!

Toying with Barcodes - a talk from 24C3

2008-01-12T00:02:00.000Z

Interesting and entertaining talk by FX of Phenoelit from 24C3 about hacking real-world systems that (mis)use barcodes. Doesn't mention BWIPP though :-P

Taming p0f by chunk processing STDIN

2007-11-30T12:08:00.000Z

P0f is a useful, but slightly outdated, tool for passive operating system fingerprinting. Unfortunately it generates fingerprint information for every identified packet matching the BPF filter provided by the user. There is no built in option to aggregate the data so that it is framed per host within a given time interval or set of contiguous records. This means that when p0f is attached to the SPAN port or tap of a high volume pipe the log files that it generates grow very quickly as they are full of repeated information.

The following little script wraps the p0f process and removes the redundant information from each 1000 record chunk of output:


#! /bin/sh

exec 3>&-
exec 2>&-
exec 1>&-
cd /

nohup p0f -i eth2 -u p0f -N -U -q -p -t -l 'src net 143.210.0.0/16' | \
sed -n -e 's/^<\([A-Za-z0-9: ]*\)> \([0-9.]\{7,15\}\):[0-9]\{1,5\} - \
\(.*\)/\2 \3/p' | gawk 'ORS=NR%1000?"\n":"\000"' | xargs -0 -i bash -c \
'date +"*** %c ***"; echo "$0" | sort | uniq' {} >> /srv/p0f/os.log &

p0f aside, the interesting part boils down to this useful Unix shell programming paradigm:

$INPUT_CMD | gawk "ORS=NR%$BLOCK_LINES?'\n':'\000'" | xargs -0 -i $PROCESS_CMD {}

It splits the streamed output of $INPUT_CMD down into chunks of $BLOCK_LINES lines which are immediately independently processed by $PROCESS_CMD. It chunks the data by replacing the ordinary line separator on every $BLOCK_LINES line into an ASCII 0 character which xargs -0 uses as the argument separator.

Apache accesslog to syslog

2007-11-09T00:48:00.000Z

Apache allows its error logs to be written to the local syslog, however it does not natively support the directing of access logs to the syslog. How frustrating!

It does however allow access logs to be written to a pipe and I have seen a number of home-brew scripts that essentially redirect the Apache access log data from STDIN to syslog.

I've yet to see anything quite as simple as the following directive that I cooked up today:

CustomLog "|/usr/bin/logger -t apache -i -p local6.notice" combined

It pipes the access log data to the BSD logger(1) utility that is installed by default on almost any Unix system. No need for any more of those STDIN wrapper scripts!

Saving the planet...

2007-10-31T12:20:00.000Z

Photo captioned "Racing at Big Green Week"

Lessons learned: Space hoppers are slower than bikes, especially uphill.

Deleting old Snort and BASE event data from MySQL

2007-09-11T17:43:00.000+01:00

The following SQL that I recently cooked up thoroughly clears out the data from events that are more than 28 days old, then reoptimises the tables. It takes some time to complete so it is probably best scheduled as a regular night time cron job.

The DELETE FROM ... USING syntax is quite appealing and powerful...

DELETE FROM event WHERE timestamp < DATE_SUB(NOW(),INTERVAL 28 DAY);

DELETE FROM data USING data LEFT OUTER JOIN event USING (sid,cid) WHERE event.sid IS NULL;
DELETE FROM iphdr USING iphdr LEFT OUTER JOIN event USING (sid,cid) WHERE event.sid IS NULL;
DELETE FROM icmphdr USING icmphdr LEFT OUTER JOIN event USING (sid,cid) WHERE event.sid IS NULL;
DELETE FROM tcphdr USING tcphdr LEFT OUTER JOIN event USING (sid,cid) WHERE event.sid IS NULL;
DELETE FROM udphdr USING udphdr LEFT OUTER JOIN event USING (sid,cid) WHERE event.sid IS NULL;
DELETE FROM opt USING opt LEFT OUTER JOIN event USING (sid,cid) WHERE event.sid IS NULL;

DELETE FROM acid_event USING acid_event LEFT OUTER JOIN event USING (sid,cid) WHERE event.sid IS NULL;
DELETE FROM ag USING acid_ag_alert AS ag LEFT OUTER JOIN event AS e ON ag.ag_sid=e.sid AND ag.ag_cid=e.cid WHERE e.sid IS NULL;

OPTIMIZE TABLE event, data, iphdr, icmphdr, tcphdr, udphdr, opt, acid_event, acid_ag_alert;

Note to self...

2007-08-29T23:27:00.000+01:00

When replacing LILO with GRUB on a Debian host, remember to run update-grub after running grub-install and before rebooting. GRUB works better when there is a menu.lst file - doh!

HTML Renderer in Pure PostScript cited in a security research paper

2007-08-25T01:23:00.000+01:00

Just noticed a passing reference to my HTML Renderer in Pure PostScript project in the extended abstract of an interesting scientific paper[1] that focuses on how weaknesses in the PostScript interpreter security model can result in information leakage, compromising the anonymity of document reviewers.

The attack vector follows the notion that the same PostScript document can be rendered differently depending upon information that can be obtained from the interpreter's context; environment variables and contents of the filesystem. Amongst other exploits this raises the possibility of creating a contract that changes after having been electronically signed.

[1] Michael Backes, Markus Durmuth, Dominique Unruh. Information Flow in the Peer-Reviewing Process. In proceedings of IEEE Symposium on Security and Privacy 2007.

Fully Automatic Installation of Debian Etch with Root on LVM using FAI

2007-08-22T19:53:00.000+01:00

At work we have been making extensive use of FAI to rapidly deploy Debian hosts. It allows us to take a virgin server and network install a fully functioning Debian system in under four minutes (less than three minutes if you discount the time our hosts spend performing their BIOS tests). It also provides a extensively customisable disaster recovery system into which you can netboot any problematic hosts to investigate and repair them. I thoroughly recommend it to anyone that manages more than a few Debian hosts and that is willing to spend the couple of days or so that it takes to configure and explore all of the options.

Unfortunately for me the current version does not natively support the Linux Logical Volume Management. There are at least two community contributed helper scripts that provide this functionality by means of well-placed hooks provided by FAI, but neither of these have support for root on LVM; in fact one actively discourages it by displaying a warning! Anyhow, placing the root filesystem on LVM is enormously beneficial since amongst other things it allows you to make a consistent point-in-time snapshot of the entire filesystem with no disruption to running services.

In the past, placing root on LVM has indeed been problematic adding extra complexity to the boot process. It requires the bootloader to pass an initial ramdisk to the kernel which contains the necessary modules, libraries, binaries and scripts to discover and enable the Logical Volume that contains the root filesystem, without which the system of no use at all. It must deal with the activation of inactive volumes and handle special volumes such as those under snapshot, mirroring, or even cryptographically secured.

Historically, support for this has been poor due to the old-style initial ramdisks built by Debian Sarge and prior being incapable of dealing with a variety of abnormal conditions, causing the startup scripts within the ramdisk to quietly and uninterruptedly hang with no simple means by which to inspect the state of the locked up system to determine the cause of failure. This periodically rendered unbootable hosts that were not cleanly shut down thus requiring the aid of a live CD distro or netboot disaster recovery image in order to fix the problems. (Usually it would be that the snapshots had failed having ran out of room, requiring that they be first removed in order for the live volume to be made accessible.)

Thankfully this has now all changed because of a vast improvement called initramfs - the new system for creating initial ramdisks in Etch. initramfs has bundled the BusyBox Bash shell clone which is launched if there is any problem in mounting the root filesystem thereby allowing you to repair it without the need for an external recovery system. Together with the big improvements that have made the startup scripts much more robust, this now mostly solves any problems with putting root on LVM. Accordingly, you can take reassurance from the fact that the new Debian Installer provides root on LVM functionality as an installation option for any new Debian system, making it no longer an exceptional setup but a well-trodden path.

Despite not supporting LVM, the FAI system can be easily modified by writing custom hook scripts that are run at designated points during the host installation process. Peter Gervai has created a script to provide support for software RAID and LVM by extending a previous script by Michal Svamberg that provides support for software RAID. The RAID and LVM hook enables the FAI installer to create the necessary LVM entities according to user provided configuration files and it is well documented in on its wiki page.

However there are some steps that must be taken in addition to the setup instructions documented in the wiki in order to successfully FAI build LVM-enabled hosts.

Firstly, ensure that the netboot installer itself is LVM capable by loading device-mapper kernel module before the partitioning task starts:

Simply append dm_mod to the kernelmodules variable in the config file class/20-hwdetect.source in the NFS-exported configuration directory.

Secondly, ensure that the fully installed target system is independantly bootable by recreating the initial ramdisk to have LVM capability and by fixing an erroneous line that finds its way into /etc/fstab due to a minor bug in the FAI partitioning scripts.

Create a new config file called package_config/SW_RAID in the NFS-exported config directory:

lvm-common
lvm2
mdadm

Also create a new config file called scripts/SW_RAID/10-misc:

#! /bin/bash
error=0 ; trap "error=$((error|1))" ERR
cat <<EOF >> $target/etc/initramfs-tools/modules
dm_mod
dm_mirror
dm_snapshot
EOF
$ROOTCMD update-initramfs -u
# Fix the incorrect /dev/ entry that SW_RAID helper makes in fstab
sed -ie '/^\/dev\/ /d' /tmp/target/etc/fstab
exit $error

Those basic changes should be all that is necessary to get any host that is part of the SW_RAID class to FAI install as a root on LVM system. Hopefully we can expect FAI to natively support LVM sometime soon, which will make the process even more painless!

Finally, here are some sample disk configuration files to give you some idea of how easy it should be to maintain.

Sample disk_config/FAIBASE file:

disk_config disk1
primary /boot 200 rw ; ext2 boot
primary swap 2000 rw
logical - 1024-

Sample class/SW_RAID.var file:

SW_RAID_CONFIG="
"

LVM_VG_CONFIG="
vg00[]=/dev/cciss/c0d0p5
"

LVM_LV_CONFIG="
vg00/root[-n root ]=2G:/:ext3:
vg00/home[-n home ]=5G:/home:ext3:
vg00/srv[-n srv ]=10G:/srv:ext3:
vg00/var[-n var ]=5G:/var:ext3:
vg00/varlog[-n varlog]=5G:/var/log:ext3:
"

PXE boot FreeDOS with MEMDISK to flash a GIGABYTE BIOS

2007-08-19T22:57:00.000+01:00

Having recently sorting through some of the junk that I have been accumulating over the past several years, I decided that I would reincarnate a three-years dormant 1GHz AMD Athlon, which has been almost stripped bare for parts, into a silent running MythTV frontend. After loading the board with a whole 256Mb of RAM, replacing the graphics card with an old NVIDIA GeForce2 MX400, replacing the existing network card with a PXE-capable 3COM 905C Fast EtherLink XL PCI NIC, and salvaging a 40Gb Seagate ST340823A HDD, I was ready to begin.

The task: Install Ubuntu using the Netboot installation image via PXE boot; a process that is usually straightforward and is adequately described in several guides such as this, http://wiki.koeln.ccc.de/index.php/Ubuntu_PXE_Install

The problem: The installer was hanging when attempting to read the partitions on the ST340823A HDD, with the IDE bus being repeatedly reset until it finally gave up. This was no surprise as the CMOS setup utility was freezing whenever I performed an IDE Auto Configuration from within BIOS configuration system, however if it was set to auto detect the drive on boot then things would proceed normally. Hence I assumed (correctly, as it happens) that my current GA-5AA F3 BIOS was restricted to a ~30GB IDE HDD size limit.

New task: Flash a legacy GIGABYTE BIOS on a machine with no external drives and with no MS-DOS boot disks or images to hand; a process that is less simple and not well documented.

GIGABYTE supply an updated BIOS version F7b(Beta) for the board. This comes in the form of a self-extracting executable that can be downloaded from http://www.giga-byte.co.uk/Support/Motherboard/BIOS_Model.aspx?ProductID=1439

The Linux file utility identifies this as an "MS-DOS executable PE for MS Windows (GUI) Intel 80386 32-bit, UPX compressed, RAR self-extracting archive" from which I figured that I could extract the contents using the rar:

$ rar x motherboard_bios_ga-5aa_f7b$beta$.exe
<...snip...>
Extracting autoexec.bat OK
Extracting flash830.exe OK
Extracting GA-5AA.F7b OK
All OK

The autoexec.bat indicates that you can flash your BIOS from MS-DOS as simply as A:\FLASH830 GA-5AA.F7b

So how do you boot into MS-DOS in order to do this? Cheat! Enter FreeDOS - an open source MS-DOS compatible operating system, which is claimed as being what most motherboard manufacturers now use in the development of their BIOSes. The project provides a raw image of a minimal boot floppy that is available from the file archive at http://www.freedos.org/freedos/files/ called fdboot.img. This needs to be amended to include the flash830.exe and GA-5AA.F7b files which can be done easily from within Linux.

mount -t vfat -o loop fdboot.img /mnt/floppy
cp flash830.exe GA-5AA.F7b /mnt/floppy
umount /mnt/floppy

Now all that is required is to boot this image and run the flash commands. MEMDISK, part of the SYSLINUX suite, provides support for booting legacy operating systems and this will work over PXELINUX, http://syslinux.zytor.com

Instructions for configuring PXELINUX are available in several places, but in summary the process involves configuring a DHCP server, configuring a TFTP server, and then copying the necessary boot images and creating the configuration files in the TFTP root.

The required pxelinux.cfg pragma for booting a FreeDOS boot disk image is as follows:

label dos
kernel memdisk
append initrd=fdboot.img

With all of this configured the PC netbooted into FreeDOS from where the flash utility was invoked as A:\FLASH830 GA-5AA.F7b. After reporting success I rebooted the PC and reassuringly the BIOS screen reported the BIOS version as being GA-5AA F7b. And most importantly, the IDE Auto Configuration feature now detected the drive correctly, without freezing!

So I had another go at installing Ubuntu using the Netboot installer... and unfortunately it hung in exactly the same place! Whilst the BIOS update had certainly resolved a particular size limit issue, it had unfortunately not solved the issue with this hard disk.

A bit of Googling revealed that the failure is due to a firmware bug in the Seagate ST340823A model that incorrectly reports the total number of addressable sectors rather than the last addressable sector (zero-based) when queried with the "stroke" feature enabled, http://bugzilla.kernel.org/show_bug.cgi?id=8816

When the Ubuntu partitioner examines the drive it attempts to read the final sector, as reported by the hard disk, which is non-existent and so causes the kernel to throw a fit, repeatedly retrying the read between forced IDE bus resets, until the channel gives up completely.

Unfortunately Seagate do not appear to have released an updated firmware for this drive so it looks like I will miss another opportunity to put FreeDOS to good use ;-)

Micro$oft's improved password policy

2007-08-02T18:19:00.000+01:00

I'm glad to see that Micro$oft are at last taking security seriously...

"Error Message: Your Password Must Be at Least 18770 Characters and Cannot Repeat Any of Your Previous 30689 Passwords"

http://support.microsoft.com/kb/276304

Barcode Writer in Pure PostScript is evil, says Google

2007-07-18T01:15:00.000+01:00

"Results 1 - 10 of about 666..."

And I thought it was just a myth.

Rapid context switching on our User Mode Linux host

2007-06-29T10:18:00.000+01:00

Now with hundreds of thousands of context switches per second I'm surprised that the box has any time to do any work in userland!

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
13 0 2760 17060 15596 3767972 0 0 76 224 354 146832 30 70 0 0
10 0 2760 16944 15596 3768020 0 0 48 0 355 135414 45 55 0 0
10 0 2760 16556 15596 3768532 0 0 28 0 349 143074 37 63 0 0
9 0 2760 17380 15596 3767776 0 0 80 0 330 141158 41 59 0 0
9 1 2760 17140 15596 3767852 0 0 20 0 356 144483 39 61 0 0
9 0 2760 17152 15596 3767888 0 0 36 160 322 145730 37 63 0 0
9 0 2760 17000 15596 3768020 0 0 108 0 354 139198 38 62 0 0
6 1 2760 17660 15596 3767264 0 0 72 0 285 129292 48 52 0 0

Using BASE Alert Groups to deploy countermeasures to Snort alerts

2007-06-27T23:04:00.000+01:00

For those who are familiar with the snort + BASE NIDS system...

I've cooked up a simple script that gives easily parsable output of the result of a simple SQL query that inspects already BASE Alert Groups to find the remote or local IP address, Alert Group name, number of alerts and last alert time for all hosts that have generated more that a given threshold of alerts within a given timeframe.

use strict;
use DBI;

my $time_period=24; # hours
my $event_threshold=0;
my $is_local='NOT'; # blank for local, 'NOT' for remote

my $sql=<<END;

SELECT \@A:=IF(iph.ip_src $is_local BETWEEN INET_ATON('192.168.1.0') AND INET_ATON('192.168.1.255'),iph.ip_src,iph.ip_dst) AS ip,
INET_NTOA(\@A) AS ip_f,
ag.ag_name,
MAX(e.timestamp) AS last_event,
COUNT(ag.ag_name) AS num_events
FROM acid_ag AS ag
JOIN acid_ag_alert AS aga USING (ag_id)
JOIN event AS e ON aga.ag_sid=e.sid AND aga.ag_cid=e.cid
JOIN iphdr AS iph USING (sid,cid)
WHERE e.timestamp>DATE_SUB(now(),INTERVAL ? HOUR)
GROUP BY ip, ag.ag_name
HAVING num_events>?
ORDER BY \@A

END

my $dbh=DBI->connect('DBI:mysql:snort','root');
my $sth=$dbh->prepare($sql);
$sth->execute($time_period,$event_threshold);

print "$_->{ag_name}\t$_->{ip_f}\t$_->{num_events}\t$_->{last_event}\n" while ($_=$sth->fetchrow_hashref);

$sth->finish;
$dbh->disconnect;

The output will look something like this:

banned 43.280.25.17 6 2007-06-27 16:07:27
banned 43.280.25.210 6 2007-06-27 05:46:20
botnet 147.67.18.2 2 2007-06-26 11:39:20
mailblock 147.67.18.2 1 2007-06-26 09:13:08
mailblock 217.21.112.5 2 2007-06-27 13:41:48

The remote host 147.67.18.2 has within the last 24 hours generated events that have been placed into two distinct Alert Groups: two events into "botnet" (the last event being seen 2007-06-26 11:39:20) and one event into "mailblock" at 2007-06-26 09:13:08.

This output can be parsed frequently by a number of cron scripts that deploy countermeasures against these events, e.g. block the "botnet" IP addresses at the firewall, or build zone files from "mailblock" IP addresses for local DNSBLs to filter mail from such hosts, etc...

Glad I live in a country that doesn't respect the DMCA...

2007-05-07T13:46:00.000+01:00

)(-F(-!!-)"-(D-&$-E£-%B-D*-$!-%^-C%-^£-%^-**-C)

The above is a representation of a work that has been protected from viewing using a trivial copy protection scheme (using my British keyboard). If you live in a jurisdiction that accepts the DMCA then you may be breaking the law to decode it!

Sign my petition - "Free TV Listings"

2007-05-04T21:05:00.000+01:00

I've posted a petition to the UK Government's E-petitions website.

Petitions with two hundred or more signatories get an official response from 10 Downing Street, so it's got be worth having a go with some of the issues that would be difficult to otherwise raise.

My particular petition has so far received just over one hundred signatures and reads as follows:

Currently anyone publishing television listings in the UK is required by law under Schedule 17 of the Broadcasting Act 1990 to pay a royalty, which BDS (Broadcasting Data Services) collects on behalf of the broadcasters.

By releasing television schedule details into the public domain many web sites will be able to integrate this information into their content, creating a tighter convergence between the web and television which will benefit advertisers, broadcasters, webmasters and importantly, the consuming public.

And my motivation for creating it is explained in this extract from a mailing list post:

There are several voluntary projects that wish to combine a database of extensive program information with an XML feed of the non-free official channel listings to create an uberfeed that can be used to schedule recordings, etc. Unfortunately they are prohibited from doing so.

I'm currently working on a mini-project in Haskell that can hook into the scheduling system to instruct MythTV to record any program that it "thinks" you will like, based its on similarity to things that you have recorded in the past (same actors, genres, etc...) and on what other users with similar interests are watching (assuming that you wish to federate your viewing information with other users). It'll also routinely record first in a series and anything else that I can think may be useful.

A detailed and accurate data feed is essential for this system to work well.

Ubuntu root on LVM from the standard Live CD

2007-04-18T20:04:00.001+01:00

These are the notes I made whilst installing Feisty Fawn from the standard Live CD, such that the root filesystem lands on LVM.

This guide does make the assumption that you know your way around Linux and is based on a rather simplified setup (for the sake of clarity) that you will probably want to amend for your specific needs.

The instructions ought to work with little or no amendment for any recent flavour of Ubuntu including Edgy Eft and Dapper Drake. If you have problems following this guide then Google is your friend!

Boot from the CD into the live desktop.

Configure networking so that we can use aptitude to obtain the necessary LVM packages.

sudo aptitude update
sudo aptitude install lvm2
modprobe dm-mod

cfdisk /dev/hda # or /dev/hdb, /dev/sda, /dev/cciss/c0d0, etc., throughout...

Write the partitions to disk something like the following:

hda1 ; Boot ; Primary ; Linux ext2 ; 200Mb # at beginning of drive for /boot
hda2 ; ; Primary ; Linux swap ; 2048Mb # at end of drive
hda3 ; ; Primary ; Linux LVM ; 120000Mb # in remaining space for root LV and other LVs

pvcreate /dev/hda3 # This may fail, see below for fix

At the time of writing there existed a bug in the packaged version of LVM that can be fixed with: ln -s /lib/lvm-200 /lib/lvm-0

vgcreate vg00 /dev/hda3
lvcreate -n root -L 10G vg00 # If this fails make sure that you have performed modprobe dm-mod

It is necessary to create filesystems on the devices in order for the ubiquity installer to correctly identify them as possible install targets.

mkfs.ext2 /dev/hda1
mkfs.ext3 /dev/mapper/vg00-root
mkswap /dev/hda2

Proceed through the installation as normal, except choose to manually prepare the disks as follows:

/dev/hda1: use as: ext2 ; mount point: /boot ; format: yes
/dev/hda2: use as: swap
/dev/mapper/vg00-root: use as: ext3 ; mount point: / ; format: yes

Allow the install to complete and click "Continue using the live CD".

mount /dev/mapper/vg00-root /target
mount /dev/hda1 /target/boot
mount -t proc proc /target/proc
mount -t sysfs sysfs /target/sys
chroot /target

aptitude update
aptitude install lvm2
ln -s /lib/lvm-200 /lib/lvm-0

vgchange -ay # Tests that LVM works from within the target system

Add the following modules to both /etc/modules and /etc/initramfs-tools/modules:
dm-mod
dm-snapshot
dm-mirror

update-initramfs -u

Exit the chroot by pressing ctrl-d.

umount /target/proc /target/sys /target/boot /target

Reboot into your new "root on LVM" Ubuntu system.

Technical Poetry... and quiet good too!

2007-04-16T23:31:00.000+01:00

"Algorhyme" by Radia Perlman

I think that I shall never see
A graph more lovely than a tree.
A tree whose crucial property
Is loop-free connectivity.

A tree which must be sure to span
So packets can reach every LAN.
First the Root must be selected
By ID it is elected.
Least cost paths from Root are traced
In the tree these paths are placed.

A mesh is made by folks like me
Then bridges find a spanning tree.

"Not a Great Loss"

2007-04-10T12:06:00.000+01:00

A friend told me this at the weekend...

Tony B'liar was visiting a primary school class when the teacher began a discussion about the meaning of the word "tragedy".

The teacher began by asking the children for an example of a tragedy.

A boy spoke up saying, "If my mum and my gran were walking down the street and a car lost control and ran them over, that would be a tragedy."

The teacher turned to the PM and asked, "What do you think Mr Blair?"

"No," replied Mr Blair, "I would call that an Accident."

A girl spoke up saying, "If a coach carrying a hundred children drove off a cliff killing everyone on board, that would be a tragedy."

"No," said Mr Blair, "I would call that a Great Loss."

Finally, another boy raised his hand saying, "If Air Force One, carrying Mr Blair and Mr Bush, was struck down by a missile and blown up, that would be a tragedy."

"That's right!" exclaimed Mr Blair, "And why do you think that would be a tragedy?"

The boy thought carefully and reasoned, "Well, it wouldn't be an Accident and it wouldn't be a Great Loss..."

SCO vs IBM

2006-12-17T21:25:00.000Z

Two very educational webcasts from the Harvard Journal of Law that give enormous insight into both sides of the SCO vs IBM battle.

Darl McBride, CEO and President, The SCO Group

Eben Moglen, General Counsel for the Free Software Foundation

Some references:

Eldred v. Ashcroft
Freeing the Mind
Zero marginal cost economics of Open Source

2006-12-02T01:43:00.000Z

I like this piece of clever code. You might too...

float InvSqrt (float x){
float xhalf = 0.5f*x;
int i = *(int*)&x;
i = 0x5f3759df - (i>>1);
x = *(float*)&i;
x = x*(1.5f - xhalf*x*x);
return x;
}

Falling into a trap with gpg

2006-11-15T17:44:00.000Z

A colleague recently had quite a scare whilst using gpg to verify the signature of a file he had amended but forgot to re-sign... it verified!!!

Had he found a collision case? He amended the file again and rechecked. It still verified, so not a collision case?!

Was gpg broken? He moved the files to my box (running a more recent version of gpg) and rechecked. Still verified!

After a little debugging all was made clear, but it revealed a frighteningly common misunderstanding of the gpg tool that is all to easy to fall into and effectively makes signature verification useless.

He wanted to sign a file creating a detached signature in Armor format. The typical use case is that one can make the file (myfile) and its signature (myfile.asc) available for download, and the recipient can then use the signer's public key to verify that myfile has not been tampered with.

From the gpg manpage the first options that seem to offer what we want are:

-s, --sign [file]
Make a signature. This command may be combined with --encrypt (for a signed and encrypted message), --symmetric (for a signed and symmetrically encrypted message), or --encrypt and --symmetric together (for a signed message that may be decrypted via a secret key or a passphrase).

-a, --armor
Create ASCII armored output.

So we proceed to make a detached signature as follows:
$ gpg -as myfile

We can verify it with:
$ gpg --verify myfile.asc
gpg: Good signature from...

Now we tamper with the original:
$ echo abc >> myfile
$ gpg --verify myfile.asc
gpg: Good signature from...

Oh! That shouldn't happen, and to somebody who is not familiar with the correct usage of gpg, the cause is far from obvious.

The are in fact two problems with the above procedure. Firstly the correct method to create a detached Armor signature for a file is to use gpg -ab myfile, not gpg -as myfile. The latter command does not generate a "detached" signature, but rather creates a self-contained signature containing the original file contents and then writes this in Armor format into a .asc file, giving you a file that is proportional in size to the original file. As can be seen in the except from the manpage, this important distinction is not made clear.

Secondly, the correct way to verify a detached signature is gpg --verify myfile.asc myfile (which always verifies the given signature against the given file), and not with gpg --verify myfile.asc as many sources wrongly quote. The latter command, given a detached signature will look for the corresponding file and validate it against the signature, however given a self-contained signed file it will validate only that file, generating output that is identical in either case, i.e. the user has no way of knowing in which of these two modes of operation the command has just run. Anybody attempting to verify a detached signature in this way is vulnerable to the following exploit: An attacker tampers with the original file and then also replaces the .asc with any self-contained signed file produced by the author using gpg -as. Since the self-contained signature has not been tampered with and will verify, the victim is falsely assured of the integrity of their file.

Armed with this knowledge it should be clear now what was going wrong with the above example. gpg was correct and myfile.asc was indeed valid, but only valid as a self-contained signed file and not as a signature for myfile, however the command did not make this clear. This would be best addressed in two ways. User education, for example amending the man page description of the gpg -s command to clearly spell out this pitfall, and by changing the behaviour of gpg to make it clear exactly what has been verified, perhaps by displaying a warning in cases where such confusion can arise.

Good security tools should make it difficult to fall into simple traps and must help to eliminate human error where at all possible. When experienced administrators are easily able to fall pray to such small oversights with such grave consequences, the tools are clearly deficient.

Non-interative package installation with APT

2006-11-15T16:39:00.000Z

One of the frequently voiced frustrations of working with Debian's APT package management tool is the difficulty of running it completely non-interactively. I've had to solve this problem recently and found none of the usual solutions to be as successful as they claim to be.

Normally, even with when specifying the "noninteractive" option, the question of whether or not to replace local configuration files with the packager's configuration files will still appear and block the script. However, the following bash command will automatically select the default option for any questions that appear during installation:


/usr/bin/yes '' | DEBIAN_FRONTEND=noninteractive \
/usr/bin/apt-get -y install package

I'd be interested to hear from anybody who finds a real situation that for which the above technique does not work.

Note: when the empty parameter '' is passed to /usr/bin/yes it outputs a stream of carriage returns, rather than the usual stream of 'y's. This causes the default, usually safe, option to be selected as the answer to the questions that any packages may ask.

Linux Format mentions barcodes in Scribus

2006-05-05T20:50:00.000+01:00

Nice to see that the Barcode Maker plugin for Scribus hasn't gone unnoticed...

From Linux Format issue 80:

Open source desktop publishing turns pro.

In many ways, Scribus is the runaway success story of open source graphics. Yes, Gimp is great, Inkscape has potential - but Scribus set itself the challenge of coming from nowhere to being a solid, professional DTP solution, and by traditional standards of software development, it's done that in next to no time.

In fact, the latest release of Scribus (1.3.3) offers features not available as standard in professional DTP software, such as the ability to generate barcodes completely within the software. It would be unthinkable for a commercial DTP product to come from zero lines of code to a reliable, professional tool in five years.