In the end I just gave up on bootstrapping my own instance from scratch and simply used one of the pre-existing Rightscale CentOS 5.4 images as a starting point. I started it up, removed almost everything from it, including the rightscale rpm. Then I snapshotted it, and converted the snapshot to an AMI using ec2-register.

Now I finally had somewhere to start from. I started up my new instance, and thanks to the instructions on this webpage I proceeded to create a blank EBS volume. I created a filesystem on it and mounted it under /mnt/ebs. Now I could rsync the running instance across. I created the /dev files, unmounted and shut down.

I now had an EBS volume from which I could create a snapshot. I then registered this snapshot again and I finally ended up with an appropriate AMI to use. Job done. Thanks once again to the webkist blog for the detailed instructions.

If I do figure out how to properly bootstrap my own EBS-based AMI image I’ll be sure to post the details here.

I was thinking of using ‘setterm’ for this, but I decided to google this first and found the perfect solution in the Arch forums. Just in case that link stops working I’m going to include the script below, hopefully it will be helpful to other people.

#!/bin/bash

###################################################
# Check if X is running or not, turn off monitor, #
# wait for a key press and turn it on again.      #
###################################################

grep_result_file=$PWD'/x_running'

# Check if X is running.
ps -e | grep -e "\bX\b" > $grep_result_file
ps -e | grep -e "\bxorg\b" >> $grep_result_file
ps -e | grep -e "\bxserver\b" >> $grep_result_file

## If you want to check result file, uncomment following lines.
#echo "===== $grep_result_file - begin ====="
#cat $grep_result_file
#echo "===== $grep_result_file -  end  ====="

if [ ! -s $grep_result_file ] || [[ $(tty) =~ tty ]] || [[ $(tty) =~ vc ]]; then
    echo 'Detected X not runnig or you are at console...'
    if [ $UID -ne 0 ]; then
        echo 'You need super user privileges to run this script at console.'
        echo 'Rerun as super user or start X and run from a terminal.'
        exit 0
    fi
    turn_off='vbetool dpms off'
    turn_on='vbetool dpms on'
else
    echo 'Detected X running...'
    turn_off='xset dpms force off'
fi

echo 'Turning off monitor...'
$turn_off

echo 'Waiting for a key press...'
read -n1 -s

echo 'Turning on monitor...'
$turn_on

rm $grep_result_file

echo 'Finished: monitor_off'
kill -9 $PPID

The superiority of OpenBSD when it comes to security is legendary. The OpenBSD community continuously do security audits of their codebase and their website proudly boasts of having only two remote holes in the default install, in more than 10 years!” PF is the default packet filter used in OpenBSD from version 3.0 onwards.

Having setup OpenBSD firewalls using PF in the past, I was interested in expanding my knowledge and this tutorial provided the perfect opportunity for me to do so. The tutorial was given by Peter M. Hansteen, who is a consultant, writer and sysadmin based in Bergen, Norway, and also the author of the excellent “The Book of PF“, published by No Starch Press.

Peter started off by answering some common questions that people might have about PF such as: Can I run it on Linux (Answer: No, but some are trying). He recommended not trusting any GUI tools, and simply using a text editor to edit pf.conf as that is simpler and faster. In addition, some tools claim to automatically convert “other” firewall rules to PF, but he recommended implementing a fresh PF config yourself.

The first firewall I ever tried to configure for my home network used iptables; when I had to implement one in PF, I found it to be a breath of fresh air. I have always found the concept of “chains” that iptables uses confusing. PF doesn’t have any concept of a chain; you simply start your rules by first “blocking everything”, then enabling the things you need, one line after the other. Although in principle this sounds exactly like what you are supposed to do with iptables, in practice the rules you generate are much simpler, and easier to understand.

Leading on from this, he showed examples of how PF should be set up in an environment where you need a “gateway” between 2 networks, and how to deal with problems faced by people who try to use FTP from behind a NAT firewall (ftp-proxy). Tables and filtering by services (http, ftp, etc) were introduced next. A table in PF is basically a list of IP addresses; listing them in a table makes it easier to apply a single rule to a collection of hosts.

Peter then moved on to the subject of dealing with the huge volume of spam that besieges us all. The two main concepts that he focused this section of the tutorial on were “tarpitting” and “greylisting“. In tarpitting, when a blacklisted host connects to you, you send replies to them very very slowly, let’s say around 1 byte at a time. When doing greylisting, you lie to unknown connecting clients using SMTP 45n errors (temporary local error). This usually thwarts spammers, who simply want to quickly connect, deliver their payload and leave. Many spammers don’t attempt to reconnect after seeing this error, while legitimate clients will automatically retry after a short period of time. These legitimate hosts are then added to a whitelist, which means that the next time they try to connect, they will no longer be given a 45n temporary error, their mail will be accepted immediately.

We then got a look at how to thwart SSH bruteforce attacks by using rate-limiting, and a short introduction to wireless networking in OpenBSD. The next thing that Peter talked about, authpf, was quite interesting. Basically users need to authenticate to authpf first; once authenticated, only then is traffic generated by these users allowed to pass through the firewall. Special rules can be setup specifically for authpf users.

The next topic was load balancing and Peter showed how to configure a “web server pool” using PF. Requests to this pool were alternated using a form of round-robin. To solve a common round-robin problem where machines in the pool go down, you can use “hoststated“, which monitors the state (up/down) of the certain specified hosts and compensates accordingly. hoststated has been renamed to relayd in OpenBSD 4.3.

You can tag incoming packets, so you can quickly pass/block packets marked with a certain tag. Setting up a OpenBSD bridge was discussed next. A bridge in this context simply refers to a transparent firewall that sits between 2 or more networks and filters packets at the link level.

Using ALTQ, you can do bandwidth allocation and traffic shaping. You can used class based queues (percent, kilo, or mega bytes), priority based or hierarchical queues. In a class based queue, you can say for example that FTP is only allowed 20% of your bandwidth.

The last major aspect of PF that Peter discussed was CARP (Common Address Redundancy Protocol) and pfsync. Put simply, CARP and pfsync allow you to setup 2 redundant firewalls instead of 1, and in case one firewall fails, everything switches over to the other firewall automatically. pfsync is used to keep the rules between the 2 firewalls in sync.

Overall, I am very pleased that I attended this tutorial. Obviously I was familiar with some of the concepts, but things like authpf, hoststated and CARP were completely new to me. I will definitely use the things I learned here when considering any OpenBSD based firewalling solutions in the future.

In any case, I was asked by Alain (UKUUG Chairman) to write a few words about one of the tutorials that I attended on the first day of the conference. It will be published in the next UKUUG newsletter. I will make sure I post a copy of what I write here.

XML Parser benchmarks: I have always had my own suspicions of which XML parser model would be faster (Sax or StaX), but I’m glad to see this benchmark done by the O’Reilly folks.

Fear and loathing at Cupertino: Jeremy Allison’s terrible experience while trying to prepare a talk for his Apple WWDC presentation. Jeremy works on Samba, along with Tridge, who they all call “the smartest man in Australia” Jeremy works at Google now. Smart man.

Three things that caught my interest today:

PowerTOP: Released by Intel, this utility builds on work done by kernel developers to make the Linux kernel power-efficient. PowerTOP gives you a snapshot of what apps are consuming the most power. Turn off these apps or modify their behavior, and you’ll notice an instant increase in the battery life.

The Linux SLAB Allocator: Traditional heap memory managers suffer from fragmentation, among other issues. The SLAB Allocator in Linux, inspired by a similar implementation for Solaris and various embedded systems, allocates memory as fixed sized objects and uses caches to reduce fragmentation. It also has options to enable hardware cache alignment which allows objects in different caches to share the same cache lines, thus improving performance.

Advanced Linux Programming: After many years of coding mostly Java, I’ve been meaning to brush up on my C, Assembly and general Unix programming skills. I found this excellent book freely available online and it seems to be getting a lot of praise from reviewers on Amazon so I downloaded it. It has a lot of topics that I’m very interested in, like IPC and threads, and it even has a few assembly oriented chapters. I will definitely be reading this one

A few people, including me, volunteered to give short presentations about certain aspects of SLED, and I was giving a short talk and demo of XGL under SLED. I demoed it to about 20 people and they were all quite impressed with it. Along they way I managed to help out several people with their linux problems.

I absolutely loved this meetup and I hope I can attend many more in the future. There are more details and pictures here.


Error: no known VMs. (check for corrupt jvm.cfg file)


I couldn’t run 'java' or 'javac' from the command-line at all! I immediately went to /System/Library/Frameworks/JavaVM.framework and looked for 'jvm.cfg'. I have 3 VMs installed on this machine, 1.3.1, 1.4.2, and 1.5.0. 1.3.1 and 1.4.2 had a proper jvm.cfg file installed but for some reason 1.5.0′s jvm.cfg was a zero-length file. Googling didn’t turn up anything useful except this tip, which wouldn’t work because in my case all my permissions were correct. Fixing permissions using Disk Utility didn’t show any permissions problems at all.

In the end, all I did was copy the 1.4.2 version over to the 1.5.0 directory and all was well. The tip above mentions that Eclipse might have had something to do with this, and there might be some truth to that, as I did update my Eclipse to 3.2 recently, but I have been using it for a week without any problems …

Boot Camp lets you install Windows XP without moving your Mac data, though you will need to bring your own copy to the table, as Apple Computer does not sell or support Microsoft Windows. Boot Camp will burn a CD of all the required drivers for Windows so you don’t have to scrounge around the Internet looking for them.

Well … I never thought I would see this day. Everyone has known that Apple wouldn’t officially support Windows on Macs and they still won’t offer support for this but it was always assumed that people would have to use hacks to get Windows to run on a Mac. With Boot Camp people will no longer have to rely on hacks and it will also burn you a CD of the required drivers for your XP system. Very very cool indeed.

So I haven’t been able to blog for 2 weeks and *so* many things have happened. narf and blanca finally managed to get Windows XP running on a Mac! This is awesome news, especially after disappointing news a while back that Vista will not support EFI either. It seems they have managed to emulate a regular BIOS for the XP on Mac competition. Soon after the announcement, Leo Laporte did a great segment for MacBreak where they install XP on a Mac Mini and that’s the first place where I saw it actually running. A lot of people can now dual-boot their new Intel Macs and have a choice of XP or OSX (and even Linux!). I’ve never seen such a versatile machine! These are interesting times indeed.

Here is a short summary of some of the announcements, linking to a Channel 9 video of each (if possible):

Sparkle – separating visual components and design of an application from the data representation.

Start.com / Gadgets – nice, clean start/home page which can be extended using “gadgets”. and these gadgets can be re-used through inheritance. The windows vista sidebar will also sport gadgets which help people get to commonly used or needed data or tasks.

Microsoft Max – a kind of a cross between iPhoto and iMovie. allows you to create rich interactive photo albums with slick effects and to export these albums so that almost anybody can view them.

LINQ – a really cool idea of trying to remove the impedance mismatch between object and relational databases. From what I have seen, manipulating XML should be easier. It’s all done in C#. Creating and populating objects after using SQL joins should be easier.

WCF or Windows Communication Founcation (formerly Indigo) – Much more than just another web-services framework, it implements a lot of the plumbing that a lot of developers usually have to manage all the way from SOAP to P2P to some other method of app-to-app or pc-to-pc integration. So if you want 2 machines to talk to one another somehow you don’t have to worry too much about the plumbing going on behind all that communication. At least that’s the easiest way I see to summarize it

One the of absolutely coolest things that Jim Allchin showed off was the ability to increase the available memory of a PC running Vista by simply plugging a USB key in. It’s ingenious and I haven’t heard of anyone doing anything like it unless I am mistaken.

WPF/E or Windows Presentation Foundation / Everywhere. WPF was formerly Avalon, a new way to build rich interactive web / client apps using XML. They showed a very cool Netflix (video rental online) demo that they ran on 4 different machines and it scaled, etc flawlessly: a desktop machine, Media Center PC, Tablet PC and a PDA. Everything being vector based makes this a lot easier. Another cool North Face demo here

A lot more stuff than this has been announced of course but it is good to see at least a few cool things and possibly 1 innovation come out of Microsoft so far! They are pretty gung-ho on security nowadays and let’s hope that Vista finally gets security right (by right I mean a lot better than XP by default). Note that I’m usually *not* very pro-Microsoft, but I do believe in competition, and I hope Apple, Google, Linux et al give Microsoft all they’ve got because it makes things a lot more interesting for the consumer and gives us more choice. That is always a good thing.

1. The OS can be made irrelevant by people writing for another OS, let’s call it WebOS.
2. People now only need to code for one platform (Java, anyone?)
3. A local web server on the client machine will mean that an app can continue on working even if it’s offline (think local app for Gmail, etc).
4. When users are back online, the app will synchronize itself with its online counterpart. Think adding Flickr pictures locally, and then uploading them.

The OS is still there whichever one it is, but it is no longer going to lock-in developers to only developing for that OS. One of the things Jason is saying is that people will no longer need to write for 3 OSes but for just one OS. This sounds just like “people will no longer have to write for 3 OSes but for just one platform: Java”.

Jason has already outlined some of the problems he envisions with this approach, such as web apps accessing local content on your hard drive. Java applets have solved this problem for a long time by using a sandbox model. The only problem is that these applets have to be downloaded and run locally for them to work. A WebOS app, however, will run both locally and remotely, it will live on your local hard drive and perhaps a sandbox model will help there as well. Are vulnerabilities detected in these local apps any more dangerous than vulnerabilities for remote apps? Perhaps there isn’t much different because at the end of the day, a security hole is the same thing whatever app it affects. However, I suppose a local security hole can do a *lot* more damage than a remote one.

Some other suggestions like: “Read newsfeeds from bloglines locally” sound no different to NetNewsWire, FeedDemon or SharpReader, etc. These are all locally accessible feed readers which, when online, update your subscriptions, etc, otherwise they work fine when offline.

I think, as [Paul Graham](http://paulgraham.com/) said, a WebOS will allow totally new kinds of applications to exist and we don’t know what they will look like at all. So this is definitely an interesting space to watch.
Some great examples of new ways of using the web are: [Backpack](http://www.backpackit.com), [Basecamp](http://www.basecamphq.com/), [Gmail](http://www.gmail.com) and [Google Maps](http://maps.google.com). Another up-and-coming app is [Hula](http://hula-project.org/Hula_Server) which has been open-sourced by Novell and is being actively worked on by many GNOME hackers. If you want to see a demo of how the future of web-calendaring might look like, take a look at [this](http://www.nat.org/2005/august/#Hula-Web-Interface) amazing demo on [Nat's](http://www.nat.org) blog.

In other news, I would love to get my hands on one of [these](http://joi.ito.com/archives/2005/08/25/wearing_firefox.html) [Firefox](http://www.getfirefox.com) T-shirts that [Joi](http://joi.ito.com) managed to pick up from the [Mozilla](http://www.mozilla.org) offices. I have ordered quite a few T-shirts from [ThinkGeek.com](http://www.thinkgeek.com) in the past, and I will order one of these if they stock them someday.

It seems the HFS+ file system on OSX is quite good at keeping file fragmentation at bay, almost eliminating it altogether in most cases! Amit Singh has a great article about it here.

A colleague at work asked me today how OSX’s Mail.app does it’s spam filtering and this appeared as if just by coincidence

Created by the engineers who bring the Japanese input method and the Speech technologies to you, Mail’s junk mail filters are outstanding. When trained for a sufficient period of time, the filters can reach 98%+ accuracy against spam and are surprisingly painless to use. In fact, this feature alone has convinced many users to switch to Mail.

The first part of that article is also pretty good, a lot of good stuff about spam, picking good email addresses, etc.

Update: These guys have released a program to fix the OSX exploit.

I must admit, saying it that way it doesn’t sound that different from just putting the java app into a directory and being done with it, but Apple have taken this a bit further than that using special features like the bundle bit and by assuming from the outset that *all* apps will be deployed like this instead of just a few that you develop. This allows you to have multiple versions of the same app in different bundles with different libraries inside them for example (as far as I understand it).

However, customizing features of the OS is not always easy. I needed to control how my app was deployed, so that users could not change the system settings for example and at first I was quite baffled how to do it. Moving the app from one location to another proved futile. OSX always knew where the app was located regardles of which directory I put it in or what I called it I later discovered that if I dig deeper into the bundle and edit the Info.plist xml file and change the BundleName, it will finally no longer run. However, this was not the solution for me since I did need to run it myself sometimes. Eventually after sometime of poking around trying to understand the system and reading hints on the excellent MacOS X Hints website, I discovered that I could simply remove privileges for the user so that he/she could no longer run System Preferences. Simple enough.

Much harder were figuring out how to remove items from the Apple menu bar as I did not want to use one since I was deploying an older app that had never used menus before and many other exotic things like replacing the Finder upon login with my own app. OS X seems to store a lot of things in databases like the NetInfo database and the ‘defaults’ database which it carries over due to it’s NeXT and OpenStep inheritance. This makes it quite convenient to change things, but only once you know where they are! Not unlike the days when I used to memorize Windows registry keys

Weather is extremely hot nowadays, usually around 30-35. The tube is horrible, just horrible, it’s baking hot in there. It’s not unusual for me to come out of it dripping with sweat. Good thing the Mayor has issued a challenge to anyone who comes up with a workable air conditioning system for the tube … Here’s hoping

My parents are coming round in about 2 weeks time, and we are both very happy to be seeing them after such a long time. We haven’t made any travel plans as such, I guess we’ll take each day as it comes …

Oh well, I used to have a perfect Slackware installation on it running Fluxbox as the window manager and Opera as the browser. Distributions such as Slackware and Debian are really very useful in situations like this, because you can install just the minimum amount necessary and get on with it. So I scrapped Mandrake and installed Debian 3.0r0. No problems so far, I’m using the same fluxbox/opera combination. A very light window manager, and quite a light browser. Opera only takes up about 14% of memory on start up and is taking up about 35% right now, which is fine as browsing is the primary purpose of this machine anyway. Oh well, lesson learned. Shiny new desktop distros just don’t belong on older and slower laptops, even with many services turned off. You’re better off going with a light-weight distro like Slackware or a minimal Debian install.