In the end I just gave up on bootstrapping my own instance from scratch and simply used one of the pre-existing Rightscale CentOS 5.4 images as a starting point. I started it up, removed almost everything from it, including the rightscale rpm. Then I snapshotted it, and converted the snapshot to an AMI using ec2-register.

Now I finally had somewhere to start from. I started up my new instance, and thanks to the instructions on this webpage I proceeded to create a blank EBS volume. I created a filesystem on it and mounted it under /mnt/ebs. Now I could rsync the running instance across. I created the /dev files, unmounted and shut down.

I now had an EBS volume from which I could create a snapshot. I then registered this snapshot again and I finally ended up with an appropriate AMI to use. Job done. Thanks once again to the webkist blog for the detailed instructions.

If I do figure out how to properly bootstrap my own EBS-based AMI image I’ll be sure to post the details here.

I was thinking of using ‘setterm’ for this, but I decided to google this first and found the perfect solution in the Arch forums. Just in case that link stops working I’m going to include the script below, hopefully it will be helpful to other people.

#!/bin/bash

###################################################
# Check if X is running or not, turn off monitor, #
# wait for a key press and turn it on again.      #
###################################################

grep_result_file=$PWD'/x_running'

# Check if X is running.
ps -e | grep -e "\bX\b" > $grep_result_file
ps -e | grep -e "\bxorg\b" >> $grep_result_file
ps -e | grep -e "\bxserver\b" >> $grep_result_file

## If you want to check result file, uncomment following lines.
#echo "===== $grep_result_file - begin ====="
#cat $grep_result_file
#echo "===== $grep_result_file -  end  ====="

if [ ! -s $grep_result_file ] || [[ $(tty) =~ tty ]] || [[ $(tty) =~ vc ]]; then
    echo 'Detected X not runnig or you are at console...'
    if [ $UID -ne 0 ]; then
        echo 'You need super user privileges to run this script at console.'
        echo 'Rerun as super user or start X and run from a terminal.'
        exit 0
    fi
    turn_off='vbetool dpms off'
    turn_on='vbetool dpms on'
else
    echo 'Detected X running...'
    turn_off='xset dpms force off'
fi

echo 'Turning off monitor...'
$turn_off

echo 'Waiting for a key press...'
read -n1 -s

echo 'Turning on monitor...'
$turn_on

rm $grep_result_file

echo 'Finished: monitor_off'
kill -9 $PPID

The most straightforward way I found to do this is as follows. Credit for this method goes to Neil Aggarwal, who posted it on the centos mailing list.

In case that link stops working someday, it is as simple as:

1. Using fdisk, I created two partitions on the drive:
/dev/sdc1 = 15 MB
/dev/sdc2 = The rest
I marked the first one bootable

2. I used dd to copy the diskboot.img to /dev/sdc1

3. Formatted /dev/sdc2 as ext3 and copied the
iso images there

4. When I boot from the usb drive, I selected hard
drive install and pointed the installer to
/dev/scd2

It worked perfectly!

Thanks, Neil!

The superiority of OpenBSD when it comes to security is legendary. The OpenBSD community continuously do security audits of their codebase and their website proudly boasts of having only two remote holes in the default install, in more than 10 years!” PF is the default packet filter used in OpenBSD from version 3.0 onwards.

Having setup OpenBSD firewalls using PF in the past, I was interested in expanding my knowledge and this tutorial provided the perfect opportunity for me to do so. The tutorial was given by Peter M. Hansteen, who is a consultant, writer and sysadmin based in Bergen, Norway, and also the author of the excellent “The Book of PF“, published by No Starch Press.

Peter started off by answering some common questions that people might have about PF such as: Can I run it on Linux (Answer: No, but some are trying). He recommended not trusting any GUI tools, and simply using a text editor to edit pf.conf as that is simpler and faster. In addition, some tools claim to automatically convert “other” firewall rules to PF, but he recommended implementing a fresh PF config yourself.

The first firewall I ever tried to configure for my home network used iptables; when I had to implement one in PF, I found it to be a breath of fresh air. I have always found the concept of “chains” that iptables uses confusing. PF doesn’t have any concept of a chain; you simply start your rules by first “blocking everything”, then enabling the things you need, one line after the other. Although in principle this sounds exactly like what you are supposed to do with iptables, in practice the rules you generate are much simpler, and easier to understand.

Leading on from this, he showed examples of how PF should be set up in an environment where you need a “gateway” between 2 networks, and how to deal with problems faced by people who try to use FTP from behind a NAT firewall (ftp-proxy). Tables and filtering by services (http, ftp, etc) were introduced next. A table in PF is basically a list of IP addresses; listing them in a table makes it easier to apply a single rule to a collection of hosts.

Peter then moved on to the subject of dealing with the huge volume of spam that besieges us all. The two main concepts that he focused this section of the tutorial on were “tarpitting” and “greylisting“. In tarpitting, when a blacklisted host connects to you, you send replies to them very very slowly, let’s say around 1 byte at a time. When doing greylisting, you lie to unknown connecting clients using SMTP 45n errors (temporary local error). This usually thwarts spammers, who simply want to quickly connect, deliver their payload and leave. Many spammers don’t attempt to reconnect after seeing this error, while legitimate clients will automatically retry after a short period of time. These legitimate hosts are then added to a whitelist, which means that the next time they try to connect, they will no longer be given a 45n temporary error, their mail will be accepted immediately.

We then got a look at how to thwart SSH bruteforce attacks by using rate-limiting, and a short introduction to wireless networking in OpenBSD. The next thing that Peter talked about, authpf, was quite interesting. Basically users need to authenticate to authpf first; once authenticated, only then is traffic generated by these users allowed to pass through the firewall. Special rules can be setup specifically for authpf users.

The next topic was load balancing and Peter showed how to configure a “web server pool” using PF. Requests to this pool were alternated using a form of round-robin. To solve a common round-robin problem where machines in the pool go down, you can use “hoststated“, which monitors the state (up/down) of the certain specified hosts and compensates accordingly. hoststated has been renamed to relayd in OpenBSD 4.3.

You can tag incoming packets, so you can quickly pass/block packets marked with a certain tag. Setting up a OpenBSD bridge was discussed next. A bridge in this context simply refers to a transparent firewall that sits between 2 or more networks and filters packets at the link level.

Using ALTQ, you can do bandwidth allocation and traffic shaping. You can used class based queues (percent, kilo, or mega bytes), priority based or hierarchical queues. In a class based queue, you can say for example that FTP is only allowed 20% of your bandwidth.

The last major aspect of PF that Peter discussed was CARP (Common Address Redundancy Protocol) and pfsync. Put simply, CARP and pfsync allow you to setup 2 redundant firewalls instead of 1, and in case one firewall fails, everything switches over to the other firewall automatically. pfsync is used to keep the rules between the 2 firewalls in sync.

Overall, I am very pleased that I attended this tutorial. Obviously I was familiar with some of the concepts, but things like authpf, hoststated and CARP were completely new to me. I will definitely use the things I learned here when considering any OpenBSD based firewalling solutions in the future.

In any case, I was asked by Alain (UKUUG Chairman) to write a few words about one of the tutorials that I attended on the first day of the conference. It will be published in the next UKUUG newsletter. I will make sure I post a copy of what I write here.

I did run into one problem, though and I couldn’t find any solution to it on either the Ubuntu Forums or anywhere else on the net. My top Gnome panel had a strange white bar under it and all my context menus had white borders. Maybe my google-fu wasn’t very good yesterday, but the only solution that I managed to find after about an hour was this on a Gentoo forum:

This is a known issue. Go to ccsm->Window Decorations and add the string !dock to the value Shadow Windows. I had to enter 2 !dock. First disabled shadows of the context menus and the tool tips, the second stops shadows for the gnome-panel.

I’m just putting this here in case it helps someone with a similar problem.

Technorati Tags: compiz fusion ubuntu beryl

Powered by ScribeFire.

The danger of a distraction depends not on how long it is, but on how much it scrambles your brain. A programmer can leave the office and go and get a sandwich without losing the code in his head. But the wrong kind of interruption can wipe your brain in 30 seconds.

This is spot-on, and I notice this a lot during my lunch break. Sometimes I can’t get a program or problem out of my head and occasionally I even come up with a solution not 10 minutes into my lunch break and then I can’t wait to get back and finish it. Other times however, especially if I have lunch with my colleagues the whole ‘problem space’ I’ve built up in my head simply vanishes. Due to the amount of work it usually takes (maybe a half-hour to an hour) to re-load my brain with the problem I was working on, a lot of post-lunch time is wasted and sometimes I can never recreate the problem fully again because I tend to be sharper in the mornings than in the lazy afternoons.

Since there’s a fixed cost each time you start working on a program, it’s more efficient to work in a few long sessions than many short ones.

I have often wanted to do this, but it’s almost impossible to do. There is always lunch, some other interruption, going home, eating dinner or something similar. On the weekends, however, I sometimes manage to stay up late and can work uninterrupted for quite a while.

Rewriting a program often yields a cleaner design.

True sometimes, but I agree with him that even the process of rewriting a program can lead to significant insights; even if the rewritten program is not a huge improvement.

Instead of summarizing the whole essay here, I highly recommend that all programmers and their managers go read it. Even non-IT staff, such mathematicians, whose work involves long-stretches of thinking, and constructing problem spaces in their heads will benefit from the advice in this essay.

I haven’t been paying attention to my RSS feeds recently and I forgot just how good some people are at writing and expressing their insights Paul Graham and Joel On Software are two blogs (journals?) that I really enjoy reading.

Technorati Tags: coding programming software

#1. Avoid using passive voice. Passive voice just means that when you say something about the world or some event that happened, make sure you identify ‘who or what’ was involved. The ‘who or what’ is commonly referred to as an agent.

For example,

Wrong:
The tea was made.
The program was written.
The computer was dropped.

Correct:
Mark made the tea.
Jane wrote this computer program.
Christopher dropped the computer.

#2. Speak directly to your reader. Never address your audience as the reader or refer to yourself as the author. You should speak directly to your reader. You should refer to her as you. If you are the sole author of a book, use I and if you have co-authors, use we. In addition, avoid using one, as in One should realize … or One has written.

#3. So, So that and Such that. Just remember the following:

So means therefore
So that means in order that
Such that means in such a way that

#4. Two or more. Use the terms between, each other, either, and a couple to refer to precisely two entities; and you should use among, one another, any one of, and several to refer to more than two entities.

I will add some more tips later on, but probably not many. The problem is that I haven’t gotten permission from the author and several of the tips have examples that are best quoted verbatim from the book.

Starting today, I will try and change that. You see, in order to motivate myself to keep this blog updated I have decided to set myself a goal and complete it. I will keep updating this blog not only with my progress, but will also write about what I have learned as I go along.

So my first goal will be ‘to improve my writing skills’. I have enjoyed writing stuff for a long time now and friends have occasionally told me that ‘you could be a really good writer’. Well I’ve finally decided I should give it a shot.

To be honest, I’ve never been good at grammar, and people find that hard to believe. I’ve simply developed a good ear for what ‘sounds right’ based on my perusal of several newspapers, magazines, novels and books over the years. Learning a lot of vocabulary while preparing for my SATs and reading ‘lots’ has really helped.

The first book I’m going to try and read and understand is going to be Bugs in Writing, which I bought several years ago, but never got around to reading. It is written mainly for people who come from a scientific or technical background and hence is perfectly suited for someone like me. Each chapter aims to analyse and fix a single ‘problem’ and you can read each chapter in whatever order you like.

With that, I bid you farewell. I believe my next post is going to be about ‘passive/active voice’. Exciting times indeed

XML Parser benchmarks: I have always had my own suspicions of which XML parser model would be faster (Sax or StaX), but I’m glad to see this benchmark done by the O’Reilly folks.

Fear and loathing at Cupertino: Jeremy Allison’s terrible experience while trying to prepare a talk for his Apple WWDC presentation. Jeremy works on Samba, along with Tridge, who they all call “the smartest man in Australia” Jeremy works at Google now. Smart man.

Three things that caught my interest today:

PowerTOP: Released by Intel, this utility builds on work done by kernel developers to make the Linux kernel power-efficient. PowerTOP gives you a snapshot of what apps are consuming the most power. Turn off these apps or modify their behavior, and you’ll notice an instant increase in the battery life.

The Linux SLAB Allocator: Traditional heap memory managers suffer from fragmentation, among other issues. The SLAB Allocator in Linux, inspired by a similar implementation for Solaris and various embedded systems, allocates memory as fixed sized objects and uses caches to reduce fragmentation. It also has options to enable hardware cache alignment which allows objects in different caches to share the same cache lines, thus improving performance.

Advanced Linux Programming: After many years of coding mostly Java, I’ve been meaning to brush up on my C, Assembly and general Unix programming skills. I found this excellent book freely available online and it seems to be getting a lot of praise from reviewers on Amazon so I downloaded it. It has a lot of topics that I’m very interested in, like IPC and threads, and it even has a few assembly oriented chapters. I will definitely be reading this one

We have just come back from a brilliant trip to Switzerland! The weather was awesome throughout our trip. We spent most of the time in Lucerne, where we stayed at the Hotel Drei Koenig. On our first day we decided to wander around Lucerne’s Old Town and see as much of it as possible. We ended up quite dreadfully tired and in the evening we just went back to the hotel and simply crashed.

We went on a trip to Mount Titlis the next day. Mount Titlis is one of the highest mountains in Switzerland, at 10,000 feet. I must admit to feeling quite terrified at that height. I was feeling quite disorientated.

The next day we made a long hiking trip up Mount Rigi-Kulm. To get to the start of the hiking point, you have to take a cruise along the river, then a trip up the mountains by train, before you get to your starting point. This is the day of our trip which I think we enjoyed the most.

We spent the rest of the trip in and around Lucerne and on the last day we made a day trip to Zurich. This allowed us to catch our flight straight from Zurich Airport back to London. All in all, a wonderful and highly recommended to anyone interested

In any case, I’m glad I finally passed, now I need to learn about cars and what to buy, etc. I’ve just never been into cars, but I feel I might become an avid car enthusiast. I do like to tinker with technical things, the more geeky the better

A few people, including me, volunteered to give short presentations about certain aspects of SLED, and I was giving a short talk and demo of XGL under SLED. I demoed it to about 20 people and they were all quite impressed with it. Along they way I managed to help out several people with their linux problems.

I absolutely loved this meetup and I hope I can attend many more in the future. There are more details and pictures here.

LinuxWorld started yesterday here in rainy London and I had a great time! It was the first time ever that I wasn’t a visitor, but was helping out at the Jokosher stand. I did several demos of the app to tons of people and we managed to distribute more than 130 Jokosher flyers to interested people.

I was quite surprised at the level of interest in the app, and we managed to solicit a great many feature requests from people, some of which will hopefully end up in Jokosher someday, making it rock even harder! I will definitely try to help out at more of these events in the future

You can find the photos I took with my camera phone at my flickr photo page.


Error: no known VMs. (check for corrupt jvm.cfg file)


I couldn’t run 'java' or 'javac' from the command-line at all! I immediately went to /System/Library/Frameworks/JavaVM.framework and looked for 'jvm.cfg'. I have 3 VMs installed on this machine, 1.3.1, 1.4.2, and 1.5.0. 1.3.1 and 1.4.2 had a proper jvm.cfg file installed but for some reason 1.5.0′s jvm.cfg was a zero-length file. Googling didn’t turn up anything useful except this tip, which wouldn’t work because in my case all my permissions were correct. Fixing permissions using Disk Utility didn’t show any permissions problems at all.

In the end, all I did was copy the 1.4.2 version over to the 1.5.0 directory and all was well. The tip above mentions that Eclipse might have had something to do with this, and there might be some truth to that, as I did update my Eclipse to 3.2 recently, but I have been using it for a week without any problems …

Thankfully I just discovered the wordpress-to-wordpress plugin which allows me to export my old blog as a WXR file which wordpress.com can then import. Now that I am back I will try to keep updating this blog more often.

It all went perfectly of course and a month later I’ve had no problems with the RAM. It’s so simple to buy it from crucial too. They have a comprehensive database of products (computers, laptops, motherboards, etc) and the type of RAM these products take. Simply make the right selections and they tell you what to buy, even how many slots you should have free by default. If even that’s too much for you and you happen to be on a Windows PC or laptop they have an ActiveX applet that will download and try to figure out the RAM type for you. This didn’t work for my system though.

Anyway, the moral of this story is: Always buy RAM from Crucial. It is crucial that you do so