Clug Park-Tech

The newest version of my IRC bot, irbie is able to evaluate python code in channel and respond with the output.

It does this in only 33 lines of RUBY code.

#!/usr/bin/env ruby
require 'rubygems'
require 'mechanize'

class Sacrilege


  def initialize
    @code_stack = Array.new
    @agent = WWW::Mechanize.new
    @agent.user_agent_alias = 'Linux Mozilla'

    page = @agent.get('http://shell.appspot.com/')
    @form = page.forms[0]
  end

  def eval(msg)
    @code_stack.push(msg) if ( msg =~ /:$/ || !(@code_stack.empty?))
    if msg == nil && !(@code_stack.empty?)
      set(@code_stack.join("\n") + "\n")
      @code_stack.clear
      page = @agent.submit(@form)
    elsif @code_stack.empty?
      set(msg)
      page = @agent.submit(@form)
    end

    return page.body.split("\n").slice(0, 15) if page
    []
  end

  def set(val)
    @form.fields.find{|f| f.name == 'statement' }.value = val
  end


end

Of course, as with everything irbie does it’s smoke and mirrors. I just call it efficient use of webservices.

If you haven’t used mechanize before, it’s also a good advert for mechanize, which, by the way, has been ported to many other languages. Python included.

• Drive 1:
• Partition 1 (35GB): Windows XP
• Partition 2 (55GB): Windows 7 RC (shiny toy)
  
• Partition 3 (rest): Linux data (/home - the serious stuff)
• Drive 2:
• Partition 1 (30GB): Ubuntu Linux 9.04
  
• Partition 2 (15GB): (to test other distros on the hardware)
  
• Partition 3 (2GB): Linux swap
• Partition 4 (rest): Windows data (C:\User - games & media)

1. Do all partitioning before installing Windows, and only use Linux to edit partitions (cfdisk or gparted).  
• The Windows 7 partition manager confuses XP because Windows 7 changes from "cylinder aligned" to "megabyte aligned" partitions.  However, both can understand what Linux writes. 
2. For Windows XP to complete installing, you have to install it before installing Windows 7. 
   
3. Let Windows 7 have the MBR on Drive1, which gives options to boot Windows 7 or XP.  Install the GRUB MBR on Drive 2, and set the BIOS to boot Drive 2 first so that GRUB runs by default.  I figure keeping the bootloader could be handy.
• Or: use GRUB only and overwrite the Windows 7 MBR with GRUB
  
• Or: install NeoGRUB on Windows 7 and add a chainloader for the Linux partition
  
4. Putting the data on the opposite drive from its host OS is to avoid user disk operations slowing the running OS.  However, the drives are not independent - you don't have "windows drive" and a "linux drive".

Seek and ye shall find…

My car stereo (a JVC that accepts USB thumb drives) is really slow at seeking to a specific position in long audio files [To be fair regarding long seek times, the JVC does seek faster off a CDR disk. However, it’s not much faster and then you’re left with a pile of disks that have been played only once.]. So, if you loose your place in a long podcast (e.g. the kids want to listen to something) you can spend up to 15 minutes holding down the fast forward button just to find your place again. It also jumbles up the play sequence for all the audio files within a folder. Both problems can be easily solved on the computer or when you put the audio files onto the USB drive.

My solution to the first problem is to split the audio file into multiple files, each file being 5 minutes long. mp3splt is a free utility that splits mp3 or ogg files without decoding the file into another format first, so there is no lossy decryption and encryption step. It is also pretty neat in that mp3splt tries to split on silent areas of the file (in between words), and the newer versions can preserve the ID3 tags within the file.

Here is the incantation that I’ve worked out to call mp3splt:

mp3splt -g %[@o,@N=1] -t 5.00 -a -o @n_@t YOUR_AUDIO_FILENAME

A brief explanation of the various command line switches:

Note that the ‘-g’ clause of the above was not available from the Debian Lenny repository version of mp3splt (that version is very old).

Optional extra: On BSD/Unix/Linux systems instead of remembering that whole string, I would recommend putting the above incantation line into a file with executable permissions in the /bin folder called say /bin/mp3s. Replace the YOUR_AUDIO_FILE part with $1. So you could then easily split an audio file called SOME_FILE simply by typing: mp3s SOME_FILE and it’s not necessary to remember the complex switches as above.

The solution to the second problem (scrambled audio files within a folder) where files seem to be playing in some random order is really trivial. It seems that windows users have never encountered the problem because when they copy an entire folder to the USB drive, the file modification information is in the same sequence as the filenames. For instance, if a windows user were to copy a folder with three tracks in it

track01.ogg
track02.ogg
track03.ogg

to the USB disk the top file would be the oldest one and the bottom file would be the youngest one. If you did the same thing on a Linux workstation, say by copying a folder with

cp -r somefolder /mnt/usb/

The bottom file from the list might not necessarily be the youngest one (in other words the file modification time is not in the same sequence as the alphabetical sequence). The solution is really trivial, copy your files like this:

cp somefolder/* /mnt/usb/somefolder/

The asterisk (*) gets expanded by the shell to a file list in the same order as you see when you list your files sorted by filename.

Seek and ye shall find…

My car stereo (a JVC that accepts USB thumb drives) is really slow at seeking to a specific position in long audio files [To be fair regarding long seek times, the JVC does seek faster off a CDR disk. However, it’s not much faster and then you’re left with a pile of disks that have been played only once.]. So, if you loose your place in a long podcast (e.g. the kids want to listen to something) you can spend up to 15 minutes holding down the fast forward button just to find your place again. It also jumbles up the play sequence for all the audio files within a folder. Both problems can be easily solved on the computer or when you put the audio files onto the USB drive.

My solution to the first problem is to split the audio file into multiple files, each file being 5 minutes long. mp3splt is a free utility that splits mp3 or ogg files without decoding the file into another format first, so there is no lossy decryption and encryption step. It is also pretty neat in that mp3splt tries to split on silent areas of the file (in between words), and the newer versions can preserve the ID3 tags within the file.

Here is the incantation that I’ve worked out to call mp3splt:

mp3splt -g %[@o,@N=1] -t 5.00 -a -o @n_@t YOUR_AUDIO_FILENAME

A brief explanation of the various command line switches:

Note that the ‘-g’ clause of the above was not available from the Debian Lenny repository version of mp3splt (that version is very old).

Optional extra: On BSD/Unix/Linux systems instead of remembering that whole string, I would recommend putting the above incantation line into a file with executable permissions in the /bin folder called say /bin/mp3s. Replace the YOUR_AUDIO_FILE part with $1. So you could then easily split an audio file called SOME_FILE simply by typing: mp3s SOME_FILE and it’s not necessary to remember the complex switches as above.

The solution to the second problem (scrambled audio files within a folder) where files seem to be playing in some random order is really trivial. It seems that windows users have never encountered the problem because when they copy an entire folder to the USB drive, the file modification information is in the same sequence as the filenames. For instance, if a windows user were to copy a folder with three tracks in it

track01.ogg
track02.ogg
track03.ogg

to the USB disk the top file would be the oldest one and the bottom file would be the youngest one. If you did the same thing on a Linux workstation, say by copying a folder with

cp -r somefolder /mnt/usb/

The bottom file from the list might not necessarily be the youngest one (in other words the file modification time is not in the same sequence as the alphabetical sequence). The solution is really trivial, copy your files like this:

cp somefolder/* /mnt/usb/somefolder/

The asterisk (*) gets expanded by the shell to a file list in the same order as you see when you list your files sorted by filename.

SuperGenPass and Password Composer are password generators, which generate different passwords for each site you use based on a single master password. This gives you the convenience of only remembering one password as well as the security of using different (and strong) passwords for each site. This means that you won't have all your accounts compromised when¹ one of them is compromised.

Most password generators are implemented as browser extensions or bookmarklets, since they are most frequently needed in a web browser. I've been wanting to start using a password generator, but I wanted to be sure that I could access my accounts even if I didn't have a web browser accessible. The two situations I could think of were a command line only system (e.g. SSH) and my cellphone².

Surprisingly, I couldn't find a command line implementation of SuperGenPass, so I wrote one in Python. I also couldn't find any J2ME or Symbian implementations, and so wrote my own one in J2ME. They both support subdomain stripping and configurable password lengths. They don't support salted passwords.

I chose SuperGenPass over Password Composer because it uses a better scheme. Password Composer only uses hex characters, whereas SuperGenPass uses a base64 encoded hash. SuperGenPass also hashes the password multiple times (which would slow down a brute force attack to find the master password) and imposes complexity requirements on the generated password (which reduces the chances that the generated password can be brute forced).

• Buy for a 5 year lifespan. Less than 5 years makes a computer one expensive consumable. Even at my R10,000 price point, computing is a R2000 per year expense.
  
• Skip generations. I am buying a Core 2 Duo, and my next PC will be the successor to the Core i7 (but only once its successor comes out - see "best of 18 months ago")
  
• Buy wholesale parts. I thoroughly recommend DC3 Distributors - the powerful box below comes to the same R8000 that one would normally pay for underpowered entry level crud at Incredible Connection. Be prepared to spend hours reading hardware reviews, benchmarks and spec sheets to be sure you are getting reliable, performing, mutually-compatible parts.
  
• Buy the best of 18 months ago. After the new generation arrives (give it 18 months), the previous generation becomes mid-range, and yields the best price/performance ratio.
  
• It will always cost around R10,000. The "best of 18 months ago" desktop always comes to around R10,000 (including monitor). This principle, stated by my father, has held between 1987 (remember the 80386?) and 2009 (Core 2 Duo / Phenom II)

• Processor (R1411): Intel Core 2 Duo 2.8GHz, 3MB L2 cache
  
• Motherboard (R1165): ASUS-P5QL Pro Intel P43 chipset, 1066MHz FSB
  
• RAM (2xR298): 2 x Apacer 2GB DDR2 PC6400 (400MHz, 800MT/s)
  
• Graphics (R1233): Zotac Nvidia 9800GT 512MB GDDR3, at 660MHz
• Disk (2xR679): 2 x Western Digital Caviar Green 500GB / 16MB cache
• Optical (R350): Sonu DRU-830C (have it already)
• Wifi (R254): DLink 54Mbps WiFi
• Case (R439): CoolerMaster Elite 335 ATX
• PSU (R450): Gigabyte ODIN 470W
  
• Mouse (R225): Microsoft Comfort Optical 3000
• Keyboard (R600): Logitech Wave
• Monitor (R2300): Samsung 2494HS, 23.6 inch 16:9,1920x1080
• Assembly (R114): Because frying the motherboard costs a lot more.
  

• PSU (R1234): CoolerMaster Silent Pro M500
• Chassis (R819): CoolerMaster Sileo 500
• Graphics (R1233): Zotac Geforce 9800GT Eco

From time to time, I've talked about my open source lyrics projection application, openlp.org, which is written in Python and Qt4 using the PyQt4 bridge. One of the core features of openlp.org is to display lyrics on a second monitor using the current "theme," which is a collection of settings for displaying the lyrics (background, font, etc).

Recently, we've hit a bit of a snag to do with the lyrics display. Functions in the Renderer class, which performs the drawing, were taking incredibly long to execute, which meant that the application was almost hanging whenever you wanted to load a song (be it into the preview or live slide controllers). So we started doing some investigation.

Tim Bentley managed to figure out that the QPixmap.scaled() function was the main problem. When creating the small preview image, we resize the background image (using the scaled function) and then paint the words on there. Martin Thompson then e-mailed the developer mailing list with some really handy info - a Python profiler.

First, if you're on an Ubuntu or Debian based like me, you need to install the python-profiler package. Once that is done, run your application like so:

$ python -m cProfile -o openlp.prof openlp.pyw

Where "openlp.prof" is the profiler's output file, and "openlp.pyw" is your application's main file. 

Then create a small script (I called mine "profiler.py" ) with the following code in it:

import pstats
p = pstats.Stats('openlp.prof')
p.sort_stats('total').print_stats(20)

Run this file after you have run your application, and it'll read the profiler file and create a summary with the top 20 results, ordered by the internal time each function took.

This helped a lot. I was able to time my functions as I played around with various Qt4 classes. 

While reading up on various rendering techniques and Qt4 classes, I came across the following sentence on the QImage page:

QImage is designed and optimized for I/O, and for direct pixel access and manipulation, while QPixmap is designed and optimized for showing images on screen.

When I replaced the QPixmap that we used to scale the background image down with to a QImage, the scaled() function dropped from 3 seconds to 0.635 seconds. A marked improvement!

However, the QPixmap.fromImage() function was taking a while, so I replaced ALL the QPixmaps with QImages, and suddenly my total rendering time went down to less than 1 second, instead of 3 - 4.

In the mean time, Tim Bentley had another look at the rendering functions himself, and through reducing the number of calls to a few functions, managed to significantly reduce the rendering time even further!

For both of us, the Python profiler was invaluable.

Fishbowl Sessions

This morning I attended a session on burnout. It was different to usual sessions in that the table has been removed from the room and the chairs were arranged in such a way that there is an inner circle and an outer circle. People who are more interested in the topic sit in the inner circle and people who are interested in keeping up to date or who might contribute more casually sit in the outer circle. This experiment is called fishbowl sessions, it’s happening with all the sessions in room 11 today.

Burn Out

We had a really good discussion about burn-out, what causes it, coping mechanisms and how to avoid it.

This is some of the things that were mentioned:

Burn-out sometimes tend to happen in cycles, and it can also easily be triggered by external factors, like burning out at work or even when your boss runs into burn-out. Negative influences seem to make burn-out worse, while spending quality time with people who share your views seem to  cool the burn-out somewhat. Many people used to do free software as a hobby and now doing it as a job as well. It’s important to get new hobbies, go work at the zoo. Having a girlfriend helps a lot in terms of grounding and having someone to talk to. Otherwise having a friend that knows you well and understands you can work just as well. Perhaps putting together a talk on burn-out that could be presented to loco teams would be of much value. Mike Basinger mentioned that he’s come accross members who have even been suicidal and that it helped a lot when they were refered to a professional. Can a whole team burn out? Some people have seen some situations where that has happened. A burn-out / health session on communities will probably be held at future UDS’s as well.

Taking on too much - Don’t kill yourself trying to be the next Colin Watson

In a project such as Ubuntu, there are lots of people we look up to and try to aspire to be like. It happens regularly that someone works really hard trying to catch up to someone elses skill level and they end up doing more damage to themselves than good. Many people actively take on too much, finding themselves to juggle too much and not doing enough leading up to more frustration. Jono mentioned that Canonical is an interesting company in the sense that managers actively have to tell their team members to stop working. It was also mentioned that it’s important to let people know that Ubuntu is like a big machine and there are lots of big and small coggs and that if a small one breaks, it can have massive implications for the bigger machine and that they are also important.

Mark Shuttleworth Shares Tips on Burn-Out

Mark often walks into sessions for a few minutes. I think he takes just enough time to gauge what the discussions are about and if it’s going into the right direction and tone. He said that what works well for him is to get to bed and get some decent sleep, and then get some good excercise when he wakes up. Other people in the session confirmed that excercise has indeed helped them in feeling good and being more productive.

So, there you have it. If you want to be a good developer or contributer and make Ubuntu as good as it could possibly be, then take care of yourself, stay healthy and as Jono said earlier this week… eat your vegetables

This week there’s been lots of interest in Moblin at the Ubuntu Developer Summit. It surely is very shiny and flashy, and I’m sure it will make a big impact on the mobile world when it is ready for mass consumption.

According to Engadget, Linpus Linux, which is distributed on Acer Aspire One netbooks, is working on a Moblin desktop for netbooks. I would actually expect Ubuntu do be the first distribution to do this, but from what I heard Moblin is really just not ready yet. Perhaps something to consider in the Karmic+1 release cycle?

These are just some points I took down during todays session at the Ubuntu Developer Summit. Better notes may be available by the time you read this via Gobby on the gobby.ubuntu.com server.

Architecture of a Directory Infrastructure

Blueprint: https://blueprints.edge.launchpad.net/ubuntu/+spec/server-karmic-directory-architecture/

• Recap: Yesterday kerberos was discussed as a default solution for authentication in Ubuntu, dns, openldap, etc
• Brief discussion on how slave/masters should be selected
• DHCP in OpenLDAP will probably not be required for the Karmic, DNS could still be configured via DHCP
• Discussion on various patches that may be appropriate for kerberising services
• Should DNS be built-in to the directory server infrastructure or would it be ok relying on an external one? Karmic release won’t be so much about integrating with current infrastructure, but will be revisited for future release.
• Discussion around PTR records and other nigly DNS and DHCP issues, race conditions and avoiding spoofing.
• Discussion of relevant OpenLDAP features
• Password changes and password synchronization is still an issue with a Samba/OpenLDAP integration, Howard has a patch that could provide a solution for MIT OpenLDAP.

Improving Loco Marketing

• Paulo Sammicheli introduced ubuntu-it and explained that there are usually differences between English and non-English speaking Loco teams.
• “Evangalism” is possibly not a good word to use when refering to advocacy.
• Some loco teams work very much like LUGs, are too passive.
• 250 CD’s for loco team is by far not enough for Loco team areas that covers millions of people.
• Discussion on printed CD’s vs self-burned. Some people feel that local burned CD’s are harmful and the printed CD’s are the only way to go. Printed CD’s isn’t an option for many because of limited quantity, timing, etc.
• Spread Ubuntu should be promoted to loco teams more.
• Paulo played a “Stand by me” video that was quite good that could be used as an inspirational video. It features street artists from all over the world, something similar that combines loco teams from all over the world would be awesome.
• A follow-up session will be held on IRC and be announced soon.

Loco Council Review

• Some people don’t know what the Loco Council does, it may need to be communicated better.
• It does conflict resolution, team approvals and team reviews.
• When a leader isn’t contactable for team review, the council should attempt to contact the most popular posters of the loco list, a team shouldn’t be deactivated/punishes when the leader is unavailable for whatever reason.
• Ubuntu-ZA had a good experience with the Loco Council review, perhaps loco teams should provide more feedback on the review process.
• Loco Council would like some more feedback and ideas.

Build a very light desktop based for Ubuntu with LXDE

• LXDE is a very light desktop environment widely used in Asia with a large user community, runs fast in 64MB memory.
• It’s themable, adjustable, has a light filemanager (although horrible usability, maybe something worth investigating)
• How much of gconf, etc is required to integrate network manager, update manager, etc? Seems like gcong may still be required.
• A .xsession file may be required to start update manager, notify-osd, network manager, etc since LXDE does not start the xdg autostart applications automatically.
• wiqd and conman has been mentioned as possible alternatives to network-manager.
• e17 isn’t a good alternative candidate, it’s not released yet.
• Do we want an image, or just a meta-package? Ogra suggests only a meta-package for now. People in the session generally agrees.

Pleniary Sessions

• Daniel Cheng talked about Ubuntu’s audio stack.
  • How ALSA and OSS came about.
  • Why Pulse Audio is necessary.
• Canonical vs Community - An Outside Study.
  • Findings from university students studying Canonical’s business model
• Moblin and Android
  • A demo of Moblin and Android running on Ubuntu. Moblin is really awesome.

Edubuntu Karmic

• Add-on release such as previous releases, the work will begin to make Edubuntu a full release again for Ubuntu 10.04 (Karmic+1)
• Universe will be enabled for builds so that applications from universe can be included.
• The Sugar interface will become part of the Edubuntu software bundle in Karmic and users will be able to install it as a optional packages. Integration of tasks from sugar in the desktop menus will be investigated.
• Localised menu support, currently some districts, most notably in Spain and in Canada requires the menus to follow a manu structure that is alligned to the curriculum and locality. Guadalinex currently addresses this by modifying 43 packages in Ubuntu for their system. We’ll attempt to produce a mechanism that would reduce their workload with regards to the menu implementation drastically.

LoCo Team Conference Packs

• The t-shirts that say “Ubuntu Guru” might be better without the “Guru” part- “Ubuntu Guru” sounds terrible in Italian.
• Language is an issue, in many countries the brochures in English aren’t of any use.
• Mozilla provides big banners for their local chapters that can be re-used, that could work for Ubuntu as well.
• Positioning text in the middle for banners/brochures is good practice, accommodates cultures that read from left to right and from right to left.
• Some people have ordered converence packs directly without consulting the loco team which creates problems.

Update (April 2009): Holy crap! The IEC really did get their site updated for the elections. I almost fell off my chair. The election results are inexplicably only available as PDFs that you have to download, but hey — baby steps.

So, if you live in South Africa and have The Internet, you probably already know that the IEC website is crap. It has been crap for years. It used to be bizarrely malformed in anything that isn’t IE, and lots of people complained.

Recently the IEC dramatically improved the situation by adding a browser check to their main page, and redirecting any browsers that don’t identify themselves as IE to an apologetic note which explains that the site doesn’t work in anything except IE. Please note that they had time to add Google Chrome to the list of other browsers — but not to actually fix the damn site; something which you might think is not rocket science, or very expensive to do, especially in this age of out-of-the-box CMSes and web development frameworks.

Of course the site still works in Firefox, exactly as badly as it used to — and the browser check is trivially circumventable. All you have to do to see it in its full malformed glory is navigate to any internal page. If you’re feeling energetic, you can make your browser lie in its user agent string. Be prepared to reload frequently — not only is the site atrociously designed and basically unmaintained (how long is that <\table> going to be there?); the server is a bit dodgy.

Now, people have been complaining about this crap for years, to little effect. Nobody seems to be particularly interested in fixing the problem.

Earlier today several people I know posted links to DigitalApartheid.com, a new site created by someone who is fed up with the state of the IEC website. As much as I agree with the purported goals of the site, I am not impressed with the way it has gone about achieving them, for several reasons.

One: the site instructs visitors to email or fax a form letter complaining about the site to various employees of the IEC. Form letters are crap. Form letters say “I’m not capable of articulating an intelligent opinion about this; I’m with that guy, so I copied what he said.” I wouldn’t be surprised if they were forwarded straight to /dev/dustbin at the IEC; I know that’s what I’d do.

Two: the form letter is full of bad punctuation and grammar. Badly written complaints make you look stupid.

Three: the form letter compares the site’s exclusion of non-IE users to apartheid, and states that the writer is contemplating not voting unless the site is fixed. Here’s where we go off the deep end.

Seriously? You really think that the inconvenience that you experience at this site because of your (admirable and sensible) choice to use a browser other than IE is comparable to decades of racist government oppression?

Dude. Maybe you should get some perspective.

Most people in South Africa don’t have access to computers. The IEC website is not the only — or indeed the primary — source of information about the elections. This information is not being denied to you — if you can’t access it in your browser (and you can, really), you have the ability to get it in some other way, just like all those computerless people.

I do want the IEC to fix their site — but bombarding them with ridiculous hyperbole isn’t going to make them do it. There are plenty of intelligent things to say about standards compliance and FOSS, and why they are important. If you’re going to send a complaint, please do it in a way which doesn’t make you — and by association everyone else who uses an alternative browser — look like a raving nutjob.

URLs shouldn't really contain file extensions (like .html, .png) since they are supposed to identify a resource and not a particular representation/format thereof. The format is indicated by the Content-Type header sent in the response. Modern CMSs do this already (for example, the URL of this page doesn't include .html).

Doing the same for static files (i.e. files served directly by the webserver) isn't straightforward because most webservers use the file extension to determine the MIME type to send in the Content-Type header. This means that simply removing the file extension from the filename (or even creating a symlink without a file extension) will cause the webserver to send the wrong Content-Type header.

I decided to try find a solution to this for my webserver of choice, Lighttpd. Lighttpd has a module which embeds a Lua interpreter and allows you to write scripts which modify (or even handle) requests. So I wrote a script which searches the directory for files with the same name as requested but with an extension. This means that any file can be accessed with the file extension removed from the URL while still having the correct Content-Type.

The script currently chooses the first matching file, which means that having multiple files with the same name but different extensions doesn't do anything useful. The proper method however is to actually do content negotiation, which chooses the format based on the preferences indicated by the HTTP client in the Accept header.

To use this script, download it and save it somewhere (I use /etc/lighttpd/). Enable mod_magnet, and add the following line to the site definition.

magnet.attract-physical-path-to = ("/etc/lighttpd/extension.lua")

Following on from yesterday's post, I decided to try implement proper content negotiation. After a fair amount of time spent getting to grips with Lua, I got a script which works very nicely. It implements server driven content negotiation for media types.

The basic idea of content negotiation is that a resource (e.g., this graph) exists in multiple formats (in this case, SVG, PNG and GIF). When a user agent requests the resource, it indicates which formats it understands by listing them in the Accept header. The server compares these to the available formats and sends the best one. So a browser which can display SVG will receive the diagram in SVG format, while a browser which can't will receive it in PNG (or GIF) format.

(The following description assumes knowledge of the Accept header format.)

The script works by searching the directory for files with the requested name but with an additional extension (each of which is a variant). The media type is inferred from the extension using /etc/mime.types, and the quality of the type is set by a hardcoded table in the script. Each variant is checked against the acceptable types sent by the user agent, and the overall quality calculated by multiplying the quality with the q parameter in the Accept header. The variant with the highest overall quality is then chosen.

Some browsers include wildcard entries such as image/* and */* in the Accept header without specifying a q parameter. This parameter defaults to 1 (the highest value), which means that no preference is actually indicated. The script implements the same hack that Apache does in order to compensate for this. It also handles directory index files by defaulting to files named "index".

To install the script, download and save it somewhere (such as /etc/lighttpd/). Then add the following to the site definition.

magnet.attract-physical-path-to = ("/etc/lighttpd/negotiate.lua")

I though it might be interesting to try and graph the Reserve Bank’s prime rate data… It goes back a long way. I used Python to scrape and collate the data and PyCha to generate the graph.

UPDATE: I’ve replaced my graphs with new versions made by Russell who corrected my original code by interpolating the data correctly over the y axis.

This is the narrow version.

And this is the wide version (click to download the actual 10000px wide png)

Interestingly enough, todays rate cut *was* on that page earlier today, but now I see it’s gone… so I inserted it manually ;)

So Facebook has changed their layout again.

And people are protesting again.

And what's going to happen (again)?

Nothing. So why moan?

To be perfectly honest, when it comes to Facebook changing their layout, I really don't care. As long as I can use it to interact with my friends, I'm happy.

(Stop reading now if you don't want to hear me moan about moaning.)

One would think that folks who use Facebook and other social sites like Youtube and MySpace would be used to change. Afterall, isn't that what the Internet is all about?

And yet they moan when change happens. Folks even create Facebook groups (which I've been invited to on many occassions) to try to raise a protest. They even "threaten" not to use Facebook anymore - as if Facebook really cares. The number of folks who actually stop using Facebook in protest is so small when compared with the rest of the users, that they make no impact whatsoever. And even if they did, I seriously doubt Facebook would listen to them.

So please folks, give it up. Take that energy that you would have spent on negativity, and spend it on being positive towards your friends on Facebook. Is there a friend who is having a tough time? Why not give them some encouragement? That's a better way to spend your energy.

Yesterday I was thinking about looking for something in my Quassel logs, and I didn't really feel like trying to scroll through the last 2 weeks' worth of backlog to find it. Also, at this stage Quassel unfortunately doesn't have a search feature either (Quassel devs, count that as a feature request ).

So I hauled out my latest trusty tool: Python.

Quassel stores it's data in an SQLite 3 database, so I opened the database and had a look at it's structure, and then wrote a Python script to search the database.

Therefore, I proudly present Quassel Search 0.1 for Quassel 0.4.

How to use it:

• Download the script
• Set it's executable bit:

  chmod a+x quassel-search.py

• Run it with the help option to see it's options:

  ./quassel-search.py -h

Notes:

• I have only tested this on Python 2.5. If you're using 2.4 you'll need to install the SQLite 3 module.
• You probably want to download your database to a local machine before searching. Also, rather make a copy of the database before you download it, because otherwise you will most likely end up with a corrupted database.

Update

I've been told there *is* a search function in Quassel, but it only seems to search the current text in the current buffer, not the backlog. I could, once again, be wrong though.

• Home row keys (arstdhneio) put the 10 most frequent letters in English under your fingertips
• Pinky finger is used only rarely
• Loads of "hand-roll combos" where you type 2, 3 or even 4 keys in one smooth motion.
• 10 keys stay where they are in QWERTY (namely Q,A,Z,X,C,V,B,H,M)
• Most windows keyboard shortcuts stay the same (see above)
  
• All keys except E and P are typed with the same finger or same hand as on QWERTY.
  

A while ago I blogged about a weird comment I had received on one of my blog posts.

In summary, there is an SEO company called SEO Results (aka BizSearch, aka NetAge) that gets its staff to trawl blogs and write comments with the Author URL set to the url of one of their SEO clients.

Author : PMM (IP: 165.146.34.239 , dsl-146-34-239.telkomadsl.co.za)
E-mail : kim@bizsearch.co.za
URL : http://www.pmmproperties.co.za
Comment:
Wow what a difference it looks fantastic, great job done

One would think that after the first run in I had with these spammers they would have avoided my blog?

Anyway, to make sure it’s clear: SEO Results are spammers and black hat SEO idiots… Using them is likely to get you bad mouthed on the internet (like this) and perhaps worse, blacklisted on google.

First let me say that I like the idea of a new, fresh news site… EWN could quickly become a serious player in the news arena, but before they do so they’re going to need to fix a few issues.

I sent an email listing some of these issues to the Primedia team. I know it got there because people who know people said there was some flapping and urgent updating that happened as a result of the email… However, I’m yet to get any form of reply whatsoever… which I think is just rude.

(update: A few things (like the comments about Mandela) have been fixed, but the overwhelming majority is still as it was when I wrote this list a few days ago. The site however seems to be suffering from lots and lots of timeouts now.)

This list is by no means exhaustive…

1. You need to add a DNS record for ewn.co.za (so that http://ewn.co.za actually works)

2. You need to add RSS, preferably ATOM, with a number of sub feeds, geographic locality etc.

3. You need to remove your stupid comments from your html source… not only is it dumb, but people WILL take offence.

<!–<li><a href=”#”>Mandela Gives Birth to a Gorilla </a><span class=”timeadded”>2&nbsp;days&nbsp;ago </span> </li><li><a href=”#”>Prengant Child attacks Mandela</a><span class=”timeadded”>3&nbsp;days&nbsp;ago </span></li><li><a href=”#”>Tourists Can’t Give Enough Birth </a><span class=”timeadded”>1&nbsp;day&nbsp;ago&nbsp;</span></li>–>

etc

4. You need to make sure all your templates actually work… for instance this one is a little too concise –
http://www.ewn.co.za/story.aspx?id=4013

5. You need to protect yourself from SQL injection and handle any attempts gracefully.
ie. http://www.ewn.co.za/articleprog.aspx?id=40%2709

6. You should probably consider looking into better urls for your articles, specifically for SEO purposes.

7. You should also probably add meta descriptions (and possibly tags) to your article pages. This will help display relevant content in search engine results.

8. Your pages do not even come close to validating XHTML transitional.

9. You need a mobile version! This is easy to implement!

10. That logo… It’s very 90’s.

11. Bonus Tip: One of my biggest gripes with the other news sites is how they never allow you to view larger versions of their images. Implementing Lightbox2 over you existing site will be easy and help
differentiate yourselves from the other players.

12. Your site search is broken in Firefox and Safari and is unstable in IE6 and 7.

13. Your server errors (timeouts etc) need to be handled more gracefully. At the moment your site displays the default .NET error pages, which is something that only the developers should be seeing.

14. Your comment form gives no indication that it hasn’t submitted due to invalid data. This will confuse users.

15. Besides the SQL Injection issues, users who search for any string that contains an apostrophe will be greeted by a rather ugly error page. Try search for o’grady.

16. You need to remove all your test data from your database. http://www.ewn.co.z/articleprog.aspx?id=183 etc

17. You should add a clearfix after your pull-out-quote on your article pages. This will ensure that articles that start with single character words like “A” don’t end up displaying the first character to the right of the pull-out with the rest of the article below the pull-out. See http://www.ewn.co.za/articleprog.aspx?id=4021

18. Your logo should be a link to your landing page. This has become a web standard and a lot of users will expect it to do so.

19. You should sanitise your article source before your editors submit it so that you don’t end up with styling imported from MS Word which can break your layout. ie. 

<p class=”MsoNormal” style=”MARGIN: 0cm 0cm 0pt; LINE-HEIGHT: 12pt; tab-stops: 18.0pt 91.6pt 137.4pt 183.2pt 229.0pt 274.8pt 320.6pt 366.4pt 412.2pt 458.0pt 503.8pt 549.6pt 595.4pt 641.2pt 687.0pt 732.8pt”>

Not only is it ugly but it will repeatedly break your validation.

eg. on http://www.ewn.co.za/articleprog.aspx?id=4033

20. While it’s debatable whether this is a true bug, there is a fair amount of functionality on your site that is broken when the user disables javascript.

21. As I’m browsing the site I am hitting a lot of timeouts. This indicates that your server is probably struggling. Most likely due to bad coding and/or a database that isn’t tuned properly.

22. Your cache control is not good. You should probably add far future expire headers to all your static resources. This will speed up the site for regular users. Also, combine and gzip your js. This will also decrease load on your site and help with all the timeouts.

I’m gradually sorting out my web presence. Something I’ve been meaning to do for a really long time is put all the little bits of code I write in an easily accessible public place, so that other people can use them. Hodgestar and I have set up Locustforge to be a site for our shared projects. So far it has a wiki (which still needs cleanup) and an svn repository for our code.

I intend to move all my code — which is presentable enough to be shown in public — in there. So far, I’ve added the PmWiki skin we use on our wiki, and some RSS feed filters which can be used with Liferea (and probably other readers too).

Recently I upgraded my phone to a Nokia N81. I set up my Zyb account, and sync'd my contacts, and realised that I could not schedule syncs like I had with my SE w810i. Until this morning, when Russell pointed out on IRC that there was a tool for that exact thing: Swim is a utility for automatic periodic synchronization of data with internet servers. It is part of the opensource project called Bergemot. The only hurdle is that you have to get the SIS file signed by the Open Signed Online and when you install it you are presented with a warning that your phone may turn into a lump of metal because you are using Software In Development. Works fine for me though.

In a week we are moving houses. We have a list of things that need to be sorted before we move. So, one of the things on the list is moving our phone line/ADSL. I have a one price ADSL from cybersmart so I can take advantage of their Night Rider plan. Little did I know, paying for our ADSL portion from another supplier was something Telkom Just Couldn't Handle during a transfer.

I phoned Telkom's 10219 number, and was told I needed to call a random number to have the transfer done, because there was an ADSL linked to it. So we tried the cybersmart route. Mandy phoned and asked if we could have the line moved. Cybersmart wasn't so sure. So, we tried Telkom again. After some lengthly discussions Telkom came back with the point that the ISP has to initiate the move. So, back to cybersmart. This time cybersmart was helpful, and said I just needed to fax them my details and request, and they could go ahead, provided they had my signature. Cybersmart phoned back a few days later to tell me the fax was quite light, so they couldn't make out much. After chatting to the very helpful cybersmarter, she said she could transfer the ADSL portion back to Telkom, and then I could handle the transfer, and once it was complete Cybersmart could migrate the line back. Convoluted, but Telkom understandable. Took a few days, but today the migration went through, and round 2 started.

I phoned Telkom, and the very helpful call centre person started to sort me out. No charges, just time. I thought I was on the home run. Then I pointed out that the line had an ADSL on it, and I wouldn't mind having the number changed. Apparently it costs R543.23 to transfer an ADSL line between premises. WHAT? But, doing a self-install of ADSL costs R0.00. So, after some protracted negotiations, the path became clear:

Cancel my ADSL portion ("downgrade" my line), transfer to the new premises (and because there is no ADSL associated I can roll in a phone number change), once the line is active in the new house, then I can re-initiate the ADSL portion for free. WOW.

It's like peeling layers off an onion, and then putting the onion back together. Through all of this the call centre people were lank nice and helpful, but the system gets me down.

Just to re-cap, to have ADSL and a phone line moved to my new house (which is about 1km away from my old one) I have to:
Instruct Cybersmart to migrate my ADSL back to Telkom, Ask Telkom to cancel my ADSL, Ask Telkom to move my phone line to my new house, Wait for the new line, Ask Telkom to "upgrade" my line to include ADSL, transfer the ADSL back to Cybersmart. Awesome system guys.

Yesterday I posted about how a local web company recently sold the work of a well known, local, independent web professional. What they did broke netiquette, ethics, and definitely copyright law. Much geek froth and outrage occurred, and a tiny storm broke out on the internet. I jumped on the bandwagon with my post because:

1. I have SIWOTI syndrome
2. Content climbs higher on google when lots of reputable sources link to what is considered to be definitive text. Whijo is considered to be somewhat of a reputable source by google, and because I use the best CMS on the internet google likes reading what I am writing, so I wanted to contribute to improving Coda's rank on google for this subject
3. My goal is to improve the quality of the South African web, and improvement/evolution comes when the economy/environment favours better products, and denounces poor product. I denounced poor product

The only problem is that sometimes success quickly exceeds expectation, and in this case, after google crawled whijo.net, my article (then entitled 'Do not use the services of ClickThinking') landed on the first page of results. Coda's much more democratic 'What were you thinking, ClickThinking?' reached higher on the first page of results, as it should have. So, geek-google-penis aside, the weight of what I had done (measured in the internet based ISO standard of LOLCATS, or Lc) struck me. I know it is all a storm in a tea cup, but a post with a title as venomous as that sticks around, and ultimately may take business from them (and I am in no position to decide if their poor form deserves to take business away from them). I had behaved like a Journalist (well, one who didn't do too well in the media and ethics course). I thought up a catchy headline, and published it with a self-congratulatory click. I guess it comes back to thinking before doing, and not being a turd on the internet. So I changed the title, and when the site is re-crawled it will have a new title which is a lot closer to the heart of the matter at hand, and a lot less sensationalist.

Sometimes my powerful Sense For Injustice conspires with my Sense For Bad Web Development, and I peak too soon, type before I think, and end up looking childish, and not accomplishing my goals. I am usually calm and rational, but some things short circuit over that calm, rational, ethical brain. So, in summation, I apologise to ClickThinking for going too far off the handle, but I still deplore what they did.

As an aside, should I really be able to get into the first page of results on google for a company that just won a web analytics award?

I’m constantly surprised when I come across long-time Linux users who don’t know about SysRq. The Linux Magic System Request Key Hacks are a magic set of commands that you can get the Linux kernel to follow no matter what’s going on (unless it has panicked or totally deadlocked).

Why is this useful? Well, there are many situations where you can’t shut a system down properly, but you need to reboot. Examples:

• You’ve had a kernel OOPS, which is not quite a panic but there could be memory corruption in the kernel, things are getting pretty weird, and quite honestly you don’t want to be running in that condition for any longer than necessary.
• You have reason to believe it won’t be able to shut down properly.
• Your system is almost-locked-up (i.e. the above point)
• Your UPS has about 10 seconds worth of power left
• Something is on fire (lp0 possibly?)
• …Insert other esoteric failure modes here…

In any of those situations, grab a console keyboard, and type Alt+SysRq+s (sync), Alt+SysRq+u (unmount), wait for it to have synced, and finally Alt+SysRq+b (reboot NOW!). If you don’t have a handy keyboard attached to said machine, or are on another continent, you can

In my books, the useful SysRq commands are:

b

Reboot

f

Call the oom_killer

h

Display SysRq help

l

Print a kernel stacktrace

o

Power Off

r

Set your keyboard to RAW mode (required after some X breakages)

s

Sync all filesystems

u

Remount all filesystems read-only

0-9

Change console logging level

In fact, read the rest of the SysRq documentation, print it out, and tape it above your bed. Next time you reach for the reset switch on a Linux box, stop your self, type the S,U,B sequence, and watch your system come up as if nothing untoward has happened.

Update: I previously recommended U,S,B but after a bit of digging, I think S,U,B may be correct.