Giving Unity Another Go

Yesterday I installed Unity5.0 and I was pleasantly surprised by some of its new features:

• I can set the panel background colour. By default, the Unity panel adapts itself to match the wallpaper colour. This doesn’t always work out, and with certain background colours it looks really horrible with the icons on it. I set mine to a none-harsh, dark grey and can now see my icons without any desire to fork out my eyes.
• I can set the launcher panel to be ever present. I have plenty of horizontal screen space and I find it annoying not having a window list present on my display. When I have to hover my mouse to the left edge and wait a few hundred milliseconds before I even see the list of open apps and where they are positioned, it just annoys me. Having them always on-screen is just so much easier.
• It’s fast and more stable. Unity 5.0 is noticeably more snappy than it’s predecessors. It also feels less buggy. What drove me away from Unity on Oneiric was that the window placement snapping got horribly confused now and again and the only way out of it was to kill Compiz or otherwise restart Unity. My session is 24 hours old already and still going strong

Some Areas that could do with Improvement

Update: I thought it’s worth mentioning that removing the Gwibber lens removed close to *500MB* of that extra 1GB RAM that was used. There also seems to be an issue where gdbus and dconf worker are way more busy than they should be (at least on my machine). I’m figuring it out and will file bugs if I can confirm them. When they behave better then memory usage in Unity and Gnome Fallback shouldn’t be that far apart.

• Global menus still get confused about running apps. Sometimes I’d get a Thunderbird title in the menu space and Thunderbird has already been closed. This is kind of weird when you’re not aware of the bug.
• Memory usage is high. I’m currently using around 1GB more memory than I typically would when using the Gnome 3 Fallback session with the same software running. I’m hoping that it stays there and that it won’t continue to rise due to memory leaks and other memory issues. This is a deal breaker on application servers.
• The Dash isn’t very pretty or user friendly. I guess the dash didn’t get much work or research done due to the focus on getting bugs fixed, so it’s probably not all that bad. At least you can right-click on the Ubuntu icon now and get a list of installed Unity lenses. The Dash home should really be customisable, and I’m not sure how users are supposed to do some rudimentary tasks like connect to a network share.

Overall Thoughts

Unity has improved a lot recently. I feel that I can continue using it if it’s memory consumption stays under control. I’m testing it on Ubuntu 12.04 which is currently in an early pre-release state. Unity crashed twice while writing this blog entry so I hope it’s just some underlying bugs that will be solved by the time Ubuntu 12.04 hits release.

As for deploying it at client sites, I don’t think I could recommend that until it’s memory issues are resolved. Losing 1GB of RAM is a lot. Simple day to day tasks should be more intuitive (finding recent docs, accessing menus, accessing what used to be known as ‘Places’, etc), and it would help a lot if the Dash home were customisable (I couldn’t find a way to do it from within Unity or anything about it in the documentation). The Gnome 3 Fallback session is very solid and very familiar and I think I’ll continue to recommend it for the typical user desktop. At the rate that Unity is improving though, that might soon change.

This year I just want to get more stuff done.

Motto for 2012:

The Principles Meme

This morning I read the following in a blog entry: “I applaud him for sticking to his principles, and not compromising“. The person who wrote it didn’t even agree with the person he was referring too, and yet he was congratulating him for sticking to his principles. I’ve seen a bunch of similar statements recently. There’s also a similar, more self-congratulating meme where people are very proud that they are unwaveringly sticking to their principles no matter what.

I’m not sure where this comes from, perhaps it stems from religious roots? Perhaps from people who are afraid to admit that they are flawed in any way? Perhaps they have some agenda that they want to push?

What if no one ever compromised on their principles? What if, in South Africa during the Apartheid years no white person were willing to consider that anyone with a different skin colour could be considered an equal? What if people could never see women as equals and they could never get voting rights or other equal rights? There are many, many more concepts in the past that were rooted as moral principles and with the hindsight we have now, we can see that they were clearly wrong. Sticking to those principles would have been harmful. The sad thing is that today still, many concepts in society is flawed. So why do we choose to applaud people who are inflexible, unscientific and in my opinion, irrational?

The Scientists

I applaud the scientist types, the ones who are able to look at new information or evidence and are able to take a step back and say “Hey, maybe I should re-think this!”. I respect those who are willing to say “Perhaps I was wrong” and share they’re experiences with others to get wider feedback rather than the person who will relentlessly defend their position, typically using some absolutes to try to prove their point.

I switched from a desktop feed reader (Liferea) to Google Reader about a year ago, because my bandwidth is now sufficient for it not to be laggy, and because keeping Liferea synced on two machines was a huge daily waste of time.¹

I was annoyed that you couldn’t filter feeds unless you installed a somewhat obsolete user script and heavily modified it (and kept your filters synced by hand), but the feed and read item syncing was a major bonus. I can serve arbitrarily modified feeds from my own webserver if I really want to, but I recently unsubscribed from most of my noisiest feeds anyway.

Then Google redesigned Reader’s layout. By now, the wavefront of caps lock rage has circled the Earth several times. A lot of people are leaving Reader, and a lot of people are continuing to use it while complaining a lot. These are not your only two options — thanks to the wonder of user scripts and user styles, you don’t have to suffer other people’s poor layout decisions.

User styles

A lot can be accomplished with a static custom stylesheet — a large collection is available at userstyles.org. To install these styles on Firefox you need the Stylish extension. Chrome apparently has some limited support for styles now, but there is also a Stylish for Chrome. I hear it’s a bit crashy. Stylish lets you toggle stylesheets on and off easily. The order in which stylesheets are applied is random, which can cause some weirdness.

I currently use this redesign with my own special sauce tweaks. I also like switching to two columns when reading long text feeds on a wide screen, but that doesn’t work so well for image-heavy posts — I’m thinking of writing a feed-specific script for this.

User scripts

Changes which require manipulation of page elements need Javascript. You can find a wide variety of scripts at userscripts.org. To use them on Firefox you need Greasemonkey or Scriptish, and Chrome has native support.

I use a script which adds favicons to the entry list — useful when you’re viewing multiple feeds in a folder. The script also allows you to specify custom favicons, but I think this functionality is broken in the new design.

1. Yes, I know Liferea can sync with Reader. It gives you a huge flat list of all your Reader feeds, which you can’t re-order, arrange in sub-folders or filter. This is completely useless.

This solution is blindingly obvious in retrospect, but I didn’t think of it until after I bought a new keyboard, because that’s when I got a good look at the underside. I hope to spare others the bother of replacing a keyboard unnecessarily.

My right arrow key broke (I blame playing too many platformers). Upon closer examination, I discovered that the problem wasn’t with the plastic scissor mechanism attached to the key — if it had been, I could have tried to obtain a replacement kit on the intertubes. The problem was that the tips of two of the metal hooks on the base of the keyboard onto which the scissor mechanism clips had snapped off. They were the ones on the bottom, so whenever I pressed the key near the top the bottom would pop off.

So I got a new keyboard. It may seem silly to replace a whole keyboard because of a broken right arrow key, but I really like playing platformers.

When I got a new keyboard, I saw that the metal underside isn’t solid metal — there are holes next to all the hooks (because it gets punched out of a flat sheet, and the hooks are bent inwards). I tried to think of a cunning fix for the broken key, but all my solutions involved replicating the properties of the hooks with wire and glue, and nothing really seemed like a good idea.

Tonight (thanks, people on clug-tech discussing Dvorak!) I finally realised that I could run a piece of thread through the holes in the scissor mechanism and the holes in the back, thereby tying the bottom of the key onto the keyboard permanently. So I did. And it worked. Seriously, that’s all it took. I did have to fiddle around with needles and tweezers to thread the thread through the scissor mechanism without taking the key right off, but that was the only hard part. It doesn’t look like the thread is ever under a lot of strain, so I used a single loop and a reef knot on the back of the board.

Now I have a spare keyboard. Which I might rearrange into a Dvorak layout.

Caveats:

• This worked for a recent Fujitsu Siemens keyboard, but I think a lot of laptop keyboards are pretty similar. If you’re not sure, see what the scissor mechanism and the back of the keyboard look like.
• My right arrow is on the edge of the board, and so are the two affected hooks. If you play platformers with WASD, fixing your key may require more tweezer acrobatics and/or the temporary removal of surrounding keys.

My home network has a somewhat complicated setup where I have multiple PPPoE sessions across my ADSL connection, with various different ISPs. This allows me to take advantage of varying ISP properties such as cost and latency, by routing different traffic over different connections. Naturally, each of these connections only affords me a single IPv4 address, so I make use of NAT to allow the rest of my network access to the Internet. A potential problem arises, however, when connections go down and come back up. In the simple case, with only one connection, MASQUERADE takes care of all the details; when the interface goes down, all of the NAT entries associated with the connection are removed, so when it comes back up, it’s not a problem that your IP address has changed, because all of the NAT entries associated with the old address are gone. This works just as well in the multiple connections scenario; if an interface goes down resulting in traffic being routed over another interface, all of the old NAT entries have been dropped, so new ones will be established associated with the interface they are now travelling over. The problem arises when the interface that went down comes back up; traffic will now be routed over the first interface again, while still being rewritten to the second interface’s address, and this traffic is almost guaranteed to be dropped by either your ISP, or their upstream provider.

What’s the solution? Well, if you absolutely definitely want to start routing traffic over the first interface as soon as it comes back up, you’re going to need to flush the associated conntrack NAT entries as soon as it comes up, and let all your users reconnect (since their connections will be interrupted); I’m not entirely sure how to do this. In my case, however, I’m more concerned with maintaining existing connections without interruption, even if that means continuing to route them over the “wrong” interface. This also applies to incoming connections; ordinarily if somebody tries to establish a connection to the public IP address of one of your connections, they will need to connect to the same interface that outbound traffic to them would be routed over, which can be somewhat inconvenient.

My solution is something I’m going to call “connection pinning”. The idea is that once an outbound interface has been selected for a particular connection (by the Linux routing table), we “pin” the connection to that interface, so that traffic associated with that connection always travels over that interface even if the routing table changes. In order to achieve this, we can use a combination of Linux policy routing (ip rule), as well as firewall / conntrack packet marking. When a connection is first established, we set a connmark, which is a value stored in the conntrack table entry for that connection. In the case of an incoming connection, we set the mark based on the interface the packet arrived on; in the case of an outgoing connection, we set the mark in POSTROUTING based on the outbound interface already selected by the routing table. Then, for future outgoing traffic associated with that connection (as determined by conntrack), we set an fwmark based on the connmark, and bypass the normal routing table using policy rules for traffic marked thusly.

This is implemented in three parts. Firewall rules added using iptables, for the netfilter/conntrack bits; an ip-up script for establishing policy rules and routes when a PPP connection is established; and an ip-down script for flushing them again when the PPP connection is terminated.

First, the firewall rules (using the excellent ferm tool):

@def $DEV_PRIVATE = eth0;
@def $NET_PRIVATE_V4 = 10.0.0.0/24;

domain ip table mangle {
    # Only match new connections; established connections should
    # already have a connmark, which should not be overwritten.
    chain (INPUT FORWARD) {
        # Unfortunately the set-mark rules need to be duplicated for
        # each ppp interface we have.
        mod conntrack ctstate NEW {
            interface ppp0 CONNMARK set-mark 1;
            interface ppp1 CONNMARK set-mark 2;
            interface ppp2 CONNMARK set-mark 3;
            interface ppp3 CONNMARK set-mark 4;
            interface ppp4 CONNMARK set-mark 5;
        }
    }
    chain POSTROUTING {
        mod conntrack ctstate NEW {
            outerface ppp0 CONNMARK set-mark 1;
            outerface ppp1 CONNMARK set-mark 2;
            outerface ppp2 CONNMARK set-mark 3;
            outerface ppp3 CONNMARK set-mark 4;
            outerface ppp4 CONNMARK set-mark 5;
        }
    }
    chain PREROUTING {
        # Copy the connmark to the fwmark in order to activate the
        # policy rules for connection pinning. Only do this for
        # traffic originating from the local network; other traffic
        # (such as traffic going *to* the local network) should be
        # left unmodified, to allow return traffic to be routed over
        # the correct interface.

        interface $DEV_PRIVATE daddr ! $NET_PRIVATE_V4 CONNMARK restore-mark;
    }
    chain OUTPUT {
        # Same as above, but for locally originating traffic.

        daddr ! $NET_PRIVATE_V4 CONNMARK restore-mark;
    }
}

# I am assuming you already have something like this:
domain ip table nat {
    chain POSTROUTING outerface (ppp0 ppp1 ppp2 ppp3 ppp4) MASQUERADE;
}

If you’re not using ferm, here’s what the raw iptables commands would be (these are exactly what ferm will install given the above, so this is just more verbose):

iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp0 --jump CONNMARK --set-mark 1
iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp1 --jump CONNMARK --set-mark 2
iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp2 --jump CONNMARK --set-mark 3
iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp3 --jump CONNMARK --set-mark 4
iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp4 --jump CONNMARK --set-mark 5
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp0 --jump CONNMARK --set-mark 1
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp1 --jump CONNMARK --set-mark 2
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp2 --jump CONNMARK --set-mark 3
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp3 --jump CONNMARK --set-mark 4
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp4 --jump CONNMARK --set-mark 5
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp0 --jump CONNMARK --set-mark 1
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp1 --jump CONNMARK --set-mark 2
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp2 --jump CONNMARK --set-mark 3
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp3 --jump CONNMARK --set-mark 4
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp4 --jump CONNMARK --set-mark 5
iptables -t mangle -A PREROUTING --in-interface eth0 ! --destination 10.0.0.0/24 --jump CONNMARK --restore-mark
iptables -t mangle -A OUTPUT ! --destination 10.0.0.0/24 --jump CONNMARK --restore-mark

iptables -t nat -A POSTROUTING --out-interface ppp0 --jump MASQUERADE
iptables -t nat -A POSTROUTING --out-interface ppp1 --jump MASQUERADE
iptables -t nat -A POSTROUTING --out-interface ppp2 --jump MASQUERADE
iptables -t nat -A POSTROUTING --out-interface ppp3 --jump MASQUERADE
iptables -t nat -A POSTROUTING --out-interface ppp4 --jump MASQUERADE

Next, the ip-up script (to be placed in /etc/ppp/ip-up.d/ and made executable):

#!/bin/sh
TABLE="$PPP_IFACE"
MARK=$((${PPP_IFACE##ppp} + 1))
ip rule del lookup "$TABLE"
ip route flush table "$TABLE"
ip route add default dev "$PPP_IFACE" table "$TABLE"
ip rule add fwmark "$MARK" table "$TABLE"

Finally, the ip-down script (to be placed in /etc/ppp/ip-down.d/ and made executable):

#!/bin/sh
TABLE="$PPP_IFACE"
ip rule del lookup "$TABLE"
ip route flush table "$TABLE"

There are a couple of changes you will need to make to adapt these for your own network. In particular, you’ll need to duplicate the pppN iptables rules for each of the PPP interfaces you want to apply this to. Also, if you are already doing packet marking for some other reason, you’ll need to change the fwmark values I’ve used to ones that don’t interfere with your existing marks. I suspect there’s a better way to only mark outbound traffic than what I do above, but I wasn’t able to figure it out. If you have any improvements to suggest, feel free to mention them in the comments; I will try to keep this post updated with any improvements I make (either on my own, or based on other people’s suggestions).

Many years ago I worked at company that sold widgets. These widgets were very complicated and required lots of customisation. The company had developed a pretty large piece of software to help their sales people build complex widget quotes with lots of line items.

This company also had a big off the shelf enterprise accounting system that handled their real accounts.

I had worked at the company for almost 2 years as a software developer when one day I found myself sitting in the accounts department helping Danny with something unrelated. It was then that I learnt what Danny from Accounts actually did.

Every morning Danny would print out the previous days ‘accepted’ quotes from the quoting software resulting in a small pile of paper, one for each customer, with hundreds of line items, for every day. Then, using a ruler and pen to scratch out the lines, he would manually re-enter all of the customer data and their quote information, line item by line item, into the big accounting system. This process took him most of the day, sometimes more if business was good. He occasionally made mistakes that either cost the company lots of money or pissed off the customers.

As a software developer I knew that both systems ran off MSSQL databases. I knew that all the relevent information probably already existed to do the “job” programmatically. I knew that it would probably take a day or two to write a piece of software that did Danny’s job, perfectly every time, in a few milliseconds.

Danny had been doing that job for almost 6 years.

Since that day, whenever I start working with a new company, I try my best to meet everyone and get an idea for what they do and how they do it before I put my head down and start trying to solve any problems. That habit has served me well. In a team of ba/tech/strat/arch people I’m often the only one who knows how the accounts actually work, or how the stock is really procured, or what the weird hippies on the third floor do. (They’re always copywriters.)

But I’m not trying to pretend I have special powers. My point is that you can never assume that other people will have looked at problems like you do, with your knowledge-set. Most of the time other people won’t even see something like that as a “problem”. Danny’s boss never thought to question the process that admittedly pre-dated him. They all have no idea what SQL is and neither should they need to. It’s not their job. It’s yours. (Assuming you’re in a tech field)

What really excites me is how this kind of technology-discovery can be applied to people who traditionally live without the exposure to technology that we do. We now live in world where mobile phones can do things that sometimes even I think are quite magical (think SoundHound and Shazaam). I don’t know what “Danny the capturer of the world” situations exist in an under-resourced high school in a Soweto. I don’t know what efficiencies might just be waiting to be discovered in a clinic in Khayelitsha. I am however convinced that if a large corporate focused solely on profits with a really good, international, management team and a chartered accountant CFO all couldn’t spot that Danny was unintentionally wasting his time (and their money), then I can only imagine what amazing, albeit probably simple, tech-opportunities are waiting to be discovered in the “real” world.

I may not be ready to tackle the townships just yet, and I’m by no means assuming that there aren’t already smart people doing this kind of stuff, but I do look forward to one day being able to spend a few weeks immersed in the daily grind of a township school teacher or a minimum-wage worker, and maybe finding some way to bring a little bit of technological awesomeness and efficiency to their lives.

I know you’re wondering. I did write that software and Danny did need to click a button every morning and watch as the script whizzed by in less than a second, but he didn’t lose his job, instead he was able to move on to tackling more challenging things that actually needed his accounting skills. Everyone’s a winner.

RFC 3749 defines a mechanism for compressing a TLS connection using the DEFLATE compression mechanism. When used in conjunction with https, this fills a similar role to that of Content-Encoding: gzip, except that headers benefit from compression too (as the whole connection is compressed), and I suspect there is less chance of weird proxy / caching bugs. I decided to do some quick tests to see which browsers actually support this, as I found approximately zero information on the subject on the internet; the results, unfortunately, are rather dismal and depressing:

• Chrome: supported (apparently since Chrome 9, but I only tested Chrome 15 on the dev channel)
• Firefox: not supported (tested 8.0 Aurora)
• Safari: not supported
• Internet Explorer 9 on Windows 7: not supported
• Android 2.3 default browser: not supported

Introduction

In case you missed the news, Google Maps Navigation (Beta) for Mobile is now available in South Africa. I tried it out briefly, and thought I’d offer some thoughts on how it compares to the primary navigation software I use, Waze. Note that as an Android user (HTC Desire HD), this review will be fairly Android-specific. If you’re using an iPhone, please write your own review and let me know; if you’re using something else, please join the rest of us in the 21st century.

Integration

Navigation is part of the Google Maps application; as one of the core Google apps that virtually every Android phone ships with, it’s well integrated with the rest of the system, and in particular, with Maps / Places / People. Waze isn’t quite as well integrated; for example, there’s no easy way to grab a location out of  Places and have Waze navigate to it.

Voice prompts

These are generated entirely via TTS, using the system-configured speech engine. This is great for road names, not so great for actually being able to comprehend the prompts; I wish they had taken the Garmin approach, and only used TTS for the road names and such, not everything. By comparison, Waze only uses pre-recorded voice prompts, although they are currently testing out TTS functionality as well (I don’t have access to that, so I don’t yet know how it compares). As far as the actual prompting goes, they seem to be about equal in terms of usefulness.

Map data

This is a bit harder to quantify, as the quality and coverage of map data for both Waze and Google Maps varies drastically depending on where exactly you are in the country. In general, Google Maps has much more complete data; on the other hand, it tends to be several years or more out of date. In areas with active Area Managers on Waze, coverage is likely to be far more accurate, even going so far as to include temporary road detours during construction and so on. As such, your mileage may (and very likely will) vary.

Routing

Waze definitely wins this one, assuming you’re navigating in an area where the map is actually sufficiently complete to allow for sensible routing. Waze tracks the average travel time along each road segment and uses this as part of its routing calculations. In addition, if there is sufficient data, it seems that this will even be broken down by day of week / time of day, so Waze knows that what might be a crawling disaster at 4pm is actually smooth sailing at lunchtime. In addition, speed data is also handled in real-time; so if there’s a traffic jam right now, and some Waze users are stuck in it, it’ll detect that the average speed *right now* is much lower than it usually is, and route you around the problematic road segments if appropriate. Google Maps, by comparison, has a real-time traffic layer which can be used for routing decisions, but there currently seems to be no traffic data for South Africa, and I’m not sure if this information is used at all for long-term routing decisions. Even if it is, it’ll take a while for them to catch up with the existing data that has already been built up by Waze, so I guess we’ll have to see how that works out.

Display / UI

Google Maps wins this one. While driving, you can have the satellite layer displayed for the map, not just the road layer, which makes it a lot easier to match the map to what you’re seeing out of your windscreen, assuming the map isn’t horribly out of date. In addition, once you’re making your final approach to the destination, it will show you a Street View image of the destination, making it much easier to find the exact place you’re looking for, instead of trying to estimate distances on a map. By comparison, Waze offers only the usual abstract road map; this works, of course, but could be better.

Crowdsourcing

In addition to using average speed data collected from users for routing decisions, Waze also does things like adjusting road segment positions, and toggling road directions (1-way to 2-way and vice-versa) automatically, based on collected driving data. Occasionally this results in errors, but it mostly saves a lot of work on the part of map editors who would otherwise have to manually fix the map up. There’s also the ability to report accidents, traffic cameras, potholes, speed traps, roadblocks, and so on, which other users will be able to see and avoid. Google Maps doesn’t really have anything similar, other than the real-time traffic info.

Conclusion

I’ll be sticking with Waze, for now; the real-time and routing functionality, as well as the ability to fix up the map myself, easily makes up for any of the other disadvantages. If Google Maps grows to encompass the functionality and userbase of Waze, this would definitely tip the scales in its favour, but that doesn’t look like it’s going to happen any time soon.

I used to launch update-notifier from my fluxbox startup file, but after the Maverick upgrade it stopped escalating privileges when launching update-manager and would just fail silently when I clicked the button to install upgrades. So I kept launching update-manager manually. Now I’ve got a proper fix:

update-notifier --force-use-gksu &

As the name of the flag suggests, this forces update-notifier to use gksu when launching subcommands. I don’t know why it stopped doing it by itself.

Since Jaunty, the default behaviour of update-notifier is to pop up update-manager when it wants to notify you of updates, which I find pretty obnoxious. To restore it to its pre-Jaunty behaviour (a notification in the system tray, which you can click on at your leisure), do this:

gconftool -s --type bool /apps/update-notifier/auto_launch false

In related news, I set the GTK theme with gtk-chtheme and killed gnome-settings-daemon to see what would break. Update manager didn’t display the right theme. This is because it runs with gksu — you have to set root’s theme to fix it. I did this by copying my .gtkrc-2.0 to /root.

A while back I was chatting with Brad and said “Someone should create a Twitter feed of popular Stack Overflow questions“.  I think he said “Yes, someone should“.

Later that night I created @pystack and @djangoverflow.

Sometimes you just have to code.

Recently a friend who’s in the magazine industry was complaining about how their company (who is a very large media company) continually cuts the budget of the magazine people (ie. Those who actually produce content) while at the same time spending gob-loads of money on their “Online” and “Mobile” people. I’m hearing stories of games rooms and guys walking down the passage with an iPad in one hand and a Macbook Pro in the other, while just down the passage there are magazine teams running on 10 year old macs.

The print-media industry is no doubt floundering. Seeing demand for its product dropping by significant numbers every year (We’re talking overall sales figures around 20% of what they were 10 years ago) while ad-sales are becoming more and more brutal due to the “global economy” and, probably more scarily, losing ad sales to online channels. Fewer people want to buy their stuff and they’re making less and less off that reduced number.

So you can imagine the kind of pressure the industry is in and how incredibly easy it would be to come to the very foolish conclusion that the correct remedy is to spend those gob-loads on “Online” or “Mobile” to the detriment of the content producers.

My father was a printer, technically an offset lithography “machine minder”. Practically that meant that he was badly paid, worked long shifts, went to work in blue overalls and came home covered in ink. The work was tough. You needed to have an expert eye, understand some of the chemistry, have delicate hands and be able to perform running repairs on the printers. We’re talking about giant room sized machines and having the ability to hear that the third roller bearing on the transfer shaft dingle dangle needed oil in the next 30 minutes or the machine would fail. (I’m paraphrasing)

The reason my dad was badly paid even though his job required so much skill was because lithography was an old technology. The mystique had been removed from the process about 100 years earlier and the machines looked after themselves just enough to be able to have an unskilled worker become fully skilled in 3 years of on the job training.

The technology was mature and there was solid competition in the market. That drove the printing prices down, which pushed the salaries down, which meant that eventually it was only slightly more attractive as a career than something like panel beating.

In the last 30 years printing has evolved to the point where the machines are easier to use, faster and even more reliable. Instead of hiring one or two guys per machine you can now have a few roaming engineers for an entire factory of printers. Putting ink on paper has never been cheaper.

My father moved to the publishing world about 30 years ago and has been wearing chinos to work ever since… Though I’m pretty sure he would still prefer to deal with machines than colleagues.

The cost and skill involved in delivering content will always drop. Technology takes care of that. However we will never have Artificial Intelligence that can honestly go to Darling and write about an Evita Bezuidenhout show, take photographs of the flowers in the Karoo, write about swimming with dolphins on a cool Sunday morning or editorialise about crappy Egyptian presidents.

100 years ago quality content made money… That is still the situation and it is unlikely to ever change. How that content is delivered should never become more important than the content itself.

You might be able to wow people with your swanky iPad application with annoying faux-turning-pages animations, but eventually the technology will mature and everyone will have swanky iPad apps. The cost involved in building those apps will drop and the big boys will be consistently competing against small, leaner, startup content producers. And as if to amplify the situation the technology is changing much much faster. It took hundreds of years to get the cost of printing on paper so low that we could afford to print a daily newspaper and sell it to the masses. The cost of producing an Ipad app halves every six month and, as the technology evolves, it becomes trivially easy for anyone with some good ideas and camera to create something that other people want… and god forbid, would actually pay money for.

So, if you happen to be the CEO of some big ass media giant, spare a thought for Gutenberg and then Google “ios and android development frameworks” before deciding not to buy your content producers some decent computers. You could even do it on your iPad.

After over a year of development, we have finally released Ibid. Ibid is a general purpose chat bot written in Python. We've suffered from a bit of feature creep, so despite being a 0.1 release it can talk 7 messaging protocols and has over 100 features provided by plugins. I think we also have an excellent architecture and very developer friendly plugin API. The 0.1.0 release can be downloaded from Launchpad or installed from our PPA.

I used to work for OLPC, whose mission is to distribute low cost laptops for education, without necessarily the connectivity with the outside world. Now at Praekelt we’re focusing on using connectivity as the power – harnessing the deployed base of mobile phones in Africa without requiring them to be smartphones or computing devices. As part of this Praekelt Foundation and Vodacom are hosting the Betavine Social Exchange Cape Town Developer Day 2010.

I’ve asked Steve Wolak to tell us more about Betavine and the event.

Who is Steve Wolak?

Stephen Wolak, Founder and Head of Betavine, has worked in mobile technology and software since graduating from Imperial College, London. Stephen joined Vodafone Group R&D in 2000 and in 2006 put forward the idea of an open platform for engaging the wider technology community with R&D activities.  The rest, as they say, is history.

MC: I first became aware of Betavine when looking for 3G drivers for Linux, but I’m sure there is more to it than that. What is Betavine and how did it start?

SW: Betavine was launched in January 2007 and has been evolving ever since, with new features being added in response to new requirements and feedback from the user base.  One area of success has been the linux software for the Vodafone Mobile Connect (VMC) card which has been downloaded over 750,000 times and has created a lively community around it.

We have also run a number of successful competitions on the website and created a lively Widget Zone. The website continues to evolve and we try out new things.

MC: What is the Betavine Social Exchange?

SW: This is our latest idea.. creating “social and sustainable” mobile solutions.  The Betavine Social Exchange seeks to bring together 3 communities; the mobile community, the social sector and the entrepreuners.  Together these communities can create mobile solutions for the social sector.  Community organisations create challenges on the website and mobile developers / social entrepreneurs create solutions. The website then supports the deployment of these solutions on the ground.

MC: The BSX’s success certainly depends on connecting the right people: those with needs – the NGOs and community organisations – and the developers. How do you publicise the BSX to reach them?

SW: We are running our pilot in South Africa and so we are working with Sangonet to help us get in touch with South African NGOs.  We are running a developer day in Cape Town to help us engage with the local developer community.

MC: What do the resulting solutions include – are they apps for mobile phones, mobi websites, SMS solutions or all of the above?

SW: All of the above.  It is important that the solution is appropriate for the challenge and the local community that will ultimately use the solution.

MC: What can developers expect from participating in the BSX?

SW: They can find real world challenges that people are seeking solutions to.  They can meet other developers and find useful resources to help them create a business.  The full resources section is coming soon.

MC: Which leads us on to the Developer Day being hosted in Cape Town next month. What’s going to be happening at the event, and how does it tie in with the BSX?

SW: We are keen to encourage mobile developers based in South Africa to engage with the real challenges that have been posted on the Betavine Social Exchange.  The developer day will include presentations on mobile technology and some exciting mobile solutions plus a developer competition and lots of creative energy and networking.

MC: You’re going to be speaking at the event. Who would you like to see there?

SW: I would like to see mobile developers plus those with design skills and a passion for using mobile technology for social change.

MC: We’re having a developer competition on the day. Can you tell us anything about the prizes/incentives you’re planning to bring?

SW: Well, the developer day is free and includes lots of food and drink plus some beer at the end of the day … :-)  We also intend to offer a few prizes for the competition winners .. But we have not decided exactly what yet.  You will have to come along and see but tickets are going fast!

Developer Day details

Date: Wednesday, March 10, 2010 from 9:30 AM – 7:00 PM

Location: The Lightbox Daylight Studio, Green Point, Cape Town

More information and free tickets are available at eventbrite. Due to the demand, the event has been expanded to 70 people.

SuperGenPass and Password Composer are password generators, which generate different passwords for each site you use based on a single master password. This gives you the convenience of only remembering one password as well as the security of using different (and strong) passwords for each site. This means that you won't have all your accounts compromised when¹ one of them is compromised.

Most password generators are implemented as browser extensions or bookmarklets, since they are most frequently needed in a web browser. I've been wanting to start using a password generator, but I wanted to be sure that I could access my accounts even if I didn't have a web browser accessible. The two situations I could think of were a command line only system (e.g. SSH) and my cellphone².

Surprisingly, I couldn't find a command line implementation of SuperGenPass, so I wrote one in Python. I also couldn't find any J2ME or Symbian implementations, and so wrote my own one in J2ME. They both support subdomain stripping and configurable password lengths. They don't support salted passwords.

I chose SuperGenPass over Password Composer because it uses a better scheme. Password Composer only uses hex characters, whereas SuperGenPass uses a base64 encoded hash. SuperGenPass also hashes the password multiple times (which would slow down a brute force attack to find the master password) and imposes complexity requirements on the generated password (which reduces the chances that the generated password can be brute forced).

Deploying a new server at work – a dedicated server hosted at Hetzner. Fortunately Jaunty (Ubuntu 9.04) was released before we had anything hosted on the machine, so I took the decision to upgrade it before we do serious deployment.

One of the shiny new features of Ubuntu Server 9.04 is etckeeper, documented here by Thierry Carrez. In particular, on 9.04 etckeeper plays well with bzr and shows the real user who typed “sudo etckeeper commit” in the bzr log, not just “root”.

As we have a (small but distributed) team adminning the server, this will help a great deal to keep track of who did what when.

Some observations on the last n releases:

Throughout the Ubuntu development cycle, there are daily “snapshot” CD images produced. If you’re fortunate to live in a country where most of the “broadband” online population are not capped at 1GB per month (and a presidential hopeful who doesn’t keep singing “bring me my machine gun“) then you can download these during the development cycle to boot (daily-live) or install (perhaps in a virtual machine) to check on the progress or help with testing. These culminate in the actual “gold” release image.

Therefore, if you have one of these images from near the end of the development cycle, such as the release candidate, you can rsync to the latest image available on release day, and that will download the differences between the iso you have, and the final daily image – which will be identical to the release image, even though the daily image will be named something like jaunty-desktop-i386.iso and the corresponding release image named ubuntu-9.04-desktop-i386.iso. Rename it, and you’re done!

(Check the MD5SUMS after the release is announced, to be 100% sure you have it. There is always a small chance of a change to the ISOs on release day if some major “ate all my data” bug is found – so if you do have problems, remember that it comes with no warranty…)

Now, for kicks, go and lurk on IRC in #ubuntu-release-party and watch the masses rocking up to ask “Is it out yet?” Note Alan Pope’s list of Things Not To Say, and don’t go gloating that you have it already – you’ll only be kicked from the channel by the ironically named partybot.

Instead, burn write it to a USB stick (CDs are so early 2008) and get installing!

URLs shouldn't really contain file extensions (like .html, .png) since they are supposed to identify a resource and not a particular representation/format thereof. The format is indicated by the Content-Type header sent in the response. Modern CMSs do this already (for example, the URL of this page doesn't include .html).

Doing the same for static files (i.e. files served directly by the webserver) isn't straightforward because most webservers use the file extension to determine the MIME type to send in the Content-Type header. This means that simply removing the file extension from the filename (or even creating a symlink without a file extension) will cause the webserver to send the wrong Content-Type header.

I decided to try find a solution to this for my webserver of choice, Lighttpd. Lighttpd has a module which embeds a Lua interpreter and allows you to write scripts which modify (or even handle) requests. So I wrote a script which searches the directory for files with the same name as requested but with an extension. This means that any file can be accessed with the file extension removed from the URL while still having the correct Content-Type.

The script currently chooses the first matching file, which means that having multiple files with the same name but different extensions doesn't do anything useful. The proper method however is to actually do content negotiation, which chooses the format based on the preferences indicated by the HTTP client in the Accept header.

To use this script, download it and save it somewhere (I use /etc/lighttpd/). Enable mod_magnet, and add the following line to the site definition.

magnet.attract-physical-path-to = ("/etc/lighttpd/extension.lua")