Clug Park-Tech

Giving Unity Another Go

Yesterday I installed Unity5.0 and I was pleasantly surprised by some of its new features:

• I can set the panel background colour. By default, the Unity panel adapts itself to match the wallpaper colour. This doesn’t always work out, and with certain background colours it looks really horrible with the icons on it. I set mine to a none-harsh, dark grey and can now see my icons without any desire to fork out my eyes.
• I can set the launcher panel to be ever present. I have plenty of horizontal screen space and I find it annoying not having a window list present on my display. When I have to hover my mouse to the left edge and wait a few hundred milliseconds before I even see the list of open apps and where they are positioned, it just annoys me. Having them always on-screen is just so much easier.
• It’s fast and more stable. Unity 5.0 is noticeably more snappy than it’s predecessors. It also feels less buggy. What drove me away from Unity on Oneiric was that the window placement snapping got horribly confused now and again and the only way out of it was to kill Compiz or otherwise restart Unity. My session is 24 hours old already and still going strong

Some Areas that could do with Improvement

Update: I thought it’s worth mentioning that removing the Gwibber lens removed close to *500MB* of that extra 1GB RAM that was used. There also seems to be an issue where gdbus and dconf worker are way more busy than they should be (at least on my machine). I’m figuring it out and will file bugs if I can confirm them. When they behave better then memory usage in Unity and Gnome Fallback shouldn’t be that far apart.

• Global menus still get confused about running apps. Sometimes I’d get a Thunderbird title in the menu space and Thunderbird has already been closed. This is kind of weird when you’re not aware of the bug.
• Memory usage is high. I’m currently using around 1GB more memory than I typically would when using the Gnome 3 Fallback session with the same software running. I’m hoping that it stays there and that it won’t continue to rise due to memory leaks and other memory issues. This is a deal breaker on application servers.
• The Dash isn’t very pretty or user friendly. I guess the dash didn’t get much work or research done due to the focus on getting bugs fixed, so it’s probably not all that bad. At least you can right-click on the Ubuntu icon now and get a list of installed Unity lenses. The Dash home should really be customisable, and I’m not sure how users are supposed to do some rudimentary tasks like connect to a network share.

Overall Thoughts

Unity has improved a lot recently. I feel that I can continue using it if it’s memory consumption stays under control. I’m testing it on Ubuntu 12.04 which is currently in an early pre-release state. Unity crashed twice while writing this blog entry so I hope it’s just some underlying bugs that will be solved by the time Ubuntu 12.04 hits release.

As for deploying it at client sites, I don’t think I could recommend that until it’s memory issues are resolved. Losing 1GB of RAM is a lot. Simple day to day tasks should be more intuitive (finding recent docs, accessing menus, accessing what used to be known as ‘Places’, etc), and it would help a lot if the Dash home were customisable (I couldn’t find a way to do it from within Unity or anything about it in the documentation). The Gnome 3 Fallback session is very solid and very familiar and I think I’ll continue to recommend it for the typical user desktop. At the rate that Unity is improving though, that might soon change.

This year I just want to get more stuff done.

Motto for 2012:

The Principles Meme

This morning I read the following in a blog entry: “I applaud him for sticking to his principles, and not compromising“. The person who wrote it didn’t even agree with the person he was referring too, and yet he was congratulating him for sticking to his principles. I’ve seen a bunch of similar statements recently. There’s also a similar, more self-congratulating meme where people are very proud that they are unwaveringly sticking to their principles no matter what.

I’m not sure where this comes from, perhaps it stems from religious roots? Perhaps from people who are afraid to admit that they are flawed in any way? Perhaps they have some agenda that they want to push?

What if no one ever compromised on their principles? What if, in South Africa during the Apartheid years no white person were willing to consider that anyone with a different skin colour could be considered an equal? What if people could never see women as equals and they could never get voting rights or other equal rights? There are many, many more concepts in the past that were rooted as moral principles and with the hindsight we have now, we can see that they were clearly wrong. Sticking to those principles would have been harmful. The sad thing is that today still, many concepts in society is flawed. So why do we choose to applaud people who are inflexible, unscientific and in my opinion, irrational?

The Scientists

I applaud the scientist types, the ones who are able to look at new information or evidence and are able to take a step back and say “Hey, maybe I should re-think this!”. I respect those who are willing to say “Perhaps I was wrong” and share they’re experiences with others to get wider feedback rather than the person who will relentlessly defend their position, typically using some absolutes to try to prove their point.

I switched from a desktop feed reader (Liferea) to Google Reader about a year ago, because my bandwidth is now sufficient for it not to be laggy, and because keeping Liferea synced on two machines was a huge daily waste of time.¹

I was annoyed that you couldn’t filter feeds unless you installed a somewhat obsolete user script and heavily modified it (and kept your filters synced by hand), but the feed and read item syncing was a major bonus. I can serve arbitrarily modified feeds from my own webserver if I really want to, but I recently unsubscribed from most of my noisiest feeds anyway.

Then Google redesigned Reader’s layout. By now, the wavefront of caps lock rage has circled the Earth several times. A lot of people are leaving Reader, and a lot of people are continuing to use it while complaining a lot. These are not your only two options — thanks to the wonder of user scripts and user styles, you don’t have to suffer other people’s poor layout decisions.

User styles

A lot can be accomplished with a static custom stylesheet — a large collection is available at userstyles.org. To install these styles on Firefox you need the Stylish extension. Chrome apparently has some limited support for styles now, but there is also a Stylish for Chrome. I hear it’s a bit crashy. Stylish lets you toggle stylesheets on and off easily. The order in which stylesheets are applied is random, which can cause some weirdness.

I currently use this redesign with my own special sauce tweaks. I also like switching to two columns when reading long text feeds on a wide screen, but that doesn’t work so well for image-heavy posts — I’m thinking of writing a feed-specific script for this.

User scripts

Changes which require manipulation of page elements need Javascript. You can find a wide variety of scripts at userscripts.org. To use them on Firefox you need Greasemonkey or Scriptish, and Chrome has native support.

I use a script which adds favicons to the entry list — useful when you’re viewing multiple feeds in a folder. The script also allows you to specify custom favicons, but I think this functionality is broken in the new design.

1. Yes, I know Liferea can sync with Reader. It gives you a huge flat list of all your Reader feeds, which you can’t re-order, arrange in sub-folders or filter. This is completely useless.

This solution is blindingly obvious in retrospect, but I didn’t think of it until after I bought a new keyboard, because that’s when I got a good look at the underside. I hope to spare others the bother of replacing a keyboard unnecessarily.

My right arrow key broke (I blame playing too many platformers). Upon closer examination, I discovered that the problem wasn’t with the plastic scissor mechanism attached to the key — if it had been, I could have tried to obtain a replacement kit on the intertubes. The problem was that the tips of two of the metal hooks on the base of the keyboard onto which the scissor mechanism clips had snapped off. They were the ones on the bottom, so whenever I pressed the key near the top the bottom would pop off.

So I got a new keyboard. It may seem silly to replace a whole keyboard because of a broken right arrow key, but I really like playing platformers.

When I got a new keyboard, I saw that the metal underside isn’t solid metal — there are holes next to all the hooks (because it gets punched out of a flat sheet, and the hooks are bent inwards). I tried to think of a cunning fix for the broken key, but all my solutions involved replicating the properties of the hooks with wire and glue, and nothing really seemed like a good idea.

Tonight (thanks, people on clug-tech discussing Dvorak!) I finally realised that I could run a piece of thread through the holes in the scissor mechanism and the holes in the back, thereby tying the bottom of the key onto the keyboard permanently. So I did. And it worked. Seriously, that’s all it took. I did have to fiddle around with needles and tweezers to thread the thread through the scissor mechanism without taking the key right off, but that was the only hard part. It doesn’t look like the thread is ever under a lot of strain, so I used a single loop and a reef knot on the back of the board.

Now I have a spare keyboard. Which I might rearrange into a Dvorak layout.

Caveats:

• This worked for a recent Fujitsu Siemens keyboard, but I think a lot of laptop keyboards are pretty similar. If you’re not sure, see what the scissor mechanism and the back of the keyboard look like.
• My right arrow is on the edge of the board, and so are the two affected hooks. If you play platformers with WASD, fixing your key may require more tweezer acrobatics and/or the temporary removal of surrounding keys.

My home network has a somewhat complicated setup where I have multiple PPPoE sessions across my ADSL connection, with various different ISPs. This allows me to take advantage of varying ISP properties such as cost and latency, by routing different traffic over different connections. Naturally, each of these connections only affords me a single IPv4 address, so I make use of NAT to allow the rest of my network access to the Internet. A potential problem arises, however, when connections go down and come back up. In the simple case, with only one connection, MASQUERADE takes care of all the details; when the interface goes down, all of the NAT entries associated with the connection are removed, so when it comes back up, it’s not a problem that your IP address has changed, because all of the NAT entries associated with the old address are gone. This works just as well in the multiple connections scenario; if an interface goes down resulting in traffic being routed over another interface, all of the old NAT entries have been dropped, so new ones will be established associated with the interface they are now travelling over. The problem arises when the interface that went down comes back up; traffic will now be routed over the first interface again, while still being rewritten to the second interface’s address, and this traffic is almost guaranteed to be dropped by either your ISP, or their upstream provider.

What’s the solution? Well, if you absolutely definitely want to start routing traffic over the first interface as soon as it comes back up, you’re going to need to flush the associated conntrack NAT entries as soon as it comes up, and let all your users reconnect (since their connections will be interrupted); I’m not entirely sure how to do this. In my case, however, I’m more concerned with maintaining existing connections without interruption, even if that means continuing to route them over the “wrong” interface. This also applies to incoming connections; ordinarily if somebody tries to establish a connection to the public IP address of one of your connections, they will need to connect to the same interface that outbound traffic to them would be routed over, which can be somewhat inconvenient.

My solution is something I’m going to call “connection pinning”. The idea is that once an outbound interface has been selected for a particular connection (by the Linux routing table), we “pin” the connection to that interface, so that traffic associated with that connection always travels over that interface even if the routing table changes. In order to achieve this, we can use a combination of Linux policy routing (ip rule), as well as firewall / conntrack packet marking. When a connection is first established, we set a connmark, which is a value stored in the conntrack table entry for that connection. In the case of an incoming connection, we set the mark based on the interface the packet arrived on; in the case of an outgoing connection, we set the mark in POSTROUTING based on the outbound interface already selected by the routing table. Then, for future outgoing traffic associated with that connection (as determined by conntrack), we set an fwmark based on the connmark, and bypass the normal routing table using policy rules for traffic marked thusly.

This is implemented in three parts. Firewall rules added using iptables, for the netfilter/conntrack bits; an ip-up script for establishing policy rules and routes when a PPP connection is established; and an ip-down script for flushing them again when the PPP connection is terminated.

First, the firewall rules (using the excellent ferm tool):

@def $DEV_PRIVATE = eth0;
@def $NET_PRIVATE_V4 = 10.0.0.0/24;

domain ip table mangle {
    # Only match new connections; established connections should
    # already have a connmark, which should not be overwritten.
    chain (INPUT FORWARD) {
        # Unfortunately the set-mark rules need to be duplicated for
        # each ppp interface we have.
        mod conntrack ctstate NEW {
            interface ppp0 CONNMARK set-mark 1;
            interface ppp1 CONNMARK set-mark 2;
            interface ppp2 CONNMARK set-mark 3;
            interface ppp3 CONNMARK set-mark 4;
            interface ppp4 CONNMARK set-mark 5;
        }
    }
    chain POSTROUTING {
        mod conntrack ctstate NEW {
            outerface ppp0 CONNMARK set-mark 1;
            outerface ppp1 CONNMARK set-mark 2;
            outerface ppp2 CONNMARK set-mark 3;
            outerface ppp3 CONNMARK set-mark 4;
            outerface ppp4 CONNMARK set-mark 5;
        }
    }
    chain PREROUTING {
        # Copy the connmark to the fwmark in order to activate the
        # policy rules for connection pinning. Only do this for
        # traffic originating from the local network; other traffic
        # (such as traffic going *to* the local network) should be
        # left unmodified, to allow return traffic to be routed over
        # the correct interface.

        interface $DEV_PRIVATE daddr ! $NET_PRIVATE_V4 CONNMARK restore-mark;
    }
    chain OUTPUT {
        # Same as above, but for locally originating traffic.

        daddr ! $NET_PRIVATE_V4 CONNMARK restore-mark;
    }
}

# I am assuming you already have something like this:
domain ip table nat {
    chain POSTROUTING outerface (ppp0 ppp1 ppp2 ppp3 ppp4) MASQUERADE;
}

If you’re not using ferm, here’s what the raw iptables commands would be (these are exactly what ferm will install given the above, so this is just more verbose):

iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp0 --jump CONNMARK --set-mark 1
iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp1 --jump CONNMARK --set-mark 2
iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp2 --jump CONNMARK --set-mark 3
iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp3 --jump CONNMARK --set-mark 4
iptables -t mangle -A FORWARD --match conntrack --ctstate NEW --in-interface ppp4 --jump CONNMARK --set-mark 5
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp0 --jump CONNMARK --set-mark 1
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp1 --jump CONNMARK --set-mark 2
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp2 --jump CONNMARK --set-mark 3
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp3 --jump CONNMARK --set-mark 4
iptables -t mangle -A INPUT --match conntrack --ctstate NEW --in-interface ppp4 --jump CONNMARK --set-mark 5
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp0 --jump CONNMARK --set-mark 1
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp1 --jump CONNMARK --set-mark 2
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp2 --jump CONNMARK --set-mark 3
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp3 --jump CONNMARK --set-mark 4
iptables -t mangle -A POSTROUTING --match conntrack --ctstate NEW --out-interface ppp4 --jump CONNMARK --set-mark 5
iptables -t mangle -A PREROUTING --in-interface eth0 ! --destination 10.0.0.0/24 --jump CONNMARK --restore-mark
iptables -t mangle -A OUTPUT ! --destination 10.0.0.0/24 --jump CONNMARK --restore-mark

iptables -t nat -A POSTROUTING --out-interface ppp0 --jump MASQUERADE
iptables -t nat -A POSTROUTING --out-interface ppp1 --jump MASQUERADE
iptables -t nat -A POSTROUTING --out-interface ppp2 --jump MASQUERADE
iptables -t nat -A POSTROUTING --out-interface ppp3 --jump MASQUERADE
iptables -t nat -A POSTROUTING --out-interface ppp4 --jump MASQUERADE

Next, the ip-up script (to be placed in /etc/ppp/ip-up.d/ and made executable):

#!/bin/sh
TABLE="$PPP_IFACE"
MARK=$((${PPP_IFACE##ppp} + 1))
ip rule del lookup "$TABLE"
ip route flush table "$TABLE"
ip route add default dev "$PPP_IFACE" table "$TABLE"
ip rule add fwmark "$MARK" table "$TABLE"

Finally, the ip-down script (to be placed in /etc/ppp/ip-down.d/ and made executable):

#!/bin/sh
TABLE="$PPP_IFACE"
ip rule del lookup "$TABLE"
ip route flush table "$TABLE"

There are a couple of changes you will need to make to adapt these for your own network. In particular, you’ll need to duplicate the pppN iptables rules for each of the PPP interfaces you want to apply this to. Also, if you are already doing packet marking for some other reason, you’ll need to change the fwmark values I’ve used to ones that don’t interfere with your existing marks. I suspect there’s a better way to only mark outbound traffic than what I do above, but I wasn’t able to figure it out. If you have any improvements to suggest, feel free to mention them in the comments; I will try to keep this post updated with any improvements I make (either on my own, or based on other people’s suggestions).

Many years ago I worked at company that sold widgets. These widgets were very complicated and required lots of customisation. The company had developed a pretty large piece of software to help their sales people build complex widget quotes with lots of line items.

This company also had a big off the shelf enterprise accounting system that handled their real accounts.

I had worked at the company for almost 2 years as a software developer when one day I found myself sitting in the accounts department helping Danny with something unrelated. It was then that I learnt what Danny from Accounts actually did.

Every morning Danny would print out the previous days ‘accepted’ quotes from the quoting software resulting in a small pile of paper, one for each customer, with hundreds of line items, for every day. Then, using a ruler and pen to scratch out the lines, he would manually re-enter all of the customer data and their quote information, line item by line item, into the big accounting system. This process took him most of the day, sometimes more if business was good. He occasionally made mistakes that either cost the company lots of money or pissed off the customers.

As a software developer I knew that both systems ran off MSSQL databases. I knew that all the relevent information probably already existed to do the “job” programmatically. I knew that it would probably take a day or two to write a piece of software that did Danny’s job, perfectly every time, in a few milliseconds.

Danny had been doing that job for almost 6 years.

Since that day, whenever I start working with a new company, I try my best to meet everyone and get an idea for what they do and how they do it before I put my head down and start trying to solve any problems. That habit has served me well. In a team of ba/tech/strat/arch people I’m often the only one who knows how the accounts actually work, or how the stock is really procured, or what the weird hippies on the third floor do. (They’re always copywriters.)

But I’m not trying to pretend I have special powers. My point is that you can never assume that other people will have looked at problems like you do, with your knowledge-set. Most of the time other people won’t even see something like that as a “problem”. Danny’s boss never thought to question the process that admittedly pre-dated him. They all have no idea what SQL is and neither should they need to. It’s not their job. It’s yours. (Assuming you’re in a tech field)

What really excites me is how this kind of technology-discovery can be applied to people who traditionally live without the exposure to technology that we do. We now live in world where mobile phones can do things that sometimes even I think are quite magical (think SoundHound and Shazaam). I don’t know what “Danny the capturer of the world” situations exist in an under-resourced high school in a Soweto. I don’t know what efficiencies might just be waiting to be discovered in a clinic in Khayelitsha. I am however convinced that if a large corporate focused solely on profits with a really good, international, management team and a chartered accountant CFO all couldn’t spot that Danny was unintentionally wasting his time (and their money), then I can only imagine what amazing, albeit probably simple, tech-opportunities are waiting to be discovered in the “real” world.

I may not be ready to tackle the townships just yet, and I’m by no means assuming that there aren’t already smart people doing this kind of stuff, but I do look forward to one day being able to spend a few weeks immersed in the daily grind of a township school teacher or a minimum-wage worker, and maybe finding some way to bring a little bit of technological awesomeness and efficiency to their lives.

I know you’re wondering. I did write that software and Danny did need to click a button every morning and watch as the script whizzed by in less than a second, but he didn’t lose his job, instead he was able to move on to tackling more challenging things that actually needed his accounting skills. Everyone’s a winner.

RFC 3749 defines a mechanism for compressing a TLS connection using the DEFLATE compression mechanism. When used in conjunction with https, this fills a similar role to that of Content-Encoding: gzip, except that headers benefit from compression too (as the whole connection is compressed), and I suspect there is less chance of weird proxy / caching bugs. I decided to do some quick tests to see which browsers actually support this, as I found approximately zero information on the subject on the internet; the results, unfortunately, are rather dismal and depressing:

• Chrome: supported (apparently since Chrome 9, but I only tested Chrome 15 on the dev channel)
• Firefox: not supported (tested 8.0 Aurora)
• Safari: not supported
• Internet Explorer 9 on Windows 7: not supported
• Android 2.3 default browser: not supported

This afternoon I came across a blog post entitled The True Cost of Open Source, in which the blogger tries to dissuade churches from using open source solutions, and thereby promote their own proprietary CMS.

I commented on their blog post, but I don't know if they'll post it, because I show them up for what they really seem to be doing. So with this in mind, I've posted my full, unedited reply below:

Why is this so reminiscent of those Microsoft-funded studies showing how open source software is more expensive than proprietary software?

Ah... I see what you're doing, but first I'm going to reply to your blog post, and then I'm going to conclude with my answer to the real reason you blogged this.

"When a project gets mature enough to be widely useful, it can be released as open source."

Uh, most open source projects are open source from the get-go, even if only one person is currently using it, and that person is the developer.

"You may not be paying software licensing fees or monthly costs, but you will be paying for labor, expertise, or suffering through long nights and weekends without a geek who can make sense of this stuff."

This is a half-truth. All three content management systems in the roundup you highlighted have one or more commercial companies that provide installations, support and other services around the open source app. On top of that you seem to assume that most churches implement websites themselves, whereas in my experience, as an IT professional, most churches ask a geek to do it for them (and then said geek uses the CMS of their choice, be it open source or not).

"The research by DeviousMedia shows that the average setup costs for any open source system can easily top $15,000. Monthly maintenance costs are usually $250 or more."

Of course what you don't point out is that DeviousMedia uses Drupal themselves, and are just doing a comparison of open source CMSes, they are not trying to explain how open source is "more expensive" or something else like that. They are simply giving their visitors an accurate comparison of open source systems.

Also, note that those setup costs are what commercial companies (that build their businesses around open source software) charge, not the REAL cost a church might incur. So this is not a true cost, just an estimated cost if you ask a company to do it for you.

1. Lack of support

This is such a red herring. I've had more and better support from the open source community than I've had from commercial companies. Sure there's no telephone number you can call, but a most of them offer either forums or IRC (and a lot of open source projects now have a web IRC client on their websites so that you don't even need to download an IRC client), or even both. Some even have a support e-mail address.

I have spoken to a number of folks about this, and they all agree with me, they've gotten more support out of open source projects than commercial companies.

2. Requirements for ongoing maintenance

While you say that this is true for all software, you make this out to be something that commercial companies are not affected by. I can tell you from personal experience that this is not the case. We're upgrading the commercial issue tracking system at work for the 2nd time in less than 2 years, and we'll still be a release behind after the upgrade.

3. No accountability from the vendor

Have you read the EULA from commercial products? Have you ever read a EULA? They are far worse than the open source licenses.

Commercial companies completely absolve themselves of any and all responsibility of your data. So if your commercial CMS messes up your database, you're up the creek without a paddle, and (as I mentioned in the "support" section) you are without help. At least the open source projects will try to help you recover your data. Try getting a commercial company to do that.

4. Forced workarounds and duct-tape solutions

While this may be true about some open source systems, it's just as true about commercial CMSes. In addition to that, there are some open source CMSes and open source church CMSes that do not have this issue.

All blankets statements are wrong. (See what I did there?)

5. Not customer-focused

This has got to be the biggest myth I've ever heard. How many open source projects have you interviewed and studied to determine this?

Time and time again I have dealt with open source projects I have found them to be exceedingly user-focused. They have gone out their way to help their users, they accept feature requests from the users and implement them. Few commercial companies will implement feature requests from users unless it is in line with their product strategy.

Additionally, most commercial companies I have dealt with are mostly concerned about the money. Want support? You need a support contract.

Lastly, as I said earlier, I'm going to answer the reason you have blogged this.

I saw your link in the post and your little note at the bottom of the blog, and went to your site, and noticed that lo and behold, you're selling a proprietary content management system! No wonder this is reminiscent of those Microsoft studies. I'm guessing that you've had a number of potential customers use open source systems instead of yours, and you're trying to counter that.

With this in mind you pulled an infographic that you thought you could use to bolster your argument, even though that infographic actually has nothing to do with your argument.

However, the truth is that even with all the hidden costs, open source systems are still cheaper than commercial systems.

Update: they have replied to my comment.

Introduction

In case you missed the news, Google Maps Navigation (Beta) for Mobile is now available in South Africa. I tried it out briefly, and thought I’d offer some thoughts on how it compares to the primary navigation software I use, Waze. Note that as an Android user (HTC Desire HD), this review will be fairly Android-specific. If you’re using an iPhone, please write your own review and let me know; if you’re using something else, please join the rest of us in the 21st century.

Integration

Navigation is part of the Google Maps application; as one of the core Google apps that virtually every Android phone ships with, it’s well integrated with the rest of the system, and in particular, with Maps / Places / People. Waze isn’t quite as well integrated; for example, there’s no easy way to grab a location out of  Places and have Waze navigate to it.

Voice prompts

These are generated entirely via TTS, using the system-configured speech engine. This is great for road names, not so great for actually being able to comprehend the prompts; I wish they had taken the Garmin approach, and only used TTS for the road names and such, not everything. By comparison, Waze only uses pre-recorded voice prompts, although they are currently testing out TTS functionality as well (I don’t have access to that, so I don’t yet know how it compares). As far as the actual prompting goes, they seem to be about equal in terms of usefulness.

Map data

This is a bit harder to quantify, as the quality and coverage of map data for both Waze and Google Maps varies drastically depending on where exactly you are in the country. In general, Google Maps has much more complete data; on the other hand, it tends to be several years or more out of date. In areas with active Area Managers on Waze, coverage is likely to be far more accurate, even going so far as to include temporary road detours during construction and so on. As such, your mileage may (and very likely will) vary.

Routing

Waze definitely wins this one, assuming you’re navigating in an area where the map is actually sufficiently complete to allow for sensible routing. Waze tracks the average travel time along each road segment and uses this as part of its routing calculations. In addition, if there is sufficient data, it seems that this will even be broken down by day of week / time of day, so Waze knows that what might be a crawling disaster at 4pm is actually smooth sailing at lunchtime. In addition, speed data is also handled in real-time; so if there’s a traffic jam right now, and some Waze users are stuck in it, it’ll detect that the average speed *right now* is much lower than it usually is, and route you around the problematic road segments if appropriate. Google Maps, by comparison, has a real-time traffic layer which can be used for routing decisions, but there currently seems to be no traffic data for South Africa, and I’m not sure if this information is used at all for long-term routing decisions. Even if it is, it’ll take a while for them to catch up with the existing data that has already been built up by Waze, so I guess we’ll have to see how that works out.

Display / UI

Google Maps wins this one. While driving, you can have the satellite layer displayed for the map, not just the road layer, which makes it a lot easier to match the map to what you’re seeing out of your windscreen, assuming the map isn’t horribly out of date. In addition, once you’re making your final approach to the destination, it will show you a Street View image of the destination, making it much easier to find the exact place you’re looking for, instead of trying to estimate distances on a map. By comparison, Waze offers only the usual abstract road map; this works, of course, but could be better.

Crowdsourcing

In addition to using average speed data collected from users for routing decisions, Waze also does things like adjusting road segment positions, and toggling road directions (1-way to 2-way and vice-versa) automatically, based on collected driving data. Occasionally this results in errors, but it mostly saves a lot of work on the part of map editors who would otherwise have to manually fix the map up. There’s also the ability to report accidents, traffic cameras, potholes, speed traps, roadblocks, and so on, which other users will be able to see and avoid. Google Maps doesn’t really have anything similar, other than the real-time traffic info.

Conclusion

I’ll be sticking with Waze, for now; the real-time and routing functionality, as well as the ability to fix up the map myself, easily makes up for any of the other disadvantages. If Google Maps grows to encompass the functionality and userbase of Waze, this would definitely tip the scales in its favour, but that doesn’t look like it’s going to happen any time soon.

Choqok seems to be about the only native microblogging client for KDE. It's actually a rather nice app, but I've found it to be rather buggy when it comes to Twitter authentication. For the last few months it has moaned about not being able to authenticate, after which it crashes. Starting the app again does the same thing, which means it crashes constantly, and you're left without a Twitter client.

The latest version of Choqok, version 1.1, seems to have fixed this bug. With this in mind, I looked for a PPA with version 1.1, but didn't find anything. I decided to build it myself, including a package so that I can install and uninstall it cleanly on my system, and so that it would work with the existing packages.

If you're compiling software on Ubuntu, there's a nice little application called "checkinstall" that will build a package file for you, based on the instructions in the Makefile for the "make install" step. By setting the parameters of the package correctly, you can ensure that it replaces the current package correctly, and when you upgrade to the next release of Ubuntu it will be removed in favour of the version in Ubuntu.

First, let's install some of the packages we'll need to compile Choqok:

$ sudo apt-get install build-essential checkinstall kdelibs5-dev libqjson-dev \
libattica-dev libqca2-dev libindicate-qt-dev libqoauth-dev

Now that we have those, we need the latest version of Choqok itself. Download the bzip2ed tarball from SourceForge.net. Once you have that, you need to extract it:

$ tar -xjvf choqok-1.1.tar.bz2

Once the files are extracted, you can compile Choqok:

$ cd choqok-1.1
$ mkdir build
$ cd build
$ cmake -DCMAKE_INSTALL_PREFIX=`kde4-config --prefix` ..
$ make

Now comes the magical part where you create a package. Normally one would run checkinstall as root so that it can install the package. I didn't want to do that, because I wanted to manually install the package myself via gdebi-kde, so I ran it as my user, and let it complain that it couldn't install the package.

$ checkinstall

When you first run checkinstall, it will ask you to confirm the package details. You need to change the following fields:

• Maintainer: Your name and e-mail address
• Summary: Choqok is a fast, efficient and simple to use micro-blogging client for KDE. It currently supports the twitter.com and identi.ca microblogging services.
• Name: choqok
• Version: 1.1
• Release: 0ubuntu1~natty1
• Group: kde
• Provides: choqok
• Replaces: choqok

Here's what it should look like:

This package will be built according to these values: 

0 -  Maintainer: [ Your Name <you@example.com> ]
1 -  Summary: [ Choqok is a fast, efficient and simple to use micro-blogging
 client for KDE. It currently supports the twitter.com and identi.ca
 microblogging services. ]
2 -  Name:    [ choqok ]
3 -  Version: [ 1.1 ]
4 -  Release: [ 0ubuntu1~natty` ]
5 -  License: [ GPL ]
6 -  Group:   [ kde ]
7 -  Architecture: [ amd64 ]
8 -  Source location: [ build ]
9 -  Alternate source location: [  ]
10 - Requires: [  ]
11 - Provides: [ choqok ]
12 - Conflicts: [  ]
13 - Replaces: [ choqok ]

After this checkinstall will run, but will ask you a few more questions. Just read the questions carefully, and do what checkinstall suggests. Once this is done, a package will be left in the "build" directory you created earlier. It should be called "choqok_1.1-0ubuntu1~natty1_i386.deb" or "choqok_1.1-0ubuntu1~natty1_amd64.deb" depending on your platform.

Now run gdebi-kde to install it:

$ gdebi-kde choqok_1.1-0ubuntu1~natty1_i386.deb

This will notify you that there is an older version in a channnel, which is the version we're wanting to replace. This is fine, so just continue with the installation.

Once that is complete, you can start up Choqok, and you should have a working microblogging client!

I used to launch update-notifier from my fluxbox startup file, but after the Maverick upgrade it stopped escalating privileges when launching update-manager and would just fail silently when I clicked the button to install upgrades. So I kept launching update-manager manually. Now I’ve got a proper fix:

update-notifier --force-use-gksu &

As the name of the flag suggests, this forces update-notifier to use gksu when launching subcommands. I don’t know why it stopped doing it by itself.

Since Jaunty, the default behaviour of update-notifier is to pop up update-manager when it wants to notify you of updates, which I find pretty obnoxious. To restore it to its pre-Jaunty behaviour (a notification in the system tray, which you can click on at your leisure), do this:

gconftool -s --type bool /apps/update-notifier/auto_launch false

In related news, I set the GTK theme with gtk-chtheme and killed gnome-settings-daemon to see what would break. Update manager didn’t display the right theme. This is because it runs with gksu — you have to set root’s theme to fix it. I did this by copying my .gtkrc-2.0 to /root.

One of the complaints I've heard fairly often from users of OpenLP is that our forums lack somewhat in usability and features. With this in mind, as part of the OpenLP 2.0 release, I wanted to see if I can move the forums over to some alternative forum software. However, I have a few important prerequisites:

1. I don't want users to have to re-register on the forums
2. I want users to be able to move seamlessly between the site and the forums without needing to log in again
3. I want to move my forum posts across

Current Situation

The web site currently runs on Drupal, which is very extensible, and I had no doubt that it would be able to provide the necessary to perform its side of the job, but I wasn't so sure if any of the forum software would be able to keep its end of the bargain. And so I started my Googling...

The open source heavyweight in this category is phpBB, but, as everybody knows they have security issues coming out of their ears. Actually, from what I've heard, version 3 of phpBB is leaps and bounds ahead of phpBB2 in terms of security. That said, after looking at it again, I still wasn't happy with it. It didn't seem to provide any real integration capabilities.

A few years ago I had also heard of Vanilla Forums, which is not just new forum software, but a new type of forum software. Their addons are simply dropped into the code base to be installed, by far the easiest plugin system yet. I went to go and have a look at Vanilla again, and lo and behold they'd already released version 2.0.

A Solution

It looked to me like Vanilla would be my best bet. However, I still wasn't sure how to provide single sign-on for my users. With this in mind I chatted to a friend of mine who deals with these sorts of web sites on a day-to-day basis, and asked him about my forum dilemma. No problem, he said, I should use the Vanilla Proxyconnect plugin. Just what I needed!

When we originally chatted about this, there was no Drupal module which could provide the integration needed from Drupal's end. My friend was happy to write one for me, but since we didn't get any further with the move to Vanilla at the time of the discussion, he hadn't done much. Fortunately for both of us, someone has written a module, named Orchid, for Drupal since, and I was able to download it and install it in no time.

Problems Encountered

At this point, having a local instance of the site running on my development server, I set up Vanilla on my server, and proceeded to configure the seamless integration. It was pretty easy, and after about 20 minutes I had things working. But I had two problems:

• If you click Sign Out on the forums, nothing happens
• If you click Sign In on Vanilla, which takes you to the Drupal login page, the Drupal login page does not take you back to Vanilla

More Googling, and after staring at code for a few hours, I had figured out the problems. Firstly, there was a missing line of code in the Proxyconnect plugin for Vanilla, and secondly, Drupal needs to be told to redirect back to Vanilla.

Fixing The Logout

The logout fix was fairly simple, I just needed to add a line of code similar to the following line to the EntryController_SignOut_Handler function:

Redirect('http://www.yoursite.com/logout.php', 302);

What I ended up with was something like this, which takes the configured logout URL into consideration (based on the login code):

public function EntryController_SignOut_Handler(&$Sender) {
   if (!Gdn::Authenticator()->IsPrimary('proxy')) return;
   $Redirect = Gdn::Request()->GetValue('HTTP_REFERER');
   $SignoutURL = Gdn::Authenticator()->GetURL(Gdn_Authenticator::URL_REMOTE_SIGNOUT, $Redirect);
   Redirect($SignoutURL, 302);
}

Fixing The Redirection: Part 1

The Proxyconnect doucmentation suggests telling the CMS that you came from the forums via a simple "?forums=1" type of link. This was not good enough for me, I want the user to be redirected back to the exact page that they were on!

Having a look again at the login and logout code, I had an idea: What if I add a token in the configurable URLs which is replaced with the URL to redirect to?

As you can see in the logout code above, I actually already have the redirect (or 'referer') URL - I just need to use it. With that in mind, I changed the logout method to look like this:

public function EntryController_SignOut_Handler(&$Sender) {
   if (!Gdn::Authenticator()->IsPrimary('proxy')) return;
   $Redirect = Gdn::Request()->GetValue('HTTP_REFERER');
   $SignoutURL = Gdn::Authenticator()->GetURL(Gdn_Authenticator::URL_REMOTE_SIGNOUT, $Redirect);
   $SignoutURL = str_replace('{REDIRECT_URL}', urlencode($Redirect), $SignoutURL);
   Redirect($SignoutURL, 302);
}

Once I'd done that, I edited the code in the login method, so that it looked like this:

$Redirect = Gdn::Request()->GetValue('HTTP_REFERER');
$SigninURL = Gdn::Authenticator()->GetURL(Gdn_Authenticator::URL_REMOTE_SIGNIN, $Redirect);
$SigninURL = str_replace('{REDIRECT_URL}', urlencode($Redirect), $SigninURL);
Redirect($SigninURL, 302);

Fixing The Redirection: Part 2

Drupal provides a "destination" parameter, which directs any given back to the specified URL, so I thought I'd take advantage of that. As a first course of action, I used the above-mentioned token in the "sign in url" configuration in Vanilla. That did not work as Drupal told me that URL did not exist. It seems that the desintation parameter only works on internal Drupal URLs. Time to make use of Drupal's module hooks system!

Since I was fixing Vanilla integration I decided to just do the Drupal side of things in the Orchid module itself. Orchid already uses the hook_user hook, and specifically deals with the logging in and logging out, so I just tagged along with that, and forced Drupal to redirect if there was a "destination" GET parameter. In the orchid_user function, I added these lines in the bottom of the if statement:

if ($_GET['destination']) {
  drupal_goto($_GET['destination']);
}

w00t! My seamless integration works!

Exporting The Data

Unfortunately, that still left me with all my forum posts sitting in Drupal, and nothing in Vanilla. So my next step was to figure out how to export the data from Drupal and then import it into Vanilla.

Some more browsing and Googling brought me to the vanilla2export.php script, which can export data from a number of different forum software packages so that you can import them into Vanilla. Once again, Drupal was unsupported, but after looking at the exporter code and seeing how pluggable it is, I had a Drupal exporter up and running in less than an hour.

The only problem I had with the exporter was that Drupal's user roles don't match well to Vanilla. On top of that, the default roles in Vanilla are deleted when you import data from the vanilla2export.php script. I dropped my Vanilla database and installed Vanilla again. With a new, clean install, I extracted the permissions into an SQL script, and added a few extra tweaks to it, to assign everyone the "Member" role.

A while back I was chatting with Brad and said “Someone should create a Twitter feed of popular Stack Overflow questions“.  I think he said “Yes, someone should“.

Later that night I created @pystack and @djangoverflow.

Sometimes you just have to code.

Recently a friend who’s in the magazine industry was complaining about how their company (who is a very large media company) continually cuts the budget of the magazine people (ie. Those who actually produce content) while at the same time spending gob-loads of money on their “Online” and “Mobile” people. I’m hearing stories of games rooms and guys walking down the passage with an iPad in one hand and a Macbook Pro in the other, while just down the passage there are magazine teams running on 10 year old macs.

The print-media industry is no doubt floundering. Seeing demand for its product dropping by significant numbers every year (We’re talking overall sales figures around 20% of what they were 10 years ago) while ad-sales are becoming more and more brutal due to the “global economy” and, probably more scarily, losing ad sales to online channels. Fewer people want to buy their stuff and they’re making less and less off that reduced number.

So you can imagine the kind of pressure the industry is in and how incredibly easy it would be to come to the very foolish conclusion that the correct remedy is to spend those gob-loads on “Online” or “Mobile” to the detriment of the content producers.

My father was a printer, technically an offset lithography “machine minder”. Practically that meant that he was badly paid, worked long shifts, went to work in blue overalls and came home covered in ink. The work was tough. You needed to have an expert eye, understand some of the chemistry, have delicate hands and be able to perform running repairs on the printers. We’re talking about giant room sized machines and having the ability to hear that the third roller bearing on the transfer shaft dingle dangle needed oil in the next 30 minutes or the machine would fail. (I’m paraphrasing)

The reason my dad was badly paid even though his job required so much skill was because lithography was an old technology. The mystique had been removed from the process about 100 years earlier and the machines looked after themselves just enough to be able to have an unskilled worker become fully skilled in 3 years of on the job training.

The technology was mature and there was solid competition in the market. That drove the printing prices down, which pushed the salaries down, which meant that eventually it was only slightly more attractive as a career than something like panel beating.

In the last 30 years printing has evolved to the point where the machines are easier to use, faster and even more reliable. Instead of hiring one or two guys per machine you can now have a few roaming engineers for an entire factory of printers. Putting ink on paper has never been cheaper.

My father moved to the publishing world about 30 years ago and has been wearing chinos to work ever since… Though I’m pretty sure he would still prefer to deal with machines than colleagues.

The cost and skill involved in delivering content will always drop. Technology takes care of that. However we will never have Artificial Intelligence that can honestly go to Darling and write about an Evita Bezuidenhout show, take photographs of the flowers in the Karoo, write about swimming with dolphins on a cool Sunday morning or editorialise about crappy Egyptian presidents.

100 years ago quality content made money… That is still the situation and it is unlikely to ever change. How that content is delivered should never become more important than the content itself.

You might be able to wow people with your swanky iPad application with annoying faux-turning-pages animations, but eventually the technology will mature and everyone will have swanky iPad apps. The cost involved in building those apps will drop and the big boys will be consistently competing against small, leaner, startup content producers. And as if to amplify the situation the technology is changing much much faster. It took hundreds of years to get the cost of printing on paper so low that we could afford to print a daily newspaper and sell it to the masses. The cost of producing an Ipad app halves every six month and, as the technology evolves, it becomes trivially easy for anyone with some good ideas and camera to create something that other people want… and god forbid, would actually pay money for.

So, if you happen to be the CEO of some big ass media giant, spare a thought for Gutenberg and then Google “ios and android development frameworks” before deciding not to buy your content producers some decent computers. You could even do it on your iPad.

Many people want to add subtitles to a DVD in their own language, and often DVD disks don’t come with subtitles in a language you want. For instance, if you’re staying in a foreign country and want English subtitles added to your rental movie. Or in my case, the library at the local Alliance Française has a lot of interesting movies completely in French, without English subtitles, and my french isn’t yet good enough to fully understand everything that is being said.

What is interesting is that there are sites on the internet that offer downloadable subtitle files for many movies. Simply type the name of your movie, plus the word “subtitle” into google and provided the movie is popular or well known enough you should be able to find a site offering the subtitle file for download. Mostly it seems that these subtitles were extracted from a DVD owned by someone else with the right languages. In other cases, it is purely fans of the movie who loved it so much and wanted to translate the words of the movie into a language that they know so that others can also enjoy the movie. I’m not so sure about the legality of the subtitles obtained in the first case, but certainly an argument may be made that since you own or have legally rented a copy of the movie it does seem to fall within “fair usage” to watch this movie with subtitles in a language you require.

So, you may be lucky. You may find the (fan-written) subtitle file that exactly matched your movie. By this I mean that the person who created your file had exactly the same version of the DVD, so that the lead in at the beginning of the movie is exactly the same length and in this case the subtitles remain perfectly in synch with the video.

However, don’t expect to watch the movie in your living room just yet. Most stand-alone DVD players that I’ve used do not have a facility to specify an external subtitles file, but almost all of the software based players on your computer do have this facility. So provided you got lucky, and got the correct subtitles file that matched your DVD, and are also content to watch movies in front of your computer that will be sufficient and you don’t need to read further.

Mostly though, the subtitles file doesn’t have the right timings for your DVD, or you might really really want to watch it on your TV in the lounge rather than with bowls of popcorn balanced on your lap in front of the computer. The guide that follows is for people who are running Linux, are comfortable with the command line, and who wish to hardcode ¹ subtitles into a XVID file for watching on a (living-room) player that can play DIVX movies. If this is not exactly what you want there still may be something for you in this blog post. I’ll outline my six-step method:

1. Rip (extract) contents of the DVD disk.
2. Attempt to play DVD file with subtitle files on desktop.
3. Create a XVID file without subtitles.
4. Transform the subtitle file so that subtitles are synchronized with the video.
5. Create a XVID file with subtitles hardcoded in.
6. Break up this file into smaller segments for players that cannot handle large files.

I myself don’t follow the above recipe every time – depending on the circumstances some of these steps may not be needed. And you may very well want a different outcome – it may be enough for you just to align the subtitles with the video, or you may want more such as recreating a DVD with subtitles. You may even despise XVID files with harcoded subtitles in them!

If you are running Debian/Ubuntu you’ll need a lot of packages to be installed – at least: mplayer mencoder ogmtools libdvdcss dvdbackup gaupol ffmpeg gstreamer0.10-x plus dependancies and probably gnome-codec-install.

The principal tools involved are the wonderful mplayer/mencoder command line tools :- they give fine control, are scriptable, and are compatible with each other in terms of command line flags.

For step 1, extract the DVD to your hard disk:

dvdbackup -i /dev/dvd -M -o name_of_movie

This will create a folder called name_of_movie in the current directory. If your DVD-device is not /dev/dvd put the correct device name there.

In step 2, you’ll need to work out which of the titles on the DVD disk contains the movie, start by typing:

mplayer -v -dvd-device name_of_movie/VIDEO_TS/ dvd://1

Keep on increasing the number at the end – from 1 to 2 to 3 etc until you find the actual movie and not just the adverts and bonus features. In the examples that follow I refer to title 1, you will need to replace that with your correct title number.

The next thing to do is test the downloaded subtitle file:

mplayer -v -dvd-device name_of_movie/VIDEO_TS/ dvd://1 -sub downloaded_sub.srt

The subtitle file doesn’t have to end in .srt – mplayer supports multiple formats. Make a careful note if the subtitles are in synch, namely do the spoken words match the subtitles. Press the right arrow key on your keyboard to advance the video. Ensure that subtitles remain in synch until the end of the movie.

For step 3, create an initial XVID file without subtitles. Yes, I know this can be lossy. And it really is not a required step if your subtitles are already in synch, if they are you can move straight on to step 5. However, if you do need to adjust the subtitle file to match the video, then in order to do so it may be necessary to work with the entire movie as a single file, instead of multiple individual VOB files.

So, here is a simple two-pass recipe for encoding a movie to XVID without subtitles:

mencoder -idx -dvd-device name_of_movie/VIDEO_TS/ dvd://1 -ovc xvid -oac pcm -xvidencopts pass=1:bitrate=1200:aspect=16/9 -o /dev/null

mencoder -idx -dvd-device name_of_movie/VIDEO_TS/ dvd://1 -ovc xvid -oac pcm -xvidencopts pass=2:bitrate=1200:aspect=16/9 -o movie.avi

Note the bitrate in the above call (1200) can be adjusted as required (higher numbers mean higher quality but also larger files). You will obviously need to make sure this call has the correct path for your VIDEO_TS folder, and also for the title track from the DVD (replace the dvd://1 with the appropriate title). Experienced mencoder users will possibly also want to add extra mencoder flags. Transcoding to XVID can take some time, depending on the speed of your computer.

Next up is step 4 – adjusting the subtitle file if that is needed. To do this, I prefer using the gaupol application under Linux. So, open gaupol and within gaupol select your subtitle file, and at the bottom of the gaupol screen select your .avi video file that you produced in the previous step. The trick to adjusting is to note exactly the time at which the first useful spoken words occur for which you have a matching subtitle, and the same for the last spoken words (unfortunately, knowing the last spoken words may spoil the end of the movie for you!). So, if you click the green play icon in gaupol it will play the video file with a clock at the top of the screen which you can use for noting when the spoken words occur. Based on the above, in an example I worked out that someone said subtitle #2 (you should be able to recognise the words based on context, even if you don’t speak the language) at 00:02:54.000 and that the one of the last spoken texts occurred for subtitle #613 at 01:29:17.000. The important thing is to get the timing of the first subtitle as accurate as possible, for the last subtitle it doesn’t matter if you’re a second or two out. To do the actual synchronization choose the “Transform Positions” tool from the subtitle menu (see screenshot below), specify the times you observed and the positions of the other subtitles will be adjusted proportionally.

To test the subtitle file is now correct, play the middle of the movie and see if the subtitles are fully in sync there as well. Repeat, if necessary with other time points, and if necessary adjusting only part of the file (“Transform selected subtitles”) rather than the entire file (“Transform current project”).

Now, for step 5 we are going to hardcode the subtitles into a new XVID file which we produce. We can choose to either start with the source coming from the original DVD-rip folder produced by step one, or we can choose to work with the XVID file produced in step three as the source. Obviously, working with the XVID file will be lossy, as one can expect some picture degradation using the XVID file as a source file. However, if this file were to be used, you would know for sure that your subtitles will sync up with the file. Think back to the DVD extract step with dvdbackup – did it report any errors at all during the extract from the DVD? For instance, if there was a scratch on the DVD disk dvdbackup would still be able to work around it by block filling in the file produced, but I have observed such a “repaired error” might cause problems with the timing and sync of subtitles in the step to follow if the original DVD extract is used as the source. Generally, because of this I prefer to work with the XVID file as the source, even though it may be lossy.

Before starting, remove any file in the current folder called “divx2pass.log”, this file will have been produced if you did a two-pass encoding as above and will interfere with the two-pass encode we are about to do.

To hardcode in the subtitles, using the divx file produced in step three as the source, here are the commands that will perform the two-pass encode for you:

mencoder -idx movie.avi -ovc xvid -sub subtitles_adjusted.srt -subpos 96 -oac pcm -xvidencopts pass=1:bitrate=1200:aspect=16/9 -o /dev/null

mencoder -idx movie.avi -ovc xvid -sub subtitles_adjusted.srt -subpos 96 -oac pcm -xvidencopts pass=2:bitrate=1200:aspect=16/9 -o hardcoded.avi

Now, if you wanted to work with the original DVD-rip folder as the source, rather than perform the lossy re-encoding of the DIVX file, then replace “movie.avi” in the above with “-dvd-device name_of_movie/VIDEO_TS/ dvd://1” adjusted where appropriate.

Also, note in the above that “subtitles_adjusted.srt” refers to your subtitle file that you produced with gaupol, and also note that I prefer specifying that my subtitles appear at position 96 (you can leave this out if it doesn’t bother you).

And also note that there is another way to hardcode in subtitles but this way can result in enormous files.

Finally, in step 6 I break up the resultant DIVX file (which would be called “harcoded.avi” if you followed step 5 exactly). Why do I do this? For two possible reasons:

1. If you are playing off a VFAT formatted memory stick files cannot be larger than 2GB in size.
2. My player can play DIVX files, but once the file exceeds 1GB in size the playback simply stops.

Now, there are two possible workarounds. You could have simply lowered the bitrate so that the final XVID file was less than 1GB in size. Or you could chop up a big file as I do below. Of course, if your player can play big DIVX files and you’re not working with a VFAT memory stick then there is no need to perform this step.

In this example I am working with a 3.1G file. Firstly, to determine how long the video is, I issue:

ffmpeg -i hardcoded.avi

In my case, it tells me that my video has

Duration: 02:05:16.88.

So, I’d like to split that into four parts, so that each part will come to less than 1GB in size (roughly) – so after some quick mental arithmetic I think I’d like each piece to be 32 minutes long.

Based on the above example, these are the four mencoder commands to splice the video into four 32 minute chunks (the last chunk will be less than 32 minutes):

mencoder -ovc copy -oac copy -endpos 0:32:00 -o hardcoded1.avi hardcoded.avi
mencoder -ovc copy -oac copy -ss 0:32:00 -endpos 0:32:00 -o hardcoded2.avi hardcoded.avi
mencoder -ovc copy -oac copy -ss 1:04:00 -endpos 0:32:00 -o hardcoded3.avi hardcoded.avi
mencoder -ovc copy -oac copy -ss 1:36:00  -o hardcoded4.avi hardcoded.avi

Granted, this is an extreme example (normally you don’t let the XVID file get so big!) but it does show how to splice up a video using the -ss and -endpos flags. Unlike the transcoding to XVID steps, this step executes very quickly as all it does is copy the video and audio streams.

And voilà, copy the split files to your memory stick, and take it to the lounge to enjoy with your popcorn, knowing that you’ve managed to render a previously unwatchable video into something you’ll enjoy, and somehow all this extra work makes the video more enjoyable too.

¹ Hardcoding means the actual video file is altered so that the subtitle is in the video stream. It’s a little nasty, and I agree softcoding (keeping the subtitles separate from the video stream) is better. The only problem is that my player refuses to accept files where the subtitles have been muxed in. If you’re curious about softcoding, I can recommend AVI-Mux GUI which does run under wine.

Geeks often need to access their *nix computers from work. Doesn’t everyone want to do that? True geeks control their computers strictly using the command-line, of course, and the tool that is used to control a remote command-line session is ssh.

What one usually does is use a tool like corkscrew to send ssh traffic through an HTTP proxy.

At one place of employment, a known trick of using corkscrew to tunnel out using the work proxy failed, with this message:

Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy service is denied. )

I tried all combinations of DOMAIN\USERNAME:PASSWORD in my corkscrew auth file but nothing worked.

If you see this message have no fear! What you need is a utility that can negotiate NTLM authorization with the proxy.

There are several open source tools that can do NTLM, of these I chose cntlm. Often ntlmaps is recommended as a utility to negotiate the authorization. The cntlm man page indicates that cntlm is far more efficient than ntlmaps, both in terms of memory and CPU usage.

One oddity about cntlm relative to other software that you may have worked with is that configuration is a two-step procedure: firstly you configure the software with a default config file, like the following (the settings that need to be configured for now in the first-step are in the first paragraph: username, domain, password, proxy, proxy-port):

#
# Cntlm Authentication Proxy Configuration
#
# NOTE: all values are parsed literally, do NOT escape spaces,
# do not quote. Use 0600 perms if you use plaintext password.
#


Username	__username__
Domain		__domain__
Password	__password__		# Use hashes instead (-H)
#Workstation	netbios_hostname	# Should be auto-guessed


Proxy		__PROXY__:__PROXY_PORT__


#
# This is the port number where Cntlm will listen
#
Listen		3128


#
# If you wish to use the SOCKS5 proxy feature as well, uncomment
# the following option, SOCKS5. It can be used several times
# to have SOCKS5 on more than one port or on different network
# interfaces (specify explicit source address for that).
#
# WARNING: The service accepts all requests, unless you use
# SOCKS5User and make authentication mandatory. SOCKS5User
# can be used repeatedly for a whole bunch of individual accounts.
#
#SOCKS5Proxy	8010
#SOCKS5User	dave:password


#
# Use -M first to detect the best NTLM settings for your proxy.
# Default is to use the only secure hash, NTLMv2, but it is not
# as available as the older stuff.
#
# This example is the most universal setup known to man, but it
# uses the weakest hash ever. I won't have it's usage on my
# conscience. :) Really, try -M first.
#
#

Auth		LM
#Flags		0x06820000
#
# Enable to allow access from other computers
#
#Gateway	yes


#
# Useful in Gateway mode to allow/restrict certain IPs
#
#Allow		127.0.0.1
#Deny		0/0


#
# GFI WebMonitor-handling plugin parameters, disabled by default
#
#ISAScannerSize	1024
#ISAScannerAgent	Wget/
#ISAScannerAgent	APT-HTTP/
#ISAScannerAgent	Yum/


#
# Headers which should be replaced if present in the request
#
#Header		User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)


#
# Tunnels mapping local port to a machine behind the proxy
# 
Tunnel		11443:__OUTSIDE_HOST.COM__:443

Then, run
cntlm -v -M http://google.com (or any other external site)

Cntlm will use this to assess the type of auth your proxy can handle. In my case I got back something like the following output:

Config profile  1/11... OK (HTTP code: 301)
----------------------------[ Profile  0 ]------
Auth            NTLMv2
PassNTLMv2      AE1234567890123234567890123456C4
------------------------------------------------

For the second configuration step, these two lines need to be pasted back into your configuration file replacing the line that said “Auth LM” (and you must do this for your own situation, you can’t reuse my lines).

Then, startup the cntlm daemon:

cntlm

Let’s test if it works. Note that in the above config file I have a tunnel defined (the last line of the config file). Now, in order to ssh to port 443 of host __outside_host.com__ which is outside the proxy, one can do so using:

ssh -p 11443 localhost

The above assumes that outside_host.com has an sshd listening on port 443.

Cntlm also works fine under windows. If you don’t have “administrator” authorisation under windows, you can still run the cntlm executable, but need to specify which cntlm.ini file to use, in other words, something like:

cntlm.exe -c cntlm.ini

Of course, if you’re running on windows you won’t have an ssh command line client, but putty can be used nicely for this purpose. Bear in mind that putty needs to be configured (for the default cntlm configuration above) to use an HTTP proxy on host “localhost” listening on port 3128.

Since January 2010, Google docs has allowed you to store any type of file, even arbitrary binary files. However, there are a couple of gotchas: one cannot upload files greater than 1GB in size, and you may want to encrypt your files so that not just anyone can read them (for instance server backup files).

The two bash scripts below provide a solution for the above. I call them the ‘mince’ scripts ‘cos they slice and dice your files and hopefully you’ll get hamburgers back at the end of the day. These scripts depend on you having a fairly new version of bash on your unix-like system, the ‘split’ utility and gnupg (GPG) which is used for the encryption/decryption. If you’re unsure of GPG, a good getting started guide can be found here.

It must be said that google docs is (in my opinion) currently not the best way to store your files in the cloud. In fact, I wrote another blog post describing the “google storage” options in greater depth.

The encrypt&split script is mince.sh and it takes two parameters, the first one a directory or archive, the second the email address for an already imported public key:

#!/bin/bash
# gpg encrypt archive and split into chunks
# $1 specifies base directory or compressed archive to encrypt.
# $2 is the recipients public key, eg. 'myfriend@his.isp.net'
set -e


CHUNK_SIZE=1000000000 #1000000000==1GB (not 1GiB!)
SCRATCH_DIR=~/scratch_space
TAR_REGEX='\.tar'


usage() {
    echo "ERROR: "
    echo " $*"
    echo "USAGE: "
    echo " mince.sh [DIRECTORY|ARCHIVE] PUBLIC_KEY_NAME"
    echo "EXAMPLE: "
    echo " ./mince.sh directory myfriend@her.isp.net"
    echo "FURTHER COMMENTS: "
    echo " if an ARCHIVE is supplied instead of a directory, it must have a name like file.tar or file.tar.gz or file.tar.bz2 "
}


#check parameters entered are valid
[ $# -ne 2 ] && usage "Two parameters are required." && exit 1
if [ ! -d "$1" ] && [[ ! $1 =~ $TAR_REGEX ]]; then
    usage "$1 is not a directory or tar/tar.gz/tar.bz2 archive."
    exit 1
fi


#if 1st parameter is a directory, then tar it up in the scratch space
if [ -d "$1" ]; then
    absolute="`cd $1; pwd` "
    mkdir -p $SCRATCH_DIR
    cd $SCRATCH_DIR
    nameonly=${absolute##*/}
    nameonly=${nameonly/ /} #remove trailing spaces
    tar -cf $nameonly.tar $absolute
    arch="${SCRATCH_DIR}/${nameonly}.tar"
    created=true
    echo "Created temporary archive $arch"
else
    arch="`readlink -f $1`"
    created=false
    echo "Working with existing archive $arch"
fi


#call for GPG encryption and compression of the archive
target="${arch##*/}.gpg"
name=${target%\.gpg}
mkdir -p $SCRATCH_DIR
cd $SCRATCH_DIR
echo "Commencing GPG encryption, please be patient"
gpg --bzip2-compress-level 6 --compress-algo bzip2 --output $target --encrypt --recipient $2 $arch


#split .gpg file into chunks of size CHUNK_SIZE
outdir="${SCRATCH_DIR}/output"
mkdir -p "$outdir"
mkdir -p "$outdir/$name"
cd $outdir/$name && rm -f $name*
echo "Splitting files"
split -b $CHUNK_SIZE "${SCRATCH_DIR}/$target"
for x in *
do
  mv $x "../${name}__$x"
done


#clean up - remove .gpg and temporary archive and temporary directory
cd $SCRATCH_DIR
rmdir "$outdir/$name"
rm $target
if [ $created == true ]; then
    echo "Removing temporary archive $arch"
    rm $arch
fi

echo "All file splits produced placed in $outdir"

Download link for mince.sh

The bash script to reconstitute the file is called unmince.sh and takes one parameter – the name of the first file downloaded from google docs:

#!/bin/bash
# reassemble an archive from chunks of a file that have been gpg-encrypted
# $1 specifies the first file produced from the mincing process, eg, file__xaa
set -e


SCRATCH_DIR=~/scratch_space
REGEX='__xaa'


usage() {
    echo "ERROR: "
    echo " $*"
    echo "USAGE: "
    echo " unmince.sh file$REGEX"
    echo "WHERE: "
    echo " file$REGEX is the first file produced by the mince script"
}


#check parameters
[ $# -ne 1 ] && usage "Only one parameter is required" && exit 1
if [ -d "$1" ] || [[ ! $1 =~ '_xaa' ]]; then
    usage "$1 cannot be a directory and must end in _xaa."
    exit 1
fi


#combine all chunks of the file
sourcepath="`readlink -f $1`"
pathonly="`dirname $sourcepath`"


just=${sourcepath##*/}
basenam=${just%$REGEX}
indir="${SCRATCH_DIR}/reconstituted"
mkdir -p $indir
cd $indir
[ -e $indir/combined.gpg ] && rm $indir/combined.gpg
for x in $pathonly/$basenam*
do
    cat $x >> combined.gpg
done


#decrypt the .gpg file
echo "Commencing GPG decryption, please be patient"
gpg --output $basenam --decrypt combined.gpg


#tidy up - remove the gpg file
rm combined.gpg


echo "The reconstituted archive $indir/$basenam was created"

Download link for unmince.sh

Since I might still tinker and improved these scripts, to get the newest version of these files take a look at my github repo at http://github.com/eyesonly/kenwood (Named after the Kenwood Chef, a famous mincer!)

After over a year of development, we have finally released Ibid. Ibid is a general purpose chat bot written in Python. We've suffered from a bit of feature creep, so despite being a 0.1 release it can talk 7 messaging protocols and has over 100 features provided by plugins. I think we also have an excellent architecture and very developer friendly plugin API. The 0.1.0 release can be downloaded from Launchpad or installed from our PPA.

I used to work for OLPC, whose mission is to distribute low cost laptops for education, without necessarily the connectivity with the outside world. Now at Praekelt we’re focusing on using connectivity as the power – harnessing the deployed base of mobile phones in Africa without requiring them to be smartphones or computing devices. As part of this Praekelt Foundation and Vodacom are hosting the Betavine Social Exchange Cape Town Developer Day 2010.

I’ve asked Steve Wolak to tell us more about Betavine and the event.

Who is Steve Wolak?

Stephen Wolak, Founder and Head of Betavine, has worked in mobile technology and software since graduating from Imperial College, London. Stephen joined Vodafone Group R&D in 2000 and in 2006 put forward the idea of an open platform for engaging the wider technology community with R&D activities.  The rest, as they say, is history.

MC: I first became aware of Betavine when looking for 3G drivers for Linux, but I’m sure there is more to it than that. What is Betavine and how did it start?

SW: Betavine was launched in January 2007 and has been evolving ever since, with new features being added in response to new requirements and feedback from the user base.  One area of success has been the linux software for the Vodafone Mobile Connect (VMC) card which has been downloaded over 750,000 times and has created a lively community around it.

We have also run a number of successful competitions on the website and created a lively Widget Zone. The website continues to evolve and we try out new things.

MC: What is the Betavine Social Exchange?

SW: This is our latest idea.. creating “social and sustainable” mobile solutions.  The Betavine Social Exchange seeks to bring together 3 communities; the mobile community, the social sector and the entrepreuners.  Together these communities can create mobile solutions for the social sector.  Community organisations create challenges on the website and mobile developers / social entrepreneurs create solutions. The website then supports the deployment of these solutions on the ground.

MC: The BSX’s success certainly depends on connecting the right people: those with needs – the NGOs and community organisations – and the developers. How do you publicise the BSX to reach them?

SW: We are running our pilot in South Africa and so we are working with Sangonet to help us get in touch with South African NGOs.  We are running a developer day in Cape Town to help us engage with the local developer community.

MC: What do the resulting solutions include – are they apps for mobile phones, mobi websites, SMS solutions or all of the above?

SW: All of the above.  It is important that the solution is appropriate for the challenge and the local community that will ultimately use the solution.

MC: What can developers expect from participating in the BSX?

SW: They can find real world challenges that people are seeking solutions to.  They can meet other developers and find useful resources to help them create a business.  The full resources section is coming soon.

MC: Which leads us on to the Developer Day being hosted in Cape Town next month. What’s going to be happening at the event, and how does it tie in with the BSX?

SW: We are keen to encourage mobile developers based in South Africa to engage with the real challenges that have been posted on the Betavine Social Exchange.  The developer day will include presentations on mobile technology and some exciting mobile solutions plus a developer competition and lots of creative energy and networking.

MC: You’re going to be speaking at the event. Who would you like to see there?

SW: I would like to see mobile developers plus those with design skills and a passion for using mobile technology for social change.

MC: We’re having a developer competition on the day. Can you tell us anything about the prizes/incentives you’re planning to bring?

SW: Well, the developer day is free and includes lots of food and drink plus some beer at the end of the day … :-)  We also intend to offer a few prizes for the competition winners .. But we have not decided exactly what yet.  You will have to come along and see but tickets are going fast!

Developer Day details

Date: Wednesday, March 10, 2010 from 9:30 AM – 7:00 PM

Location: The Lightbox Daylight Studio, Green Point, Cape Town

More information and free tickets are available at eventbrite. Due to the demand, the event has been expanded to 70 people.

SuperGenPass and Password Composer are password generators, which generate different passwords for each site you use based on a single master password. This gives you the convenience of only remembering one password as well as the security of using different (and strong) passwords for each site. This means that you won't have all your accounts compromised when¹ one of them is compromised.

Most password generators are implemented as browser extensions or bookmarklets, since they are most frequently needed in a web browser. I've been wanting to start using a password generator, but I wanted to be sure that I could access my accounts even if I didn't have a web browser accessible. The two situations I could think of were a command line only system (e.g. SSH) and my cellphone².

Surprisingly, I couldn't find a command line implementation of SuperGenPass, so I wrote one in Python. I also couldn't find any J2ME or Symbian implementations, and so wrote my own one in J2ME. They both support subdomain stripping and configurable password lengths. They don't support salted passwords.

I chose SuperGenPass over Password Composer because it uses a better scheme. Password Composer only uses hex characters, whereas SuperGenPass uses a base64 encoded hash. SuperGenPass also hashes the password multiple times (which would slow down a brute force attack to find the master password) and imposes complexity requirements on the generated password (which reduces the chances that the generated password can be brute forced).

Deploying a new server at work – a dedicated server hosted at Hetzner. Fortunately Jaunty (Ubuntu 9.04) was released before we had anything hosted on the machine, so I took the decision to upgrade it before we do serious deployment.

One of the shiny new features of Ubuntu Server 9.04 is etckeeper, documented here by Thierry Carrez. In particular, on 9.04 etckeeper plays well with bzr and shows the real user who typed “sudo etckeeper commit” in the bzr log, not just “root”.

As we have a (small but distributed) team adminning the server, this will help a great deal to keep track of who did what when.

Some observations on the last n releases:

Throughout the Ubuntu development cycle, there are daily “snapshot” CD images produced. If you’re fortunate to live in a country where most of the “broadband” online population are not capped at 1GB per month (and a presidential hopeful who doesn’t keep singing “bring me my machine gun“) then you can download these during the development cycle to boot (daily-live) or install (perhaps in a virtual machine) to check on the progress or help with testing. These culminate in the actual “gold” release image.

Therefore, if you have one of these images from near the end of the development cycle, such as the release candidate, you can rsync to the latest image available on release day, and that will download the differences between the iso you have, and the final daily image – which will be identical to the release image, even though the daily image will be named something like jaunty-desktop-i386.iso and the corresponding release image named ubuntu-9.04-desktop-i386.iso. Rename it, and you’re done!

(Check the MD5SUMS after the release is announced, to be 100% sure you have it. There is always a small chance of a change to the ISOs on release day if some major “ate all my data” bug is found – so if you do have problems, remember that it comes with no warranty…)

Now, for kicks, go and lurk on IRC in #ubuntu-release-party and watch the masses rocking up to ask “Is it out yet?” Note Alan Pope’s list of Things Not To Say, and don’t go gloating that you have it already – you’ll only be kicked from the channel by the ironically named partybot.

Instead, burn write it to a USB stick (CDs are so early 2008) and get installing!

URLs shouldn't really contain file extensions (like .html, .png) since they are supposed to identify a resource and not a particular representation/format thereof. The format is indicated by the Content-Type header sent in the response. Modern CMSs do this already (for example, the URL of this page doesn't include .html).

Doing the same for static files (i.e. files served directly by the webserver) isn't straightforward because most webservers use the file extension to determine the MIME type to send in the Content-Type header. This means that simply removing the file extension from the filename (or even creating a symlink without a file extension) will cause the webserver to send the wrong Content-Type header.

I decided to try find a solution to this for my webserver of choice, Lighttpd. Lighttpd has a module which embeds a Lua interpreter and allows you to write scripts which modify (or even handle) requests. So I wrote a script which searches the directory for files with the same name as requested but with an extension. This means that any file can be accessed with the file extension removed from the URL while still having the correct Content-Type.

The script currently chooses the first matching file, which means that having multiple files with the same name but different extensions doesn't do anything useful. The proper method however is to actually do content negotiation, which chooses the format based on the preferences indicated by the HTTP client in the Accept header.

To use this script, download it and save it somewhere (I use /etc/lighttpd/). Enable mod_magnet, and add the following line to the site definition.

magnet.attract-physical-path-to = ("/etc/lighttpd/extension.lua")

I’m constantly surprised when I come across long-time Linux users who don’t know about SysRq. The Linux Magic System Request Key Hacks are a magic set of commands that you can get the Linux kernel to follow no matter what’s going on (unless it has panicked or totally deadlocked).

Why is this useful? Well, there are many situations where you can’t shut a system down properly, but you need to reboot. Examples:

• You’ve had a kernel OOPS, which is not quite a panic but there could be memory corruption in the kernel, things are getting pretty weird, and quite honestly you don’t want to be running in that condition for any longer than necessary.
• You have reason to believe it won’t be able to shut down properly.
• Your system is almost-locked-up (i.e. the above point)
• Your UPS has about 10 seconds worth of power left
• Something is on fire (lp0 possibly?)
• …Insert other esoteric failure modes here…

In any of those situations, grab a console keyboard, and type Alt+SysRq+s (sync), Alt+SysRq+u (unmount), wait for it to have synced, and finally Alt+SysRq+b (reboot NOW!). If you don’t have a handy keyboard attached to said machine, or are on another continent, you can

In my books, the useful SysRq commands are:

b

Reboot

f

Call the oom_killer

h

Display SysRq help

l

Print a kernel stacktrace

o

Power Off

r

Set your keyboard to RAW mode (required after some X breakages)

s

Sync all filesystems

u

Remount all filesystems read-only

0-9

Change console logging level

In fact, read the rest of the SysRq documentation, print it out, and tape it above your bed. Next time you reach for the reset switch on a Linux box, stop your self, type the S,U,B sequence, and watch your system come up as if nothing untoward has happened.

Update: I previously recommended U,S,B but after a bit of digging, I think S,U,B may be correct.