Clug Park-Tech

06 January 2016

Michael Gorven (cocooncrash)

Memory optimised decompressor for Pebble Classic

TLDR: DEFLATE decompressor in 3K of RAM

For a Pebble app I've been writing, I need to send images from the phone to the watch and cache them in persistent storage on the watch. Since the persistent storage is very limited (and the Bluetooth connection is relatively slow) I need these to be as small as possible, and so my original plan was to use the PNG format and gbitmap_create_from_png_data(). However, I discovered that this function is not supported on the earlier firmware used by the Pebble Classic. Since PNGs are essentially DEFLATE compressed bitmaps, my next approach was to manually compress the bitmap data. This meant that I needed a decompressor implementation ("inflater") on the watch.

The constraint

The major constraint for Pebble watchapps is memory. On Pebble Classic apps have 24K of RAM available for the compiled code (.text), global and static variables (.data and .bss) and heap (malloc()). There is an additional 2K for the stack (local variables). The decompressor implementation needed to have both small code size and variable usage. I discovered tinf which seemed to fit the bill, and tried to get it working.

Initially, trying to decompress something simply crashed the app. It took some debug prints to determine that code in tinf_uncompress() wasn't even being executed, and I realised that it was exceeding the 2K stack limit. I changed the TINF_DATA struct to be allocated on the heap to get past this. At this stage it was using 1.2K of .text, 1.4K of .bss, 1K of stack, and 1.2K of heap (total 4.8K). I set about optimising the implementation for memory usage.

Huffman trees

Huffman coding is a method to represent frequently used symbols with fewer bits. It uses a tree (otherwise referred to as a dictionary) to convert symbols to bits and vice versa. DEFLATE can use Huffman coding in two modes: dynamic and fixed. In dynamic mode, the compressor constructs an optimal tree based on the data being compressed. This results in the smallest representation of the actual input data; however, it has to include the computed tree in the output in order for a decompressor to know how to decode the data. In some cases the space used to serialise the tree negates the improvement in the input representation. In this case the compressor can used fixed mode, where it uses a static tree defined by the DEFLATE spec. Since the decompressor knows what this static tree is, it doesn't need to be serialised in the output.

The original tinf implementation builds this fixed tree in tinf_init() and caches it in global variables. Whenever it encounters a block using the fixed tree it has the tree immediately available. This makes sense when you have memory to spare, but in this case we can make another tradeoff. Instead we can store the fixed tree in the same space used for the dynamic tree, and rebuild it every time it is needed. This saves 1.2K of .bss at the expense of some additional CPU usage.

The dynamic trees are themselves serialised using Huffman encoding (yo dawg). tinf_decode_trees() needs to first build the code tree used to deserialise the dynamic tree, which the original implementation loads into a local variable on the stack. There is an intermediate step between the code tree and dynamic tree however (the bit length array), and so we can borrow the space for the dynamic instead of using a new local variable. This saves 0.6K of stack.

The result

With the stack saving I was able to move the heap allocation back to the stack. (Since the stack memory can't be used for anything else it's kind of free because it allows the non-stack memory to be used for something else.) The end result is 1.2K of .text, 0.2K of .bss and 1.6K of stack (total 3.0K), with only 1.4K counting against the 24K limit. That stack usage is pretty tight though (trying to use app_log() inside tinf causes a crash) and is going to depend on the caller using limited stack. My modified implementation will allocate 1.2K on the heap by default, unless you define TINF_NO_MALLOC. Using zlib or gzip adds 0.4K of .text. You can find the code on bitbucket.

by mgorven at 06 January 2016 06:28 AM

07 June 2015

Tristan Seligmann (mithrandi)

Adventures in deployment with Propellor, Docker, and Fabric

Preface

After playing around with Docker a bit, I decided that it would make an ideal deployment platform for my work services (previously we were using some ad-hoc isolation using unix users and not much else). While Docker’s security is…suspect…compared to a complete virtualization solution (see Xen), I’m not so much worried about complete isolation between my services, as things like easy resource limits and imaging. You can build this yourself out of cgroups, chroot, etc. but in the end you’re just reinventing the Docker wheel, so I went with Docker instead.

However, Docker by itself is not a complete solution. You still need some way to configure the Docker host, and you also need to build Docker images, so I added Propellor (which I recently discovered) and Fabric to the mix.

Propellor

Propellor is a configuration management system (in the sense of Puppet, Chef, Salt, et al.) written in Haskell, where your configuration itself is Haskell code. For someone coming from a systems administration background, the flexibility and breadth offered by a real programming language like Haskell may be quite daunting, but as a programmer, I find it far more straightforward to just write code that does what I want, extracting common pieces into functions and so on. Our previous incarnation of things used Puppet for configuration management, but it always felt very awkward to work with; another problem is that Puppet was introduced after a bunch of the infrastructure was in place, meaning a lot of things were not actually managed by Puppet because somebody forgot. Propellor was used to configure a new server from scratch, ensuring that nothing was done ad-hoc, and while I won’t go into too much detail about Propellor, I am liking it a lot so far.

The role of Propellor in the new order is to configure things to provide the expected platform. This includes installing Docker, installing admin user accounts, SSH keys, groups, and so on.

Docker

The Docker workflow I adopted is based on the one described by Glyph. I would strongly recommend you go read his excellent post for the long explanation, but the short version is that instead of building a single container image, you instead build three: A “build” container used to produce the built artifacts from your sources (eg. Python wheels, Java/Clojure JARs), a “run” container which is built by installing the artifacts produced by running the “build” container, and thus does not need to contain your toolchain and -dev packages (keeping the size down), and a “base” container which contains the things shared by the “build” and “run” containers, allowing for even more efficiency of disk usage.

While I can’t show the Docker bits for our proprietary codebases, you can see the bits for one of our free software codebases, including instructions for building and running the images. The relative simplicity of the .docker files is no accident; rather than trying to shoehorn any complex build processes into the Docker image build, all of the heavy lifting is done by standard build and install tools (in the case of Documint: apt/dpkg, pip, and setuptools). Following this principal will save you a lot of pain and tears.

Fabric

The steps outlined for building the Docker images are relatively straightforward, but copy/pasting shell command lines from a README into a terminal is still not a great experience. In addition, our developers are typically working from internet connections where downloading multi-megabyte Docker images / packages / etc. is a somewhat tedious experience, and uploading the resulting images is ten times worse (literally ten times worse; my connection at home is 10M down / 1M up ADSL, for example). Rather than doing this locally, this should instead run on one of our servers which has much better connectivity and a stable / well-defined platform configuration (thanks to Propellor). So now the process would be “copy/paste shell command lines from a README into an ssh session” — no thanks. (For comparison, our current deployment processes use some ad-hoc shell scripts lying around on the relevant servers; a bit better than copy/pasting into an ssh session, but not by much.)

At this point, froztbyte reminded me of Fabric (which I knew about previously, but hadn’t thoughto f in this context). So instead I wrote some fairly simple Fabric tasks to automate the process of building new containers, and also deploying them. For final production use, I will probably be setting up a scheduled task that automatically deploys from our “prod” branch (much like our current workflow does), but for testing purposes, we want a deploy to happen whenever somebody merges something into our testing release branch (eventually I’d like to deploy test environments on demand for separate branches, but this poses some challenges which are outside of the scope of this blog post). I could build some automated deployment system triggered by webhooks from BitBucket (where our private source code is hosted), but since everyone with access to merge things into that branch also has direct SSH access to our servers, Fabric was the easiest solution; no need to add another pile of moving parts to the system.

My Fabric tasks look like this (censored slightly to remove hostnames):

@hosts('my.uat.host')
def build_uat_documint():
    with settings(warn_only=True):
        if run('test -d /srv/build/documint').failed:
            run('git clone --quiet -- https://github.com/fusionapp/documint.git /srv/build/documint')
    with cd('/srv/build/documint'):
        run('git pull --quiet')
        run('docker build --tag=fusionapp/documint-base --file=docker/base.docker .')
        run('docker build --tag=fusionapp/documint-build --file=docker/build.docker .')
        run('docker run --rm --tty --interactive --volume="/srv/build/documint:/application" --volume="/srv/build/documint/wheelhouse:/wheelhouse" fusionapp/documint-build')
        run('cp /srv/build/clj-neon/src/target/uberjar/clj-neon-*-standalone.jar bin/clj-neon.jar')
        run('docker build --tag=fusionapp/documint --file=docker/run.docker .')


@hosts('my.uat.host')
def deploy_uat_documint():
    with settings(warn_only=True):
        run('docker stop --time=30 documint')
        run('docker rm --volumes --force documint')
    run('docker run --detach --restart=always --name=documint --publish=8750:8750 fusionapp/documint')

Developers can now deploy a new version of Documint (for example) by simply running fab build_uat_documint deploy_uat_documint. Incidentally, the unit tests are run during the container build (from the .docker file), so deploying a busted code version by accident shouldn’t happen.

by mithrandi at 07 June 2015 10:57 PM

07 March 2015

Tristan Seligmann (mithrandi)

Axiom benchmark results on PyPy

EDIT: Updated version now available.

EDIT: Fixed the issue with the store-opening benchmark

Axiom conveniently includes a few microbenchmarks; I thought I’d use them to give an idea of the speed increase made possible by running Axiom on PyPy. In order to do this, however, I’m going to have to modify the benchmarks a little. To understand why this is necessary, one has to understand how PyPy achieves the speed it does: namely, through the use of JIT (Just-In-Time) compilation techniques. In short, these techniques mean that PyPy is compiling code during the execution of a program; it does this “just in time” to run the code (or actually, if I understand correctly, in some cases only after the code has been run). This means that when a PyPy program has just started up, there is a lot of performance overhead in the form of the time taken up by JIT compilation running, as well as time taken up by code being interpreted slowly because it has not yet been compiled. While this performance hit is quite significant for command-line tools and other short-lived programs, many applications making use of Axiom are long-lived server processes; for these, any startup overhead is mostly unimportant, the performance that interests us is the performance achieved once the startup cost has already been paid. The Axiom microbenchmarks mostly take the form of performing a certain operation N times, recording the time taken, then dividing that time by N to get an average time per single operation. I have made two modifications to the microbenchmarks in order to demonstrate the performance on PyPy; first, I have increased the value of “N”; second, I have modified the benchmarks to run the entire benchmark twice, throwing away the results from the first run and only reporting the second run. This serves to exclude startup/”warmup” costs from the benchmark.

All of the results below are from my desktop machine running Debian unstable on amd64, CPython 2.7.5, and PyPy 2.1.0 on a Core i7-2600K running at 3.40GHz. I tried to keep the system mostly quiet during benchmarking, but I did have a web browser and other typical desktop applications running at the same time. Here’s a graph of the results; see the rest of the post for the details, especially regarding the store-opening benchmark (which is actually slower on PyPy).

[graph removed, see the new post instead]

To get an example of how much of a difference this makes, let’s take a look at the first benchmark I’m going to run, item-creation 15. This benchmark constructs an Item type with 15 integer attributes, then runs 10 transactions where each transaction creates 1000 items of that type. In its initial form, the results look like this:

mithrandi@lorien> python item-creation 15
0.000164939785004
mithrandi@lorien> pypy item-creation 15
0.000301389718056

That’s about 165µs per item creation on CPython, and 301µs on PyPy, nearly 83% slower; not exactly what we were hoping for. If I increase the length of the outer loop (number of transactions) from 10 to 1000, and introduce the double benchmark run, the results look a lot more encouraging:

mithrandi@lorien> python item-creation 15
0.000159110188484
mithrandi@lorien> pypy item-creation 15
8.7410929203e-05

That’s about 159µs per item creation on CPython, and only 87µs on PyPy; that’s a 45% speed increase. The PyPy speed-up is welcome, but it’s also interesting to note that CPython benefits slightly from the changes to the benchmark. I don’t have any immediate explanation for why this might be, but the difference is only about 3%, so it doesn’t matter too much.

The second benchmark is inmemory-setting. This benchmark constructs 10,000 items with 5 inmemory attributes (actually, the number of attributes is hardcoded, due to a limitation in the benchmark code), and then times how long it takes to set all 5 attributes to new values on each of the 10,000 items. I decreased the number of items to 1000, wrapped a loop around the attribute setting to repeat it 1000 times, and introduced the double benchmark run:

mithrandi@lorien> python inmemory-setting
4.86490821838e-07
mithrandi@lorien> pypy inmemory-setting
1.28742599487e-07

That’s 486ns to set an attribute on CPython, and 129ns on PyPy, for a 74% speed increase. Note that this benchmark is extremely sensitive to small fluctuations since the operation being measured is such a fast one, so the results can vary a fair amount between benchmarks run. For interest’s sake, I repeated the benchmark except with a normal Python class substituted for Item, in order to compare the overhead of setting an inmemory attribute as compared with normal Python attribute access. The result was 61ns to set an attribute on CPython (making an inmemory attribute about 700% slower), and 2ns on PyPy (inmemory is 5700% slower). The speed difference on PyPy is more to do with how fast setting a normal attribute is on PyPy, than to do with Axiom being slow.

The third benchmark is integer-setting. This benchmark is similar to inmemory-setting except that it uses integer attributes instead of inmemory attributes. I performed the same modifications, except with an outer loop of 100 iterations:

mithrandi@lorien> python integer-setting
1.23480038643e-05
mithrandi@lorien> pypy integer-setting
3.80326986313e-06

That’s 12.3µs to set an attribute on CPython, and 3.8µs on PyPy, a 69% speed increase.

The fourth benchmark is item-loading 15. This benchmark creates 10,000 items with 15 integer attributes each, then times how long it takes to load an item from the database. On CPython, the items are deallocated and removed from the item cache immediately thanks to refcounting, but on PyPy a gc.collect() after creating the items is necessary to force them to be garbage collected. In addition, I increased the number of items to 100,000 and introduced the double benchmark run:

mithrandi@lorien> python item-loading 15
9.09668397903e-05
mithrandi@lorien> pypy item-loading 15
5.70205903053e-05

That’s 90µs to load an item on CPython, and 57µs on PyPy, for a modest 37% speed increase.

The fifth benchmark is multiquery-creation 5 15. This benchmark constructs (but does not run) an Axiom query involving 5 different types, each with 15 attributes (such a query requires Axiom to construct SQL that mentions each item table, and each column in those tables) 10,000 times. I increased the number of queries constructed to 100,000 and introduced the double benchmark run:

mithrandi@lorien> python multiquery-creation 5 15
5.5426299572e-05
mithrandi@lorien> pypy multiquery-creation 5 15
7.98981904984e-06

55µs to construct a query on CPython; 8µs on PyPy; 86% speed increase.

The sixth benchmark is query-creation 15. This benchmark is the same as multiquery-creation, except for queries involving only a single item type. I increased the number of queries constructed to 1,000,000 and introduced the double benchmark run:

mithrandi@lorien> python query-creation 15
1.548528409e-05
mithrandi@lorien> pypy query-creation 15
1.56546807289e-06

15.5µs to construct a query on CPython; 1.6µs on PyPy; 90% speed increase.

The final benchmark is store-opening 20 15. This benchmark simply times how long it takes to open a store containing 20 different item types, each with 15 attributes (opening a store requires Axiom to load the schema from the database, among other things). I increased the number of iterations from 100 to 10,000; due to a bug in Axiom, the benchmark will run out of file descriptors partway, so I had to work around this. I also introduced the double benchmark run:

mithrandi@lorien> python store-opening 20 15
0.00140788140297
mithrandi@lorien> pypy store-opening 20 15
0.00202187280655

1.41ms to open a store on CPython; 2.02ms on PyPy; 44% slowdown. I’m not sure what the cause of the slowdown is.

A bzr branch containing all of my modifications is available at lp:~mithrandi/divmod.org/pypy-benchmarking.

by mithrandi at 07 March 2015 11:24 AM

Axiom benchmark results on PyPy 2.5.0

This is a followup to a post I made about 1.5 years ago, benchmarking Axiom on PyPy 2.1.0. Not too much has changed in Axiom since then (we fixed two nasty bugs that mainly affected PyPy, but I don’t expect those changes to have had much impact on performance), but PyPy (now at 2.5.0) has had plenty of work done on it since then, so let’s see what that means for Axiom performance!

Unlike my previous post, I’m basically just going to show the results here without much commentary:

Graph of Axiom performance

A few notes:

  • I didn’t redo the old benchmark results, but the hardware/software I ran the benchmarks on is not significantly different, so I think the results are still valid as far as broad comparisons go (as you can see, the CPython results match fairly closely).
  • The benchmark harness I’m using now is improved over the last time, using some statistical techniques to determine how long to run the benchmark, rather than relying on some hardcoded values to achieve JIT warmup and performance stability. Still could use some work (eg. outputting a kernel density estimate / error bars, rather than just a single mean time value).
  • There is one new benchmark relative to the last time, powerup-loading; PyPy really shines here, cutting out a ton of overhead. There’s still room for a few more benchmarks of critical functions such as actually running and loading query results (as opposed to just constructing query objects).
  • The branch I used to run these benchmarks is available on Github.
  • The horizontal axis is cut off at 1.0 so you can’t actually see how store-opening lines up, but the raw data shows that PyPy 2.1.0 took about 53% longer on this benchmark, whil PyPy 2.5.0 only takes about 2% longer.

by mithrandi at 07 March 2015 11:22 AM

22 August 2014

Adrianna Pińska (Confluence)

Yet another way to play videos in an external player from Firefox

I spent some time today trying to figure out how to get Firefox to play embedded videos using anything other than Flash, which is an awful, stuttering, resource-devouring mess. The internet is littered with old browser extensions and user scripts which allegedly make this possible (e.g. by forcing sites like YouTube to use the vlc media plugin instead), but I was unable to get any of them to work correctly.

Here’s a quick hack for playing videos from YouTube and any other site that youtube-dl can handle in an external mplayer window. It’s based on several existing HOWTOs, and has the benefit of utilising a browser extension which isn’t dead yet, Open With, which was designed for opening websites in another browser from inside Firefox.

I wrote a little script which uses youtube-dl to download the video and write it to stdout, and pipes this output to mplayer, which plays it. Open With is the glue which sends URLs to the script. You can configure the extension to add this option to various context menus in the browser — for example, I can see it if I right-click on an URL or on an open page. You may find this less convenient than overriding the behaviour of the embedded video on the page, but I prefer to play my videos full-screen anyway.

This is the script:

#!/bin/sh
youtube-dl -o - $1 | mplayer -

Make it executable. Now open Open With’s preferences, add this executable, and give it a nicer name if you like. Enjoy your stutter-free videos. :)

(Obviously you need to have youtube-dl and mplayer installed in order for this to work. You can adapt this to other media players — just check what you need to do to get them to read from stdin.)

by confluence at 22 August 2014 02:41 PM

17 April 2013

Raoul Snyman (superfly)

Nomanini at ScaleConf 2013

For those who didn't know, I started working for a company called Nomanini at the beginning of last year. Nomanini manufactures an airtime point of sale terminal for use in the informal sector (taxis, spaza shops, etc).

I'm excited to say that my boss will be speaking at ScaleConf 2013, which kicks off tomorrow, and myself and my colleagues are going to be there too. See you there!

by raoul at 17 April 2013 02:02 PM

23 February 2013

Raoul Snyman (superfly)

How NOT to conduct yourself online

Many of you know that I am the project leader of an open source worship presentation program called OpenLP. Yesterday evening, as I was doing something on OpenLP's project page on SourceForge.net (a popular repository for open source projects), I glanced at the reviews page and the one below caught my eye.

Review

Now, I don't normally worry about negative reviews (they lend your project a certain amount of authenticity), I couldn't help but notice the reference AND link to a commercial product. I then simply did a search for the reviewer on Google and one of the first few hits was the reviewer's blog. I went to the About page, and sure enough, not only was it the same person, I found out some interesting facts about him.

Bio

Establishing that the blog did indeed belong to the reviewer was easy. Firstly, the reviewer didn't even bother to use a different photo for his profile on SF.net, and secondly his domain name, Twitter account and SF.net username were all identical. Lastly, the product he was promoting over OpenLP was the very same product he is the executive director (or somesuch inflated title) of.

I just couldn't believe my eyes. Here is what I would presume is an upstanding Christian man, and he directly slates a competitor's product without so much as disclosing who he is. I don't go around telling everyone how awful EasyWorship, ProPresenter and MediaShout are. It just is not something you do. If your product is better than the others then let it speak for itself.

This reminds me of Microsoft a few years ago who didn't know how to handle open source software, so they paid people to conduct absurd studies to prove how great they were.

Seriously, are you so scared of OpenLP that you have to pretend to be an unhappy user? I highly doubt you've ever even downloaded OpenLP, nevermind actually used it.

Incidentally, I marked his comment as spam, so it is no longer visible.

by raoul at 23 February 2013 06:35 AM

14 January 2013

Raoul Snyman (superfly)

RE: Ubuntu Loco Games 2013

LoCo Games

The newly-approved Ubuntu Mexico LoCo has decided to celebrate their status as an official LoCo by hosting an online gaming event. They will be hosting matches for AssaultCube, Urban Terror and Battle for Wesnoth on the 9th of Feb from 17:00 SAST (9:00 CST) to midnight (16:00 CST) on the 10th of Feb. They have 2 LoCo's signed up so far, so why don't you get together and add yours!

For more information, check out their blog post or the event on the LoCo site.

by raoul at 14 January 2013 09:56 AM

30 October 2012

Michael Gorven (cocooncrash)

XBMC packages for Raspberry Pi running Raspbian

I recently got a Raspberry Pi, which is an ARM based device which runs Linux. My goal is to use this as my HTPC running XBMC, so that I can move the fileserver out the lounge.

Edit: I've moved the latest information and updates about these packages to this page.

Building

I installed Raspbian on my RPi, which is basically a rebuild of Debian Wheezy specifically for the RPi (targeting the ARMv6 architecture with hard float). I found various instructions on how to build XBMC for Raspbian, but none of them were in the form of deb packages, and installing software without packages just makes me queezy. So I went off and built it myself.

Since the RPi is relatively low powered, I built the package on my laptop using qemu-user, which emulates binaries with a different architecture. I based the packaging on the XBMC package in Wheezy, and the source is from the xbmc-rbp branch on GitHub. I made the modifications to the source as per this forum post and added an initscript so that it can automatically start at bootup.

Installing

The easiest way to install the package is to add my archive to your system. To do this, store the following in /etc/apt/sources.list.d/mene.list:

deb http://archive.mene.za.net/raspbian wheezy contrib

and then import the archive signing key:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-key 5243CDED

You can then install it as you would with any other package, for example, with apt-get:

sudo apt-get install xbmc

(If you don't want to configure my archive you can download the packages manually, but you'll have to deal with all the dependencies. Note that it requires a newer libcec package which is also in my archive.)

The user which you're going to run XBMC as needs to be a member of the following groups:

audio video input dialout plugdev

Running

To run XBMC, run xbmc-standalone from a VT (i.e. not under X). XBMC accesses the display directly and not via Xorg.

If you want XBMC to automatically start when the system boots, edit /etc/default/xbmc and change ENABLED to 1:

ENABLED=1

You also need to set the user which XBMC should run as (the xbmc user is not automatically created at the moment). Run sudo service xbmc start to test this.

Configuration

The following settings in advancedsettings.xml decreases the CPU usage while showing the UI. Disabling the RSS feeds also helps with this.

<advancedsettings>
    <gui>
        <algorithmdirtyregions>3</algorithmdirtyregions>
        <nofliptimeout>0</nofliptimeout>
    </gui>
</advancedsettings>

Rebuilding

If you want to rebuild this package with a different source (e.g. a later Git revision), you need to prepare the source by running ./bootstrap before creating the orig.tar.gz. You obviously need all the build dependencies (which should be listed in the packaging), as well as the VideoCoreIV libraries in /opt. You probably want to set DEB_BUILD_OPTIONS=nocheck parallel=4 to disable the tests (they failed to build for me), and speed up the build by using more than one core. You can find the packaging in my archive or my Bazaar repository.

by mgorven at 30 October 2012 10:35 PM

Building Raspbian images for Raspberry Pi

I recently bought a Raspberry Pi, which is a credit card sized computer with an ARM processor. I'm using it as my TV frontend, running Raspbian and XBMC. I'm building my own packages for XBMC since it requires the latest development version.

I initially installed my Pi with the foundation image, but found that it included a lot of packages which I didn't need. Since I have a slight obsession about doing things as efficiently as possible, I decided to build my own image with XBMC from scratch.

I implemented a script in Bash, mkraspbianxbmc.sh which does this. It uses debootstrap to install a minimal Raspbian system in a chroot. It then installs XBMC and a couple extra packages, and does some necessary configuration. Finally it creates an image file with the necessary partitions, creates the filesystems, and copies the installation into the image file. The resultant image fits onto a 1GiB SD card. You can download a pre-built image from this page.

The script can be modified to build images with different packages, or even a very minimal image which fits onto a 512MiB SD card.

by mgorven at 30 October 2012 10:34 PM

30 August 2012

Adrianna Pińska (Confluence)

Finding articles on Twitter

I have strong opinions about retweeting, which seem to be the opposite of many other people’s opinions about retweeting. I like new-style retweets. If I retweet someone, I want it to be clear that they are the original source of the tweet. If someone else tweets something terribly insightful, I don’t want to see fifty old-style retweets of it. Finally, I would like “RT @joe RT @bob OMG, totally RT @alice I agree! RT @cheesemonger OMG you guys I loled!!1 RT @somedude this is lank insightful RT @dave The” to die in a fire.

Which it mostly has. When new-style retweeting was first introduced there was a lot of wailing and gnashing of teeth as some people were outraged that they were seeing tweets from people they weren’t directly following in their timeline. Today, nobody seems to care anymore, and I seldom see a manual old-style retweet. Unfortunately, I see a similar problem with the automated sharing of articles.

As far as I’m concerned, the best way to share an article on site Foo on Twitter is this: 1) go to Foo’s Twitter stream, 2) find the tweet which links to the article, 3) retweet it. Unfortunately, what most “share this article on Twitter” buttons that sites display (and encourage you to use) actually do is make you create an old-style retweet. It usually automatically includes the site’s Twitter username, and of course it’s a link to the site, so credit is not really a problem — but duplication still is, and in my OCD opinion this form of sharing is far less elegant than retweeting the original, official tweet.

I made a bookmarklet which searches Twitter for the title of the page you’re currently reading. It tries to autodetect and chop off the site name if it exists, and it loads the results in a new window (or tab, depending on your browser preferences). It currently doesn’t limit the search to the site’s Twitter account, because determining what that is is less trivial than extracting the title from the page. I may write a more complicated bookmarklet when I have more time. In the meantime, if you’re as obsessive about Twitter etiquette as I am, enjoy:

Find on Twitter

by confluence at 30 August 2012 01:27 PM

13 August 2012

Adrianna Pińska (Confluence)

How to hack GTK’s “Recently Used” virtual folder

Lots of people hate GTK’s new file chooser behaviour, which sets the location to a virtual folder with recently used files unless the app overrides it to something more useful. Some apps don’t. It is currently not possible to change this default.

Some people hate this behaviour because they don’t want their gran to see all their recently viewed donkey porn when saving a spreadsheet. If I let my gran use my computer I would give her a guest account, so I just hate it because it punches my workflow in the stomach, and stabs it repeatedly in the face while it’s down.

The privacy advocates can solve their problem by clearing the file in which recently used files are recorded and making it read-only. That doesn’t help me have something that isn’t utterly useless as the first thing I see in a file chooser. I looked more closely at the file to see how easy it would be to hard-code it to something useful, like a list of commonly used directories.

The file is ~/.local/share/recently-used.xbel. It’s XML, so it’s fugly, but reasonably editable. A single bookmark entry looks like this:

<bookmark href="file:///home/username/path/to/some.file.html" added="2012-08-13T23:07:20Z" modified="2012-08-13T23:07:20Z" visited="2012-08-13T23:07:20.868364Z">
    <info>
        <metadata owner="http://freedesktop.org">
            <mime:mime-type type="text/html"/>
            <bookmark:applications>
                <bookmark:application name="Firefox" exec="&apos;firefox %u&apos;" modified="2012-08-13T23:07:20Z" count="1"/>
            </bookmark:applications>
        </metadata>
    </info>
</bookmark>

So if you change the path to a directory you want to see in the list, and set the appropriate mimetype, everything should Just Work, right?

Well, yes, actually. It is that simple. Today I learned that directories have a mimetype: inode/directory. I haven’t checked what happens if you set the incorrect mimetype; if you do and it’s something hilarious, let me know in the comments.

If you opened the file in multiple applications, you’ll see multiple applications listed in the application section. This information does not seem to affect which files appear in which apps — the same list is used everywhere. You can remove the whole section, and the file chooser doesn’t seem to care.

So I added some directories I use frequently, removed all the other bookmarks, and made the file read-only. Now I have a sane default. I was considering writing a script to make editing the file more pleasant, but most GTK apps I use sensibly override the default to the most recently used directory, so I don’t really need this solution to be that advanced.

(P.S. Please don’t comment just to tell me to “use a different window manager”, which I’ve already been doing for years. This is a toolkit issue, and I’m not going to stop using GTK apps because the file chooser is doing something stupid.)

by confluence at 13 August 2012 11:33 PM

05 December 2010

Jonathan Groll (eyesonly)

Add subtitles to a DVD


Many people want to add subtitles to a DVD in their own language, and often DVD disks don’t come with subtitles in a language you want. For instance, if you’re staying in a foreign country and want English subtitles added to your rental movie. Or in my case, the library at the local Alliance Française has a lot of interesting movies completely in French, without English subtitles, and my french isn’t yet good enough to fully understand everything that is being said.

What is interesting is that there are sites on the internet that offer downloadable subtitle files for many movies. Simply type the name of your movie, plus the word “subtitle” into google and provided the movie is popular or well known enough you should be able to find a site offering the subtitle file for download. Mostly it seems that these subtitles were extracted from a DVD owned by someone else with the right languages. In other cases, it is purely fans of the movie who loved it so much and wanted to translate the words of the movie into a language that they know so that others can also enjoy the movie. I’m not so sure about the legality of the subtitles obtained in the first case, but certainly an argument may be made that since you own or have legally rented a copy of the movie it does seem to fall within “fair usage” to watch this movie with subtitles in a language you require.

So, you may be lucky. You may find the (fan-written) subtitle file that exactly matched your movie. By this I mean that the person who created your file had exactly the same version of the DVD, so that the lead in at the beginning of the movie is exactly the same length and in this case the subtitles remain perfectly in synch with the video.

However, don’t expect to watch the movie in your living room just yet. Most stand-alone DVD players that I’ve used do not have a facility to specify an external subtitles file, but almost all of the software based players on your computer do have this facility. So provided you got lucky, and got the correct subtitles file that matched your DVD, and are also content to watch movies in front of your computer that will be sufficient and you don’t need to read further.

Mostly though, the subtitles file doesn’t have the right timings for your DVD, or you might really really want to watch it on your TV in the lounge rather than with bowls of popcorn balanced on your lap in front of the computer. The guide that follows is for people who are running Linux, are comfortable with the command line, and who wish to hardcode 1 subtitles into a XVID file for watching on a (living-room) player that can play DIVX movies. If this is not exactly what you want there still may be something for you in this blog post. I’ll outline my six-step method:

  1. Rip (extract) contents of the DVD disk.
  2. Attempt to play DVD file with subtitle files on desktop.
  3. Create a XVID file without subtitles.
  4. Transform the subtitle file so that subtitles are synchronized with the video.
  5. Create a XVID file with subtitles hardcoded in.
  6. Break up this file into smaller segments for players that cannot handle large files.

I myself don’t follow the above recipe every time – depending on the circumstances some of these steps may not be needed. And you may very well want a different outcome – it may be enough for you just to align the subtitles with the video, or you may want more such as recreating a DVD with subtitles. You may even despise XVID files with harcoded subtitles in them!

If you are running Debian/Ubuntu you’ll need a lot of packages to be installed – at least: mplayer mencoder ogmtools libdvdcss dvdbackup gaupol ffmpeg gstreamer0.10-x plus dependancies and probably gnome-codec-install.

The principal tools involved are the wonderful mplayer/mencoder command line tools :- they give fine control, are scriptable, and are compatible with each other in terms of command line flags.

For step 1, extract the DVD to your hard disk:

dvdbackup -i /dev/dvd -M -o name_of_movie

This will create a folder called name_of_movie in the current directory. If your DVD-device is not /dev/dvd put the correct device name there.

In step 2, you’ll need to work out which of the titles on the DVD disk contains the movie, start by typing:

mplayer -v -dvd-device name_of_movie/VIDEO_TS/ dvd://1

Keep on increasing the number at the end – from 1 to 2 to 3 etc until you find the actual movie and not just the adverts and bonus features. In the examples that follow I refer to title 1, you will need to replace that with your correct title number.

The next thing to do is test the downloaded subtitle file:

mplayer -v -dvd-device name_of_movie/VIDEO_TS/ dvd://1 -sub downloaded_sub.srt

The subtitle file doesn’t have to end in .srt – mplayer supports multiple formats. Make a careful note if the subtitles are in synch, namely do the spoken words match the subtitles. Press the right arrow key on your keyboard to advance the video. Ensure that subtitles remain in synch until the end of the movie.

For step 3, create an initial XVID file without subtitles. Yes, I know this can be lossy. And it really is not a required step if your subtitles are already in synch, if they are you can move straight on to step 5. However, if you do need to adjust the subtitle file to match the video, then in order to do so it may be necessary to work with the entire movie as a single file, instead of multiple individual VOB files.

So, here is a simple two-pass recipe for encoding a movie to XVID without subtitles:

mencoder -idx -dvd-device name_of_movie/VIDEO_TS/ dvd://1 -ovc xvid -oac pcm -xvidencopts pass=1:bitrate=1200:aspect=16/9 -o /dev/null
mencoder -idx -dvd-device name_of_movie/VIDEO_TS/ dvd://1 -ovc xvid -oac pcm -xvidencopts pass=2:bitrate=1200:aspect=16/9 -o movie.avi

Note the bitrate in the above call (1200) can be adjusted as required (higher numbers mean higher quality but also larger files). You will obviously need to make sure this call has the correct path for your VIDEO_TS folder, and also for the title track from the DVD (replace the dvd://1 with the appropriate title). Experienced mencoder users will possibly also want to add extra mencoder flags. Transcoding to XVID can take some time, depending on the speed of your computer.

Next up is step 4 – adjusting the subtitle file if that is needed. To do this, I prefer using the gaupol application under Linux. So, open gaupol and within gaupol select your subtitle file, and at the bottom of the gaupol screen select your .avi video file that you produced in the previous step. The trick to adjusting is to note exactly the time at which the first useful spoken words occur for which you have a matching subtitle, and the same for the last spoken words (unfortunately, knowing the last spoken words may spoil the end of the movie for you!). So, if you click the green play icon in gaupol it will play the video file with a clock at the top of the screen which you can use for noting when the spoken words occur. Based on the above, in an example I worked out that someone said subtitle #2 (you should be able to recognise the words based on context, even if you don’t speak the language) at 00:02:54.000 and that the one of the last spoken texts occurred for subtitle #613 at 01:29:17.000. The important thing is to get the timing of the first subtitle as accurate as possible, for the last subtitle it doesn’t matter if you’re a second or two out. To do the actual synchronization choose the “Transform Positions” tool from the subtitle menu (see screenshot below), specify the times you observed and the positions of the other subtitles will be adjusted proportionally.

To test the subtitle file is now correct, play the middle of the movie and see if the subtitles are fully in sync there as well. Repeat, if necessary with other time points, and if necessary adjusting only part of the file (“Transform selected subtitles”) rather than the entire file (“Transform current project”).

Now, for step 5 we are going to hardcode the subtitles into a new XVID file which we produce. We can choose to either start with the source coming from the original DVD-rip folder produced by step one, or we can choose to work with the XVID file produced in step three as the source. Obviously, working with the XVID file will be lossy, as one can expect some picture degradation using the XVID file as a source file. However, if this file were to be used, you would know for sure that your subtitles will sync up with the file. Think back to the DVD extract step with dvdbackup – did it report any errors at all during the extract from the DVD? For instance, if there was a scratch on the DVD disk dvdbackup would still be able to work around it by block filling in the file produced, but I have observed such a “repaired error” might cause problems with the timing and sync of subtitles in the step to follow if the original DVD extract is used as the source. Generally, because of this I prefer to work with the XVID file as the source, even though it may be lossy.

Before starting, remove any file in the current folder called “divx2pass.log”, this file will have been produced if you did a two-pass encoding as above and will interfere with the two-pass encode we are about to do.

To hardcode in the subtitles, using the divx file produced in step three as the source, here are the commands that will perform the two-pass encode for you:

mencoder -idx movie.avi -ovc xvid -sub subtitles_adjusted.srt -subpos 96 -oac pcm -xvidencopts pass=1:bitrate=1200:aspect=16/9 -o /dev/null
mencoder -idx movie.avi -ovc xvid -sub subtitles_adjusted.srt -subpos 96 -oac pcm -xvidencopts pass=2:bitrate=1200:aspect=16/9 -o hardcoded.avi

Now, if you wanted to work with the original DVD-rip folder as the source, rather than perform the lossy re-encoding of the DIVX file, then replace “movie.avi” in the above with “-dvd-device name_of_movie/VIDEO_TS/ dvd://1” adjusted where appropriate.

Also, note in the above that “subtitles_adjusted.srt” refers to your subtitle file that you produced with gaupol, and also note that I prefer specifying that my subtitles appear at position 96 (you can leave this out if it doesn’t bother you).

And also note that there is another way to hardcode in subtitles but this way can result in enormous files.

Finally, in step 6 I break up the resultant DIVX file (which would be called “harcoded.avi” if you followed step 5 exactly). Why do I do this? For two possible reasons:

  1. If you are playing off a VFAT formatted memory stick files cannot be larger than 2GB in size.
  2. My player can play DIVX files, but once the file exceeds 1GB in size the playback simply stops.

Now, there are two possible workarounds. You could have simply lowered the bitrate so that the final XVID file was less than 1GB in size. Or you could chop up a big file as I do below. Of course, if your player can play big DIVX files and you’re not working with a VFAT memory stick then there is no need to perform this step.

In this example I am working with a 3.1G file. Firstly, to determine how long the video is, I issue:

ffmpeg -i hardcoded.avi

In my case, it tells me that my video has

Duration: 02:05:16.88.

So, I’d like to split that into four parts, so that each part will come to less than 1GB in size (roughly) – so after some quick mental arithmetic I think I’d like each piece to be 32 minutes long.

Based on the above example, these are the four mencoder commands to splice the video into four 32 minute chunks (the last chunk will be less than 32 minutes):

mencoder -ovc copy -oac copy -endpos 0:32:00 -o hardcoded1.avi hardcoded.avi
mencoder -ovc copy -oac copy -ss 0:32:00 -endpos 0:32:00 -o hardcoded2.avi hardcoded.avi
mencoder -ovc copy -oac copy -ss 1:04:00 -endpos 0:32:00 -o hardcoded3.avi hardcoded.avi
mencoder -ovc copy -oac copy -ss 1:36:00  -o hardcoded4.avi hardcoded.avi

Granted, this is an extreme example (normally you don’t let the XVID file get so big!) but it does show how to splice up a video using the -ss and -endpos flags. Unlike the transcoding to XVID steps, this step executes very quickly as all it does is copy the video and audio streams.

And voilà, copy the split files to your memory stick, and take it to the lounge to enjoy with your popcorn, knowing that you’ve managed to render a previously unwatchable video into something you’ll enjoy, and somehow all this extra work makes the video more enjoyable too.

1 Hardcoding means the actual video file is altered so that the subtitle is in the video stream. It’s a little nasty, and I agree softcoding (keeping the subtitles separate from the video stream) is better. The only problem is that my player refuses to accept files where the subtitles have been muxed in. If you’re curious about softcoding, I can recommend AVI-Mux GUI which does run under wine.

05 December 2010 03:29 PM

03 November 2010

Jonathan Groll (eyesonly)

Tunnel through ISA proxy

Image of tunnel attributed to Richard Freeman Geeks often need to access their *nix computers from work. Doesn’t everyone want to do that? True geeks control their computers strictly using the command-line, of course, and the tool that is used to control a remote command-line session is ssh.

What one usually does is use a tool like corkscrew to send ssh traffic through an HTTP proxy.

At one place of employment, a known trick of using corkscrew to tunnel out using the work proxy failed, with this message:

Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy service is denied. )

I tried all combinations of DOMAIN\USERNAME:PASSWORD in my corkscrew auth file but nothing worked.

If you see this message have no fear! What you need is a utility that can negotiate NTLM authorization with the proxy.

There are several open source tools that can do NTLM, of these I chose cntlm. Often ntlmaps is recommended as a utility to negotiate the authorization. The cntlm man page indicates that cntlm is far more efficient than ntlmaps, both in terms of memory and CPU usage.

One oddity about cntlm relative to other software that you may have worked with is that configuration is a two-step procedure: firstly you configure the software with a default config file, like the following (the settings that need to be configured for now in the first-step are in the first paragraph: username, domain, password, proxy, proxy-port):

#
# Cntlm Authentication Proxy Configuration
#
# NOTE: all values are parsed literally, do NOT escape spaces,
# do not quote. Use 0600 perms if you use plaintext password.
#

Username __username__ Domain __domain__ Password __password__ # Use hashes instead (-H) #Workstation netbios_hostname # Should be auto-guessed
Proxy __PROXY__:__PROXY_PORT__
# # This is the port number where Cntlm will listen # Listen 3128
# # If you wish to use the SOCKS5 proxy feature as well, uncomment # the following option, SOCKS5. It can be used several times # to have SOCKS5 on more than one port or on different network # interfaces (specify explicit source address for that). # # WARNING: The service accepts all requests, unless you use # SOCKS5User and make authentication mandatory. SOCKS5User # can be used repeatedly for a whole bunch of individual accounts. # #SOCKS5Proxy 8010 #SOCKS5User dave:password
# # Use -M first to detect the best NTLM settings for your proxy. # Default is to use the only secure hash, NTLMv2, but it is not # as available as the older stuff. # # This example is the most universal setup known to man, but it # uses the weakest hash ever. I won't have it's usage on my # conscience. :) Really, try -M first. # #
Auth LM #Flags 0x06820000 # # Enable to allow access from other computers # #Gateway yes
# # Useful in Gateway mode to allow/restrict certain IPs # #Allow 127.0.0.1 #Deny 0/0
# # GFI WebMonitor-handling plugin parameters, disabled by default # #ISAScannerSize 1024 #ISAScannerAgent Wget/ #ISAScannerAgent APT-HTTP/ #ISAScannerAgent Yum/
# # Headers which should be replaced if present in the request # #Header User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)
# # Tunnels mapping local port to a machine behind the proxy # Tunnel 11443:__OUTSIDE_HOST.COM__:443

Then, run
cntlm -v -M http://google.com (or any other external site)

Cntlm will use this to assess the type of auth your proxy can handle. In my case I got back something like the following output:

Config profile  1/11... OK (HTTP code: 301)
----------------------------[ Profile  0 ]------
Auth            NTLMv2
PassNTLMv2      AE1234567890123234567890123456C4
------------------------------------------------

For the second configuration step, these two lines need to be pasted back into your configuration file replacing the line that said “Auth LM” (and you must do this for your own situation, you can’t reuse my lines).

Then, startup the cntlm daemon:


cntlm

Let’s test if it works. Note that in the above config file I have a tunnel defined (the last line of the config file). Now, in order to ssh to port 443 of host __outside_host.com__ which is outside the proxy, one can do so using:


ssh -p 11443 localhost

The above assumes that outside_host.com has an sshd listening on port 443.

Cntlm also works fine under windows. If you don’t have “administrator” authorisation under windows, you can still run the cntlm executable, but need to specify which cntlm.ini file to use, in other words, something like:


cntlm.exe -c cntlm.ini

Of course, if you’re running on windows you won’t have an ssh command line client, but putty can be used nicely for this purpose. Bear in mind that putty needs to be configured (for the default cntlm configuration above) to use an HTTP proxy on host “localhost” listening on port 3128.

03 November 2010 02:30 PM

29 October 2010

Jonathan Groll (eyesonly)

Split and encrypt files for google docs

Creative commons image of meat grinder from http://www.flickr.com/photos/amayu/3629064681/Since January 2010, Google docs has allowed you to store any type of file, even arbitrary binary files. However, there are a couple of gotchas: one cannot upload files greater than 1GB in size, and you may want to encrypt your files so that not just anyone can read them (for instance server backup files).

The two bash scripts below provide a solution for the above. I call them the ‘mince’ scripts ‘cos they slice and dice your files and hopefully you’ll get hamburgers back at the end of the day. These scripts depend on you having a fairly new version of bash on your unix-like system, the ‘split’ utility and gnupg (GPG) which is used for the encryption/decryption. If you’re unsure of GPG, a good getting started guide can be found here.

It must be said that google docs is (in my opinion) currently not the best way to store your files in the cloud. In fact, I wrote another blog post describing the “google storage” options in greater depth.

The encrypt&split script is mince.sh and it takes two parameters, the first one a directory or archive, the second the email address for an already imported public key:

#!/bin/bash
# gpg encrypt archive and split into chunks
# $1 specifies base directory or compressed archive to encrypt.
# $2 is the recipients public key, eg. 'myfriend@his.isp.net'
set -e

CHUNK_SIZE=1000000000 #1000000000==1GB (not 1GiB!) SCRATCH_DIR=~/scratch_space TAR_REGEX='\.tar'
usage() { echo "ERROR: " echo " $*" echo "USAGE: " echo " mince.sh [DIRECTORY|ARCHIVE] PUBLIC_KEY_NAME" echo "EXAMPLE: " echo " ./mince.sh directory myfriend@her.isp.net" echo "FURTHER COMMENTS: " echo " if an ARCHIVE is supplied instead of a directory, it must have a name like file.tar or file.tar.gz or file.tar.bz2 " }
#check parameters entered are valid [ $# -ne 2 ] && usage "Two parameters are required." && exit 1 if [ ! -d "$1" ] && [[ ! $1 =~ $TAR_REGEX ]]; then usage "$1 is not a directory or tar/tar.gz/tar.bz2 archive." exit 1 fi
#if 1st parameter is a directory, then tar it up in the scratch space if [ -d "$1" ]; then absolute="`cd $1; pwd` " mkdir -p $SCRATCH_DIR cd $SCRATCH_DIR nameonly=${absolute##*/} nameonly=${nameonly/ /} #remove trailing spaces tar -cf $nameonly.tar $absolute arch="${SCRATCH_DIR}/${nameonly}.tar" created=true echo "Created temporary archive $arch" else arch="`readlink -f $1`" created=false echo "Working with existing archive $arch" fi
#call for GPG encryption and compression of the archive target="${arch##*/}.gpg" name=${target%\.gpg} mkdir -p $SCRATCH_DIR cd $SCRATCH_DIR echo "Commencing GPG encryption, please be patient" gpg --bzip2-compress-level 6 --compress-algo bzip2 --output $target --encrypt --recipient $2 $arch
#split .gpg file into chunks of size CHUNK_SIZE outdir="${SCRATCH_DIR}/output" mkdir -p "$outdir" mkdir -p "$outdir/$name" cd $outdir/$name && rm -f $name* echo "Splitting files" split -b $CHUNK_SIZE "${SCRATCH_DIR}/$target" for x in * do mv $x "../${name}__$x" done
#clean up - remove .gpg and temporary archive and temporary directory cd $SCRATCH_DIR rmdir "$outdir/$name" rm $target if [ $created == true ]; then echo "Removing temporary archive $arch" rm $arch fi
echo "All file splits produced placed in $outdir"

Download link for mince.sh

The bash script to reconstitute the file is called unmince.sh and takes one parameter – the name of the first file downloaded from google docs:

#!/bin/bash
# reassemble an archive from chunks of a file that have been gpg-encrypted
# $1 specifies the first file produced from the mincing process, eg, file__xaa
set -e

SCRATCH_DIR=~/scratch_space REGEX='__xaa'
usage() { echo "ERROR: " echo " $*" echo "USAGE: " echo " unmince.sh file$REGEX" echo "WHERE: " echo " file$REGEX is the first file produced by the mince script" }
#check parameters [ $# -ne 1 ] && usage "Only one parameter is required" && exit 1 if [ -d "$1" ] || [[ ! $1 =~ '_xaa' ]]; then usage "$1 cannot be a directory and must end in _xaa." exit 1 fi
#combine all chunks of the file sourcepath="`readlink -f $1`" pathonly="`dirname $sourcepath`"
just=${sourcepath##*/} basenam=${just%$REGEX} indir="${SCRATCH_DIR}/reconstituted" mkdir -p $indir cd $indir [ -e $indir/combined.gpg ] && rm $indir/combined.gpg for x in $pathonly/$basenam* do cat $x >> combined.gpg done
#decrypt the .gpg file echo "Commencing GPG decryption, please be patient" gpg --output $basenam --decrypt combined.gpg
#tidy up - remove the gpg file rm combined.gpg
echo "The reconstituted archive $indir/$basenam was created"

Download link for unmince.sh

Since I might still tinker and improved these scripts, to get the newest version of these files take a look at my github repo at http://github.com/eyesonly/kenwood (Named after the Kenwood Chef, a famous mincer!)

29 October 2010 01:25 PM

05 February 2010

Morgan Collett (morgs)

Betavine Cape Town Developer Day 2010

I used to work for OLPC, whose mission is to distribute low cost laptops for education, without necessarily the connectivity with the outside world. Now at Praekelt we’re focusing on using connectivity as the power – harnessing the deployed base of mobile phones in Africa without requiring them to be smartphones or computing devices. As part of this Praekelt Foundation and Vodacom are hosting the Betavine Social Exchange Cape Town Developer Day 2010.

I’ve asked Steve Wolak to tell us more about Betavine and the event.

Who is Steve Wolak?

Stephen Wolak, Founder and Head of Betavine, has worked in mobile technology and software since graduating from Imperial College, London. Stephen joined Vodafone Group R&D in 2000 and in 2006 put forward the idea of an open platform for engaging the wider technology community with R&D activities.  The rest, as they say, is history.

MC: I first became aware of Betavine when looking for 3G drivers for Linux, but I’m sure there is more to it than that. What is Betavine and how did it start?

SW: Betavine was launched in January 2007 and has been evolving ever since, with new features being added in response to new requirements and feedback from the user base.  One area of success has been the linux software for the Vodafone Mobile Connect (VMC) card which has been downloaded over 750,000 times and has created a lively community around it.

We have also run a number of successful competitions on the website and created a lively Widget Zone. The website continues to evolve and we try out new things.

MC: What is the Betavine Social Exchange?

SW: This is our latest idea.. creating “social and sustainable” mobile solutions.  The Betavine Social Exchange seeks to bring together 3 communities; the mobile community, the social sector and the entrepreuners.  Together these communities can create mobile solutions for the social sector.  Community organisations create challenges on the website and mobile developers / social entrepreneurs create solutions. The website then supports the deployment of these solutions on the ground.

MC: The BSX’s success certainly depends on connecting the right people: those with needs – the NGOs and community organisations – and the developers. How do you publicise the BSX to reach them?

SW: We are running our pilot in South Africa and so we are working with Sangonet to help us get in touch with South African NGOs.  We are running a developer day in Cape Town to help us engage with the local developer community.

MC: What do the resulting solutions include – are they apps for mobile phones, mobi websites, SMS solutions or all of the above?

SW: All of the above.  It is important that the solution is appropriate for the challenge and the local community that will ultimately use the solution.

MC: What can developers expect from participating in the BSX?

SW: They can find real world challenges that people are seeking solutions to.  They can meet other developers and find useful resources to help them create a business.  The full resources section is coming soon.

MC: Which leads us on to the Developer Day being hosted in Cape Town next month. What’s going to be happening at the event, and how does it tie in with the BSX?

SW: We are keen to encourage mobile developers based in South Africa to engage with the real challenges that have been posted on the Betavine Social Exchange.  The developer day will include presentations on mobile technology and some exciting mobile solutions plus a developer competition and lots of creative energy and networking.

MC: You’re going to be speaking at the event. Who would you like to see there?

SW: I would like to see mobile developers plus those with design skills and a passion for using mobile technology for social change.

MC: We’re having a developer competition on the day. Can you tell us anything about the prizes/incentives you’re planning to bring?

SW: Well, the developer day is free and includes lots of food and drink plus some beer at the end of the day … :-)  We also intend to offer a few prizes for the competition winners .. But we have not decided exactly what yet.  You will have to come along and see but tickets are going fast!

Developer Day details

Date: Wednesday, March 10, 2010 from 9:30 AM – 7:00 PM

Location: The Lightbox Daylight Studio, Green Point, Cape Town

More information and free tickets are available at eventbrite. Due to the demand, the event has been expanded to 70 people.


by Morgan at 05 February 2010 12:13 PM

13 May 2009

Morgan Collett (morgs)

Ubuntu Server: Versioning /etc with etckeeper rocks!

Deploying a new server at work – a dedicated server hosted at Hetzner. Fortunately Jaunty (Ubuntu 9.04) was released before we had anything hosted on the machine, so I took the decision to upgrade it before we do serious deployment.

One of the shiny new features of Ubuntu Server 9.04 is etckeeper, documented here by Thierry Carrez. In particular, on 9.04 etckeeper plays well with bzr and shows the real user who typed “sudo etckeeper commit” in the bzr log, not just “root”.

As we have a (small but distributed) team adminning the server, this will help a great deal to keep track of who did what when.


by Morgan at 13 May 2009 07:43 PM

23 April 2009

Morgan Collett (morgs)

Surviving an Ubuntu Release Day

Some observations on the last n releases:

Throughout the Ubuntu development cycle, there are daily “snapshot” CD images produced. If you’re fortunate to live in a country where most of the “broadband” online population are not capped at 1GB per month (and a presidential hopeful who doesn’t keep singing “bring me my machine gun“) then you can download these during the development cycle to boot (daily-live) or install (perhaps in a virtual machine) to check on the progress or help with testing. These culminate in the actual “gold” release image.

Therefore, if you have one of these images from near the end of the development cycle, such as the release candidate, you can rsync to the latest image available on release day, and that will download the differences between the iso you have, and the final daily image – which will be identical to the release image, even though the daily image will be named something like jaunty-desktop-i386.iso and the corresponding release image named ubuntu-9.04-desktop-i386.iso. Rename it, and you’re done!

(Check the MD5SUMS after the release is announced, to be 100% sure you have it. There is always a small chance of a change to the ISOs on release day if some major “ate all my data” bug is found – so if you do have problems, remember that it comes with no warranty…)

Now, for kicks, go and lurk on IRC in #ubuntu-release-party and watch the masses rocking up to ask “Is it out yet?” Note Alan Pope’s list of Things Not To Say, and don’t go gloating that you have it already – you’ll only be kicked from the channel by the ironically named partybot.

Instead, burn write it to a USB stick (CDs are so early 2008) and get installing!


by Morgan at 23 April 2009 06:46 PM

06 October 2008

Stefano Rivera (tumbleweed)

The joy that is SysRq

I’m constantly surprised when I come across long-time Linux users who don’t know about SysRq. The Linux Magic System Request Key Hacks are a magic set of commands that you can get the Linux kernel to follow no matter what’s going on (unless it has panicked or totally deadlocked).

Why is this useful? Well, there are many situations where you can’t shut a system down properly, but you need to reboot. Examples:

  • You’ve had a kernel OOPS, which is not quite a panic but there could be memory corruption in the kernel, things are getting pretty weird, and quite honestly you don’t want to be running in that condition for any longer than necessary.
  • You have reason to believe it won’t be able to shut down properly.
  • Your system is almost-locked-up (i.e. the above point)
  • Your UPS has about 10 seconds worth of power left
  • Something is on fire (lp0 possibly?)
  • …Insert other esoteric failure modes here…

In any of those situations, grab a console keyboard, and type Alt+SysRq+s (sync), Alt+SysRq+u (unmount), wait for it to have synced, and finally Alt+SysRq+b (reboot NOW!). If you don’t have a handy keyboard attached to said machine, or are on another continent, you can

# echo u > /proc/sysrq-trigger

In my books, the useful SysRq commands are:

b
Reboot
f
Call the oom_killer
h
Display SysRq help
l
Print a kernel stacktrace
o
Power Off
r
Set your keyboard to RAW mode (required after some X breakages)
s
Sync all filesystems
u
Remount all filesystems read-only
0-9
Change console logging level

In fact, read the rest of the SysRq documentation, print it out, and tape it above your bed. Next time you reach for the reset switch on a Linux box, stop your self, type the S,U,B sequence, and watch your system come up as if nothing untoward has happened.

Update: I previously recommended U,S,B but after a bit of digging, I think S,U,B may be correct.

by tumbleweed at 06 October 2008 11:31 AM