Clug Park-Tech

19 May 2018

Tristan Seligmann (mithrandi)

Build artifacts with pex

For Continuous Integration in my Python application (as opposed to library) projects, I generally want to run my tests as well as build a Docker image, making use of the same artifacts and environment for both: I want to test what I'm actually deploying, rather than something merely similar. Previously this meant doing pip install twice; once into a virtualenv to run the tests, then again in the Docker image build. Sharing the pip cache between these steps speeds things up a lot, but this still ends up using quite a bit of time on network roundtrips etc.

Now that pex works with pypy, I have developed a slightly better workflow for this. Briefly speaking, Pex is a tool for assembling a Python application into a single runnable file that embeds the dependencies of the application; at runtime, the dependencies are ziploaded or extracted to a temporary location as necessary to run the application.

The workflow:

  1. Build a pex file.
  2. Run the tests against the pex file.
  3. Copy the pex into the Docker image.

This is similar to what I would do with a language like Go or Haskell that would produce a single executable as part of the build process.

by mithrandi at 19 May 2018 08:01 AM

19 January 2018

Tristan Seligmann (mithrandi)

Adventures in deployment, redux

Just a brief update: I've moved on from the hackish deployment strategy described in an older post; we now have Drone for CI building Docker images, pushing them to a registry, and deploying them on Rancher. You can see an example of the Drone pipeline to do this in our GitHub repos.

This is all I'm going to say on the topic, because that's all there really is to it. If you're using Kubernetes, I strongly suggest you look into Weave Flux for deployment; this is eventually where I want to end up, but migrating to Kubernetes is hard for us at the moment.

by mithrandi at 19 January 2018 09:52 PM

10 November 2016

Tristan Seligmann (mithrandi)

Running away again

[I originally posted this on Facebook; I've disabled comments here because I don't think I can cope with them over here on my blog, but if you really need to comment you can find me on Facebook or elsewhere]

I've run away several times in my life.

The very first time was as a teenager; I don't even recall exactly what it was that triggered it, I think it was anxiety over a compulsory school camp that I didn't want to attend. I didn't have a plan, I didn't even have any real intention to actually run away for real, I just felt like I was trapped and needed to escape / lash out somehow. "Running away" amounted to declaring my intent, leaving the house, hiding somewhere nearby while everyone frantically searched for me, and eventually returning a few hours later once I had calmed down a bit. Of course, at the time, I wasn't really capable of fully understanding the situation; I didn't understand the destructive mental paths that lead to being unable to cope, I didn't really care how much emotional stress I was causing to others, and I didn't realise that had I been able to communicate better about how I was feeling, I could probably have dealt with things much better. In short: I was wrong to run away this time. Then again, I was just a teenager, so I think I can be forgiven for failing at those things.

Later on in life, running away took on a more sophisticated form. Once I gained sufficient control over my life (both in the legal and the financial sense), I took steps to cut certain people out of it. I have few regrets about the process (which was staggered, not something I did all at once); some relationships and interactions are simply not healthy or beneficial for the parties involved, and not having to deal with the constant stress of those interactions definitely improved my mental health and my life in general by quite a bit. In short: I think I was right to run away this time. But in the end, it was still running away; I couldn't cope with the situation, so I chose not to cope with it.

And now…recent events surrounding the 2016 US elections are slowly producing that rising sense of panic and inability to cope again. The elections themselves are not the trigger; rather, it is the conversations they have sparked which has lead me to realise certain things about relatives, acquaintainces, and friends. As most of you probably know, I'm no stranger to intellectual debate / conflict; this isn't about that. I can quite happily handle interactions with someone I disagree with on politics, economics, or what have you; most of the time, I'm happy to discuss / argue the matter for hours, unpacking our disagreement into differences in assumptions vs. differences in reasoning, and ending with a better understanding of both sides even if my mind remains unchanged.

But this is different: this is a conflict over whether or not we should care about other people. Caring about other people is something I learned as an incontrovertible truth from childhood, not a selective privilege to be extended to only some and not others. I now realise that many around me do not feel the same way; they feel that some people are not deserving of their care, for whatever reason, and…I don't know how to deal with this.

I feel like running away this time would be a mistake, that I need to stay and engage with these people, attempt to change their minds; but I have no idea how to do this, and I fear the poisoning of either my own mind, or my relationships, as a result of this. I fear that to maintain these relationships as-is, with no condemnation, will result in sending the message that I support something I do not. I fear that such condemnation would inevitably lead to the destruction of those relationships anyway, accomplishing nothing good in the end. I fear that by running away, I am losing the opportunity to do something about all of this. I feel guilty because of my fear; there are others who do not even have the option of running away, others who are the direct target of the uncaring and the hatred, rather than mere bystanders who can leave at any other time. How much worse must their fear be than mine?

And so here I am, trapped in a seemingly endless mental loop, afraid to run and afraid not to run.

by mithrandi at 10 November 2016 11:12 PM

06 January 2016

Michael Gorven (cocooncrash)

Memory optimised decompressor for Pebble Classic

TLDR: DEFLATE decompressor in 3K of RAM

For a Pebble app I've been writing, I need to send images from the phone to the watch and cache them in persistent storage on the watch. Since the persistent storage is very limited (and the Bluetooth connection is relatively slow) I need these to be as small as possible, and so my original plan was to use the PNG format and gbitmap_create_from_png_data(). However, I discovered that this function is not supported on the earlier firmware used by the Pebble Classic. Since PNGs are essentially DEFLATE compressed bitmaps, my next approach was to manually compress the bitmap data. This meant that I needed a decompressor implementation ("inflater") on the watch.

The constraint

The major constraint for Pebble watchapps is memory. On Pebble Classic apps have 24K of RAM available for the compiled code (.text), global and static variables (.data and .bss) and heap (malloc()). There is an additional 2K for the stack (local variables). The decompressor implementation needed to have both small code size and variable usage. I discovered tinf which seemed to fit the bill, and tried to get it working.

Initially, trying to decompress something simply crashed the app. It took some debug prints to determine that code in tinf_uncompress() wasn't even being executed, and I realised that it was exceeding the 2K stack limit. I changed the TINF_DATA struct to be allocated on the heap to get past this. At this stage it was using 1.2K of .text, 1.4K of .bss, 1K of stack, and 1.2K of heap (total 4.8K). I set about optimising the implementation for memory usage.

Huffman trees

Huffman coding is a method to represent frequently used symbols with fewer bits. It uses a tree (otherwise referred to as a dictionary) to convert symbols to bits and vice versa. DEFLATE can use Huffman coding in two modes: dynamic and fixed. In dynamic mode, the compressor constructs an optimal tree based on the data being compressed. This results in the smallest representation of the actual input data; however, it has to include the computed tree in the output in order for a decompressor to know how to decode the data. In some cases the space used to serialise the tree negates the improvement in the input representation. In this case the compressor can used fixed mode, where it uses a static tree defined by the DEFLATE spec. Since the decompressor knows what this static tree is, it doesn't need to be serialised in the output.

The original tinf implementation builds this fixed tree in tinf_init() and caches it in global variables. Whenever it encounters a block using the fixed tree it has the tree immediately available. This makes sense when you have memory to spare, but in this case we can make another tradeoff. Instead we can store the fixed tree in the same space used for the dynamic tree, and rebuild it every time it is needed. This saves 1.2K of .bss at the expense of some additional CPU usage.

The dynamic trees are themselves serialised using Huffman encoding (yo dawg). tinf_decode_trees() needs to first build the code tree used to deserialise the dynamic tree, which the original implementation loads into a local variable on the stack. There is an intermediate step between the code tree and dynamic tree however (the bit length array), and so we can borrow the space for the dynamic instead of using a new local variable. This saves 0.6K of stack.

The result

With the stack saving I was able to move the heap allocation back to the stack. (Since the stack memory can't be used for anything else it's kind of free because it allows the non-stack memory to be used for something else.) The end result is 1.2K of .text, 0.2K of .bss and 1.6K of stack (total 3.0K), with only 1.4K counting against the 24K limit. That stack usage is pretty tight though (trying to use app_log() inside tinf causes a crash) and is going to depend on the caller using limited stack. My modified implementation will allocate 1.2K on the heap by default, unless you define TINF_NO_MALLOC. Using zlib or gzip adds 0.4K of .text. You can find the code on bitbucket.

by mgorven at 06 January 2016 06:28 AM

22 August 2014

Adrianna Pińska (Confluence)

Yet another way to play videos in an external player from Firefox

I spent some time today trying to figure out how to get Firefox to play embedded videos using anything other than Flash, which is an awful, stuttering, resource-devouring mess. The internet is littered with old browser extensions and user scripts which allegedly make this possible (e.g. by forcing sites like YouTube to use the vlc media plugin instead), but I was unable to get any of them to work correctly.

Here’s a quick hack for playing videos from YouTube and any other site that youtube-dl can handle in an external mplayer window. It’s based on several existing HOWTOs, and has the benefit of utilising a browser extension which isn’t dead yet, Open With, which was designed for opening websites in another browser from inside Firefox.

I wrote a little script which uses youtube-dl to download the video and write it to stdout, and pipes this output to mplayer, which plays it. Open With is the glue which sends URLs to the script. You can configure the extension to add this option to various context menus in the browser — for example, I can see it if I right-click on an URL or on an open page. You may find this less convenient than overriding the behaviour of the embedded video on the page, but I prefer to play my videos full-screen anyway.

This is the script:

#!/bin/sh
youtube-dl -o - $1 | mplayer -

Make it executable. Now open Open With’s preferences, add this executable, and give it a nicer name if you like. Enjoy your stutter-free videos. :)

(Obviously you need to have youtube-dl and mplayer installed in order for this to work. You can adapt this to other media players — just check what you need to do to get them to read from stdin.)

by confluence at 22 August 2014 02:41 PM

17 April 2013

Raoul Snyman (superfly)

Nomanini at ScaleConf 2013

For those who didn't know, I started working for a company called Nomanini at the beginning of last year. Nomanini manufactures an airtime point of sale terminal for use in the informal sector (taxis, spaza shops, etc).

I'm excited to say that my boss will be speaking at ScaleConf 2013, which kicks off tomorrow, and myself and my colleagues are going to be there too. See you there!

by raoul at 17 April 2013 02:02 PM

23 February 2013

Raoul Snyman (superfly)

How NOT to conduct yourself online

Many of you know that I am the project leader of an open source worship presentation program called OpenLP. Yesterday evening, as I was doing something on OpenLP's project page on SourceForge.net (a popular repository for open source projects), I glanced at the reviews page and the one below caught my eye.

Review

Now, I don't normally worry about negative reviews (they lend your project a certain amount of authenticity), I couldn't help but notice the reference AND link to a commercial product. I then simply did a search for the reviewer on Google and one of the first few hits was the reviewer's blog. I went to the About page, and sure enough, not only was it the same person, I found out some interesting facts about him.

Bio

Establishing that the blog did indeed belong to the reviewer was easy. Firstly, the reviewer didn't even bother to use a different photo for his profile on SF.net, and secondly his domain name, Twitter account and SF.net username were all identical. Lastly, the product he was promoting over OpenLP was the very same product he is the executive director (or somesuch inflated title) of.

I just couldn't believe my eyes. Here is what I would presume is an upstanding Christian man, and he directly slates a competitor's product without so much as disclosing who he is. I don't go around telling everyone how awful EasyWorship, ProPresenter and MediaShout are. It just is not something you do. If your product is better than the others then let it speak for itself.

This reminds me of Microsoft a few years ago who didn't know how to handle open source software, so they paid people to conduct absurd studies to prove how great they were.

Seriously, are you so scared of OpenLP that you have to pretend to be an unhappy user? I highly doubt you've ever even downloaded OpenLP, nevermind actually used it.

Incidentally, I marked his comment as spam, so it is no longer visible.

by raoul at 23 February 2013 06:35 AM

14 January 2013

Raoul Snyman (superfly)

RE: Ubuntu Loco Games 2013

LoCo Games

The newly-approved Ubuntu Mexico LoCo has decided to celebrate their status as an official LoCo by hosting an online gaming event. They will be hosting matches for AssaultCube, Urban Terror and Battle for Wesnoth on the 9th of Feb from 17:00 SAST (9:00 CST) to midnight (16:00 CST) on the 10th of Feb. They have 2 LoCo's signed up so far, so why don't you get together and add yours!

For more information, check out their blog post or the event on the LoCo site.

by raoul at 14 January 2013 09:56 AM

30 October 2012

Michael Gorven (cocooncrash)

XBMC packages for Raspberry Pi running Raspbian

I recently got a Raspberry Pi, which is an ARM based device which runs Linux. My goal is to use this as my HTPC running XBMC, so that I can move the fileserver out the lounge.

Edit: I've moved the latest information and updates about these packages to this page.

Building

I installed Raspbian on my RPi, which is basically a rebuild of Debian Wheezy specifically for the RPi (targeting the ARMv6 architecture with hard float). I found various instructions on how to build XBMC for Raspbian, but none of them were in the form of deb packages, and installing software without packages just makes me queezy. So I went off and built it myself.

Since the RPi is relatively low powered, I built the package on my laptop using qemu-user, which emulates binaries with a different architecture. I based the packaging on the XBMC package in Wheezy, and the source is from the xbmc-rbp branch on GitHub. I made the modifications to the source as per this forum post and added an initscript so that it can automatically start at bootup.

Installing

The easiest way to install the package is to add my archive to your system. To do this, store the following in /etc/apt/sources.list.d/mene.list:

deb http://archive.mene.za.net/raspbian wheezy contrib

and then import the archive signing key:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-key 5243CDED

You can then install it as you would with any other package, for example, with apt-get:

sudo apt-get install xbmc

(If you don't want to configure my archive you can download the packages manually, but you'll have to deal with all the dependencies. Note that it requires a newer libcec package which is also in my archive.)

The user which you're going to run XBMC as needs to be a member of the following groups:

audio video input dialout plugdev

Running

To run XBMC, run xbmc-standalone from a VT (i.e. not under X). XBMC accesses the display directly and not via Xorg.

If you want XBMC to automatically start when the system boots, edit /etc/default/xbmc and change ENABLED to 1:

ENABLED=1

You also need to set the user which XBMC should run as (the xbmc user is not automatically created at the moment). Run sudo service xbmc start to test this.

Configuration

The following settings in advancedsettings.xml decreases the CPU usage while showing the UI. Disabling the RSS feeds also helps with this.

<advancedsettings>
    <gui>
        <algorithmdirtyregions>3</algorithmdirtyregions>
        <nofliptimeout>0</nofliptimeout>
    </gui>
</advancedsettings>

Rebuilding

If you want to rebuild this package with a different source (e.g. a later Git revision), you need to prepare the source by running ./bootstrap before creating the orig.tar.gz. You obviously need all the build dependencies (which should be listed in the packaging), as well as the VideoCoreIV libraries in /opt. You probably want to set DEB_BUILD_OPTIONS=nocheck parallel=4 to disable the tests (they failed to build for me), and speed up the build by using more than one core. You can find the packaging in my archive or my Bazaar repository.

by mgorven at 30 October 2012 10:35 PM

Building Raspbian images for Raspberry Pi

I recently bought a Raspberry Pi, which is a credit card sized computer with an ARM processor. I'm using it as my TV frontend, running Raspbian and XBMC. I'm building my own packages for XBMC since it requires the latest development version.

I initially installed my Pi with the foundation image, but found that it included a lot of packages which I didn't need. Since I have a slight obsession about doing things as efficiently as possible, I decided to build my own image with XBMC from scratch.

I implemented a script in Bash, mkraspbianxbmc.sh which does this. It uses debootstrap to install a minimal Raspbian system in a chroot. It then installs XBMC and a couple extra packages, and does some necessary configuration. Finally it creates an image file with the necessary partitions, creates the filesystems, and copies the installation into the image file. The resultant image fits onto a 1GiB SD card. You can download a pre-built image from this page.

The script can be modified to build images with different packages, or even a very minimal image which fits onto a 512MiB SD card.

by mgorven at 30 October 2012 10:34 PM

30 August 2012

Adrianna Pińska (Confluence)

Finding articles on Twitter

I have strong opinions about retweeting, which seem to be the opposite of many other people’s opinions about retweeting. I like new-style retweets. If I retweet someone, I want it to be clear that they are the original source of the tweet. If someone else tweets something terribly insightful, I don’t want to see fifty old-style retweets of it. Finally, I would like “RT @joe RT @bob OMG, totally RT @alice I agree! RT @cheesemonger OMG you guys I loled!!1 RT @somedude this is lank insightful RT @dave The” to die in a fire.

Which it mostly has. When new-style retweeting was first introduced there was a lot of wailing and gnashing of teeth as some people were outraged that they were seeing tweets from people they weren’t directly following in their timeline. Today, nobody seems to care anymore, and I seldom see a manual old-style retweet. Unfortunately, I see a similar problem with the automated sharing of articles.

As far as I’m concerned, the best way to share an article on site Foo on Twitter is this: 1) go to Foo’s Twitter stream, 2) find the tweet which links to the article, 3) retweet it. Unfortunately, what most “share this article on Twitter” buttons that sites display (and encourage you to use) actually do is make you create an old-style retweet. It usually automatically includes the site’s Twitter username, and of course it’s a link to the site, so credit is not really a problem — but duplication still is, and in my OCD opinion this form of sharing is far less elegant than retweeting the original, official tweet.

I made a bookmarklet which searches Twitter for the title of the page you’re currently reading. It tries to autodetect and chop off the site name if it exists, and it loads the results in a new window (or tab, depending on your browser preferences). It currently doesn’t limit the search to the site’s Twitter account, because determining what that is is less trivial than extracting the title from the page. I may write a more complicated bookmarklet when I have more time. In the meantime, if you’re as obsessive about Twitter etiquette as I am, enjoy:

Find on Twitter

by confluence at 30 August 2012 01:27 PM

13 August 2012

Adrianna Pińska (Confluence)

How to hack GTK’s “Recently Used” virtual folder

Lots of people hate GTK’s new file chooser behaviour, which sets the location to a virtual folder with recently used files unless the app overrides it to something more useful. Some apps don’t. It is currently not possible to change this default.

Some people hate this behaviour because they don’t want their gran to see all their recently viewed donkey porn when saving a spreadsheet. If I let my gran use my computer I would give her a guest account, so I just hate it because it punches my workflow in the stomach, and stabs it repeatedly in the face while it’s down.

The privacy advocates can solve their problem by clearing the file in which recently used files are recorded and making it read-only. That doesn’t help me have something that isn’t utterly useless as the first thing I see in a file chooser. I looked more closely at the file to see how easy it would be to hard-code it to something useful, like a list of commonly used directories.

The file is ~/.local/share/recently-used.xbel. It’s XML, so it’s fugly, but reasonably editable. A single bookmark entry looks like this:

<bookmark href="file:///home/username/path/to/some.file.html" added="2012-08-13T23:07:20Z" modified="2012-08-13T23:07:20Z" visited="2012-08-13T23:07:20.868364Z">
    <info>
        <metadata owner="http://freedesktop.org">
            <mime:mime-type type="text/html"/>
            <bookmark:applications>
                <bookmark:application name="Firefox" exec="&apos;firefox %u&apos;" modified="2012-08-13T23:07:20Z" count="1"/>
            </bookmark:applications>
        </metadata>
    </info>
</bookmark>

So if you change the path to a directory you want to see in the list, and set the appropriate mimetype, everything should Just Work, right?

Well, yes, actually. It is that simple. Today I learned that directories have a mimetype: inode/directory. I haven’t checked what happens if you set the incorrect mimetype; if you do and it’s something hilarious, let me know in the comments.

If you opened the file in multiple applications, you’ll see multiple applications listed in the application section. This information does not seem to affect which files appear in which apps — the same list is used everywhere. You can remove the whole section, and the file chooser doesn’t seem to care.

So I added some directories I use frequently, removed all the other bookmarks, and made the file read-only. Now I have a sane default. I was considering writing a script to make editing the file more pleasant, but most GTK apps I use sensibly override the default to the most recently used directory, so I don’t really need this solution to be that advanced.

(P.S. Please don’t comment just to tell me to “use a different window manager”, which I’ve already been doing for years. This is a toolkit issue, and I’m not going to stop using GTK apps because the file chooser is doing something stupid.)

by confluence at 13 August 2012 11:33 PM

05 December 2010

Jonathan Groll (eyesonly)

Add subtitles to a DVD


Many people want to add subtitles to a DVD in their own language, and often DVD disks don’t come with subtitles in a language you want. For instance, if you’re staying in a foreign country and want English subtitles added to your rental movie. Or in my case, the library at the local Alliance Française has a lot of interesting movies completely in French, without English subtitles, and my french isn’t yet good enough to fully understand everything that is being said.

What is interesting is that there are sites on the internet that offer downloadable subtitle files for many movies. Simply type the name of your movie, plus the word “subtitle” into google and provided the movie is popular or well known enough you should be able to find a site offering the subtitle file for download. Mostly it seems that these subtitles were extracted from a DVD owned by someone else with the right languages. In other cases, it is purely fans of the movie who loved it so much and wanted to translate the words of the movie into a language that they know so that others can also enjoy the movie. I’m not so sure about the legality of the subtitles obtained in the first case, but certainly an argument may be made that since you own or have legally rented a copy of the movie it does seem to fall within “fair usage” to watch this movie with subtitles in a language you require.

So, you may be lucky. You may find the (fan-written) subtitle file that exactly matched your movie. By this I mean that the person who created your file had exactly the same version of the DVD, so that the lead in at the beginning of the movie is exactly the same length and in this case the subtitles remain perfectly in synch with the video.

However, don’t expect to watch the movie in your living room just yet. Most stand-alone DVD players that I’ve used do not have a facility to specify an external subtitles file, but almost all of the software based players on your computer do have this facility. So provided you got lucky, and got the correct subtitles file that matched your DVD, and are also content to watch movies in front of your computer that will be sufficient and you don’t need to read further.

Mostly though, the subtitles file doesn’t have the right timings for your DVD, or you might really really want to watch it on your TV in the lounge rather than with bowls of popcorn balanced on your lap in front of the computer. The guide that follows is for people who are running Linux, are comfortable with the command line, and who wish to hardcode 1 subtitles into a XVID file for watching on a (living-room) player that can play DIVX movies. If this is not exactly what you want there still may be something for you in this blog post. I’ll outline my six-step method:

  1. Rip (extract) contents of the DVD disk.
  2. Attempt to play DVD file with subtitle files on desktop.
  3. Create a XVID file without subtitles.
  4. Transform the subtitle file so that subtitles are synchronized with the video.
  5. Create a XVID file with subtitles hardcoded in.
  6. Break up this file into smaller segments for players that cannot handle large files.

I myself don’t follow the above recipe every time – depending on the circumstances some of these steps may not be needed. And you may very well want a different outcome – it may be enough for you just to align the subtitles with the video, or you may want more such as recreating a DVD with subtitles. You may even despise XVID files with harcoded subtitles in them!

If you are running Debian/Ubuntu you’ll need a lot of packages to be installed – at least: mplayer mencoder ogmtools libdvdcss dvdbackup gaupol ffmpeg gstreamer0.10-x plus dependancies and probably gnome-codec-install.

The principal tools involved are the wonderful mplayer/mencoder command line tools :- they give fine control, are scriptable, and are compatible with each other in terms of command line flags.

For step 1, extract the DVD to your hard disk:

dvdbackup -i /dev/dvd -M -o name_of_movie

This will create a folder called name_of_movie in the current directory. If your DVD-device is not /dev/dvd put the correct device name there.

In step 2, you’ll need to work out which of the titles on the DVD disk contains the movie, start by typing:

mplayer -v -dvd-device name_of_movie/VIDEO_TS/ dvd://1

Keep on increasing the number at the end – from 1 to 2 to 3 etc until you find the actual movie and not just the adverts and bonus features. In the examples that follow I refer to title 1, you will need to replace that with your correct title number.

The next thing to do is test the downloaded subtitle file:

mplayer -v -dvd-device name_of_movie/VIDEO_TS/ dvd://1 -sub downloaded_sub.srt

The subtitle file doesn’t have to end in .srt – mplayer supports multiple formats. Make a careful note if the subtitles are in synch, namely do the spoken words match the subtitles. Press the right arrow key on your keyboard to advance the video. Ensure that subtitles remain in synch until the end of the movie.

For step 3, create an initial XVID file without subtitles. Yes, I know this can be lossy. And it really is not a required step if your subtitles are already in synch, if they are you can move straight on to step 5. However, if you do need to adjust the subtitle file to match the video, then in order to do so it may be necessary to work with the entire movie as a single file, instead of multiple individual VOB files.

So, here is a simple two-pass recipe for encoding a movie to XVID without subtitles:

mencoder -idx -dvd-device name_of_movie/VIDEO_TS/ dvd://1 -ovc xvid -oac pcm -xvidencopts pass=1:bitrate=1200:aspect=16/9 -o /dev/null
mencoder -idx -dvd-device name_of_movie/VIDEO_TS/ dvd://1 -ovc xvid -oac pcm -xvidencopts pass=2:bitrate=1200:aspect=16/9 -o movie.avi

Note the bitrate in the above call (1200) can be adjusted as required (higher numbers mean higher quality but also larger files). You will obviously need to make sure this call has the correct path for your VIDEO_TS folder, and also for the title track from the DVD (replace the dvd://1 with the appropriate title). Experienced mencoder users will possibly also want to add extra mencoder flags. Transcoding to XVID can take some time, depending on the speed of your computer.

Next up is step 4 – adjusting the subtitle file if that is needed. To do this, I prefer using the gaupol application under Linux. So, open gaupol and within gaupol select your subtitle file, and at the bottom of the gaupol screen select your .avi video file that you produced in the previous step. The trick to adjusting is to note exactly the time at which the first useful spoken words occur for which you have a matching subtitle, and the same for the last spoken words (unfortunately, knowing the last spoken words may spoil the end of the movie for you!). So, if you click the green play icon in gaupol it will play the video file with a clock at the top of the screen which you can use for noting when the spoken words occur. Based on the above, in an example I worked out that someone said subtitle #2 (you should be able to recognise the words based on context, even if you don’t speak the language) at 00:02:54.000 and that the one of the last spoken texts occurred for subtitle #613 at 01:29:17.000. The important thing is to get the timing of the first subtitle as accurate as possible, for the last subtitle it doesn’t matter if you’re a second or two out. To do the actual synchronization choose the “Transform Positions” tool from the subtitle menu (see screenshot below), specify the times you observed and the positions of the other subtitles will be adjusted proportionally.

To test the subtitle file is now correct, play the middle of the movie and see if the subtitles are fully in sync there as well. Repeat, if necessary with other time points, and if necessary adjusting only part of the file (“Transform selected subtitles”) rather than the entire file (“Transform current project”).

Now, for step 5 we are going to hardcode the subtitles into a new XVID file which we produce. We can choose to either start with the source coming from the original DVD-rip folder produced by step one, or we can choose to work with the XVID file produced in step three as the source. Obviously, working with the XVID file will be lossy, as one can expect some picture degradation using the XVID file as a source file. However, if this file were to be used, you would know for sure that your subtitles will sync up with the file. Think back to the DVD extract step with dvdbackup – did it report any errors at all during the extract from the DVD? For instance, if there was a scratch on the DVD disk dvdbackup would still be able to work around it by block filling in the file produced, but I have observed such a “repaired error” might cause problems with the timing and sync of subtitles in the step to follow if the original DVD extract is used as the source. Generally, because of this I prefer to work with the XVID file as the source, even though it may be lossy.

Before starting, remove any file in the current folder called “divx2pass.log”, this file will have been produced if you did a two-pass encoding as above and will interfere with the two-pass encode we are about to do.

To hardcode in the subtitles, using the divx file produced in step three as the source, here are the commands that will perform the two-pass encode for you:

mencoder -idx movie.avi -ovc xvid -sub subtitles_adjusted.srt -subpos 96 -oac pcm -xvidencopts pass=1:bitrate=1200:aspect=16/9 -o /dev/null
mencoder -idx movie.avi -ovc xvid -sub subtitles_adjusted.srt -subpos 96 -oac pcm -xvidencopts pass=2:bitrate=1200:aspect=16/9 -o hardcoded.avi

Now, if you wanted to work with the original DVD-rip folder as the source, rather than perform the lossy re-encoding of the DIVX file, then replace “movie.avi” in the above with “-dvd-device name_of_movie/VIDEO_TS/ dvd://1” adjusted where appropriate.

Also, note in the above that “subtitles_adjusted.srt” refers to your subtitle file that you produced with gaupol, and also note that I prefer specifying that my subtitles appear at position 96 (you can leave this out if it doesn’t bother you).

And also note that there is another way to hardcode in subtitles but this way can result in enormous files.

Finally, in step 6 I break up the resultant DIVX file (which would be called “harcoded.avi” if you followed step 5 exactly). Why do I do this? For two possible reasons:

  1. If you are playing off a VFAT formatted memory stick files cannot be larger than 2GB in size.
  2. My player can play DIVX files, but once the file exceeds 1GB in size the playback simply stops.

Now, there are two possible workarounds. You could have simply lowered the bitrate so that the final XVID file was less than 1GB in size. Or you could chop up a big file as I do below. Of course, if your player can play big DIVX files and you’re not working with a VFAT memory stick then there is no need to perform this step.

In this example I am working with a 3.1G file. Firstly, to determine how long the video is, I issue:

ffmpeg -i hardcoded.avi

In my case, it tells me that my video has

Duration: 02:05:16.88.

So, I’d like to split that into four parts, so that each part will come to less than 1GB in size (roughly) – so after some quick mental arithmetic I think I’d like each piece to be 32 minutes long.

Based on the above example, these are the four mencoder commands to splice the video into four 32 minute chunks (the last chunk will be less than 32 minutes):

mencoder -ovc copy -oac copy -endpos 0:32:00 -o hardcoded1.avi hardcoded.avi
mencoder -ovc copy -oac copy -ss 0:32:00 -endpos 0:32:00 -o hardcoded2.avi hardcoded.avi
mencoder -ovc copy -oac copy -ss 1:04:00 -endpos 0:32:00 -o hardcoded3.avi hardcoded.avi
mencoder -ovc copy -oac copy -ss 1:36:00  -o hardcoded4.avi hardcoded.avi

Granted, this is an extreme example (normally you don’t let the XVID file get so big!) but it does show how to splice up a video using the -ss and -endpos flags. Unlike the transcoding to XVID steps, this step executes very quickly as all it does is copy the video and audio streams.

And voilà, copy the split files to your memory stick, and take it to the lounge to enjoy with your popcorn, knowing that you’ve managed to render a previously unwatchable video into something you’ll enjoy, and somehow all this extra work makes the video more enjoyable too.

1 Hardcoding means the actual video file is altered so that the subtitle is in the video stream. It’s a little nasty, and I agree softcoding (keeping the subtitles separate from the video stream) is better. The only problem is that my player refuses to accept files where the subtitles have been muxed in. If you’re curious about softcoding, I can recommend AVI-Mux GUI which does run under wine.

05 December 2010 03:29 PM

03 November 2010

Jonathan Groll (eyesonly)

Tunnel through ISA proxy

Image of tunnel attributed to Richard Freeman Geeks often need to access their *nix computers from work. Doesn’t everyone want to do that? True geeks control their computers strictly using the command-line, of course, and the tool that is used to control a remote command-line session is ssh.

What one usually does is use a tool like corkscrew to send ssh traffic through an HTTP proxy.

At one place of employment, a known trick of using corkscrew to tunnel out using the work proxy failed, with this message:

Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy service is denied. )

I tried all combinations of DOMAIN\USERNAME:PASSWORD in my corkscrew auth file but nothing worked.

If you see this message have no fear! What you need is a utility that can negotiate NTLM authorization with the proxy.

There are several open source tools that can do NTLM, of these I chose cntlm. Often ntlmaps is recommended as a utility to negotiate the authorization. The cntlm man page indicates that cntlm is far more efficient than ntlmaps, both in terms of memory and CPU usage.

One oddity about cntlm relative to other software that you may have worked with is that configuration is a two-step procedure: firstly you configure the software with a default config file, like the following (the settings that need to be configured for now in the first-step are in the first paragraph: username, domain, password, proxy, proxy-port):

#
# Cntlm Authentication Proxy Configuration
#
# NOTE: all values are parsed literally, do NOT escape spaces,
# do not quote. Use 0600 perms if you use plaintext password.
#

Username __username__ Domain __domain__ Password __password__ # Use hashes instead (-H) #Workstation netbios_hostname # Should be auto-guessed
Proxy __PROXY__:__PROXY_PORT__
# # This is the port number where Cntlm will listen # Listen 3128
# # If you wish to use the SOCKS5 proxy feature as well, uncomment # the following option, SOCKS5. It can be used several times # to have SOCKS5 on more than one port or on different network # interfaces (specify explicit source address for that). # # WARNING: The service accepts all requests, unless you use # SOCKS5User and make authentication mandatory. SOCKS5User # can be used repeatedly for a whole bunch of individual accounts. # #SOCKS5Proxy 8010 #SOCKS5User dave:password
# # Use -M first to detect the best NTLM settings for your proxy. # Default is to use the only secure hash, NTLMv2, but it is not # as available as the older stuff. # # This example is the most universal setup known to man, but it # uses the weakest hash ever. I won't have it's usage on my # conscience. :) Really, try -M first. # #
Auth LM #Flags 0x06820000 # # Enable to allow access from other computers # #Gateway yes
# # Useful in Gateway mode to allow/restrict certain IPs # #Allow 127.0.0.1 #Deny 0/0
# # GFI WebMonitor-handling plugin parameters, disabled by default # #ISAScannerSize 1024 #ISAScannerAgent Wget/ #ISAScannerAgent APT-HTTP/ #ISAScannerAgent Yum/
# # Headers which should be replaced if present in the request # #Header User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)
# # Tunnels mapping local port to a machine behind the proxy # Tunnel 11443:__OUTSIDE_HOST.COM__:443

Then, run
cntlm -v -M http://google.com (or any other external site)

Cntlm will use this to assess the type of auth your proxy can handle. In my case I got back something like the following output:

Config profile  1/11... OK (HTTP code: 301)
----------------------------[ Profile  0 ]------
Auth            NTLMv2
PassNTLMv2      AE1234567890123234567890123456C4
------------------------------------------------

For the second configuration step, these two lines need to be pasted back into your configuration file replacing the line that said “Auth LM” (and you must do this for your own situation, you can’t reuse my lines).

Then, startup the cntlm daemon:


cntlm

Let’s test if it works. Note that in the above config file I have a tunnel defined (the last line of the config file). Now, in order to ssh to port 443 of host __outside_host.com__ which is outside the proxy, one can do so using:


ssh -p 11443 localhost

The above assumes that outside_host.com has an sshd listening on port 443.

Cntlm also works fine under windows. If you don’t have “administrator” authorisation under windows, you can still run the cntlm executable, but need to specify which cntlm.ini file to use, in other words, something like:


cntlm.exe -c cntlm.ini

Of course, if you’re running on windows you won’t have an ssh command line client, but putty can be used nicely for this purpose. Bear in mind that putty needs to be configured (for the default cntlm configuration above) to use an HTTP proxy on host “localhost” listening on port 3128.

03 November 2010 02:30 PM

29 October 2010

Jonathan Groll (eyesonly)

Split and encrypt files for google docs

Creative commons image of meat grinder from http://www.flickr.com/photos/amayu/3629064681/Since January 2010, Google docs has allowed you to store any type of file, even arbitrary binary files. However, there are a couple of gotchas: one cannot upload files greater than 1GB in size, and you may want to encrypt your files so that not just anyone can read them (for instance server backup files).

The two bash scripts below provide a solution for the above. I call them the ‘mince’ scripts ‘cos they slice and dice your files and hopefully you’ll get hamburgers back at the end of the day. These scripts depend on you having a fairly new version of bash on your unix-like system, the ‘split’ utility and gnupg (GPG) which is used for the encryption/decryption. If you’re unsure of GPG, a good getting started guide can be found here.

It must be said that google docs is (in my opinion) currently not the best way to store your files in the cloud. In fact, I wrote another blog post describing the “google storage” options in greater depth.

The encrypt&split script is mince.sh and it takes two parameters, the first one a directory or archive, the second the email address for an already imported public key:

#!/bin/bash
# gpg encrypt archive and split into chunks
# $1 specifies base directory or compressed archive to encrypt.
# $2 is the recipients public key, eg. 'myfriend@his.isp.net'
set -e

CHUNK_SIZE=1000000000 #1000000000==1GB (not 1GiB!) SCRATCH_DIR=~/scratch_space TAR_REGEX='\.tar'
usage() { echo "ERROR: " echo " $*" echo "USAGE: " echo " mince.sh [DIRECTORY|ARCHIVE] PUBLIC_KEY_NAME" echo "EXAMPLE: " echo " ./mince.sh directory myfriend@her.isp.net" echo "FURTHER COMMENTS: " echo " if an ARCHIVE is supplied instead of a directory, it must have a name like file.tar or file.tar.gz or file.tar.bz2 " }
#check parameters entered are valid [ $# -ne 2 ] && usage "Two parameters are required." && exit 1 if [ ! -d "$1" ] && [[ ! $1 =~ $TAR_REGEX ]]; then usage "$1 is not a directory or tar/tar.gz/tar.bz2 archive." exit 1 fi
#if 1st parameter is a directory, then tar it up in the scratch space if [ -d "$1" ]; then absolute="`cd $1; pwd` " mkdir -p $SCRATCH_DIR cd $SCRATCH_DIR nameonly=${absolute##*/} nameonly=${nameonly/ /} #remove trailing spaces tar -cf $nameonly.tar $absolute arch="${SCRATCH_DIR}/${nameonly}.tar" created=true echo "Created temporary archive $arch" else arch="`readlink -f $1`" created=false echo "Working with existing archive $arch" fi
#call for GPG encryption and compression of the archive target="${arch##*/}.gpg" name=${target%\.gpg} mkdir -p $SCRATCH_DIR cd $SCRATCH_DIR echo "Commencing GPG encryption, please be patient" gpg --bzip2-compress-level 6 --compress-algo bzip2 --output $target --encrypt --recipient $2 $arch
#split .gpg file into chunks of size CHUNK_SIZE outdir="${SCRATCH_DIR}/output" mkdir -p "$outdir" mkdir -p "$outdir/$name" cd $outdir/$name && rm -f $name* echo "Splitting files" split -b $CHUNK_SIZE "${SCRATCH_DIR}/$target" for x in * do mv $x "../${name}__$x" done
#clean up - remove .gpg and temporary archive and temporary directory cd $SCRATCH_DIR rmdir "$outdir/$name" rm $target if [ $created == true ]; then echo "Removing temporary archive $arch" rm $arch fi
echo "All file splits produced placed in $outdir"

Download link for mince.sh

The bash script to reconstitute the file is called unmince.sh and takes one parameter – the name of the first file downloaded from google docs:

#!/bin/bash
# reassemble an archive from chunks of a file that have been gpg-encrypted
# $1 specifies the first file produced from the mincing process, eg, file__xaa
set -e

SCRATCH_DIR=~/scratch_space REGEX='__xaa'
usage() { echo "ERROR: " echo " $*" echo "USAGE: " echo " unmince.sh file$REGEX" echo "WHERE: " echo " file$REGEX is the first file produced by the mince script" }
#check parameters [ $# -ne 1 ] && usage "Only one parameter is required" && exit 1 if [ -d "$1" ] || [[ ! $1 =~ '_xaa' ]]; then usage "$1 cannot be a directory and must end in _xaa." exit 1 fi
#combine all chunks of the file sourcepath="`readlink -f $1`" pathonly="`dirname $sourcepath`"
just=${sourcepath##*/} basenam=${just%$REGEX} indir="${SCRATCH_DIR}/reconstituted" mkdir -p $indir cd $indir [ -e $indir/combined.gpg ] && rm $indir/combined.gpg for x in $pathonly/$basenam* do cat $x >> combined.gpg done
#decrypt the .gpg file echo "Commencing GPG decryption, please be patient" gpg --output $basenam --decrypt combined.gpg
#tidy up - remove the gpg file rm combined.gpg
echo "The reconstituted archive $indir/$basenam was created"

Download link for unmince.sh

Since I might still tinker and improved these scripts, to get the newest version of these files take a look at my github repo at http://github.com/eyesonly/kenwood (Named after the Kenwood Chef, a famous mincer!)

29 October 2010 01:25 PM

05 February 2010

Morgan Collett (morgs)

Betavine Cape Town Developer Day 2010

I used to work for OLPC, whose mission is to distribute low cost laptops for education, without necessarily the connectivity with the outside world. Now at Praekelt we’re focusing on using connectivity as the power – harnessing the deployed base of mobile phones in Africa without requiring them to be smartphones or computing devices. As part of this Praekelt Foundation and Vodacom are hosting the Betavine Social Exchange Cape Town Developer Day 2010.

I’ve asked Steve Wolak to tell us more about Betavine and the event.

Who is Steve Wolak?

Stephen Wolak, Founder and Head of Betavine, has worked in mobile technology and software since graduating from Imperial College, London. Stephen joined Vodafone Group R&D in 2000 and in 2006 put forward the idea of an open platform for engaging the wider technology community with R&D activities.  The rest, as they say, is history.

MC: I first became aware of Betavine when looking for 3G drivers for Linux, but I’m sure there is more to it than that. What is Betavine and how did it start?

SW: Betavine was launched in January 2007 and has been evolving ever since, with new features being added in response to new requirements and feedback from the user base.  One area of success has been the linux software for the Vodafone Mobile Connect (VMC) card which has been downloaded over 750,000 times and has created a lively community around it.

We have also run a number of successful competitions on the website and created a lively Widget Zone. The website continues to evolve and we try out new things.

MC: What is the Betavine Social Exchange?

SW: This is our latest idea.. creating “social and sustainable” mobile solutions.  The Betavine Social Exchange seeks to bring together 3 communities; the mobile community, the social sector and the entrepreuners.  Together these communities can create mobile solutions for the social sector.  Community organisations create challenges on the website and mobile developers / social entrepreneurs create solutions. The website then supports the deployment of these solutions on the ground.

MC: The BSX’s success certainly depends on connecting the right people: those with needs – the NGOs and community organisations – and the developers. How do you publicise the BSX to reach them?

SW: We are running our pilot in South Africa and so we are working with Sangonet to help us get in touch with South African NGOs.  We are running a developer day in Cape Town to help us engage with the local developer community.

MC: What do the resulting solutions include – are they apps for mobile phones, mobi websites, SMS solutions or all of the above?

SW: All of the above.  It is important that the solution is appropriate for the challenge and the local community that will ultimately use the solution.

MC: What can developers expect from participating in the BSX?

SW: They can find real world challenges that people are seeking solutions to.  They can meet other developers and find useful resources to help them create a business.  The full resources section is coming soon.

MC: Which leads us on to the Developer Day being hosted in Cape Town next month. What’s going to be happening at the event, and how does it tie in with the BSX?

SW: We are keen to encourage mobile developers based in South Africa to engage with the real challenges that have been posted on the Betavine Social Exchange.  The developer day will include presentations on mobile technology and some exciting mobile solutions plus a developer competition and lots of creative energy and networking.

MC: You’re going to be speaking at the event. Who would you like to see there?

SW: I would like to see mobile developers plus those with design skills and a passion for using mobile technology for social change.

MC: We’re having a developer competition on the day. Can you tell us anything about the prizes/incentives you’re planning to bring?

SW: Well, the developer day is free and includes lots of food and drink plus some beer at the end of the day … :-)  We also intend to offer a few prizes for the competition winners .. But we have not decided exactly what yet.  You will have to come along and see but tickets are going fast!

Developer Day details

Date: Wednesday, March 10, 2010 from 9:30 AM – 7:00 PM

Location: The Lightbox Daylight Studio, Green Point, Cape Town

More information and free tickets are available at eventbrite. Due to the demand, the event has been expanded to 70 people.

by Morgan at 05 February 2010 12:13 PM

13 May 2009

Morgan Collett (morgs)

Ubuntu Server: Versioning /etc with etckeeper rocks!

Deploying a new server at work – a dedicated server hosted at Hetzner. Fortunately Jaunty (Ubuntu 9.04) was released before we had anything hosted on the machine, so I took the decision to upgrade it before we do serious deployment.

One of the shiny new features of Ubuntu Server 9.04 is etckeeper, documented here by Thierry Carrez. In particular, on 9.04 etckeeper plays well with bzr and shows the real user who typed “sudo etckeeper commit” in the bzr log, not just “root”.

As we have a (small but distributed) team adminning the server, this will help a great deal to keep track of who did what when.

by Morgan at 13 May 2009 07:43 PM

23 April 2009

Morgan Collett (morgs)

Surviving an Ubuntu Release Day

Some observations on the last n releases:

Throughout the Ubuntu development cycle, there are daily “snapshot” CD images produced. If you’re fortunate to live in a country where most of the “broadband” online population are not capped at 1GB per month (and a presidential hopeful who doesn’t keep singing “bring me my machine gun“) then you can download these during the development cycle to boot (daily-live) or install (perhaps in a virtual machine) to check on the progress or help with testing. These culminate in the actual “gold” release image.

Therefore, if you have one of these images from near the end of the development cycle, such as the release candidate, you can rsync to the latest image available on release day, and that will download the differences between the iso you have, and the final daily image – which will be identical to the release image, even though the daily image will be named something like jaunty-desktop-i386.iso and the corresponding release image named ubuntu-9.04-desktop-i386.iso. Rename it, and you’re done!

(Check the MD5SUMS after the release is announced, to be 100% sure you have it. There is always a small chance of a change to the ISOs on release day if some major “ate all my data” bug is found – so if you do have problems, remember that it comes with no warranty…)

Now, for kicks, go and lurk on IRC in #ubuntu-release-party and watch the masses rocking up to ask “Is it out yet?” Note Alan Pope’s list of Things Not To Say, and don’t go gloating that you have it already – you’ll only be kicked from the channel by the ironically named partybot.

Instead, burn write it to a USB stick (CDs are so early 2008) and get installing!

by Morgan at 23 April 2009 06:46 PM

06 October 2008

Stefano Rivera (tumbleweed)

The joy that is SysRq

I’m constantly surprised when I come across long-time Linux users who don’t know about SysRq. The Linux Magic System Request Key Hacks are a magic set of commands that you can get the Linux kernel to follow no matter what’s going on (unless it has panicked or totally deadlocked).

Why is this useful? Well, there are many situations where you can’t shut a system down properly, but you need to reboot. Examples:

  • You’ve had a kernel OOPS, which is not quite a panic but there could be memory corruption in the kernel, things are getting pretty weird, and quite honestly you don’t want to be running in that condition for any longer than necessary.
  • You have reason to believe it won’t be able to shut down properly.
  • Your system is almost-locked-up (i.e. the above point)
  • Your UPS has about 10 seconds worth of power left
  • Something is on fire (lp0 possibly?)
  • …Insert other esoteric failure modes here…

In any of those situations, grab a console keyboard, and type Alt+SysRq+s (sync), Alt+SysRq+u (unmount), wait for it to have synced, and finally Alt+SysRq+b (reboot NOW!). If you don’t have a handy keyboard attached to said machine, or are on another continent, you can

# echo u > /proc/sysrq-trigger

In my books, the useful SysRq commands are:

b
Reboot
f
Call the oom_killer
h
Display SysRq help
l
Print a kernel stacktrace
o
Power Off
r
Set your keyboard to RAW mode (required after some X breakages)
s
Sync all filesystems
u
Remount all filesystems read-only
0-9
Change console logging level

In fact, read the rest of the SysRq documentation, print it out, and tape it above your bed. Next time you reach for the reset switch on a Linux box, stop your self, type the S,U,B sequence, and watch your system come up as if nothing untoward has happened.

Update: I previously recommended U,S,B but after a bit of digging, I think S,U,B may be correct.

by tumbleweed at 06 October 2008 11:31 AM