Sometimes a little deadlining does miracles.

So here we go. I will be officially launching a private beta of Half Price Tuesdays on the 29th of September at the September Geekdinner, final details of which will be available shortly.

Unfortunately, if you’re not already on the list for the event you won’t be because it’s full, but I will open the beta to others shortly thereafter.

If you are coming to the dinner, please don’t expect fireworks or rocket-science; hpt was meant to be coded over a weekend, but I got carried away in the design phase.

As you might have read in my previous blog post, my computer died the other day, and I've been going through the process of geting new hardware.

On Wednesday evening I searched through the product listings on the sites of local computer dealers, and on Thursday I placed my order for a new motherboard, graphics card, case (with PSU) and processor (my existing hard drive was quite new, and the motherboard contains all the other things like LAN and sound card).

On Friday morning I was as excited as a child at Christmas. This was the day my hardware was to arrive. Which it did. That afternoon when I got home, I went about putting everything together.

After I had assembled the computer, hard drive plugged in, I booted up and waited in eager anticipation for my "new" computer to show it's face. I was also expecting to run into a few minor blips along the way.

Nothing happened.

Wait, let me rephrase that: Nothing bad happened.

My computer just booted up as per usual, as though it had always been using this new hardware. Sure, I'm running a 32-bit kernel on my new 64-bit processor, but that didn't bother Linux. It didn't bother linx that I had a new sound card and a new network card either. It just worked.

I was impressed. I mean, I had half expected that to happen (and I had sincerely hoped it would), but when it did happen I was still incredibly impressed. Hannah was also impressed that Amarok just continued from the where it had left off when my PC previously had shutdown, in the middle of a song.

So, as per my title (which I got from Rob Miller's Linux.com article), Linux is boring. No problems, no hitches, it just works.

I got home last night and tried to switch on my PC, and it didn't want to boot up. So I restarted it, and continued trying to get it to boot for an hour, but I eventually gave up. My PC has been giving me problems recently (not booting up), so this was not completely unexpected. However, I hadn't expected it to happen so soon.

I even tried to boot up with a LiveCD, to chech that it wasn't my hard drive that was the problem (which I doubted, since the hard drive itself is fairly new), but the LiveCD didn't work either. So, as far as I can see, it looks like my motherboard has packed up.

So today I'll be getting quotes for components, and I'll see how much the "damage" is. I'm hoping that I can actually order the parts today and collect them tomorrow.

5 years is not bad going for a PC.

*sigh* So that's the (bad) news for the day...

So, I have been on teh internets for a while, I have been a member of many mailinglists, chatted on irc, etc. etc. So maybe I have a more attuned sense of the social norms of the internet. Years ago a bunch of script kiddies had a mailing list called KiDDiESoc, KiDDiESoc was great, at it's peak it had traffic in the region of 100 messages per day. Between maybe 15 people. Many of these messages were what can only be called spam. Spam and baggy pantsing. We had arguments about top, bottom, and inline posting. We argued, we joked, but mostly, (in my opinion) it was a space to share ideas, and explore the social norms of electronically mediated discussion and socialisation. There were other good things, but for me, we all definitely learned how to be better internet citizens for it. Lessons like:

• Be as clear as possible in what you write (especially when arguing) because a lot of subtlety is lost when reading characters on a blinking terminal
• Be respectful of people's boundaries, because it is a lot easier to overstep the mark
• Try to remain calm and don't vent against someone personally if you are annoyed at their argument
• Try at all times to keep the signal to noise ratio as high as possible

There are many more that exist, are unspoken, or are jotted down in RFCs and random pages on the intertubes. What I am getting at is that, like my finely tuned sense of justice, I have a finely tuned sense of the social norms that dictate my interactions on the web. In a few days time the Digital Citizens Indaba is happening in Grahamstown. Last year I was involved in the indaba, and so my name would have been associated with it (quite possibly on a facebook group or something). So a few weeks ago a guy by the name of With Held* sent me a friend request, with a note saying "Hi there ,I'M looking forward in seeing you at the Digital Citizen Indaba this year.I will be coming with Ramon Thomas.". Now, I am not a fan of Ramon Thomas you are welcome to google him, and look at his website, and even meet him to create your own opinions. Either way, that friend request annoyed me, because it was unsolicited, and, if we deconstruct "friend request" we see that it contains the word "friend". According to that wikipedia link, "'Friendship' is a term used to denote co-operative and supportive behavior between two or more beings". I do not claim that every "friend" on my "friend list" is exactly in that box, but both people involved in those transactions have deemed the relationship as mutually beneficial, and justified. I have never met With Held, I probably will never meet With Held, I am not friends with the person he mentions, and I am not attending the event he mentions (and I personally know a few other people who got the same friend request from With Held, who are in exactly the same position as me). So, if we look at it, he is sending unsolicited, unjustified, non-mutually beneficial, "friend requests" to Random people on facebook. In a way, that sounds like something else I am familiar with. SPAM. Gracefully, I ignored his friend request. Then, yesterday, he sent another one. Which pissed me off. I vented, I channeled, and I may have broken some of the rules I mentioned above, when I sent him this message (Amanda said I should have stopped after the second sentence or so, but I needed to get my point across):

Hi,

I am not going to the DCI, and, while Ramon Thomas may think it is a good idea to "network" with facebook, you are using tactics that are commonly used by spammers. I am not your friend, I have not ever met you, and I probably won't ever meet you. You are breaking social norms, and if everyone on the internet broke accepted social norms, then no one would want to use the internet. It is quite ironic that you are attending the digital citizens indaba, and yet you clearly have no idea what social norms exist in the digital medium.

I know multiple people you have tried to friend, and your behaviour verges on harassment. When I IGNORED your friend request, please RESPECT ME, and back THE FUCK OFF.

So, I feel like the last sentence probably was a little harsh, but, hands up who is tired of spammers and people who think the internet has vastly different rules that govern it? * Names changed to protect the identity of the person

These days, you can hardly get any service without having to sign some kind of service contract. I make a point of reading every single one of them. I hate signing them, because they often have clauses in them that I consider unreasonable. Yet, I put up with it and sign it, since it would just be too big a deal to argue against those clauses, especially when I need to have the service right now.

These corporations also bend and twist and mangle the laws to such an extent that the laws are essentially defeated, and as consumers our rights are compromised.

I was happy to read on Slashdot today that the Washington State Supreme Court ruled that AT&T’s service agreement may not waiver a client’s right to file a lawsuit against the company.

“AT&T’s Consumer Services Agreement is substantively unconscionable and therefore unenforceable to the extent that it purports to waive the right to class actions, require confidentiality, shorten the Washington Consumer Protection Act statute of limitations, and limit availability of attorney fees. … Courts will not be easily deceived by attempts to unilaterally strip away consumer protections and remedies by efforts to cloak the waiver of important rights under an arbitration clause.”

The full ruling is available in PDF format.

I think that the big corporates who attempt to downplay our rights are scumbags, and the people who take the time to deal with them and bring them to justice are my heroes.

(rant)

I’m so frustrated with SARS (South African Revenue Service). They sent me an e-mail to notify me that I’ve received communication from them that require my attention. I click on the link which takes me to their website, enter my username and password, and it takes me to a page where I have to download the contents of the documentation in a PDF file. So, I do so.

When I opened the file with my PDF viewer, and it displayed that I require the latest version of Adobe Reader. This was already a bit frustrating, having to do so much work and still not being able to read my message. At least Adobe reader is packaged in Medibuntu, so I add the repositories and install it. 20 minutes worth of downloads later, I open the PDF. Yet again, it displayed “You need the latest version of Adobe reader to open this file”. Grrr.

I head over to Adobe.com, saw a link that says “Adobe Reader download”, where it also said “Latest Version”, and downloaded it. Turns out it is also Version 8.1.2, the same version available from Medibuntu, and it also doesn’t want to open the document.

After trying all different kinds of things, I’m still unable to open the PDF. Thanks Adobe, for implementing such horrible technologies in our government, and thank you SARS, for absolutely not caring about free software users, even when our government has a mandate to move over to free software itself.

Then again, it’s not like any government ever had a good reputation for keeping its promises. I don’t know how I expected ours to be any different.

If you run a decent OS (linux) you should be able to:

sudo apt-get install fcrackzip

Then read the man page but know that there is one little gotcha. fcrackzip’s default brute force starting length is 5 characters and by default it will run up to 6 characters. The gotcha of course is that if your password is 4 characters long you will never find it. So always run it with the -l flag and start at 1.

jonathan@jonathan:~/Desktop$ fcrackzip secret.zip –verbose -b -l 1-10 -u
found file ‘fool.swf’, (size cp/uc  87763/172969, flags 9, chk 6136)
found file ‘file.exe’, (size cp/uc 632452/1176497, flags 9, chk 614d)
found file ‘logo.jpg’, (size cp/uc  49916/ 51346, flags 9, chk 6113)
found file ‘code.txt’, (size cp/uc   5661/ 34639, flags 9, chk 6141)
checking pw g:*~

PASSWORD FOUND!!!!: pw == idea

Easy peasy. It finds that password in 0.796 seconds.

Someone owes me a beer :)

Akismet

By the time I write this, Akismet will have caught a quarter of a million of spam entries on my blog. Thanks again, Akismet!

Michelle Obama

This morning while driving to work, I heard excerpts from Michelle Obama’s speech that she delivered in Denver yesterday. I haven’t cried in nearly 13 years, but every now and again something happens that brings me close, that speech got brought me one of those moments. I think Michelle Obama will make an awesome first lady.

Monopoly World Edition

Hasbro (the company behind the popular game, Monopoly), will be releasing a new version of the game that contains popular cities around the world. The cities were chosen by votes on their website, and they have made the results public. My home town, Cape Town, got in at number 3. Awesome.

Well my review finally when live today. It's a printer review with a difference... it's reviewed on Linux. I have a Windows computer at home that I log onto every now and then for developing openlp.org, but for this review I was determined not to go there. I was largely pleased with the printer, but you can read more about it in my review on IOL Technology.

I have a little theory I have been working on, it is probably dead obvious to some people, but it only crystalised in my head relatively recently. I have had a few ideas in my time (so far none have made it to the real world), and in mentally stepping through these ideas I have started to refine my methods of evaluating an idea. One of the mental tests I have been using recently is: Does this idea:

1. Help some business/cross section of people do what they do, more efficiently/economically?
2. Would the technology associated to my idea help me act more efficiently in an environment of competition?
3. Or, does this idea earn me income by facilitating other people earning income

So, Point 1 and point 3 interact in an interesting way, since helping people operate more efficiently/economically in their business, I am effectively providing a service which facilitates other people earning income. There is a subtle difference, but I guess point 1 could be considered a further classification of point 3. Basically, for me, the pencil test of an idea, deciding on its value and functionality, is "Does this make other people money", and that is the difference that social applications have brought to the table. Generally the oft touted value in a web app is "create eyeballs, and sell advertising", and it is valid, but it is not the only golden egg. Facebook has pretty much gotten the lion-share of eyeballs in recent time, and they have a powerful and subtle distinction over other eyeball generators: targeted advertising. If you look over the questions you answer on your profile it reads just like (but, is disguised well enough not to seem like) the kind of thing you see on marketing surveys. They offer a fantastic platform to advertisers: you can choose your segment _exactly_, pay per advert, etc. etc. Either way, facebook is successful on two fronts, it helps people be more efficient in their social networking (i.e. something they do normally), and provides a platform for companies to advertise efficiently (lets not get into effectiveness). So, they win on point 1, and they win on point 3. They also perform well on point 2, because they used technology well to achieve better efficiency in a competitive market. But I didn't come here to praise just the facebook, I was only trying to illustrate my point. So, how well do the ideas floating round your head fit into my picture? If you hypothetically test your idea, does it honestly do well against those criteria? And, if it doesn't could you modify your idea to fit into that framework? I think too many web applications are hyped up (the hype curve, duh), and brush over these things. There are lots of apps floating around which I know I will try out, say "neat" and never use again, because they don't help me be more efficient or help me make money (not to say that said app may not be good for other people). The other further classification is "does this make it harder, or easier for someone to do said task" and that must take into account factors like bandwidth, computer availability etc. Good ideas meld into our lifestyles. A good example is 3D, interactive displays, they are a cute concept, and wow the audience on the bourne identity, but they are unrealistic until certain things happen that make them _easier_ to use than our current interactions with the tech around us. Its for that reason that people bleat about the iPhone not having a keyboard, because, well, the keyboard is a banging idea (still), and quite hard to beat. A recent example of this was the new South African foray into cellphone based money transactions called "wiWallet". Their site claims it is "payment simplified". They are trying to crack the nut that people have been trying to crack for ages...turning your cellphone into a "wallet". But IMHO they have fared poorly. I personally have moved from using cash to using my credit card for everything, so I could get away with sticky taping my CC onto the back of my phone, and have a better opportunity of being able to pay almost anywhere. The idea of coding your CC onto your phone removes that physical token, but doesn't seem to add much value beyond the romantic idea of not having to carry a CC around. So your target market is now: People who have credit cards, and are willing to pay more money per month, for the chance that they will be able to pay for something without carrying their wallet (if their vendor has a machine which can do it). If they removed the need for having a credit card (like taking the money out of your airtime, etc.) then they would be moving into an interesting territory. Since I have to carry my driver's license (because I drive most of the places I am going to shop), carrying my credit card does not cost me anything extra. wiWallet does not make it more efficient for me to make a payment, it does facilitate making people money, but it doesn't improve on the model the credit card already provides. To me it feels like a hi-tech solution for a low-tech problem. Just my 2c. Also, for real, genuine success, in the words of J.D. Hitchcock "Make your (website|application) awesome".

I used Google Apps to host mail for this domain for a while, and wanted to close down the account since I don't use it anymore. Before I did that I wanted to move all the data onto my server. Transferring the emails was fairly straightforward using POP3, but I couldn't find a way to download the Google Talk logs. Gmail handles the logs as emails, but they aren't accessible using either POP3 or IMAP.

I therefore wrote a Python script which downloads the logs via the web interface. On Jeremy's suggestion I used BeautifulSoup to parse the HTML this time, which worked very well. The script works with both Google Apps and normal Gmail, although my account got locked twice while trying to download the 3500 logs in my account.

Due to a very restrictive firewall at the CHPC, I need to run a VPN to get access to things like email, Jabber and SSH. This however degrades my web browsing experience, since that gets tunnelled as well. I therefore wanted a setup where only ports which are blocked get tunnelled through the VPN, while everything else goes out normally.

The routing part was fairly straightforward, which consists of an iptables rule to mark certain packets, and an alternate routing table for these marked packets. I first created a name for the new table by adding the following to /etc/iproute2/rt_tables.

10  vpn

I then added a default route to the new table specifying the IP address of the VPN server and the VPN interface, and a rule to use this table for packets marked by iptables.

ip route add default via 10.8.0.3 dev tun0 table vpn
ip rule add fwmark 0x1 table vpn

The following iptables rule will mark packets destined to the listed port numbers. Note that this is for packets originating from the firewall host — if you want this to apply to packets forwarded for other hosts it must be in the PREROUTING chain.

iptables -t mangle -A OUTPUT -p tcp -m multiport --dports 22,995,587,5223 -j MARK --set-mark 0x1

The actual routing worked, but packets were being sent with the wrong source IP. I therefore needed to NAT packets going out on the VPN interface (the IP address is the local IP of the VPN connection).

iptables -t nat -A POSTROUTING -o tun0 -j SNAT --to 10.8.0.4

I could then see packets going out on the VPN interface with the correct source IP as well as the replies, but it still wasn't working. I eventually discovered that rp_filter must be disabled in order for this to work.

echo 0 > /proc/sys/net/ipv4/conf/tun0/rp_filter

The one thing missing from my posting by email setup was support for images. The Mailsave module has finally been updated for Drupal 6, and so I can now submit attachments with email posts. The one shortcoming is that files are simply added to posts as normal attachments, and so images aren't automatically displayed. I therefore have to manually insert images in the body of the post, but I actually prefer this since it's a simpler system and gives me more control.

I also needed a way of resizing images on my phone since they are too big. I found the Nokia Image Editor¹ which seems to work fairly well, although it only allows resizing to specific resolutions.

Richard Feynman is one of my core inspirations. This seemingly simple man was not only a Nobel winning quantum physicist, but also a man who claimed that anyone who said they understood quantum physics was a liar. He also played the bongo drums, picked locks and worked with Einstein on the atomic bomb.

Feyman says:

I can live with doubt and uncertainty and not knowing… I think it’s much more interesting to live not knowing than to have answers that might be wrong.

I don’t feel frightened by not knowing things, by being lost in the mysterious universe without having any purpose… which is the way it really is as far as I can tell.

Watch the video:

Because I am on the internet, and because the apps I use offer statistics, I figure I would share some of them, feel free to do the same, if you feel the urge.

Google Reader

Last.fm

16,205 plays since 8 Aug 2006

Twitter

• Following 30
• Followers 51
• Favorites 0
• Direct Messages 5
• Updates 547

My brother was talking about buying my Canon Digital IXUS 750 Camera off me. (or PowerShot SD550 for Americans) He had an identical camera and waterproof housing for it (this costs significantly more than the camera). But said waterproof housing had not been properly closed once…

My camera wasn’t having any of that, and the next time I turned it on, it half-opened the lens, groaned, and said “E18”. Bugger. Googling E18 turned up a few sites showing other people with the same problem, lots of other people: e18error.com, E18 Error on Wikipedia. It appears to be a generic error for lens problems in Canon cameras, and occurs so often that class action suits have been filed against Canon.

I read the tales of woe, and tried the suggested remedies of shaking, banging, prodding, and otherwise mauling my poor camera. Nothing helped. I put it in a pelican case and forgot about it for a few weeks.

Reading on, I discovered a few tales of brave owners disassembling and repairing their cameras, mostly successfully. As a geek, I knew I was going to have to give this a shot. I’ve taken things apart since I learned how to use a screwdriver, so I can normally put them back together again (these days), and they normally still work.

Eventually, I got around to this, last week. Nobody has posted disassembly instructions for any camera near my model, so I had to work it out for myself. Now, let’s remedy that:

My Symptoms

My camera’s lens was open, and wouldn’t move at all. Turning it on gave an E18 error.

Preparation

I’d recommend the following:

• An afternoon & evening to yourself
• A large, empty desk (preferably with a lip, to catch dropped screws)
• A lino floor (carpets can lose dropped screws)
• A good desk light
• Lens tissue (or better yet, the wet-wipe version)
• Meths and ear-buds (or other solvent of choice)
• Superglue (in case you break something or something is broken)
• Tweezers, pliers, leatherman, etc. (you are dealing with lots of little things)
• Screwdrivers: small philips-head drivers for screws, and a few tiny flat ones for prying.
• A torch (to help you find dropped screws)
• A third-hand (or at least its magnifying glass)
• A blower/brush (to get rid of dirt)
• A working camera (to document the procedure, so you can put it back together)
• Patience - dropped screws can be hard to find

Warnings: You need to have a willingness to part with your patient’s life. You also need to be aware that camera flash assemblies contain high-voltage capacitors, that usually hold a small residual charge. Stay well clear of them and their circuitry. If possible, discharge it as soon as you see it, with a heavy-duty resistor.

Tips: Lay out the removed parts in the order you disassembled them, together with their screws. That way you won’t have the “left-over screw” problem or put things together in the wrong order.

Disassembly

Remove the battery and SD card.

To remove the case, you need to undo all the exterior screws: 3 on the base, 2 on the left, and 2 on the right (one is under flap C). The side plate A is loose, and B is a plastic sheet that can be pulled out, revealing an additional screw. Flap C is attached to the body, not B. When reassembling, take care to insert lip D under the back panel.

There are no clips on the bottom or sides, but there are 3 along the top, between the front and back halves. One to the right of the shutter, two to the left. Pry up on the front half.

There should be a black O-ring on the outer part of the lens. Lift it off and store.

The three main modules are now visible. Motherboard and battery (A), Flash unit (B), and Optics (C). While we won’t undo these connectors quite yet, as the LCD is currently attached to both sides, but this is a good opportunity to explain the connectors that you’ll be encountering.

The ribbon cable E plugs into the white connector with a black lid. The black lid needs to be folded back for the ribbon to be removed. It simply pulls out along it’s axis. To re-insert: open, push in ribbon as far as it’ll go, and close. These connectors are quite delicate, be careful.

There is another type of ribbon connector which simply relies on friction. The LCD back-light cable is an example. You just pull it out with tweezers, and push it back with tweezers (without bending it, if possible).

The flash power lead D must be pulled up, away from the camera. Insert a tiny screwdriver underneath the wires at the point indicated, and pry up.

The buttons are a loose piece of rubber. Lift off.

The LCD needs to be removed first. Pull out the backlight power ribbon (A). Unscrew the screw above the LCD, releasing a bar. The left side has a small clip that needs to be released, and then the LCD-backlight assembly should lift upwards. The right hand side has a lip under the keypad module, so lift the left side first. You won’t be able to disconnect the LCD ribbon until you remove the keypad plate.

Unscrew the 2 screws at the top of the keypad plate. There are a few clips holding the bottom in place (arrowed). But you should be able to pull the plate away, revealing the ribbon connectors for both units. Unplug them both.

The Flash unit can now be removed. Unplug the cables shown earlier, as well as the screw on the bottom right-corner. The ribbon plugs into the flash unit, unplug (C).

The left two screws on the back (red) will release the flash unit.

Before unscrewing the optical assembly, open the CCD ribbon connector (A). When re-attaching the module, the cable should again be inserted first, and locked last.

The three (green) screws on the metal frame will release the optical module. Beware a tiny spring hiding under B. Lift it out, and store it.

Before we can take the Optical assembly apart, the focussing LED has to be removed. Unstick ribbon A, and pry up the LED (B). Continue lifting the ribbon, unsticking the status LED section (C), too.

While we are here, the focussing servo’s cogs are under D, if you are cog-cleaning. Don’t open if you don’t need to.

Unplug (pull) the shutter-ribbon from E, and unstick E’s ribbon from the lens-body.

The lens and viewfinder assembly can now be removed from the base-plate with the CCD and motors. Unscrew the 4 screws and one on the base. The long screw comes from near F. Lift up the lens carefully. A small black cog will be loose near F. Remove and store.

The green screw gives access to the zoom servo’s cogs. Don’t open unless you need to.

On the CCD base, the sharp bit (A) activates the lens-cap mechanism in the lens, when it’s closed. The lens element (B) is for focussing, and in my case it’s sitting at an odd angle, because the short pin (circled) had broken, and had to be glued back in place. This pin passes through an IR light-switch when the lens is at a certain hight, allowing the camera to calibrate its focus.

Check that the focussing element moves up and down smoothly when you rotate the thread below A.

While you are here, blow any dust off the lens and CCD below it.

To disassemble the lens: Un-thread the ribbon. Roll the big cog on the side until it’s fully closed, and clicks, revealing the pins of the inner rings, and push the outermost interior ring of the lens backwards from the front. It should pop out.

The rings either simply pop out backwards, or have a track leading to the surface. Clean all the tracks and pins.

If you are having shutter-trouble, you can open the innermost module, but beware it’s delicate. If the lens-cap is jamming, operate it a bit with a screwdriver (wiggle), blow air at it, etc until it works cleanly.

Reassembly

Finally, if you found your problem, reassemble.

Remember to rethread the lens ribbon before you attach the outermost ring. The lens should operate smoothly when zoomed with the big cog. It’s easiest if you attach it to the CCD plate in the opened state.

The camera behaves well, and can be tested disassembled. If you are having E18 trouble, you can just connect the lens to the motherboard, insert the battery, and turn it on. If it’s working, the lens should open, and close when turned off (and the power button LED should go out promptly, if it doesn’t you haven’t found the trouble yet).

Enjoy your newly fixed camera. I am, mine.

There were a few questions about the choice of Python as a language, and whether and what languages would come next, comparisons to other existing containers, and so forth.  Guido van Rossum said it was partly because Python is one of the three big languages at Google, and because it was (relatively) easy to harden the VM.  Kevin Gibbs said they had to start somewhere, and that they were committed to others.  Paul McDonald said that the two most voted-for issues on the issue tracker are language-related, and that there were teams (ie, more than one) currently actively working on languages (ie, more than one).

A couple of questions around "maturity" - the team says they'll make it clear when it is no longer a preview, and that this will probably happen when they have the billing set up and offline processing.  They expect billing to be available "toward the end of the year".

Question about HTTPS/SSL and access to encryption within GAE code.  Answer is that it's something they want to do, but don't know when they'll get to it.  Data is "strictly" partitioned between apps in the store (BigTable).

A common thread in answers were that the Google App Engine team were very interested in people being able to get their data and code out of GAE, and they're working on making it easy to bulk output the data.  They hoped that a standard would emerge for BigTable-like storage (CouchDB, SimpleDB) so that people could write code and host it on GAE or elsewhere.   And people are already working on compatible APIs to make it possible to run on other storage systems (but may not be too efficient).

Some interesting news was delivered during the Google I/O keynote.

In terms of Google App Engine, the announcement that got the biggest applause was that it was now open to all signups - no waiting list and a few tens of thousands of developers.

Beyond that, the two new APIs were announced - the memcache API and the Image API.

Some pricing expectations for usage beyond the free chunk given to you were given:

• CPU: 5 million "average" page views free, 10-12c per core-hour thereafter
  
• Storage: 500MB free, 15-18c per GB-month thereafter.
  
• Incoming traffic: 5 million "average" page views, 11-13c/GB thereafter
  
• Outgoing traffic: 5 million "average" page views, 9-11c/GB thereafter
  

The Google Web Toolkit 1.5 release candidate was released today, which brings Java 5 language features.

In terms of OpenSocial, the 0.8 version specification was released yesterday, and that AOL has joined the OpenSocial initiative.

On Saturday (May 10th) the Cape Town Python User Group held a Python Sprint meeting as part of the Global Python Sprint weekend.  8 or so of us got together on and off from 10:30am until about 9:30pm at the SynthaSite offices around a table and worked through 10 or so issues in the Python issue database.

Thanks to The Other Neil and Simon for most of the organisation effort, and to them and Adrianna, Russell, Jonathan, Jeremy, Brad, and David for coming through and taking part.

And thanks to SynthaSite for coffee, coke, crisps, chocolates, and other goodies.

According to The Other Neil, we worked on:

• http://bugs.python.org/issue2736
• http://bugs.python.org/issue2809
• http://bugs.python.org/issue2742
• http://bugs.python.org/issue1491
• http://bugs.python.org/issue1343
• http://bugs.python.org/issue2776
• http://bugs.python.org/issue1777412
• http://bugs.python.org/issue2768
• http://bugs.python.org/issue2722
• http://bugs.python.org/issue1153769
• http://bugs.python.org/issue2746

A housemate of mine got a new Samsung phone on the weekend. Being a resident geek, I offered to transfer her contacts across rather than get her sister to manually retype 500-odd contacts.

Naturally, I thought this would be a simple problem, right? I mean, everyone updates their phones every 2 years, this must be a pretty common use case. All my Sony Ericsson phones have had a “send all contacts by Bluetooth” option since the inception of Bluetooth. Naturally, it didn’t have such a feature, it only supports sending one contact at a time. (Although, to Samsung’s credit, the new phone will be able to do for the next upgrade)

Next option: I’ll sync old phone to laptop to new phone.

The Samsung website has a helpful Windows utility that you can download to do this, however you need the cable to link the phone to the computer. The phones needed different cables, and I had neither. My laptop with a Windows partition has had broken Bluetooth ever since its motherboard got replaced. So that wasn’t an option. The phones don’t have IRDA, so there was no way to connect them with the Windows laptop.

Time to do it properly.

I tried wammu, a python-based gammu GUI. It supported the phones via the “blueat” driver, and could browse their SIM cards fine, but not their internal Phonebooks. It couldn’t back them up either. A bit of poking around with gammu on the command line showed that the internal phone books are not 0-indexed (normal computer counting, 0 to n-1) or 1-indexed (normal human counting, 1 to n), but 2-indexed. Dijkstra would turn in his grave!

At this point, I could see that I was going to have to write my own, backup utility. The output of gammu was awkable, but seeing as there are good gammu-python bindings, I decided to do it in pure Python.

Reading the address book went something like this:

import gammu, pickle
sm = gammu.StateMachine()
sm.ReadConfig(3, 0)
sm.Init()
old = []
for i in range(2, 587):
    old.append(sm.GetMemory("ME", i)

pickle.dump(old, file("phonebook.dump", "w"))

The 3 signifies gammu configuration number 3, read into position 0. 587 is the number of address book entries. “ME” means internal memory. I then pickled “old” in preparation for the next stage. Here is an example of an item in old:

{'Entries': [{'AddError': 7517792,
              'Type': 'Text_FirstName',
              'Value': u'Foo'},
             {'AddError': 796160623,
              'Type': 'Text_LastName',
              'Value': u'Bar'},
             {'AddError': 796160623,
              'SMSList': [],
              'Type': 'Number_Other',
              'Value': u'0211234567',
              'VoiceTag': 0},
             {'Type': 'Category', 'Value': 0}],
 'Location': 2,
 'MemoryType': 'ME'}

Pretty icky, but at least all the information is there. At this point, one should be able to feed it into the new phone:

sm.Terminate()
sm = gammu.StateMachine()
sm.ReadConfig(4, 0)
sm.Init()
for i in old:
    sm.AddMemory(i)

However nothing I tried worked, I always got an “Invalid Location” error. I think the 2-indexing is trumping gammu again.

Next idea, lets munge the data into vCard format and use wammu / gammu’s “import from vCard” function. (Code coming up soon) Turns out this doesn’t work either. The phone only received the First name, first phone number, and various other things that I didn’t send it (i.e. custom ring tones that it made up). Hmph!

Aha, but cellphones can normally Bluetooth vCards to each other. So I pushed it the vCard collection via obexftp. Starts transmitting, but then the phone reboots. I played around a bit, and found that if you send it more than one vCard in a vCard file, it reboots. Lovely.

So my final solution was: Extract address book with python-gammu. Transform into vCards. Send each one individually. At least the phone had a “trust this device” option so that it wouldn’t prompt the user for every vCard I sent, but just automatically import them - the first sensible feature I’ve found on it.

Here goes:

#!/usr/bin/env python
import os, pickle, time

def normalise_num(n):
    "Neaten up the phone number, internationalise, etc."
    if n.startswith("+"):
        return n
    if n.startswith("00"):
        return "+" + n[2:]
    if len(n) == 10 and n[0] == "0":
        return "+27" + n[1:]
    return n

d = pickle.load(file("phonebook.dump", "r"))

# Normalise into a sensible format:
o = []
for i in d:
    t = {}
    for j in i["Entries"]:
        if j["Type"] == "Text_FirstName":
            t["First"] = j["Value"]
        if j["Type"] == "Text_LastName":
            t["Last"] = j["Value"]
        if j["Type"] == "Number_Other":
            n = normalise_num(j["Value"])
            type = "Home"
            if n[3] in ("7", "8"):
                type = "Cell"
            if type not in t:
                t[type] = []
            t[type].append(n)
    o.append(t)

# Write & Send vCards:
for i in o:
    f = file("temp.vcf", "w")
    f.write("BEGIN:VCARD\n")
    f.write("VERSION:2.1\n")
    f.write("N:%s;%s;;;\n" % (i.get("Last", ""), i.get("First", "")))
    pref = ";PREF"
    for j in i["Cell"]:
        f.write("TEL;CELL%s:%s\n" % (pref, j))
        pref=""
    for j in i["Home"]:
        f.write("TEL;HOME%s:%s\n" % (pref, j))
        pref=""
    f.write("END:VCARD\n")
    f.close()
    os.system("obexftp -b 00:DE:AD:00:BE:EF -p temp.vcf")
    # Give the thing a chance to recover:
    time.sleep(0.1)

Yes, the normalisation could be done with list comprehensions, but it would be horrible to read. And there might by Python Obex bindings, but I couldn’t be bothered.

I got to spend an afternoon messing with dodgy Cellphones, rather than having a teenager do the job for free. I think I chose the wrong option, but at least it was fun.

Footnote: Samsung, your phones User Interface is awful. Why on earth is Bluetooth under “Applications” rather than “Settings”? I searched everywhere but there, and finally googled before I found it…

Now that I have a computer which still has all of its teeth and is good for something other than yelling “You damn kids! Get off my lawn!”*, I have set up Basilisk II, the 68k Mac emulator (when I upgrade to Hardy I will also try SheepShaver, the PowerPC emulator). It’s running MacOS 8.0, the highest version of the operating system that Basilisk will support, and is using a Quadra ROM.

MacOS 8 is prehistoric, and the emulator keeps hanging — I’m hoping that with a bit more research I’ll be able to optimise the settings and make it stop doing that . Why did I bother doing this at all? Mostly so that I could once again play the finest Breakout clone in the history of human civilisation: Akira Nagamatsu and Shizue Mouri’s MozPong.

Instead of a paddle, you have a small boy with a big head. Instead of a ball, you have “Butasan”, which seems to be a tiny flying pig (I only solved this mystery ten minutes ago, with the awesome power of Google. I always thought it was some kind of clay bottle!). Instead of bricks, you have eggs — when you break them, chickens fall out and you have to catch them for points. Some eggs contain extra Butasan, and there are also bombs (which explode). Every now and then, a dinosaur called Josephine walks onto the screen and tries to hug you, thereby preventing you from getting to your Butasan (or chickens). You get rid of her by hitting her with the Butasan and running over her while she’s down. Catching lots of chickens gives you random bonuses — most of them just give you more points, but some have interesting special effects.

This game is made of win. It makes the whole emulator setup worth it. There is apparently also a Windows port, which you may find more convenient if you’re already running the OS of Evil.

Other favourites I am looking forward to playing are Maniac (a cross between Pacman and hangman), Blobbo Lite (a puzzle game) and Bill the Demon (a gruesome little platform game) — which I will review properly at a later stage.

* Its RAM is an order of magnitude larger than that of my old computer. This means that I can now actually run things that normal people run — even all at the same time.

My post about repositories wasn’t just a little attempt to stave off work, it was part of a larger scheme.

I share the ADSL line in my digs with 3 other people. We do split-routing to save money, but we still have to divide the phone bill at the end of the month. Rather than buy a fixed cap, and have a fight over who’s fault it was when we get capped, we are running a pay-per-use system (with local use free, subsidised by me). It means you don’t have to restrain yourself for the common cap, but it also means I need to calculate who owes what.

For the first month, I used my old standby, bandwidthd. It uses pcap to count traffic, and gives you totals and graphs. For simplicity of logging, I gave each person a /28 for their machines and configured static DHCP leases. Then bandwidthd totalled up the internet use for each /28.

This was sub-optimal. bandwidthd either sees the local network, in which case it can’t see which packets went out over which link. Or it can watch the international link, but then not know which user is responsible.

I could have installed some netflow utilities at this point, but I wanted to roll my own with the correct Linux approach (ulog) rather than any pcapping. ulogd is the easy ulog solution.

Ulogd can pick up packets that you “-j ULOG” from iptables. It receives them over a netlink interface. You can tell iptables how many bytes of each packet to send, and how many to queue up before sending them. E.g.

# iptables -I INPUT 1 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 48 --ulog-prefix input

will log the first 48 bytes of any incoming packet to netlink-group 1. It will tag the packet as being “input”, and send them in batches of 50. 48 bytes is usually enough to catch any data you could want from the headers. If you were only need size, 4 bytes will do, and for source and destination as well, 20.

Now, we tell ulogd to listen for this stuff and log it. Ulogd has a pluggable architecture. IPv4 decoding is a plugin, and there are various logging plugins for “-j LOG” emulation, Text files, pcap-files, MySQL, PostgreSQL, and SQLite. For my purposes, I used MySQL as the router in question already had MySQL on it (for Cacti). Otherwise, I would have opted for SQLite. Be warned that the etch version of ulogd doesn’t automatically reconnect to the MySQL server should the connection break for any reason. I backported the lenny version to etch to get around that. (You also need to provide the reconnect and connect_timeout options.)

Besides the reconnection issue, the SQL implementations are quite nice. They have a set schema, and you just need to create a table with the columns in it that you are interested in. No other configuration (beyond connection details) is necessary.

My MySQL table:

CREATE TABLE `ulog` (
  `id` int(10) unsigned NOT NULL auto_increment,
  `oob_time_sec` int(10) unsigned NOT NULL,
  `oob_prefix` char(4) NOT NULL,
  `ip_totlen` smallint(5) unsigned NOT NULL,
  PRIMARY KEY  (`id`),
  UNIQUE KEY `id` (`id`),
  KEY `oob_prefix` (`oob_prefix`),
  KEY `oob_time_sec` (`oob_time_sec`)
);

My ulogd.conf:

[global]
# netlink multicast group (the same as the iptables --ulog-nlgroup param)
nlgroup=1    
# logfile for status messages
logfile="/var/log/ulog/ulogd.log"    
# loglevel: debug(1), info(3), notice(5), error(7) or fatal(8)
loglevel=5    
# socket receive buffer size (should be at least the size of the
# in-kernel buffer (ipt_ULOG.o 'nlbufsiz' parameter)
rmem=131071    
# libipulog/ulogd receive buffer size, should be > rmem
bufsize=150000
# ulogd_BASE.so - interpreter plugin for basic IPv4 header fields
#             you will always need this
plugin="/usr/lib/ulogd/ulogd_BASE.so"
plugin="/usr/lib/ulogd/ulogd_MYSQL.so"

[MYSQL]
table="ulog"
pass="foo"
user="ulog"
db="ulog"
host="localhost"
reconnect=5
connect_timeout=10

The relevant parts of my firewall rules:

# Count proxy usage (transparent and explicit)
iptables -A count-from-inside -p ! tcp -j RETURN
iptables -A count-from-inside -p tcp -m multiport --destination-ports ! 3128,8080 -j RETURN
iptables -A count-from-inside -s 10.0.0.16/28 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix sr-p
iptables -A count-from-inside -s 10.0.0.32/28 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix fb-p
iptables -A count-from-inside -s 10.0.0.128/25 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix gu-p

iptables -A count-to-inside -p ! tcp -j RETURN
iptables -A count-to-inside -p tcp -m multiport --source-ports ! 3128,8080 -j RETURN
iptables -A count-to-inside -d 10.0.0.16/28 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix sr-p
iptables -A count-to-inside -d 10.0.0.32/28 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix fb-p
iptables -A count-to-inside -d 10.0.0.128/25 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix gu-p

# Count forwarded traffic (excluding local internet connection - ppp2)
iptables -A count-forward-in -i ppp2 -j RETURN
iptables -A count-forward-in -d 10.0.0.16/28 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix sr-f
iptables -A count-forward-in -d 10.0.0.32/28 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix fb-f
iptables -A count-forward-in -d 10.0.0.128/25 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix gu-f

iptables -A count-forward-out -o ppp2 -j RETURN
iptables -A count-forward-out -s 10.0.0.16/28 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix sr-f
iptables -A count-forward-out -s 10.0.0.32/28 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix fb-f
iptables -A count-forward-out -s 10.0.0.128/25 -j ULOG --ulog-nlgroup 1 --ulog-qthreshold 50 --ulog-cprange 4 --ulog-prefix gu-f

# Glue
iptables -A INPUT -i eth0 -j count-from-inside
iptables -A OUTPUT  -o eth0 -j count-to-inside
iptables -A FORWARD -i ppp+ -j count-forward-in
iptables -A FORWARD -o ppp+ -j count-forward-out

So, traffic for my /28 (sr) will be counted as sr-f or sr-p so I can tally up proxy & forwarded traffic separately. (Yes, I can count traffic with squid too, but doing it all in one place is simpler.) fb is random housemate Foo Bar, and gu guest (unreserved IP addresses).

You can query the usage this month with for example:

SELECT oob_prefix, SUM(ip_totlen) FROM ulog WHERE oob_time_sec > UNIX_TIMESTAMP('2008-04-01 00:00:00') GROUP BY oob_prefix;

Your table will fill up fast. We are averaging around 200 000 rows per day. So obviously some caching is in order:

CREATE TABLE daily (
  id INT UNSIGNED NOT NULL AUTO_INCREMENT,
  time TIMESTAMP,
  oob_prefix CHAR(4) NOT NULL,
  data INT UNSIGNED NOT NULL,
  PRIMARY KEY (id),
  KEY (oob_prefix(4)),
  KEY (time)
);

And every night, run something like:

INSERT INTO daily (time, oob_prefix, data)
SELECT FROM_UNIXTIME(MAX(oob_time_sec)), oob_prefix, SUM(ip_totlen)
FROM ulog
WHERE oob_time_sec >= UNIX_TIMESTAMP('2008-04-01 00:00:00')
  AND oob_time_sec < UNIX_TIMESTAMP('2008-04-02 00:00:00')
GROUP BY oob_prefix;
DELETE FROM ulog WHERE oob_time_sec  >= UNIX_TIMESTAMP('2008-04-01 00:00:00')
  AND oob_time_sec < UNIX_TIMESTAMP('2008-04-02 00:00:00');

Finally, I have a simple little PHP script that provides reporting and calculates dues. Done.

Last weekend I went hiking with Hodgestar, Hodgestar’s Mom and Neil in the Vogelgat nature reserve near Hermanus. It’s lovely — isolated, almost entirely empty of people (it’s a private reserve with a limited membership), and full of fynbos and frogs. I have photos, which I should upload. There are several little huts where you can stay overnight; we stayed at Leopard Camp.

I have new hiking boots, which made my hiking experience a lot more pleasant than the previous one. Getting them was a bit of an adventure; they were a hurried, last-minute purchase, and the first pair I got did not fit as well at home as it seemed to in the shop and had to be exchanged. The second pair is really as waterproof as advertised, and very comfortable. I have ended up with a men’s size 8; I see that shoe sizes are exactly as reliable as clothing sizes.

I have read Grass, which is really good. I am currently reading the third Digger book (after work while waiting for Hodgestar) and His Dark Materials (at home, because it is borrowed (thanks, Katherine!)).

HDM is a lot better than the movie. All the bizarre and inexplicable things that happen in the movie happen for logical reasons in the first book. It also has an actual ending, which is a great improvement. Part of the movie’s incomprehensibility can be blamed directly on the scriptwriters’ removal of all overt religious references — since a key element of the plot (why everyone is so upset about Dust) hinges on a purely religious concept. Seriously, how did they think that they could just take that bit out, and have the whole thing still make sense? Yeah, they do sort of allude to it, but no satisfactory explanation is given.

Is this an “atheist” series? Uh, not so much, except possibly in the eyes of people who conflate atheism with criticism of religious institutions and their teachings, or with “hatred of god”, that notorious strawman belief system I’m not entirely sure can actually be found in the wild (except in isolated individuals who aren’t very sane). The books are certainly scathingly critical of the Catholic Church, which is presented as a Bad Guy throughout, and it looks like god very much exists in the setting and is turning out to be some kind of supernatural villain.

I will reserve final judgement on the series until I’ve finished the last book, but so far it’s OK, and a fun, quick read. Most people seem to find the second book a bit dull; I think I agree. The switch in protagonists is a bit jarring. I don’t find Will Parry very likeable, especially in comparison to Lyra.

At work I have installed Ubuntu Gutsy on my MacBook Pro. Almost everything worked out of the box (although there’s a lot of functionality I’ve never tried out), but I had to upgrade to the Hardy kernel to get sleep to work. Now I’m installing Debian Etch in a VM (using KVM/Qemu). So far, so good.

Hodgestar and I got ourselves a data projector for Christmas. It’s very nice. And now I’m off to watch Farscape.

ETA: I forgot — I have bought Creatures of Rokugan and Emerald Empire (recent L5R 3rd Edition sourcebooks). I haven’t had much time to read them yet, but they look pretty good. CoR is a bit dry and lacking in pictures and explanations, and both books unfortunately suffer from AEG’s usual proofreading issues, but EE looks like a really awesome setting resource. The bit I did manage to read finally clarifies the issue of meat-eating in Rokugani society, and I’m hoping for a resolution of the leather stupidity as well. Vassal families have finally been updated! (Of course, I reserve the right to ignore some of this information utterly if it josses my campaign.) The book has lots of wonderful in-character commentary in the form of extracts from the diary of Doji Barahime, a snarky Crane Clan courtier.