After over a year of development, we have finally released Ibid. Ibid is a general purpose chat bot written in Python. We've suffered from a bit of feature creep, so despite being a 0.1 release it can talk 7 messaging protocols and has over 100 features provided by plugins. I think we also have an excellent architecture and very developer friendly plugin API. The 0.1.0 release can be downloaded from Launchpad or installed from our PPA.

CUPOOTINO, California. Steven Jobless, CEO of Crapple Corp, announced his company’s long awaited tablet form-factor machine last Wednesday. Whilst sequestered in the backwoods of Tennessee last year, Mr Jobless spent a lot of time waiting for medical test results and considering the meaning of life. Whereupon he had what he described as a “number 1 experience”. It hit him – the perfect Captive Market. The new product, marketed as the iPoo is of the ideal form factor for use whilst positioned on the lavatory. Unlike the early tablet products produced by their competitors, the iPoo is not a fully fledged compooter, but follows the Jeff Raskin school of thought on what constitutes a suitable “Intuitive Interface” for the average user.

Marketing of the new product is intended to involve heavy use of anti-social media – each geek on the planet is forced into deciding whether they would buy the device or not. Since most of them think it’ll be shit, it’ll count as a ringing endorsement for the non-tech savvy public. It seems that the geek public is waiting for the next shitteration of the device.

Taking full advantage of the DRM the product provides, it comes in two models. The slightly more expensive option, “Comes with Porn” (seemingly aimed at competitor Noklue) unlocks DRM for porn on the device, and also uses a new technology known only as “3D-multitouch” (details not yet provided at time of going to press).

We spent some time after the media announcement assessing the mood amongst the company faithful, and by and large that mood was one of “Relief”. Reaction amongst the Linux community was less positive, with comments ranging from “We invented Brown” (Ubuntu’s Mark Shuttleworth), “We invented shit-casting” (Communitization manager, Jono Bacon) to “It’s Crap” (Fabian Scherschel, from the Linuxoutlaws).

Yesterday I looked at very simple theming in Pylons, and mentioned that I'd prefer loading the current theme from the database rather than the configuration file. So this evening I decided to see if I could achieve that.

Looking in environment.py, I noticed that SQLAlchemy and the models were being configured and initialised last in the load_environment method. I moved those two lines up to the top of that method, and changed "config" to "app_conf". Then I simply did a normal Session.query() to fetch my theme name out of the "variables" table in my database.

Now that I had my theme name, I constructed my theme directory using the path to the themes directory from my configuration file and the name of the theme from the database. Then I set up my static files and tempaltes directories using the new theme directory.

Lastly, I wanted to do something that Drupal, the CMS that powers this blog, does with it's theming system. Drupal has a set of fall-back templates so that if you don't provide (whether by mistake or by choice) all the templates a theme needs, the default Drupal templates are used. This was very easy to set up, because Mako's TemplateLookup class accepts a list of template paths. So all I did was add the normal Pylons template path to the list as well.

So now my file looks like this:

def load_environment(global_conf, app_conf):
    """
    Configure the Pylons environment via the ``pylons.config`` object
    """
    # Setup the SQLAlchemy database engine
    engine = engine_from_config(app_conf, 'sqlalchemy.')
    init_model(engine)
 
    # Pull out theme variable
    theme = Session.query(Variable).get(u'theme')
    if not theme:
        theme_name = u'default'
    else:
        theme_name = theme.value
 
    # Pylons paths
    root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
    theme_dir = os.path.join(app_conf[u'themes.directory'], theme_name)
    paths = dict(root=root,
                 controllers=os.path.join(root, 'controllers'),
                 static_files=os.path.join(theme_dir, 'public'),
                 templates=[os.path.join(theme_dir, 'templates'),
                            os.path.join(root, 'templates')])

As usual, any comments or suggestions are welcome

For a while I've been thinking about how to achieve themes in some of my Pylons applications, and just today I was looking around in my environment.py file, and saw the configuration of the templates and public directories. That gave me an idea, can I load a theme directory from the configuration file and use it in there?

I had a bit of a false start initially, I thought that I could just use the config object as I do elsewhere in my application, but that didn't want to work. Eventually I found the solution: the app_conf object.

This is how my code now looks:

    ...
    theme_dir = os.path.join(app_conf['themes.directory'], app_conf['themes.current'])
    paths = dict(root=root,
                 controllers=os.path.join(root, 'controllers'),
                 static_files=os.path.join(theme_dir, 'public'),
                 templates=[os.path.join(theme_dir, 'templates')]
    )
    ...

And the relevant lines in my config.ini file:

themes.directory = %(here)s/themes
themes.current = default

Each theme is simply a directory with two subdirectories, "public" where my images, styles and scripts live, and "templates" where my template files live.

I'm still looking into how I can pull the current theme from the database at this early stage of the application, but this is a start in the right direction.

Hopefully, this will help someone else. If you have any suggestions for a better or more elegant solution, I'd be keen to hear about it.

In a recent SAP CRM 2007 implementation, we had the requirement to hide a view (“partial”) in response to a change in a dropdown value.

This is easily done using javascript code on the client side, and what’s more such a solution to the problem is easy to implement and is not really restricted to SAP CRM 2007 or even SAP web development in general.

This is what we wished to achieve – hide block ‘A’ on the following screen if the value in the category dropdown ‘B’ is anything other than “corporate brands”:

This can be done using the ABAP CRM framework, but there are several potential concerns:

1. Components not previously enhanced may require enhancement
2. Components are designed to be independent, and making one component rely on another (which we want to do here) involves creating some sort of shared attribute
3. In order to suppress a view it will be necessary to re-render all views on the current frame and will involve an unnecessary round trip.

It was therefore decided to solve this problem using javascript code.

As SAP CRM 2007 is supported by Firefox (version 2, although version 3 will work with rendering glitches), it is possible to use the Firefox ‘Firebug’ extension to inspect a web page and determine the relationship between elements on screen and appropriate blocks of HTML.

From Firebug analysis (or if you have patience to do this manually) it was determined that this block of HTML code represents the Dropdown B:

<td class="th-ip-td1" style="overflow: hidden; width: 100%;">
<fieldset class="th-if-wrapper-onfocus" style="border: 0pt none ; margin-right: 3px;">
<input id="C14_W59_V60_ClassificationDdlb1" class="th-if th-if-icon" readonly="readonly" 
value="Corporate Brands" onkeydown="thtmlb_ddlbInputKeyDown(this,event);"
style="width: 100%;" size="16" onblur="thtmlb_inputBlur(this);"
onfocus="thtmlb_inputFocus(this);thtmlbSaveKeyboardFocus('C14_W59_V60_ClassificationDdlb1');"
onclick="thtmlb_inputClick(this,7);"/>
</fieldset>

The element we’re interested in detecting a change of value for is the input tag with id “C14_W59_V60_ClassificationDdlb1”

This block of HTML code represents the viewset area A that we will be hiding (ignoring a lot of the inner HTML):

<td class="th-gr-td" valign="top" rowspan="1" colspan="1">
<div id="C11_W43_V44_V48" excevt=""
intevt="c:C11_W43_V44_V48:C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do;" 
automode="true" tgt="" dhe="true" 
style="display: inline;">
<!-- Begin C11_W43_V44_V48 -->
<div id="C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do" class="th-ajax-area">
<!-- Begin C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do -->
<table ....

The element we’re interested in hiding is the div with id=“C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do”

In it’s simplest case, the following javascript, if inserted into the view somewhere after the thtmlb:gridCell tag that declares dropdown element B, does what is required – it will hide block A if the value of dropdown B changes to anything other than “Corporate Brands”:

<script for="ClassificationDdlb1" event=onchange type="text/javascript">
var dd = document.getElementById("C14_W59_V60_ClassificationDdlb1");
var hidee = document.getElementById("C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do");
if(dd.value != "Corporate Brands"){
  hidee.style.display = 'none';
}
else {
  hidee.style.display = 'inline';
}
</script>

Of course, we’re going to require something a little more complicated! The problem is that the id for the elements under consideration is not fixed – the C14_W59_V60 will be different for different roles and is affected by configuration changes. One way to solve this problem is to write a javascript function that will go through all elements and check for a regular expression match based on the contents of id. However the javascript function getElementById only accepts an exact match on id.

A custom function fuzzyElementSearch was created to find the element of interest (this function is contained within the file jonathan.js which will be listed afterwards), this is what the call to this function looks like:

<script src="/sap/bc/bsp/sap/crmcmp_ic_frame/scripts/common/jonathan.js" type="text/javascript">
</script>
<script for="ClassificationDdlb1" event=onchange type="text/javascript">

var dd       = fuzzyElementSearch("th-ip-td1","td",null,"input","ClassificationDdlb1", 0)[0];
var dd2      = dd[0];
var hide     = fuzzyElementSearch("th-gr-td","td", null, "div","productdetailviewset", 1)[0];
if ((hide) && (dd2) && (hide.length > 0)) {
  var hide2    = hide[0];

  if ((dd2.value) != "Corporate Brands") {
    hide2.style.display = 'none';
  }
  else {
    hide2.style.display = 'inline';
  }
}

</script>

The file jonathan.js contains the following javascript:-

function fuzzyElementSearch(classname, tagname, root, tag2, subbie, level) {

    // Call function from Rhino book that returns
    // an array of DOM elements that are members of the specified class,
    // have the specified tagname, and are descendants of the specified root.
    var elements = getElements(classname, tagname, root);

    // Find children (of tag2) for each of these elements
    var subfound = [];

    for(var i = 0; i < elements.length; i++) {
        var subchild  = elements[i];
        var grandkids = subchild.getElementsByTagName(tag2);
        if (grandkids) subfound.push(grandkids);
    }

    // search "level" deep for each of the above matches
    // for an element where the id contains the substring
    var pattern = new RegExp(subbie, "g");
    var matches = [];
    for(var i = 0; i < subfound.length; i++) {
        var matchee  = subfound[i];
        if (matchee.length > level) {
          var matchin = matchee[level];
          var matchid = matchin.id;
          var reslt = pattern.test(matchid);
          if(reslt == true) {
             matches.push(matchee);
          }
        }
    }

    // Note that we always return an array, even if it is empty
    return matches;


 // *** The following attribution applies to the rest of the code in this file
 // This code is from the book JavaScript: The Definitive Guide, 5th Edition,
 // by David Flanagan. Copyright 2006 O'Reilly Media, Inc. (ISBN #0596101996)
 // *** 

 /* Rest of code == EXAMPLE 15-4 From David Flanagan, "Javascript"
 * getElements(classname, tagname, root):
 * Return an array of DOM elements that are members of the specified class,
 * have the specified tagname, and are descendants of the specified root.
 *
 * If no classname is specified, elements are returned regardless of class.
 * If no tagname is specified, elements are returned regardless of tagname.
 * If no root is specified, the document object is used.  If the specified
 * root is a string, it is an element id, and the root
 * element is looked up using getElementsById()
 */
    function getElements(classname, tagname, root) {
        // If no root was specified, use the entire document
        // If a string was specified, look it up
        if (!root) root = document;
        else if (typeof root == "string") root = document.getElementById(root);

        // if no tagname was specified, use all tags
        if (!tagname) tagname = "*";

        // Find all descendants of the specified root with the specified tagname
        var all = root.getElementsByTagName(tagname);

        // If no classname was specified, we return all tags
        if (!classname) return all;

        // Otherwise, we filter the element by classname
        var elements = [];  // Start with an emtpy array
        for(var i = 0; i < all.length; i++) {
            var element = all[i];
            if (isMember(element, classname)) // isMember() is defined below
                elements.push(element);       // Add class members to our array
        }

        // Note that we always return an array, even if it is empty
        return elements;

        // Determine whether the specified element is a member of the specified
        // class.  This function is optimized for the common case in which the
        // className property contains only a single classname.  But it also
        // handles the case in which it is a list of whitespace-separated classes.
        function isMember(element, classname) {
            var classes = element.className;  // Get the list of classes
            if (!classes) return false;             // No classes defined
            if (classes == classname) return true;  // Exact match

            // We didn't match exactly, so if there is no whitespace, then
            // this element is not a member of the class
            var whitespace = /\s+/;
            if (!whitespace.test(classes)) return false;

            // If we get here, the element is a member of more than one class and
            // we've got to check them individually.
            var c = classes.split(whitespace);  // Split with whitespace delimiter
            for(var i = 0; i < c.length; i++) { // Loop through classes
                if (c[i] == classname) return true;  // and check for matches
            }

            return false;  // None of the classes matched
        }
    }
}

The function fuzzyElementSearch accepts five parameters, and is used to find the element based on it’s relation to an an outer element. In other words, we find the element we’re interested in based on the context of the surrounding HTML code.

So, if we consider the case where we are looking for the element that represents viewset A that we want to hide:

<td class="th-gr-td" valign="top" rowspan="1" colspan="1">
<div id="C11_W43_V44_V48" excevt=""
intevt="c:C11_W43_V44_V48:C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do;" 
automode="true" tgt="" dhe="true" style="display: inline;">
<!-- Begin C11_W43_V44_V48 -->
<div id="C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do" class="th-ajax-area">
<!-- Begin C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do -->
<table ....

The outer element in this case is <td class="th-gr-td" valign="top" rowspan="1" colspan="1">
The inner element (which is what we want in the end) will be <div id="C1_W1_V2_C1_W1_V2_V3_C11_W43_V44_C11_W43_V44_V48_productdetailviewset.do" class="th-ajax-area">

The following five parameters supplied to the fuzzyElementSearch are what the function uses to robustly identify the element we want:-

• classname = the CSS class of the outer element, or th-gr-td in this case.
• tagname = the tag type of the outer element, td in this case
• root = the root element from which we start searching. We’ll search the entire document so can leave this as null
• tag2 = the tag type of the inner element, or div in this case
• subbie = substring (actually a regular expression string) that will match the id of the inner element
• level = number of levels down from outer tag to inner tag. In the above case, we are interested in the 2nd div tag nested inside the td tag. Since javascript counting starts at 0 the level we are interested in is level 1.

Putting it all together, the following call to the fuzzyElementSearch gives us the inner element we want every time (it will be the first item in the array returned):

var hide     = fuzzyElementSearch("th-gr-td","td", null, "div","productdetailviewset", 1)[0];

How does fuzzyElementSearch work? It makes use of a function getElements that is from David Flanagan’s book “Javascript” (ISBN 0-596-10199-6, also known as the Rhino book) – this function allows us to search for elements based on class and/or tag name. With the results of this function similar code is used to find all the children of these elements that have the correct tag and regular expression match. I heartily endorse David Flanagan’s book, and if you are interested in learning javascript then consider visiting the book’s web site.

Yesterday I had to buy a new printer because my old one packed up over the weekend. Since I run Ubuntu/Kubuntu Linux exclusively at home, I needed a printer I could be sure would work in Linux. This means that generally I need to get either HP or Epson printers. I've had both makes of printers before, and they've both served me well, so I have no problem being "limited" to those two.

I picked up an HP Deskjet D2663, a printer similar to my last one, one of the bottom of the range of HP's. I don't need anything fancy, I already have a small photo printer (it came with the camera) and I doubt I'd use the scanner from one of those all-in-ones. The D2663 is a cheap printer, but a decent one.

My wife and I have separate computers, which means that we need a shared printer, so I attach my computer to one of the servers at home and then we both have access to it at the same time. Unfortunately the server is running Hardy, which means it doesn't have the latest driver for this brand new printer.

I went to the HPLIP site, and downloaded the tarball, and looked for a PPD file. I found the HP Deskjet D2600 series PPD, but when I loaded it, I got an error message that said that the "foomatic-rip-hplip" filter was not found. I opened the PPD file and found two lines that specified that filter. After a bit of googling, I found that I just needed to remove the "-hplip" from the filter name, and it worked!

Yay, so now I have a new printer, working perfectly on my Ubuntu Hardy server.

An oldie, but a goodie, and there is always someone out there who doesn’t know about this. Often, one encounters video files that do not have an index or have a broken index- this is almost certainly the case if your media player or PVR refuses to allow you to fast forward within the file. Easily fixed under Linux using the following mencoder incantation:

mencoder -idx input.avi -ovc copy -oac copy -o output.avi

As a bonus, this apparently also repairs broken interleaving.

Some more great mplayer/mencoder hints, including tricks such as appending multiple AVI files are here.

SuperGenPass and Password Composer are password generators, which generate different passwords for each site you use based on a single master password. This gives you the convenience of only remembering one password as well as the security of using different (and strong) passwords for each site. This means that you won't have all your accounts compromised when¹ one of them is compromised.

Most password generators are implemented as browser extensions or bookmarklets, since they are most frequently needed in a web browser. I've been wanting to start using a password generator, but I wanted to be sure that I could access my accounts even if I didn't have a web browser accessible. The two situations I could think of were a command line only system (e.g. SSH) and my cellphone².

Surprisingly, I couldn't find a command line implementation of SuperGenPass, so I wrote one in Python. I also couldn't find any J2ME or Symbian implementations, and so wrote my own one in J2ME. They both support subdomain stripping and configurable password lengths. They don't support salted passwords.

I chose SuperGenPass over Password Composer because it uses a better scheme. Password Composer only uses hex characters, whereas SuperGenPass uses a base64 encoded hash. SuperGenPass also hashes the password multiple times (which would slow down a brute force attack to find the master password) and imposes complexity requirements on the generated password (which reduces the chances that the generated password can be brute forced).

URLs shouldn't really contain file extensions (like .html, .png) since they are supposed to identify a resource and not a particular representation/format thereof. The format is indicated by the Content-Type header sent in the response. Modern CMSs do this already (for example, the URL of this page doesn't include .html).

Doing the same for static files (i.e. files served directly by the webserver) isn't straightforward because most webservers use the file extension to determine the MIME type to send in the Content-Type header. This means that simply removing the file extension from the filename (or even creating a symlink without a file extension) will cause the webserver to send the wrong Content-Type header.

I decided to try find a solution to this for my webserver of choice, Lighttpd. Lighttpd has a module which embeds a Lua interpreter and allows you to write scripts which modify (or even handle) requests. So I wrote a script which searches the directory for files with the same name as requested but with an extension. This means that any file can be accessed with the file extension removed from the URL while still having the correct Content-Type.

The script currently chooses the first matching file, which means that having multiple files with the same name but different extensions doesn't do anything useful. The proper method however is to actually do content negotiation, which chooses the format based on the preferences indicated by the HTTP client in the Accept header.

To use this script, download it and save it somewhere (I use /etc/lighttpd/). Enable mod_magnet, and add the following line to the site definition.

magnet.attract-physical-path-to = ("/etc/lighttpd/extension.lua")

Recently I upgraded my phone to a Nokia N81. I set up my Zyb account, and sync'd my contacts, and realised that I could not schedule syncs like I had with my SE w810i. Until this morning, when Russell pointed out on IRC that there was a tool for that exact thing: Swim is a utility for automatic periodic synchronization of data with internet servers. It is part of the opensource project called Bergemot. The only hurdle is that you have to get the SIS file signed by the Open Signed Online and when you install it you are presented with a warning that your phone may turn into a lump of metal because you are using Software In Development. Works fine for me though.

In a week we are moving houses. We have a list of things that need to be sorted before we move. So, one of the things on the list is moving our phone line/ADSL. I have a one price ADSL from cybersmart so I can take advantage of their Night Rider plan. Little did I know, paying for our ADSL portion from another supplier was something Telkom Just Couldn't Handle during a transfer.

I phoned Telkom's 10219 number, and was told I needed to call a random number to have the transfer done, because there was an ADSL linked to it. So we tried the cybersmart route. Mandy phoned and asked if we could have the line moved. Cybersmart wasn't so sure. So, we tried Telkom again. After some lengthly discussions Telkom came back with the point that the ISP has to initiate the move. So, back to cybersmart. This time cybersmart was helpful, and said I just needed to fax them my details and request, and they could go ahead, provided they had my signature. Cybersmart phoned back a few days later to tell me the fax was quite light, so they couldn't make out much. After chatting to the very helpful cybersmarter, she said she could transfer the ADSL portion back to Telkom, and then I could handle the transfer, and once it was complete Cybersmart could migrate the line back. Convoluted, but Telkom understandable. Took a few days, but today the migration went through, and round 2 started.

I phoned Telkom, and the very helpful call centre person started to sort me out. No charges, just time. I thought I was on the home run. Then I pointed out that the line had an ADSL on it, and I wouldn't mind having the number changed. Apparently it costs R543.23 to transfer an ADSL line between premises. WHAT? But, doing a self-install of ADSL costs R0.00. So, after some protracted negotiations, the path became clear:

Cancel my ADSL portion ("downgrade" my line), transfer to the new premises (and because there is no ADSL associated I can roll in a phone number change), once the line is active in the new house, then I can re-initiate the ADSL portion for free. WOW.

It's like peeling layers off an onion, and then putting the onion back together. Through all of this the call centre people were lank nice and helpful, but the system gets me down.

Just to re-cap, to have ADSL and a phone line moved to my new house (which is about 1km away from my old one) I have to:
Instruct Cybersmart to migrate my ADSL back to Telkom, Ask Telkom to cancel my ADSL, Ask Telkom to move my phone line to my new house, Wait for the new line, Ask Telkom to "upgrade" my line to include ADSL, transfer the ADSL back to Cybersmart. Awesome system guys.

Yesterday I posted about how a local web company recently sold the work of a well known, local, independent web professional. What they did broke netiquette, ethics, and definitely copyright law. Much geek froth and outrage occurred, and a tiny storm broke out on the internet. I jumped on the bandwagon with my post because:

1. I have SIWOTI syndrome
2. Content climbs higher on google when lots of reputable sources link to what is considered to be definitive text. Whijo is considered to be somewhat of a reputable source by google, and because I use the best CMS on the internet google likes reading what I am writing, so I wanted to contribute to improving Coda's rank on google for this subject
3. My goal is to improve the quality of the South African web, and improvement/evolution comes when the economy/environment favours better products, and denounces poor product. I denounced poor product

The only problem is that sometimes success quickly exceeds expectation, and in this case, after google crawled whijo.net, my article (then entitled 'Do not use the services of ClickThinking') landed on the first page of results. Coda's much more democratic 'What were you thinking, ClickThinking?' reached higher on the first page of results, as it should have. So, geek-google-penis aside, the weight of what I had done (measured in the internet based ISO standard of LOLCATS, or Lc) struck me. I know it is all a storm in a tea cup, but a post with a title as venomous as that sticks around, and ultimately may take business from them (and I am in no position to decide if their poor form deserves to take business away from them). I had behaved like a Journalist (well, one who didn't do too well in the media and ethics course). I thought up a catchy headline, and published it with a self-congratulatory click. I guess it comes back to thinking before doing, and not being a turd on the internet. So I changed the title, and when the site is re-crawled it will have a new title which is a lot closer to the heart of the matter at hand, and a lot less sensationalist.

Sometimes my powerful Sense For Injustice conspires with my Sense For Bad Web Development, and I peak too soon, type before I think, and end up looking childish, and not accomplishing my goals. I am usually calm and rational, but some things short circuit over that calm, rational, ethical brain. So, in summation, I apologise to ClickThinking for going too far off the handle, but I still deplore what they did.

As an aside, should I really be able to get into the first page of results on google for a company that just won a web analytics award?