One thing I see a lot of in professional Django is the importation of ugettext, Django’s internationalization library that leverages the GNU gettext project’s toolkit for generating translation catalogs. Laying the groundwork for translation is important to larger projects intended for universal appeal. Because the underscore is a valid leading character in a function name in both C and Python, it’s also a valid function name in its own right, and gettext recognizes that single function, _(), as the wrapper for a string to be recorded in the translation catalog, and for that string to be the “key” for translations into other languanges.

However, it gets a little old, after a while, typing in the same “import-as” string in module after module. So I decided, to heck with it, I’m just going to make the gettext function (or in Django’s case, ugettext) absolutely everywhere:

<__init.py__>=
from django.utils.translation import ugettext
import __builtin__
__builtin__.__dict__['_'] = ugettext

Put this into the __init.py__ file in the root of your project (the same directory level as urls.py and settings.py) and this installs _() as a global reference into the current running python VM, and now it’s as universally available as int(), map(), or str().

This is, of course, controversial.  Modifying the python global namespace to add a function can be considered maintenance-hostile.  But the gettext feature is so universal– at least to me– that __init__.py is where it belongs.

• 0 Comments

At del.icio.us (and some other sites, but I’ll use del.icio.us as my example), one of the most interesting features is that your username is one of the first-level “commands” you can send to the application: “http://del.icio.us/elfsternberg” is a valid URL, and points to my collection of bookmarks.

At the same, there are other first-level commands you might send to del.icio.us, including /save, /tag, /user, and /network, and /url. In Django, all of these commands would be rooted in url.py.

I’m going to show you a few very simple ways to make the magic of del.icio.us work in Django, and (most importantly) show how to validate new users at registration time to make sure that their usernames do not mask, and are not masked by, your otherwise RESTful URLs.

Using the Resolver to ensure uniqueness

I’m going to assume you know enough about Django to set up a demonstration sever. If you don’t, please consult the Django documentation at the Django Project.

The basic premise here is that you have some basic RESTFUL paths, and then you have your username-based paths. Let’s assume you’re using django-registration, since the real difficulty here is ensuring no invalid slugs get past your registration engine. For my purposes, the project is named “demonstration” and the application “demo”. Here is what you add to urls.py

<urls.py>=
from django.conf.urls.defaults import *
from django.contrib.auth.views import *
from django.core.urlresolvers import reverse

# Just two pages to show what we mean.
from demo.index import intro, index

# This is explained below.
from demo.forms import NewRegistrationForm

urlpatterns = patterns(",
    # Ordinary URLS understood by the system.
    url(r'^$', index, name="index"),
    url(r'^index/$', index, name="index"),
    url(r'^intro/$', intro, name="intro"),

    # Override the registration application's register method to
    # include a registration form with a URL checker.
    url(r'^accounts/register/$',
        'registration.views.register',
        { 'form_class': NewRegistrationForm },
        name = 'registration_register'),

    # URLs driven by the bookmarking application
    url(r'^(?P<user_slug>\w+)/$', user, name="user"),
    url(r'^(?P<user_slug>\w+)/(?P<object_slug>\d+)$',
        bookmark, name="addnew"),

The big keys here are the NewRegistrationForm and the URLs in which the user_slug (if you don’t know what a slug is, you can consult the Django glossary at , but the short form is this: a slug is a URL-ready string version of a label, all-lowercase, alphanumeric only, with spaces either eliminated or replaced with hyphens. If my name is “Elf Sternberg”, typical slugs would be elfsternberg or elf-sternberg. Using slugs instead of object IDs has become popular as a way of assisting with mnemonic URLs. For more, see “Cool URLs don’t change“) is exploited to find the “hub” view of the user’s experience.  Different views are provided at the same URL for those who are logged in and viewing their own page, and those who are viewing pages belonging to someone other than themselves.

Writing the user and bookmark slugs should be trivially easy. The question here becomes, how do you prevent users from choosing a name like “index” or “intro,” names that would be masked by the command syntax, since Django resolves URLs in the order in which URL patterns appear in the urls.py file

Above, in urls.py, I mentioned that we had imported NewRegistrationForm, a newly defined object within the application. I also told django-registration to use NewRegistrationForm as the form to show when going to the registration page.

NewRegistrationForm inherits from the base registration form and adds another layer of validation: confirm for me that the username being added would not conflict with an existing URL that already resolves within our application. To do that, we just use the resolve function provided by Django, by defining our own forms file under the demo/ directory:

<forms.py>=
from registration.forms import RegistrationForm
from django.forms import ValidationError
from django.core.urlresolvers import resolve, Resolver404
from urlparse import urlparse

class NewRegistrationForm(RegistrationForm):

    # Ensures than any usernames added will not
    # conflict with existing commands.

 def clean_username(self):
        username = super(NewRegistrationForm, self).clean_username()
        try:    resolve(urlparse('/' + username + '/')[2])
        except Resolver404, e:
            return username

        raise ValidationError(_(u'This username does not create '
                                u'a valid URL.  Please choose '
                                u'another'))

After importing all the necessary tools, I create a new class and override the method clean_username, which first invokes the parent class machinery for validating the username, and then attempts to resolve that username against the existing application. If resolve throws the Resolver404 exception, then we know that the username did not resolve, and is therefore valid to use! Otherwise, we raise a validation error. I always assume the presence of a gettext handler (the underscore function) and the use of unicode inside the application.

A couple of style notes: I assume gettext is installed as the underscore function, and I enjoy exploiting Python’s behavior of concatening adjacent strings into a single expression.  That’s what the text in the exception string is doing.

Big Fat Caveat

Make sure you have a very clear vision of your product’s future evolution, and that all of the commands you anticipate providing in the future have been allocated or reserved before you let your users have their hands on the system. Users are clever and will reserve usernames that might mask commands that you’ll want. Your only alternatives are to negotiate with the user for a change of name, or get out the thesaurus. Neither is terrific fun.

• 0 Comments

Today’s lesson: Always use the proper tool. I wrestled for an hour yesterday trying to the the Gimp to do “text along a curve.” It never quite came out right. The few times I got it close, the text looked awful.

This morning I tried a different tool: Inkscape, which is designed to handle vectors and curves. It looked great the first time. Well, it looked right.  Now I have to fiddle with it to get the text just right, and then fiddle some more.  I might be learning vector illustration by the time this is over, which was planned for the next project, not this one.

• 1 Comment

I got an email today from one of the super secret stealth start-ups, the one I really wanted, and they told me that they had chosen to go with another candidate. Bummer.

At one recent interview, an engineer asked me to “give him the sum of the two highest digits in an integer array. Any language.” I thought for a moment, then wrote:

array.sort()
return array[-1] + array[-2]

He said that was fine, but since it would be O(n*log_n), could I do it in O(n). He wanted a linear search instead.

After answering the question to his satisfaction, we got into a bit of a religious discussion about the merits of concentrating on performance during initial composition, versus concentrating on correctness. I’m much more a “get it working, then get it right, then profile the working program for bottlenecks and work them out.” It is possible that you can create systemically slow solutions that’ll require a re-write, but that’s actually far less likely than having I/O or CPU bottlenecks in small functions of code that can be fixed with optimization.

He later told me that, although I didn’t get the job, he voted to hire me. He liked my answers and appreciated that I was willing to stand my ground and discuss the relative merits of different approaches.

• 0 Comments

Introduction

When you roll around the web you see lots of web applications that show off some localized capability to get data from a back-end in a semi-real-time manner. The most famous is the “search suggestion” drop-down that comes off Google or Bing, and those are wonderful, but often the Javascript for the drop down and handling the div and layer and all that stuff obscures one interesting technology web developers have to master: the round-trip: as events (keypress, mousemove, timeout) happen within the browser’s javascript virtual machine, something is sent via AJAX to the server, and something is then sent back (usually via JSON) to the browser.

So I’m going to tell you how to do a round-trip. We’ll be using Django and Prototype, but this technique works with all application servers and javascript frameworks.

Setting up Django

The first thing you need to do is update settings.py. Here are the changes you make. You’ll note that we’re not doing anything with the database. We are, however, setting up a media directory for static service. You will also note the STATIC_DIR_ROOT trick; please do not deploy with that active. Use it only for development.

<settings.py>=
import os
DIRNAME = os.path.abspath(os.path.normpath(os.path.dirname(__file__)))
STATIC_DOC_ROOT = '%s/' % os.path.normpath(os.path.join(DIRNAME, 'media'))
TEMPLATE_DIRS = (
    '%s/' % os.path.normpath(os.path.join(DIRNAME, 'templates')),
)
ROOT_URLCONF = 'roundtrip.urls'

And make these changes to urls.py. There are a lot of tricks here, including the direct_to_template and static_serve methods.

<urls.py>=
from django.views.generic.simple import direct_to_template
from django.views.static import serve as static_serve
from settings import STATIC_DOC_ROOT
from rtdemo import keypress

urlpatterns = patterns('',
    url(r'^/$', direct_to_template, {template: 'page.html'}),
    url(r'^keypress/$', keypress),
    url(r'^static/(?P<path>.*)$', static_serve,
                {'document_root': STATIC_DOC_ROOT, 'show_indexes': True}),
)

The container page

Now, you need page.html. This is going to be our raw display page, and we’re not going to do anything miraculous at all. What we will be doing is importing two pieces of javascript: prototype.js and our own code, roundtrip.js. Roundtrip will contain the heart of our code.

This page also contains a textarea for the input area and a div for output area. I’ve embedded the styles just to reduce the number of files this examples needs, but in practice don’t embed styles; use style sheets. This page is utterly unremarkable, and without almost any styling at all otherwise.

<page.html>=
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
    "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
  <meta http-equiv="content-type" content="text/html; charset=us-ascii" />
  <link rel="stylesheet" href="/static/style.css" type="text/css" />
  <title>Immunity Inc</title>
</head>
<body>

<textarea id="inputarea" style="width: 30em; height: 12ex"></textarea>

<div id="outputarea"
   style="width: 30em; height: 12ex; padding: 1em; border: 1px solid green;">

</div>

<script
  type="text/javascript"
  src="/static/prototype.js">
</script>
<script
  type="text/javascript"
  src="/static/roundtrip.js">
</script>
</body>
</html>

You might have noticed that I put the javascript at the bottom.  This is becoming a pretty standard way of doing stuff; it ensures that all of the HTML content is delivered to the browser before some potentially pipe-clogging and page-wrecking javascript starts coming down.  It also delays all javascript processing until the DOM content is fully loaded.   Most modern Javascript ibraries have an ‘onload’ option to delay loading until the DOM is completely ready, but this helps ensure that delayed processing.

The Protoype part

The real magic here is in the javascript, so let’s get started. We’re going to use Prototype’s class handler, but you could do this with an ordinary javascript prototype handler.

<roundtrip.js>=
var Roundtrip = Class.create({
  initialize: function() {
        this.source = $('inputarea');
        this.target = $('outputarea');
        Event.observe(this.source, 'keypress',
                      this.updateoutput.bindAsEventListener(this));
  },

When this object is instantiated, the object now has handles to both our input and output areas, and expects a function called updateoutput that will, well, do the magic. Let’s describe that:

<roundtrip.js>+=
  updateoutput: function(e) {
     var payload = ('content=' + encodeURIComponent(this.source.value));
     new Ajax.Request('/keypress/',
                     { method: 'post',
                       postBody: payload,
                       onSuccess: this.show.bind(this)});
  },

This function reads the source object’s contents, bundles it into a key/value pair ready to send to the server, and then calls Ajax.Request(). The magic here is that we’re now expecting another new method, show(), that will handle what the server says.  By setting the method, I ensure where Django will store the content.

<roundtrip.js>+=

  show: function(response) {
    var resp = eval('(' + response.responseText + ')');
    this.target.innerHTML = resp.content;
  }
});

And that’s it. We’re gonna send the contents of the textarea to the server where it’ll do something, send back an HTML payload as a response, and show that response in the target div.

Now we have to set up this object:

<roundtrip.js>+=
Event.observe(window, 'load', function() {
    var a = new Roundtrip();
}, false);

Nothing remarkable about that.

The Django piece

The only other piece is the response. Up above we’ve defined the script named rtdemo.py, and a method keypress. Our demonstration will, for every keypress, analyze the whole content and return it backwards.

<rtdemo.py>=
from django.http import HttpResponse
from django.utils import simplejson

def keypress(request):
    content = request.POST['content']
    content = content[::-1].replace('\n', '<br>\n')
    return HttpResponse(simplejson.dumps({'content': content}))

And that’s it. Now we have the whole circuit: draw the page that the user will see, fill in the javascript that connects the input area to an event handler (keypress) that sends the content to the server, and connect another event handler (end ajax transaction) that takes the response and fills in the output area, and the server-side handler.

There are lots of things you can do with this technique. And there are lots of improvements that can be made on it: many of the “drop down” handlers will not send every keypress, but only after a suitable amount of time has passed since the last transaction, to prevent overloading the server. This example sends the whole content, but do you really need to send all of it?

Endnote: this example was developed with the excellent Noweb Literate Programming Toolkit, which made writing this all up much, much easier.

• 0 Comments

Grr.  Okay, so here’s the confession:

For the past eight-plus years, I have been working in a sealed greenhouse environment of web development, a veritable Biosphere II of HTML and HTTP.  I’ve been writing a web-based front end for a single product, working with a limited number of graphic designs handed to me by the marketing department, using a limited number of Javascript libraries and exactly one web development platform: Webware.  Now, there’s nothing wrong with Webware.  I like it.  But it’s an archaic design, based on the transaction model first published with WebObjects back in the mid-1990s.  I have had some opportunities in that time to work with other tools, deploying several PHP and Rails applications: the Journal Entries publishing toolkit is Django, these blogs are Wordpress,  I’ve done two freelance gigs involving Rails and Django,  I’ve done three “interactive” brochures in Wordpress and Joomla, and one wholly static brochure in plain ol’ HTML.  We did some in-depth analysis of what it would take to move off Webware to Django, and the results were sobering, but I’d actually made a significant first stab at creating a Django dispatcher that would correctly and succinctly manage a Webware applications, by analyzing the Django transaction model and comparing it to the Webware Emulation Layer in Pylons.

Compared to my Webware experience, everything is pretty shallow.  In the past nine weeks, I’ve been sucking down web development and design at a rate I can’t recall.  My brain hurts.  For this week, I’ve been working on a freelance job which has been an absolute blast, as it combines two of my favorite things: event-driven Javascript with realtime customer-facing updates, and an obscure programming language (if x86 assembly qualifies as “obscure”).  I did it in Django, which I’d definitely grown to love over the past year, much more so than Rails.

But one thing kept bugging the hell out of me: floats didn’t work well.  If at all.  And now I know why.

A long time ago, for miscellaneous reasons of ideological purity, we decided to go with a Strict DTD in the DOCTYPE header over at Isilon.  Our assignment was to code to IE7, Firefox 2, and Safari.  It turns out that if you don’t put one of those in the header, IE7 interprets some things in an IE6-ish fashion, including floats and margins.

How annoying.  For the first time in my life, I actually have to learn what “Quirks Mode” means. Well, now I know.

The funny thing is, we never had to worry about this at Isilon.  We’d set it once in the heirachy header, and that was that.  That was all we needed.  Even better, it was XHTML pure because we used a python-sided templating engine that made it impossible to write improperly closed HTML.

Live and learn.  One more thing for the developer/designer checklist.

• 0 Comments

There’s one piece of advice SEO mavens always give out that always sets my hair on fire.  It’s this: “Always design your web pages with CSS.  No Tables!”

This advice is pure, unmitigated bullshit.

I just designed a website for a client.  I used CSS for a lot of things.  Surprisingly enough, I used tables for– gasp– tabular data.  A table of classroom assignments.  A table of type/key/value tuples for features of the software the students are writing for each assignment.   Tabular data.  That’s what tables are for.

SEO experts who terrify newcomers into thinking they’re doing something wrong when they use a table– a perfectly useful tool intended for the liquid display of tabular information, the outer perimeter of which can be affixed to whatever grid makes you happy– do nobody any favors.  Be specific.  Say “Don’t use tables for layout.  Learn about semantic design.  Use tables for what they’re good for.”

• 0 Comments

Ruby is just an excuse for people who once loved spaghetti code, who wanted to be “the smartest guy in the room,” and who wanted to be indispensible, to get back at everyone who loved Python and thought that there was finally a decent way to write disciplined code.

Everyone talks about how nice Ruby folks are.  Ruby is vengeance with a smile.  After spending several hours digging through a Rails app trying to figure out where the hell all of these invocations are coming from, where they’re defined, and what they do (the whole “we can arbitrarily redefine typographic symbols!” thing is insane!), I’ve just gotten to the point where I want to growl, “Wanna know how I got these scars?“

• 0 Comments

I’ve been helping a close friend with her website, which is written in rails. She admits that’s a mistake, now, because it shares database tables with a PHP application, and the communication between the two has always less than stellar. I’ve been looking into her current major problem, but while I’ve been at it, I’ve been looking for easy solutions to put her back on an all-PHP track. (This seems to be the right thing to do; PHP is the only language she knows well, and the PHP component is the only thing that works reliably.)

I looked at Symfony as an easy PHP-based route off the rails.  It’s not bad, but getting it to run on my box was a nightmare. Undocumented are the number of things with which PHP must be configured to get Symfony running. For the record, to get the sandbox application to run, you need not just the stock PHP, but PHP with PDO, SimpleXML, and SQLLite. None of those are stock on Gentoo (and apparently not on Ubuntu, either).  Once I had rebuilt PHP (3 times!), it was all working.

But now it’s up and I can play with it. Oh, and I got Drupal running, but I had to use my production box to make it so. Annoying, that.

• 0 Comments

Most people encountering the Ajax components of Django for the first time usually encounter this pairing:

from django.utils import simplejson
...
return HttpResponse(simplejson.dumps(some_python_obj))

For most applications, this is enough.  But I’m especially fond of iterators, generators, functors and closures, dense components that express one thought well: the structure of a tree, or the rows of a database, to be sent to the browser.  The routine dumps() doesn’t understand any of those things, but with a simple addition, you can plug them into your code and be on your way without headache.  Dumps() just calls JSONEncoder(), and JSONEncoder has a routine for extending its functionality.

The routine is to override a method named default() (why “default?” I have no idea) and add the object types and signatures you want to send to the browser.  Normally, this exists for you to provide custom “object to JSON” handlers for your objects, but there’s nothing custom about iterators, generators, functors and closures.  They are native Python objects.

from django.utils.simplejson.encoder import JSONEncoder

class ExtJsonEncoder(JSONEncoder):
    def default(self, c):
        # Handles generators and iterators
        if hasattr(c, '__iter__'):
            return [i for i in c]

        # Handles closures and functors
        if hasattr(c, '__call__'):
            return c()

        return JSONEncoder.default(self, c)

This exploits the OO nature of Python code: everything is an object, and I’m just checking to see if the object passed in has the interface needed to treat it like an iterator or a function.  If it’s an iterator, iterate it; if it’s a function, call it.
Normally I like my code to be sensible, so I devolve this into a function, just like dumps():

from django.utils import simplejson

def json(s, **kw):
    kw.update({'cls': ExtJsonEncoder})
    return simplejson.dumps(s, **kw)

To show this in use, I’ll use the Tag() tree class I’ve been working on this week.  First, here’s the “Django Tree to Dojo Tree JSON” as a functor (a functor is a class that includes a method named __call__() in its interface, and a reference to an object of that class can be called like a function):

class TreeToJson:
    def __init__(self, roots):
        self.roots = roots

    def _index(self, nodes):
        def _item(node):
            t = { 'id': node.id,
                  'name': node.name,
                  'top': node.is_root()}
            if not node.is_leaf():
                t.update({'children': self._index(node.get_children())})
            return t

        return [_item(node) for node in nodes]

    def __call__(self):
        return self._index(self.roots)

print json(TreeToJson(Tag.get_root_nodes())

The class TreeToJson completely encapsulates the process of converting a django-treebeard tree into a dojo tree structure here. As an alternative, here it is as a generator:

def treetojson(roots):
    def _item(node):
        t = { 'id': node.id,
              'name': node.name,
              'top': node.is_root()}
        if not node.is_leaf():
            t.update({'children': _tree(node.get_children())})
        return t

    def _tree(nodes):
        for node in nodes:
            yield _item(node)

    return _tree(roots)

print json(treetojson(Tag.get_root_nodes()))

Again, the function treetojson completely encapsulates the process of converting a django-treebeard tree into a dojo tree.  Both of these work quite nicely, spewing out a Javascript Object ready to be eval’d and processed on the client.

Now, obviously, my implementation of ExtJsonEncoder has its opinions: it prioritizes iterators over functions.  But I’ve been using this implementation with this priority with Webware and EXT-JS (and a handwritten version of simplejson) and I have yet to encounter a case where a class had both __iter__() and __call__() defined or needed the latter prioritized.

• 4 Comments