If you don’t know what Gearman is, it’s a “generic application framework to farm out work to other machines or processes that are better suited to do the work.”  You have Gearman clients and workers: clients dispatch jobs to a configurable table of gearman servers, which in turn dispatch the job to any idle worker processes.  Clients and workers can be on separate machines; it is the collection of gearman servers that makes decisions about which worker to accept a given task.  This can be very useful when you have a long-running process, such as audio or video processing (or, in my case, the automatic LaTeX-ification of documentation).

As a (utterly trivial and inadequate) example, I’m going to show you how to communicate between a client triggered with Django, a worker that does some work, and then how the message gets back to Django that the process has been run.  This example is trivial because it assumes both processes are on the same machine; it is inadequate because it uses no sychronization to ensure that the message passed back to the Django process isn’t accidentally destroyed by a race condition.

Let’s start with the basics:

virtualenv --no-site-packages geardemo
cd geardemo
source bin/activate
pip install gearman django
django-admin startproject geardemo
cd geardemo
./manage.py startapp testapp
mkdir workers

Here, I’ve created a virtual environment in which we’re going to run our example, installed gearman and django, started a django project, and in that project created our app and a directory for the workers.

What I want to do is create a view in my testapp that dispatches jobs to Gearman. The view will take a single argument from its web page and pass that to Gearman. I also want to pass my session key, because I’ll be using the session object to receive notice that the process is done.  In testapp/views.py:

from django.shortcuts import render_to_response
from django.template import RequestContext
from gearman import GearmanClient, Task

from django.conf import settings

try:
    import cPickle as pickle
except ImportError:
    import pickle

def run(request):
    if request.method == "POST":
        jobname = request.POST.get('name')

        if jobname.strip():
            client = GearmanClient(["127.0.0.1"])
            req = request.COOKIES.get(settings.SESSION_COOKIE_NAME, None)
            arg = (jobname, req)
            pik = pickle.dumps(arg, pickle.HIGHEST_PROTOCOL)
            res = client.dispatch_background_task("work", pik)

        status = request.session.get('worker.status', [])
        return render_to_response('view.html', { 'status': status },
            context_instance=RequestContext(request))

The tricky part, for me at least, was figuring out how to pass multiple objects to the worker process. They have to be pickled; the gearmand servers are written in C and will blindly pass any stream of bytes from the clients to the workers, so to pass multiple objects, they must be serialized into a string using pickle.dumps before handing them to gearman.

The worker process has its own mysteries.  In the client, I get the status from the session object, which means that I have to populate the session object within the worker.  This is inadequate because there’s no synchronization process at work; for example, the django process could read the status object, the worker process could read then write, then the django process could write, and erase the worker process’s changes.  To do this, I pass the session key to the worker in the client request above.  This is likewise inadequate because there’s no guarantee the worker and the client are on the same system– but this will work if you’re using memcached as your session store.  In fact, I recommend using memcached and a smarter synchronization process, one perhaps with atomic increments, independent of sessions, for this kind of communication.

Here’s the file workers/worker.py:

from django.core.management import setup_environ
from django.utils.importlib import import_module
from gearman import GearmanWorker
import time
import sys
import os.path

try:
    import cPickle as pickle
except ImportError:
    import pickle

sys.path.insert(0, os.path.realpath(os.path.join(os.path.dirname(__file__), '..')))
import settings

setup_environ(settings)
from django.conf import settings
engine = import_module(settings.SESSION_ENGINE)

def dothework(gjob):
    job, session_key = pickle.loads(gjob.arg)
    for i in xrange(0, 6):
        time.sleep(1)

    if session_key:
        session = engine.SessionStore(session_key)
        status = session.get('worker.status', [])
        status.append('done with %s' % job)
        session['worker.status'] = status
        session.save()

    return True

worker = GearmanWorker(['127.0.0.1'])
worker.register_function("work", dothework
worker.work()

There’s a certain amount of rigamorale in importing the Django session object.  Using sys.path to put settings.py within our papth, importing the session object, setting up the environment, then re-importing the django settings object gives us access to the SESSION_ENGINE, and just like the client I need pickle to get at the argument and the session key passed from the django process.    This process merely sleeps for six seconds, then writes back to the user’s session that his process is done, with the word passed in from the client.

Note that both the client and the server use a token, “work”, to communicate which function they want run at the worker’s end.

One last file. Here’s view.html, the template for our example:



<h1>Test</h1>
<p>Current Status: </p>
{% if not status %}
<p>No recent status updates</p>
{% else %}
<ul>
{% for s in status %}
<li>{{ s }}</li>
{% endfor %}
</ul>
{% endif %}

<hr>
<form action="." method="POST">
{% csrf_token %}
<label for="name">Job Name: <input type="text" name="name"></label>
<input type="submit" value="submit">
</form>

Very simple and straightforward.  We’re not even using forms.  Just if there’s status then show it, and ask for another job to do.

To run this:

./manage syncdb
gearmand -d
cd workers
python workers.py &
cd ..
./manage runserver

If you did everything right, you can now browse to port 8000 on your server and get the view above. Pass it job names, and six seconds after you do, you’ll get a response. If you stack the jobs, the later ones will take longer because Gearman is running them in a serial queue, not in parallel.

And that’s it.

Full source code to the geardemo program is available.  You will still have to set up the virtualenv on your own.

So, my first foray into times and chimes is done.  I used some free bamboo photoshop brushes, and jQuery, but the HTML, CSS, graphics, layout and so forth is all mine.  The script is 100% HTML5, there’s a lot of CSS3, and it uses HTML5 native audio, so this thing doesn’t work at all on IE (unless you’re running IE9; I don’t know what the capabilities of IE9 are yet).  Basically it works on latest & greatest: Chrome, Safari 3, Firefox 3.6 (maybe 3.5?) and so forth.  The whole project took approximately 10 hours of development time.

Nine Amazing Hours

Critiques, comments, suggestions are all welcome.  The functionality is what I want it to be; if you want a full-fledged meditation timer well, maybe I’ll do that next.

And one beg: The idea behind these projects is to learn how to, as my design teachers put it, “design, develop, market and maintain a project.”  I have no idea how to market something like this.  How would I go about telling people about it?

How annoying.  Well, it’s a simple CSS setting to make it work as expected.

(Edit: I see, as always, A List Apart is way ahead of me on this.  By almost a year!)

First, go and create a key pair for your site. You don’t even have to give them an email address, which is nice.

Install the recaptcha client library on your site:

pip install recaptcha-client

You’ll have to override or replace any registration templates you have, and add this to the form, somewhere in the form (usually, right above the submit button):

<p>Please be a human, and not some spamming robot:</p>

<script type="text/javascript"
    src="http://www.google.com/recaptcha/api/challenge?k=YOUR_RECAPTCHA_PUBLIC_KEY{{ captcha_error }}">
</script>

<noscript>
    <iframe src="http://www.google.com/recaptcha/api/noscript?k=YOUR_RECAPTCHA_PUBLIC_KEY{{ captcha_error }}"
        height="300" width="500" frameborder="0"></iframe><br>
    <textarea name="recaptcha_challenge_field" rows="3" cols="40">
    </textarea>
    <input type="hidden" name="recaptcha_response_field"
        value="manual_challenge">
</noscript>

Add your private key to your settings.py file, and code your “accounts.view.create” this way:

def create(request, template_name='accounts/create.html',
    redirect_field_name=REDIRECT_FIELD_NAME):

    user_form = None
    captcha_error = ""
    redirect_to = request.REQUEST.get(redirect_field_name, '')

    if request.method == "POST":
        captcha_response = captcha.submit(
            request.POST.get("recaptcha_challenge_field", None),
            request.POST.get("recaptcha_response_field", None),
            settings.RECAPTCHA_PRIVATE_KEY,
            request.META.get("REMOTE_ADDR", None))

        if not captcha_response.is_valid:
            captcha_error = "&error=%s" % captcha_response.error_code
        else:
            # perform other registration checks as needed...
            # success!
            return HttpResponseRedirect(redirect_to)

    if not user_form:
        user_form = UserForm(prefix="user")

    return render_to_response(template_name, {
        'captcha_error': captcha_error,
        'user_form': user_form},
        context_instance=RequestContext(request))

And that’s it. You have ReCaptcha enabled. I see that the python library includes an HTML generator, but it’s for recaptcha.net, and I decided to use the newer google addresses.

By the way, I’m not sure why, but I much prefer the form = None sentinel method of checking for form initialization. I think it’s a lot cleaner than a metric ton of else statements.

I’m going to start with a piece of Django code that someone else wrote: Django Activity Stream, a simple piece of code that lets you track “everything” the actors in your system do: every bookmark made, every comment made, every story read in my current publishing system. If you’re an author, you want to track when your stories are read, and when comments are made on your stories. If you’re a reviewer, you want to know when someone replies to your review, or gives you a thumbs up. And so forth an so on.

The trouble with all this is that, in order for it to work, you would have to touch all of your different modules with function calls like action.send(), which would just clutter every piece of code with knowledge about this logging facility. Not cool. The activity stream is orthagonal to your business logic, but critical to your goal: this is known as a cross-cutting concern. Having to touch all of those Django applications in order to get logging becomes known as tangling.

Django provides an easier way of monitoring its own internal activity, called signals. A “Signal” is a specialized object that can publish when a certain event has occurred, and then other systems can subscribe to receive the signal when the event occurs.  

Django provides a comprehensive collection of signals associated with the database, and since most Django applications are MVC, when the model is altered a database event occurs, and we can easily intercept those and use them to our advantage.   The most common signal used is post_save, which is emitted whenever anything saves to the database.   Here’s an example that listens for the creation of a FacebookUser (which also creates a standard django User as well), and guarantees that a Profile for the user will also be generated:

from facebookuser.models import FacebookUser
from django.db.models.signals import post_save

def set_up_profile_for_remote_user(sender, **kwargs):
    if not kwargs['created']:
        return
    Profile(user = kwargs['instance'].user).save()

post_save.connect(set_up_profile_for_remote_user, sender=FacebookUser)

The post_save.connect method says “Listen for database saves, and if the sending object is a FacebookUser, call the function set_up_profile_for_remote_user.” The callback gets three arguments: the calling class (FacebookUser), the specific instance that was just saved, and a flag indicating whether or not that class was just created, or merely modified. The last two are sent as named arguments, so most signals use the kwargs syntax.

When used with the activity stream (as with logging), the database is emitting all of these events. The activity stream has a method for capturing what object emitted the event, a “verb” associated with that event, and an optional target object. All we need to do is capture the various database events, associate them with actions, and route them to the stream. For this, I have my various event generators, I have the activity stream, and I have a small project-specific application I name “hub,” into which I just put the signal routers:

def record_comment(sender, **kwargs):
    if not kwargs['created']:
        return
    comment = kwargs['instance']
    action.send(sender = comment.user, verb = 'commented',
                target = comment, public = True)

post_save.connect(record_comment, sender = Comment)

By using a simple application silo like “hub,” neither the activity stream application, nor any of the various applications being monitored, need know anything about one another, yet the cross-cutting concerns of the activity stream can be fully addressed.  This ease of dealing with such concerns is the cornerstone of Aspect-oriented programming, and is a good way of easing into an understanding of AOP.

A couple of caveats: One, Django signals are synchronous. In these examples, immediately after the database write ends the signal receivers are called sequentially within the thread of the original request event. Any signal handler that takes a significant time to resolve ought to be kicked out to a queue manager. (This fact seems to surprise a lot of people when they encounter it.) Two, the order in which callbacks are called for a given signal cannot be set by the developer. If any given signal has multiple subscribers, do not rely on the order of those subscribers to handle the signal.

There are quite a few Django signals covering pre- and post-saving, deleting, and HTTP request management, and it’s possible that you can always write your own if you feel your application is doing something unique enough the standard set won’t work for you, but must publish so other applications can listen in and get signaled as needed. But for the most part, the standard set provides you with awesome capabilities that should get you on your way.

p.s. After a while handling signals, you may notice that the callback record_comment calls action.send, which is itself a signal publisher, the generic “add an action to the Action table” handler. The idea behind activity stream is that you would have lots of publishers of actions, and only one subscriber listening for these events. By further centralizing the activity stream, we avoid the kind of clutter seen in Pinax, which is riddled with “optional” calls to the notification application.  Take this example from the microblogging application:

try:
    from notification import models as notification
except ImportError:
    notification = None
...
    if notification:
        notification.send([reply_recipient], "tweet_reply_received", {'tweet': instance,})

I learned this today because I followed some links off an article about George Carlin that were poorly coded.  Those links took me to Salon’s 404 page, which uses Salon’s standard layout, complete with ads. That is a major faux-pax; when the user is lost, the last thing he or she wants is to be sold to.  I understand that Salon needs to make money and ad impressions are its chief vehicle for doing so, but the add blare was such a major turn-off I almost gave up struggling to find what I wanted.

The 404 is incredibly unhelpful.  So are most in the galleries you might find on the Internet.  A 404 ought to make a positively heroic effort to decode what you were trying to do and help you find it.  If you recently re-arranged your website, hopefully the arrangement has some sort of algorithmic relationship: “pages named x-y-z.html can now be found in y/z/x.html,” for example.  Helping people find that relationship for themselves and understand your new layout is far better than just saying “Er, you screwed up.”

PS,  It’s “We screwed up.” Never blame the user.

As a trivial example, I’m going to add your website’s search engine to Firefox’s drop-down of search engines. I’m going to assume that you already have a search engine enabled on Django. Haystack, or Djapian, or Merquery, or something along those lines.

First, you need a file called opensearch.xml. I put this into my template base directory:

<?xml version="1.0" encoding="UTF-8" ?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
  <ShortName>Search My Website</ShortName>
  <Description>Search for my brilliance.</Description>
  <Image width="16" height="16" type="image/png">http://mywebsite.com{{ MEDIA_URL }}img/favicon.png</Image>
  <Url type="text/html" template="http://mywebsite.com/search/?q={searchTerms}"/>
</OpenSearchDescription>

Obviously, there are things to modify here: your real short name, description, image and search paths.   You know where those go.

This goes into your base urls.py (if you haven’t imported direct_to_template, now is the right time):

url(r'^opensearch.xml$', direct_to_template,
    { 'template': 'opensearch.xml',
      'mimetype': 'application/opensearchdescription+xml' },
    name="opensearch"),

Again, if you changed the names or locations of the template, you have no one but yourself to blame if stuff blows up.   The name opensearch.xml is not particularly important or required, in either the template (obviously) or in the deployed URL.   To make external spiders and browsers know about your website, this goes into your base.html or site_base.html, somewhere in the headers with your metainformation and style sheets and so forth.

<link rel="search" type="application/opensearchdescription+xml"          
    title="Indieflix Search"           
    href="{% url opensearch %}" />

And that’s it.

To test this, load up your home page in Firefox, then click on the search bar’s drop-down. You should see your “short name” offered as a search plug-in. Select it and type some search terms, and Firefox will know to use your search engine, and the results will be from your site.

Pretty cool.

Nifty.  I kept noticing, however, an odd trend: programmers attaching potentially large querysets to the context.  Take, for example, the bookmarks app: it has a context processor that connects a user’s bookmarks to the context, so when time comes to process the request if the programmer wants the list of the user’s bookmarks, there it is.

It took me three days to realize that this is not wasteful.  It says so right there in the goddamn manual: QuerySets are lazy — the act of creating a QuerySet doesn’t involve any database activity. So building up the queryset doesn’t trigger a query until you actually need the first item of the queryset.  It just sits there, a placeholder for the request, until you invoke it in, say, a template.  Which means that you can attach all manner of readers to your contexts, and they won’t do a database hit or other I/O until you drop a for loop into a template somewhere. I’ve been avoiding this technique for no good reason.

This also means that pure “display this list and provide links to controlling pages, but provide no controls of your own” pages are pure templates that can be wrapped in generic views.

Ah!  And this is why I didn’t get why generic views were such a big deal.  Information is often so context sensitive so how could a generic view possibly be useful?  The examples in the book are all tied to modifying the local context in urls.py, but that’s not really where the action is.  The action is in the context processors, which are building that context.

I feel like Yoda’s about to walk in and say, “Good, but much to learn you still have, padawan.”

1. Create a new canvas, 600 x 335, background white. Add two new guides (Image -> Guides -> New Guide), one at 285px and one at 310px.

2. Create two new layers, 600×25 pixels. Label them “upper bar” and “lower bar” (if you can’t figure out which is which, well…), and put them into the 25-pixel high gaps created by the guides.

3. Fill the top bar with a vertical gradient, light on top, dark on the bottom, #787b7d to #35393d. Fill the bottom bar with a vertical gradient, light on top, dark on bottom, #0c0c0c to #000000. Set the bars to 90% opacity, normal over the white background.

4. Draw a new line across the very top of the top bar, color #484b4d. To do this, from the top bar, select a 2-pixel high area at the very top of the bar (from the guide to guide +2) stretching the entire width of the area. Copy, paste as floating selection, then click “new layer” in the new layer dialog to make this its own layer. Fill with color. Duplicate this line and put it right under the first line; fill with color #9fa2a4.

5. Create a new layer, 2 pixels wide and 19 pixels high. This will be the top half of a separator bar for your menu. Place it so the bottom of this tiny layer rests on the second guide. Fill it with a gradient, #4d6672 #676a6d top to bottom.

Create a similar layer, 2 pixels wide and 21 pixels high. This will be the top half of the separator. Place it right under the other half, and just fill it with #43474b. You can merge this layer down with the one above, and just have a “divider.”

At this point, it might be wise to delete the guides. I found they weren’t helpful, and often misplaced the divider.

Clone the divider (Layer Dialog, duplicate button on the bottom) and place it at regular intervals, to make the menu. Fill the space between with text, color #ffffff.

6. Pick a background. I used Forest on a Foggy Day (CC Licensed), but you can surely find your own favorites from around the net. Crop and scale it to the size of our demo, and place it as the background layer. (You may choose to delete, or merely cover, the original white background.)

7. Now for the glassy see-through “Vista” effect. Click on the background layer. With the select tool, select a rectangle about 200×160, starting at 105px down and (here’s the tricky part) -15 pixels left. Select “rounded corners”, 15px radius. Copy, then Paste As New Image.

With the new image, do a Gaussian Blur, 10px radius, then up the canvas size by 10 pixels (Image -> Canvas Size), and using the same dialog center the image on the new canvas. Then, with the image, add a drop shadow (Layer -> Layer Effects -> Drop Shadow, Opacity 60%, Angle 135°, Distance 2, Spread 5, Layer knocks out drop shadow: Yes, Merge layers: Yes). Crop the image to its essentians (Image -> Autocrop).

Copy the new image, then go back to the other image and paste it. If you did it right, you left the old selection active, and the new selection will drop into place. If not, then you should paste it as a new layer and position it by hand at 0×105 (The part to the left was cut off in transit to the new image, remember?).

8. Add text to your glassy portion, and user the Layer Drop Shadow as described above on the words. You may have to mess with it a bit to get it right, but the angle must match the angle of the glass, or it won’t look right.

The free template for this Photoshop to GIMP experiment is available: Vista Style.

• How to lose weight
• Health products – Vitamins, minerals, colon blow
• Starting a home-based business
• How to make money from your existing business
• How to market on the Internet
• How to make money through investments
• Personal improvement
• How to turn a hobby into a business
• How to move to a new state/country
• How to practice your religion

I’m not sure why, but I find this list incredibly sad. There are so many decent sites out there that give this information away, and most of what’s behind the paywalls is pure bullshit not worth the pixels it’s drawn with.

This just confirms what one commenter said earlier: most people don’t want a well-made website. Well-made websites scare people into thinking there’s trickery, deceit, and too much intelligence behind them for the individual to keep up with. Stupidity sells.