Introduction

I’ve been playing with Backbone.js, a small but nifty Javascript library that provides a small Model-View-Controller framework where Models can generate events that trigger View changes, and vice versa, along with a Collections models so groups of models can cause view-level events, and a Sync library that provides a basic REST architecture for propagating client-made changes back to the server.

There are a number of good tutorials for Backbone, such as: Meta Cloud, &Yet’s Tutorial, Backbone Mobile (which is written in Coffee), and Backbone and Django. However, a couple of months ago I was attempting to learn Sammy.js, a library very similar to Backbone, and they had a nifty tutorial called The JsonStore.

In the spirit of The JSON Store, I present The Backbone Store.

Literate Program

A note: this article was written with the Literate Programming toolkit Noweb. Where you see something that looks like <this>, it’s a placeholder for code defined elsewhere in the document. Placeholders with an equal sign at the end of them indicate the place where that code is defined. The link (U->) indicates that the code you’re seeing is defined here but used later in the document, and (<-U) indicates it was used earlier but is being defined here.

The Store

The store has three features: A list of products, a product detail page, and a “shopping cart” that does nothing but tally up the number of products total that you might wish to order. The main viewport flips between a list of products and a product detail; the shopping cart quantity tally is always visible.

Let’s start by showing you the HTML that we’re going to be exploiting. As you can see, the shopping cart’s primary display is already present, with zero items showing. DOM ID “main” is empty. We’ll fill it with templated data later.

HTML

<index.html>=
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

        <title>The Backbone Store</title>
        <link rel="stylesheet" href="jsonstore.css" type="text/css" media="screen" charset="utf-8" />

        <product list template>

        <product template>

    </head>
    <body>
        <div id="container">
            <div id="header">
                <h1>
                    The Backbone Store
                </h1>

                <div class="cart-info">
                    My Cart (<span class="cart-items">0</span> items)
                </div>
            </div>

            <div id="main">
            </div>
        </div>
        <script src="jquery-1.4.4.min.js" type="text/javascript"></script>
        <script src="jquery.tmpl.min.js" type="text/javascript"></script>
        <script src="underscore.js" type="text/javascript"></script>
        <script src="backbone.js" type="text/javascript"></script>
        <script src="store.js" type="text/javascript"></script>
    </body>
</html>

This is taken, more or less, straight from The JSON Store. I’ve included one extra thing, aside from jQuery and Backbone, and that’s the jQuery Templates kit. There is also a simplified JSON file that comes in the download; it contains six record albums that the store sells. (Unlike the JSON store, these albums don’t exist; the covers were generated during a round of The Album Cover Game.)

The Program

And here’s the skeleton of the program we’re going to be writing:

<store.js>=
<product models>

<product list view>

<shopping cart models>

<shopping cart view>

<product view>

<application>

Products and Product List View

To start, I have a list of products. The basic product is just a model, with nothing to show for it; the list of products is a Backbone.Collection, with one feature, the comparator, which sorts the albums in order by album title.

<product models>= (<-U)
var Product = Backbone.Model.extend({});

var ProductCollection = Backbone.Collection.extend({
    model: Product,
    comparator: function(item) {
        return item.get('title');
    }
});

The ProductCollection is what we want to show when the user isn’t looking at a specific product. I create a Backbone.View object class ProductListView:

<product list view>= (<-U)
var ProductListView = Backbone.View.extend({
    el: $('#main'),
    indexTemplate: $("#indexTmpl").template(),

    render: function() {
        var sg = this;
        this.el.fadeOut('fast', function() {
            sg.el.empty();
            $.tmpl(sg.indexTemplate, sg.model.toArray()).appendTo(sg.el);
            sg.el.fadeIn('fast');
        });
        return this;
    }

});

Here, we’ve told this view that it’s principle element is the DOM ID “main”, allocated an indexTemplate using the jQuery template complire, and created a render function that fades out “main”, replaces its content with a rendered template, and fades it back in.

The template looks like this:

<product list template>= (<-U)
        <script id="indexTmpl" type="text/x-jquery-tmpl">
        <div class="item">
         <div class="item-image">
             <a href="#item/${cid}"><img src="${attributes.image}" alt="${attributes.title}" /></a>
         </div>
         <div class="item-artist">${attributes.artist}</div>
            <div class="item-title">${attributes.title}</div>
            <div class="item-price">$${attributes.price}</div>
        </div>
        </script>

There’s some Demeter violations going on here, in that I have to know about the attributes of a Backbone model, something that’s normally hidden within the class. But this is good enough for our purposes. The above is a jQuery template, and the \$\{\} syntax is what’s used to dereference variables within a template.

(As an aside, I think that the set and get methods of Backbone.Model are a poor access mechanism. I understand why they’re there, and I can only hope that someday Javascript Getter and Setters become so well-established as to make set and get irrelevant.)

The Shopping Cart

Before I move on to the product view, I want to go over the shopping cart.

A little rocket science here: A Cart contains CartItems. Each “item” represents a quantity of a Product. (I know, that always struck me as odd, but that’s how most online stores do it.) CartItem has an update method that allows you to add more (but not remove any– hey, the Sammy store wasn’t any smarter, and this is For Demonstration Purposes Only), and we use the set method to make sure that a “change” event is triggered.

The Cart, in turn, has a method, getByPid (“Product ID”), which is meant to assist other objects in finding the CartItem associated with a specific product. Here, I’m just using the Backbone default client id. Once we’ve found that item, we can just call update() on it.

<shopping cart models>= (<-U)
var CartItem = Backbone.Model.extend({
    update: function(amount) {
        this.set({'quantity': this.get('quantity') + amount});
    }
});

var Cart = Backbone.Collection.extend({
    model: CartItem,
    getByPid: function(pid) {
        return this.detect(function(obj) { return (obj.get('product').cid == pid); });
    },
});

The cart is represented by a little tag in the upper right-hand corner of the view; it never goes away, and its count is always the total number of Products (not CartItems) ordered. So the CartView needs to update whenever a CartItem is added or updated. And we want a nifty little animation to go with it:

<shopping cart view>= (<-U)
var CartView = Backbone.View.extend({
    el: $('.cart-info'),

    initialize: function() {
        this.model.bind('change', _.bind(this.render, this));
    },

    render: function() {
        var sum = this.model.reduce(function(m, n) { return m + n.get('quantity'); }, 0);
        this.el
            .find('.cart-items').text(sum).end()
            .animate({paddingTop: '30px'})
            .animate({paddingTop: '10px'});
    }
});

A couple of things here: the render is rebound to this to make sure it renders in the context of the view. I found that that was not always happening. Note the use of reduce, a nifty method from underscore.js that allows you to build a result out an array using an anonymous function. This reduce, obviously, sums up the total quantity of items in the cart. Also, jQuery enthusiasts could learn (I certainly did!) from the .find() and .end() methods, which push a child object onto the stack to be modified, and then pop it off after the operation has been applied.

One of the big things this illustrates is that a Backbone.View is not a full-page event; it’s a mini-application for drawing its own little universe, that may be part of a larger universe. It’s entirely possible to have lots of Views on a page.

Also, this cart does not have a template associated with it: we’re changing a single textual item on the page and animating another one that is always present.

The Product Detail View

So now we’re down to the ProductView. This is slightly more complicated. First, let me show you a skeleton of the view, similar to the one we saw for the ProductListView:

<product view>= (<-U)
var ProductView = Backbone.View.extend({
    el: $('#main'),
    itemTemplate: $("#itemTmpl").template(),

<product events>

<product view initialization>

<update product>

<render product>
});

The reason the ProductView is complex is because it’s going to interact with the shopping cart. We need to keep track of the cart. There are two ways of dealing with this: Have the ProductView track down its cart item every time, or keep a reference to an individual track item having found it once. I’m going with the first option.

<product view initialization>= (<-U)
    initialize: function(options) {
        this.cart = options.cart;
    },

Rendering is exactly the same as that for the ProductListView. In fact, it’s so similar, I’m thinking maybe I should have made this an abstract function and mixed it in afteward:

<render product>= (<-U)
    render: function() {
        var sg = this;
        this.el.fadeOut('fast', function() {
            sg.el.empty();
            $.tmpl(sg.itemTemplate, sg.model).appendTo(sg.el);
            sg.el.fadeIn('fast');
        });
        return this;
    }

The template for a ProductView, however, has some interesting qualities:

<product template>= (<-U)
        <script id="itemTmpl" type="text/x-jquery-tmpl">
        <div class="item-detail">
          <div class="item-image"><img src="${attributes.large_image}" alt="${attributes.title}" /></div>
          <div class="item-info">
            <div class="item-artist">${attributes.artist}</div>
            <div class="item-title">${attributes.title}</div>
            <div class="item-price">$${attributes.price}</div>
            <div class="item-form">
              <form action="#/cart" method="post">
                <input type="hidden" name="item_id" value="${cid}" />
                <p>
                  <label>Quantity:</label>
                  <input type="text" size="2" name="quantity" value="1" class="uqf" />
                </p>
                <p><input type="submit" value="Add to Cart" class="uq" /></p>
              </form>
            </div>
            <div class="item-link"><a href="${attributes.url}">Buy this item on Amazon</a></div>
            <div class="back-link"><a href="#">&laquo; Back to Items</a></div>
          </div>
        </div>
        </script>

Note the octothorpe used as the target link for “Home”. I kept thinking an empty link or just “/” would be appropriate, but no, it’s an octothorpe.

Also note that it has a form. (Again, note the Demeter violations.) What we want is to update the shopping cart whenever the user enters a number into the input box and either presses “Add To Cart” or the ENTER button. That gives us our methods: We’re in a view for a specific product; we must see if the customer has a CartItem for that product in the Cart, and add or update it as needed. Like so:

<update product>= (<-U)
    update: function(e) {
        e.preventDefault();
        var cart_item = this.cart.getByPid(this.model.cid);
        if (_.isUndefined(cart_item)) {
            cart_item = new CartItem({product: this.model, quantity: 0});
            this.cart.add(cart_item, {silent: true});
        }
        cart_item.update(parseInt($('.uqf').val()));
    },

    updateOnEnter: function(e) {
        if (e.keyCode == 13) {
            return this.update(e);
        }
    },

But how to do these events get triggered? Go back to the ProductView skeleton above; there’s a placeholder for “product events”, which looks like this:

<product events>= (<-U)
    events: {
        "keypress .uqf" : "updateOnEnter",
        "click .uq"     : "update",
    },

Backbone uses a curious definition of an event with an “event selector”, followed by a target method of the View class. Backbone is also limited about what events can be used here, as the following events cannot be wrapped by jQuery’s delegate method and do not work: “focus”, “blur”, “change”, “submit”, and “reset”.

We preventDefault to keep the traditional meaning of the submit button from triggering. When the CartItem is updated, it triggers a “change” event, and the CartView will update itself automatically. I added the “silent” option to keep the “change” event from triggering twice when adding a new CartItem to the Cart.

The Router

The router is a fairly straightforward component. It’s purpose is to pay attention to the “#hash” portion of your URL and, when it changes, do something. Anything, really. Backbone.History is the event listener for the hash, so it has to be activated after the application. In many ways, a Backbone “Controller” is just a big View with authority over the entire Viewport.

Here’s the skeleton of our router, along with its instantiation:

<application>= (<-U)
var Workspace = Backbone.Controller.extend({
<application variables>

<routes>

<initialization>

<index render call>

<product render call>
});

workspace = new Workspace();
Backbone.history.start();

There are two routes that we want to present: the index (our list of products) and the item (a product detail). So, using Backbone.Controller, we’re going to route the following:

<routes>= (<-U)
    routes: {
        "": "index",
        "item/:id": "item",
    },

There are a few things I want to track: the index view, the individual product views, and the shopping cart.

<application variables>= (<-U)
    _index: null,
    _products: null,
    _cart :null,

Now, we can render the index view:

<index render call>= (<-U)
    index: function() {
        this._index.render();
    },

There are two things left in our workspace, that we haven’t defined. The intialization, and the product render.

Initialization consists of getting our product list, creating a shopping cart to hold “desired” products (and in quantity!), and creating the product list view.

<initialization>= (<-U)
    initialize: function() {
        var ws = this;
        if (this._index === null) {
            $.ajax({
                url: 'data/items.json',
                dataType: 'json',
                data: {},
                success: function(data) {
                    ws._cart = new Cart();
                    new CartView({model: ws._cart});
                    ws._products = new ProductCollection(data);
                    ws._index = new ProductListView({model: ws._products});
                    Backbone.history.loadUrl();
                }
            });
            return this;
        }
        return this;
    },

Here, I load the data, and upon success create a new ProductCollection from the data, a new shopping cart, and a new ProductListView for the product collection. I then call Backbone.history.loadUrl(), which then routes us to the correct view.

Thanks to this, users can bookmark places in your site other than the home page. Yes, the bookmark will be funny and have at least one octothorpe in it, but it will work.

And now I’m down to one last thing. I haven’t defined that product render call in the application controller. The one thing I don’t want to do is have ProductViews for every product, if I don’t need them. So I want to build them as-needed, but keep them, and associate them with the local Product, so they can be recalled whenever we want. The underscore function isUndefined is excellent for this.

<product render call>= (<-U)
    item: function(id) {
        if (_.isUndefined(this._products.getByCid(id)._view)) {
            this._products.getByCid(id)._view = new ProductView({model: this._products.getByCid(id),
                                                                 cart: this._cart});
        }
        this._products.getByCid(id)._view.render();
    }

And that’s it. Put it all together, and you’ve got yourself a working Backbone Store.

This code is available at my github at The Backbone Store.

Index of code references:

Postscript:  Someone pointed out to me that using the toJSON() method that Backbone provides for models and collections would provide better results than the Demeter violations I complained about.  The problem with that is that I’m using the CID as my primary key, and toJSON doesn’t include the CID in the JSON object.  I suppose I could override the toJSON method and add it, or write an inline to decorate the JSON product with the CID, but after all that it doesn’t seem worth the effort.  For more canonical work, though, it’s something to keep in mind.

One of those problems routinely experienced by Django developers is “The Admin page of ten thousand users.” There are plenty of objects in our model that are owned by a specific user, and when we want to create or edit such a list, we’re frequently confronted with an unsorted, difficult-to-search <select> box with a randomized list of users. In this day, we use Chrome (or Firefox 4) on a machine with reasonable memory, so we have no need (yet) to segment that select box into sub-selects, but even with a well-provisioned desktop finding that one user out of a list of thousands can be a chore.

As is often the case with Django, we can work backwards from a generalized display to a more specific display. Here’s the source code I’m going to explain:

from django import forms
from django.contrib import admin
from django.contrib.admin import widgets

from models import *

class UserModelChoiceField(forms.ModelChoiceField):
    def label_from_instance(self, obj):
        return "%s, %s" % (obj.last_name, obj.first_name)

class SubscriptionAdminForm(forms.ModelForm):
    user = UserModelChoiceField(
        queryset = User.objects.order_by('last_name', 'first_name'))
    book = forms.ModelChoiceField(
        queryset = Book.objects.order_by('title'))
    purchase_date = forms.SplitDateTimeField(
        widget = widgets.AdminSplitDateTime)
    class Meta:
        model = Subscription

class SubscriptionAdmin(admin.ModelAdmin):
    search_fields = ['user__last_name', 'user__first_name', 'book__title']
    form = SubscriptionAdminForm

admin.site.register(Subscription, SubscriptionAdmin)

Starting at the bottom, I register the Subscription object with a ModelAdmin to the admin system. I override the standard ModelAdmin with two options. The first specifies the order in which objects on the Subscription list page will be searched when the user searches for anything. This helps find a specific subscription. Once you’ve found the subscription you want to manage, you go to the Subscription detail page.

It’s rare (but possible) that the administrator wants to change which user owns a specific subscription, but it’s more common that an administrator may want to comp a subscription to a user. First you have to find that user. To do that, you can let the Admin create its own generic form, but that form will be populated with users ordered by the default order specified in Auth (and there is no default order, so the order is deterministic, but useless to you and me).

Instead, I create a SubscriptionAdminForm, and tell the SubscriptionAdmin to use it instead. In that form, I first specify that this form is a ModelForm for model Subscription. “book” is made into a ModelChoiceField, but I manually specify an order for the queryset. “purchase_date” uses the SplitDateTimeField, but I use the Django admin’s AdminSplitDateTime widget, to maintain visual consistency with the rest of the admin. For the User, I could start with a standard ModelChoiceField, but that would make the labels come out “first_name last_name,” which is not search-ready via standard key searches. You’re likely to have a lot of “Roberts” scattered throughout the system, but only a few “Smiths,” and it would be nice if they were all near each other.

So I override the ModelChoiceField and change the label_from_instance() method to return something other that the standard representation; in this case, I want the format “last name, first name.”

And there you go: Books ordered by title, users ordered by Last Name, First Name (and displayed that way), and the whole thing displayed in a Django Admin friendly way.

Little bits of pieces of this stuff can be found scattered throughout the Django Book, the Django docs, and the source code, but this blog entry provides a concise summary of the basic features you might want in a customized Admin form. Consider this as a template for improving the experience for the poor schlubs who have to use your Admin page every day.

15Nov

Linux, Flash 10, and saving Flash files

Posted by Elf Sternberg as chat, Linux, personal

It used to be possible for Linux users to find and save the movie file from a Flash download with earlier versions of the Flash plug-in.  Flash put the file in the /tmp directory and if you were quick you  could copy it elsewhere.  If you had a plethora of Flash files, you could often find the one you wanted with a simple ls -lt /tmp/Flash* , which sorts the files in order from newest to oldest.

The latest version of Flash does something tricky, though.  It creates a temporary file to hold the Flash object, then, while keeping the handle to the file open, immediately deletes it.  This means that the Flash application can continue to access and use the file until you navigate away from it, but it no longer shows up in directory listings.

However, if you’re really Linux savvy, there are ways around even this little annoyance.

Looking for a Flash movie you’re looking at right now?  First, run this: lsof | grep Flash .  On my computer, the output was:

midori     9206  elf   57u      REG        8,3  10012691     64772 /tmp/FlashXXKoFQgx (deleted)

There’s that noxious little “(deleted)” flag, too.  But you have a hint.  First, your process id in 9206, and the file descriptor is 57.  Now, you can just cp /proc/9206/fd/57 /home/elf/movie.flv and there you go.  File copied.  At least, it’s worked for me the few times I’ve done it recently.

Lsof is a program that lets you identify all the open resources a running program is using. In this case, my running program is midori, a Gecko-using browser with a long history of trying to keep up. By identifying the resource, I can then wander down the /proc tree, getting at the process’s available resources directly. As a user, I’m free to access (and muck up) my own resources without risking anything or anyone else on the system– Linux is good about that.

10Nov

Blatantly looking for work…

Posted by Elf Sternberg as personal

Okay, since I’ve had the heart-to-heart with the boss, I should ask y’all for the same understanding: I’m looking for work again.  I love working where I am now, the absolute freedom, the responsibility for the entire working stack from the OS up to the CSS/JS/HTML deployment layer, but right now this is an underfunded start-up that isn’t paying anyone anything near market rates.  They know that, I know that, there’s nothing anyone can do about it right now.

So, to remind everyone, here are the basics: I’m an end-to-end developer with a strong preference for Python and its collection of web development environments.  I’ve written everything from network servers in C++ to animation tricks in jQuery, I’m good at Test Driven Development, Agile and XP, and all the other meta-development things that developers are asked to do.  I can configure MySQL, and have active interest and recent experience in gamification, social network integration, A/B testing, automated performance expansion via AWS/EC2, and vertical and horizontal sharding for data management issues.  My graphic design skills have withered without recent practice, but could be refreshed if that’s what I need to do.

My resume says the rest.

27Oct

Test From Home Under Load

Posted by Elf Sternberg as Design, web development

I learned an important lesson today: never put on hold a book from the library that’s already listed on the shelves. Just go get it. If it’s on hold, they’ll take it off the shelves and put it “into the system,” which means you won’t actually be able to get it for 24 hours.

I also learned another important lesson. Always start a few load-creating automated tests, then go to the most hostile wi-fi enabled cafe you know and try to use your web application from there.

The King County Library System recently spent a lot of money upgrading their computerized system. It looks like there was a desperate attempt to make it “more 2.0″ by throwing a thin layer of AJAX into the mix: searches now update page partials rather than whole pages, and various divs for holds, fines, checkouts, and wishlists come and go as you click on the appropriate tabs. All fairly unimpressive, by my standards. The developers, whomever they were, used Adobe’s library as their basis as the Javascript is full of Dreamweaver-related (“MM_”) tags. Where they didn’t use Dreamweaver, they wrote the code themselves; there’s a strong Not Invented Here feel to much of the Javascript they used.

And I’m sure, when they first started out, it looked okay. They tried to search for a few books, the spinner spun for a second and then the page rendered. Only now, under load, the spinner spins for a several seconds, and then the page goes blank. A long pause later, the page begins drawing again. The clue that the spinner needs to stop and the signal that the data is ready to be drawn are two different signals.

There are several examples of this, but the book search is the most egregious. All in all, the upgrade has made a lot of people mad: renderers don’t work, often fail to deliver, time out without excuse. And this doesn’t have to be hard. KCLS has slightly under a million items, and had 21 million check-ins last year, or about 3 a second during business hours. Prodigious, but seriously middle-tier as far as businesses go. A modern implementation of the website should not be as difficult as they’ve made it.

I wish I’d known this a long time ago.  Django’s request object includes a dictionary of key/value pairs passed into the request via POST or GET methods.  That dictionary, however, works in a counter-intuitive fashion.  If a URL reads http://foo.com?a=boo, then the expected content of request.GET['a'] would be 'boo', right?  And most of us who’ve used other URL parsers in the past know that http://foo.com?a=boo&a=hoo know that the expected content of request.GET['a'] would be ['boo', 'hoo'].

Except it isn’t.  It’s just 'hoo'.  Digging into the source code, I learn that in Django’s MultiValueDict, __getitem__(self, key) has been redefined to return the last item of the list.  I have no idea why.  Maybe they wanted to ensure that a scalar was always returned.  The way to get the whole list (necessary when doing an ‘in’ request) is to call request.GET.getlist('a').

Lesson learned, an hour wasted.

It drives me nuts that we in the Django community rely on Solr or Haystack to provide us with full-text search when MySQL provides a perfectly functional full-text search feature, at least at the table level and for modest projects. I understand that not every app runs on MySQL, but mine do, and I’m sure many of you are running exactly that, and could use this technique without modification.

Well, after much digging, I found an article on MercuryTide’s website covering custom QuerySets with FULLTEXT and relevance, and built this library around it.

I used this rather than Django’s internal filter keyword search, because this technique adds an additional aggregated value, the relevance of the search terms to the search. This is useful in sorting the search, something not automatically provided by the QuerySet.filter() mechanism.

You must create the indexes against which the search will be conducted. For performance reasons, if you’re importing a massive collection of data, it’s better to import all of the data and then create the index. More importantly, when you declare that a SearchManager to be used by a Model, you declare it thusly:

class Book(models.Model):
    ...
    objects = SearchManager()

When you do, you must add an index that corresponds to that list of fields:

CREATE FULLTEXT INDEX book_text_index ON books_book (title, summary)

Notice how the contents of the index correspond with the contents of the Search Manager.  Or you can automate the process with South:

    def forwards(self, orm):
        db.execute('CREATE FULLTEXT INDEX book_text_index ON books_book (title, summary)')

    def backwards(self, orm):
        db.execute('DROP INDEX book_text_index on books_book')

To use the library is fairly trivial. If there is only one index (which can encompass several columns) for any table, you call

books = Book.objects.search('The Metamorphosis').order_by('-relevance')

If there’s more than one index, you specify the index by the list of fields:

books = Book.objects.search('The Metamorphosis', ('title', 'summary')).order_by('-relevance')

Note that that’s a tuple, and must be.

If you specify fields that are not part of a FULLTEXT index, the error message will include lists of viable indices.   It will also tell you if there are no indices.  (Getting that to work was tricky, as it involved database introspection and the decoration of methods, so I’m especially proud of it.)

The library is fully available on my github account: django_mysqlfulltextsearch

13Oct

Validation!

Posted by Elf Sternberg as chat, django, personal

Last night at the Django meetup, we also talked about unit testing.  Someone mentioned continuous integration, and we all discussed our favorites.  At one point, the fellow at whose offices we were holding the meeting mentioned that his team used Hudson and pulled up an example on the overhead projector.  I mentioned that Hudson was my favorite as well, and he said, “Yes, I think it was your blog entries that led me to this solution.”

That’s the second time this year I’ve walked into a meeting and someone’s said, “I’ve used something you wrote.”  Kinda cool.  Wish it was addictive.

13Oct

Django-Zipdistance

Posted by Elf Sternberg as Design, programming, python

Recently, I had the pleasure of attending another of those Seattle Django meet-ups.  This one was a potpourri event, just people talking about what they knew and how they knew it.   I revealed that I’d written my first aggregator, and that seemed to be an impressive statement.  Apparently Django Aggregators (database conditionals that perform sub-selected summarizing or filtering events) is something of a black art, much like WordPress Treewalkers were a black art I figured out in just a few hours.

Aggregators consist of two parts: The Definition and the Implementation.  Unfortunately, Django’s idea is that these are two different objects, bound together not by inheritance but by aggregation (both the definition and the implementation are assembled in a generic context, one providing access to the ORM and the other to the SQL).   The definition is used by the ORM to track the existence and name of the aggregate, and is then used to invoke the implemenation, which in turn creates the raw SQL that will be added to the SQL string ultimately sent to the server, and ultimately parsed by the ORM.

I needed to use aggregation because I wanted to say, “For any two points’ latitude and longitude, give me the great circle distance between them,” and then say, “For a point (X, Y) on a map, give me every other place in the database within n miles great circle distance.”

The latter was not possible with Django’s Queryset.extra() feature.    You can add a WHERE clause, but not a HAVING clause, and this definitely requires a HAVING clause when running on MySQL.  Using an Aggregator with a limit forces the ORM to realize it needs a HAVING clause.  Besides, it was a good excuse to learn the basics of Aggregation. Ultimately, I was able to do what the task required: find the distance between any two US Zip Code regions without making third-party requests.

I make absolutely no promises that this code is useful to anyone else.  The Aggregator is definitely not pretty: it’s virtually a raw SQL injector.  But it was fun.  Enjoy: Django-ZipDistance.

07Oct

A significant Django tool: storages

Posted by Elf Sternberg as Uncategorized

If you work with Django for any period of time, the day comes when you’ll be accepting outside data from your users: files, images, and the like.  Django provides only two places where you can store these items: in memory, or to a file.  Django storages provides for two more critical locations: BLOBs in your SQL database (which you may want to do– never know), and most importantly, Amazon AWS S3.  For my current work with a film library and catalog, having S3 be the storage solution has always been a bit of a kludge: producers would upload films to the server, and we’ve eventually get it into S3.  Now, with AWS S3, Django Storages backed with the incredible BOTO library makes the entire process unbelievably easy.

Storages also supports storing content in the database, CouchDB, FTP, or anything else you can imagine. And the source code makes for excellent examples.

Also, if you use django-storages, consider looking at many of the branches on bitbucket, because there are variants of it for S3 that disable the HTTPS default for Cloudfront, which was important for us at Indieflix. Not everything coming out of Cloudfront has to run through SSL.

Recent Comments