I had that very experience this evening.  It’s funny what gives you insight, what makes you understand.  I’ve been trying to wrap my head around the whole idea of Haskell’s type, typeclass, data, and instance, and just not getting it.  It wasn’t clicking for me at all.  I went back to chapter 2 (!) of Real World Haskell and started working through the problems again, and I hit this example:

ghci> lines "the quick\nbrown fox\njumps"["the quick","brown fox","jumps"]

And being a good little student I meant to type:

ghci> :t lines

But being a lazy student I used keyboard shortcuts to summon the previous example. I ended up with this:

ghci> :t lines "the quick\nbrown fox\njumps"
lines "the quick\nbrown fox\njumps" :: [String]

And the lights totally went on. Because I realized that the expression to the left of the :: symbol is not “an lvalue”, it’s just a value. If you don’t understand Haskell, why that’s a big deal might not make sense to you. But I get some of it now.

I’ve been thinking about this because I have a different project that’s very RDFa-heavy, and the details of implementation have been challenging. But no, I had to come to a different conclusion:

RDFa and Backbone.js are incompatible.

RDF is a framework for self-describing data, such that the data carries along with itself tokenized indicators for how it is to be organized, presented, and edited. The idea is simple: the browser requests a page, the server sends it all: the HTML represents the structured data, the CSS the visual depiction, and the javascript detects the RDFA (the ‘A’ stands for ‘attributes’, HTML attributes) and decorates these islands of content with controls to make the content editable and saveable back to the server.  The data itself is kept in a representative semantic form of (X)HTML trees.

Backbone, in contrast, is a software framework in which the data is represented in a tree-like structure in Javascript itself, in the models and collections. The data does not come down with the page, but is requested after the fact by Backbone.  The HTML is rendered as a side-effect (my Haskell classes are rubbing off on me): it is a representation of the data-state on the client. Backbone works hard under the covers to keep the client and server state in sync, but in all cases the HTML is just a representation of the data; it is not the data itself.  The data itself is kept in Javascript trees.

These are two completely different approaches to managing data on the client. Both are valid. And both have their uses, but for the moment they don’t mesh well.

“If you have an idea and publish it on the internet, it counts as the ghost of done.”

So here’s my ghost:

We need a Backbone-like for RDFa. RDFa cannot easily be coerced into working with Backbone, but let’s face it: there are only a few things we work with on the internet: text and inputs, and lists and trees of text and inputs.

It ought to be possible to formalize a backbone-like collection of Views, Schemas and Rules, such that the transition from presentation to revision is seamless and perhaps not even distinct, and synchronization for a range of RDFa-enabled articles, forms, fieldsets, and the like is automagical. In other works, make Models and Views “know” about their data because it’s already present in the HTML, and not the other way around.  It’s the transition from page-format to server-side format that’s the pain point.

Hmm. This needs more thinking.

Well, with the latest audio standard from the W3C (note that that’s a “Raw Tip” specification, meaning that it’s completely vaporware), it is now possible. Chrome supports much of the WebAudio specification, so this in what I hacked together in the past, oh, four hours or so: HTML5 Tone Matrix.

Compared to the original, it’s not very pretty at all. I don’t understand audio. But it’s a start, and like all experiments, won’t suffer from further experimentation and understanding.

quicksort [] = []
quicksort (x:xs) =
           let smallerOrEqual = [a | a <- xs, a <= x]
               larger = [a | a <- xs, a > x]
           in quicksort smallerOrEqual ++ [x] ++ quicksort larger

I wondered what it would be like in Coffeescript. Frighteningly enough, other than moving the guard condition into the function itself, they look remarkably similar:

quicksort = (x) ->
    return [] if x.length == 0
    h = x.pop()
    smallerOrEqual = (a for a in x when a <= h)
    larger = (a for a in x when a > h)
    (quicksort smallerOrEqual).concat([h]).concat(quicksort larger)

Coffescript is a gateway drug to Haskell. If you don’t want to learn Haskell, turn back now. (On the other hand, I think learning Haskell will be a hell of a good thing for me. My brain needs the exercise.)

But not as brain-bending as Haskell.

That said, I just finished the first exercise in Seven Languages in Seven Weeks, although really I’d think it’s more like Seven Languages in Seven Days, given my ferocious language consumption– I just skipped right to the Haskell part, because I have a project that requires Haskell at the moment. Still, this was pretty cool. Don’t read further if you don’t want the answer.

The problem was “Write a function that reverses a list.” In most languages, that’s pretty simple. In Haskell, not so much. This problem requires three leaps: First, to understand that “strings are lists of characters” means exactly what it says, so the thing introduced earlier as a “String concatenation infix operator” is in fact a list concatenation infix operator, and finally take the Fibonacci example, with its helper functions and patterns, and apply that idea to this problem.

    reverseSub :: ([x], [x]) -> ([x], [x])
    reverseSub (xs, []) = (xs, [])
    reverseSub (xs, (h:t)) = reverseSub([h] ++ xs, t)

    reverse1 :: [x] -> [x]
    reverse1 [] = []
    reverse1 xs = (fst . reverseSub) ([], xs)

The function I’m defining I’m defining takes a list and returns a list. If the list is empty, return an empty list. Otherwise, return the first item in the tuple returned by reverseSub, to which I provide an empty list and my initial list.

reverseSub, in turn, takes a tuple of two lists, and returns a tuple of two lists. The first pattern says “If the second list is empty, return the tuple unchanged.” The second pattern means “Otherwise, reverseSub of two lists is “the head of the second list concatenated with the first list, and the tail of the second list,” applied recursively. It still feels totally weird to me that Haskell would automatically apply the second operation again and again until the first is met; I understand it intellectually, in a super-cool, I-love-recursion way, but it doesn’t quite resonate yet.

Still, this is way fun.

I said, “Sure. Me.” I told him about the tutorials I had written for Backbone, the current use of Backbone at ${DAY_JOB}, the stuff I had contributed to Backbone-relational, and for a while we talked about the difference between Backbone and Knockout on the one hand, and then those versus do-everything frameworks like Cappucino and Sproutcore.

But here’s what gets me wondering: is Backbone.js, a fairly small and smart library, by itself difficult enough to warrant its own recruitment category? Any competent Javascript programmer who’s had any experience writing enterprise-level code, working with any of the above frameworks (along with JavascriptMVC and Batman and… did Amber ever make it into the wild?  I can’t follow everything, you know!), should be able to get up-to-speed on Backbone within a week or so of using it.  Just write a tutorial, like I did, then re-write to figure out how silly your original try was.

Is “We need Backbone experience” even a reasonable request at this point?

So, really, your code looks like this:

backends = [
    {
        name: 'Event Hub'
        host: 'http://127.0.0.1:8024'
        paths: ['^/socket.io/']
    },
    {
        name: 'Django Server'
        host: 'http://127.0.0.1:8022'
        paths: ['^/api/']
    }
]

standard_options = (t) ->
    target:
        host: t.hostname
        port: t.port
        https: (t.protocol == 'https:')
    enable:
        xforward: true

for backend in backends
    backend.paths = (new RegExp(i) for i in backend.paths)
    backend.options = standard_options(url.parse(backend.host))

find_backend = (req) ->
    pathname = url.parse(req.url).pathname
    for backend in backends
        for path in backend.paths
            m = path.exec(pathname)
            if m
                return backend
    null

proxyserver = httpProxy.createServer (req, res, proxy) ->
    backend = find_backend(req)
    if backend
        proxy.proxyRequest(req, res, backend.options)
        return @

    # Clean up if no backend found
    try
        res.writeHead 404
        res.end()
    catch error
        console.error("res.writeHead/res.end error: %s", error.message)
    undefined

proxyserver.listen(80)

Instead of all that silliness in the original, here I define my routes in a fairly standard human-readable format, and then translate them into the target style preferred by node-http-proxy. I have a simple function for looking up the back-end depending upon URL components and returning the first match. I let node-http-proxy do all the work of creating a connection to the back-end if it doesn’t work.

All that crud from last week reduced to a couple of list comprehensions and translations. Sad, isn’t it?

I should add that there’s a bug in the current version of node-http-proxy that will cause the code above to fail. Node-http-proxy has an internal call, _getKey(), to look up that host to which it will route a request, that it should use, but does not, for the ad-hoc proxy creation. Until that bug is fixed (and yes, I have a patch submitted in), you’ll have to patch the server by hand.

There is some additional magic not shown in this example.  The rewriters is a trivial add-in, and neither example handles the case where a back-end issues an http:// re-direct, but that has to be mangled to read https:// before being forwarded to the client. Again, there’s a bug in the original that prevents this from working right, and I have a patch in for that as well.

Finally, the code as checked out of npm has a bug that prevents websocket upgrades from working properly. My work-around was to disable that in the code and do it myself, but that’s not a satisfactory hack and I haven’t sent that forward to the nodejitsu team yet.

I will say that with all of these fixes in place, I have a working solution for replacing Nginx. It proxies nicely, it proxies websockets nicely, and it proxies large-file uploads without using hideously huge memory or file buffers on the proxy server, all through SSL. We will be evaluating its performance, stability, and cross-browser issues over the next couple of weeks to ensure it’s a good solution.

I will also say that reading the Nginx source code was an eye-opening experience. The core and http modules are very well-written, clear and concise, and learning from their comments and code examples has made a big difference in how I’m approaching this proxying issue.

In theory, I could reduce Switchboard down to my routines for defining backends and frontends, add the rewriter and ad-hoc websocket upgrader.  Maybe I will.  It’ll be a heck of a lot shorter than the current edition.

I don’t do much with Cold Fusion, Flash, and obviously a beginners’ article on Coffeescript just isn’t going to impress me, so those tutorials were out. On the other hand, the central article was on responsive design, and a badge graphically associated with the “responsive design” part of the cover art promised “16 pages of how-to guides and advice!”

Which there wasn’t. Other than one article on using Drupal’s server-side client detection facility (Drupal? Server-side user agent detection? I mean, really?), there wasn’t anything in this issue that really corresponded as actionable advice.

If you want a real looksee at responsive design, Ethan Marcotte’s original article is where you start, Smashing Magazine has a decent article, and once you understand the basics, DesignModo’s 50 Examples is definitely where you see how the rubber meets the road.

Maybe I’m just done reading magazines.

tl;dr version: We recently replaced our Nginx with an HTTP reverse proxy written entirely in Node.js. We leveraged Nodejitsu’s node-http-proxy, and added our own logic to provide a useful switchboard for several small HTTP servers, each of which “does one thing well,” in the Coffeescript programming language.

There are three scenarios where a reverse proxy is useful. The first is when you have several back-end servers that all do the same thing, and you want to distribute the load across those servers; the second is when you have several different back-end servers on the same box, one per hostname, and you want to dispatch the packet to the correct handler.

The last is when you have several different services that are deployed from several different kinds of webservers, and you want all of those to be fed out of the same address/port combination. For a real-world example, we at Spiral Genetics have our core application server; an authentication and accounting server (“Who are you and do you have any money?”); a media server; an import handler for customer uploads; and a pub/sub/hub that coordinates among these, and publishes events to our web-based UI as events happen on the back-end.

Ngnix buffers requests before passing them on to the back-end service, and it doesn’t speak HTTP/1.0 natively to its back-ends, a requirement for websockets. Both of these limitations needed to be overcome.

My initial research led me to Node-HTTP-Proxy, which did about 90%of what I wanted it to do. The request-response cycle within Node-HTTP-Proxy’s HttpProxy object was exactly what I wanted, with non-buffering of the body of the HTTP request, and automatic upgrading to both HTTP/1.1 and the Websocket protocol as needed.

Node-HTTP-Proxy’s examples came in two flavors: a single route to a single back-end, multiple-routes to hostname-related back-ends, and a how-to example that showed how to route to different back-ends based on URL fragments.

It was this last example, route-by-proxy, that provided the bulk of the intelligence needed for our solution.

Segmentation

There were two things that needed to be addressed: first, since we had multiple back-end servers that would be addressed through a single hostname, segmenting those back-end servers would have to be done on a URL basis. Fortunately, Node 0.6′s request/response cycle pauses when the head of the request has been processed and allows us to process the inbound data in chunks, the same way we’d handle outbound data.

The second problem was that one of our services was “macros of instructions to the data server,” which was easier to store as a set of procedural steps somewhere other than the data server. So we needed a URL re-writer that looked like it was part of the data server’s namespace, but was actually in its own little application.

Node-HTTP-Proxy’s “proxy-by-url” example demonstrates how to do this, for the most part, but here’s what we wanted:

<configure and run the application>= (U->) [D->]
routes = [
    ['http://127.0.0.1:80', ['^/AnimatedSkills/', '^/180minutes/'], "Event Hub"],
    ['http://127.0.0.1:8002', '^/soundpodcast/', "Auth Server"]
]

This is the base configuration: For each destination, we have a list of URLs that we want to go to that destination. You’ll note that the last one in “Auth Server” goes to the home page; it’s a Django application that’s perfectly capable of figuring out if you’re logged in and serving you the application, or demanding you log in if you aren’t.

You’ll also note that the path to “Auth Server” comes before the “Data Server.” It’s a more precise path, and we want it to be analyzed first.

We also wanted to do URL rewrites after the path had been determined. So we’re going to pass that to our switchboard:

<configure and run the application>+= (U->) [<-D->]
switchboard_options =
    rewrites: [['^/soundpodcast/', '/']]

And finally we start the application, passing our switchboard to node-http-proxy’s httpProxy.createServer function:

<configure and run the application>+= (U->) [<-D]
server_options = {}
console.log("Listening on port 8120")
proxy = httpProxy.createServer(server_options, Connect.logger(), switchboard(routes, switchboard_options))
proxy.listen(8120)

The Switchboard

I’m asking three things of the switchboard: efficient look-ups based on the URL’s content, efficient rewrite of the URL to be passed to the back-end server, and a minimal number of proxy handlers to be configured in the meantime.

What I want, ultimately, is to create an array of routes that I will search, in order, for a match. First match wins, and to associate those routes with the appropriate proxy.

Using Coffeescript, let’s put that into a constructor.

<Switchboard constructor>= (U->) [D->]
constructor: (routes, routing_options = {}) ->
    @rewrites = []
    @routes = []

    for route in routes
        target = url.parse(route[0])

        options = {}
        options.target = {}
        options.target.host = target.hostname
        options.target.port = target.port
        options.target.https = (target.protocol == 'https:')

        proxy = new HttpProxy(options)
        proxy.on 'start', (req, res, target) => @emit('start', req, res, target)
        proxy.on 'end', (req, res) => @emit('end', req, res)

Here, for a given back-end, we’re creating a new HttpProxy object that knows the basic information needed to contact the back-end. That information won’t be needed until we make a request, but now it’s cached in the proxy object.

Now, for each route, the second item in each routing table object, we’re going to create a RegExp object for matching, a reference to the proxy we just created, and we’ll keep the comment. (We don’t actually do anything smart with the comment right now.)

<Switchboard constructor>+= (U->) [<-D->]
        for r in (if (route[1] instanceof Array) then route[1] else [route[1]])
            @routes.push
                match: new RegExp(r)
                proxy: proxy
                comment: route[2]

A little syntactic sugar there allows you to pass in plain paths rather than arrayed ones, but that’s about it.

And finally, we need to remember the rewriting options. Again, we care about a regexp for the inbound path.

<Switchboard constructor>+= (U->) [<-D]
    if routing_options and routing_options.rewrites?
        @rewrites = ({match: new RegExp(r[0]), rewrite: r[1]} for r in routing_options.rewrites)

You can’t tell me that Coffeescript’s comprehension’s aren’t cool.

Next, we want to be able to determine if a given request matches our switchboard. That’s a fairly straightforward RegExp match. When we get a hit, we want to return the route object associated with the match. If we don’t get a hit, return null

<Switchboard URL matcher>= (U->)
match: (path) ->
    for route in @routes
        m = route.match.exec(path)
        if m
            return route
    null

And finally, now that we’ve had a match, we want to proxy that match. The handler has all the context information we need.

<Switchboard proxy dispatch>= (U->)
proxy: (handler, req, res) ->
    path = url.parse(req.url)
    req.url = @rewrite(req.url)
    p = handler.proxy.proxyRequest req, res, null

Rewrites are probably more complicated than the proxy itself. We have to get the URL right, but we can’t afford to confuse the HttpProxy, which has its own needs about URLs. The popular convention is to replace \$1 with the first match of the RegExp, \$2 with the second and so on. Here, we use Node’s
url.parse() method to create a URL object, perform a match against our list of rewrites, and if we get a hit, we make a new URL object, copying those from the original, and then mangling the pathname. We transform this new object back into a standard URL for HttpProxy to make sense of, and return it. Otherwise, we return the original URL unchanged.

<Switchboard URL rewriter>= (U->)
rewrite: (ourl) ->
    path = url.parse(ourl)
    for check in @rewrites
        m = check.match.exec(path.pathname)
        if m
            resp = {}
            for i of path
                if path.hasOwnProperty(i)
                    resp[i] = path[i]
            resp.pathname = check.rewrite
            for i in [1..m.length - 1]
                resp.pathname = resp.pathname.replace(@substitutions[i], m[i])
            return url.format(resp)
    return ourl

The @substitutions are pre-cached regular expressions for $1 through $9:

<Substitution regexps for URL rewriting>= (U->)
substitutions: (new RegExp('\\$' + i) for i in [0..10])

And our class looks like:

<Class Switchboard>= (U->)
class Switchboard extends events.EventEmitter
      <Substitution regexps for URL rewriting>

      <Switchboard constructor>

      <Switchboard URL matcher>

      <Switchboard URL rewriter>

      <Switchboard proxy dispatch>

It’s nice and all that we have this class, but for our purposes we’re going to create a single function that pulls it all together, and handles the case where there is no match. And then we’ll export this
function so other people can use it:

<export the switchboard function>= (U->)
module.exports = (routes, options) ->
    switchboard = new Switchboard routes, options
    (req, res, next) ->
        path = url.parse(req.url)
        handler = switchboard.match(path.pathname)
        if handler
            return switchboard.proxy handler, req, res, next
        try
            res.writeHead 404
            res.end()
        catch error
            console.error("res.writeHead/res.end error: %s", er.message)
        undefined

So now our programs become, first, our library:

<switchboard.coffee>=
events = require('events')
HttpProxy = require('http-proxy').HttpProxy
url = require('url')

<Class Switchboard>

<export the switchboard function>

And secondly, our example:

<dispatchexample.coffee>=
Connect = require('connect')
httpProxy = require('http-proxy')
switchboard = require('./switchboard')

<configure and run the application>

As always, this code, in both the Literate Programming original and a fully functional NodeJS/Coffeescript library, is available at my github under elfsternberg/node-http-proxy-switchboard.

• <Class Switchboard>: D1, U2
• <configure and run the application>: D1, D2, D3, U4
• <dispatchexample.coffee>: D1
• <export the switchboard function>: D1, U2
• <Substitution regexps for URL rewriting>: D1, U2
• <Switchboard constructor>: D1, D2, D3, U4
• <Switchboard proxy dispatch>: D1, U2
• <Switchboard URL matcher>: D1, U2
• <Switchboard URL rewriter>: D1, U2
• <switchboard.coffee>: D1

I’m also fond of namespaces, and have started using the easy modules hack to manage namespaces with Coffeescript. It’s just a layer of abstraction on top of simply allocating yourself a namespace hanging off the root (in the client case, the traditional DOM window object). But it’s a nice, clear, self-documenting abstraction.

I discovered, however, that Backbone-Relational doesn’t like this idea. Since it’s an ORM, it needs to keep references to the models used for relationships, and it stores these as strings. The introspection layer is fairly dumb and not sure where to stop, so it assumes all of the names hang off the root namespace.

Bad idea. The following will not work:

@module "MyModule", ->
    class @AClass
        relations: [
            type: Backbone.HasMany
            key: 'children'
            relatedModel: @TheOtherClass
            includeInJSON: false
            reverseRelation:
                type: Backbone.HasOne
                key: 'parent'
        ]

However, because Backbone-Relational is working off string namespaces, specifying the relationships a  string works wonders:

@module "MyModule", ->
    MyModule = @
    class MyModule.AClass
        relations: [
            type: Backbone.HasMany
            key: 'children'
            relatedModel: "MyModule.TheOtherClass"
            includeInJSON: false
            reverseRelation:
                type: Backbone.HasOne
                key: 'parent'
        ]

In some respects, this is fairly ugly. You’re losing some of the shorthand advantages of using the Module hack by repetitiously (if you’re using the namespace in more than one file) re-assigning the module’s this object to a local name.  In other respects, it’s self-documenting, allows Backbone-Relational to work well, and is well-behaved.