I tried using the event driver (evdev) settings, both “Axes Swap” and “Axis Inversion”. It didn’t help. I got desperate, and tried to teach myself the Coordinate Transformation Matrix, but that link is one brutally nasty “If you can’t do matrix math, I’m not stupid enough to help you” bit of writing. I am knowledgeable enough to do matrix math, but his explanation is as clear as landfill, and about as useful.

It turns out that none of my confusion was my fault. There’s a bug xf86-input-evdev-2.7, the version that ships with Ubuntu 12 (Quantal). It doesn’t handle rotation or inversion well on the ELAN Touchscreens that ship with the Yoga. It’s somewhat understandable– the Yoga is a very new piece of hardware– but damn, it’s annoying.

I downloaded, compiled and installed xf86-input-evdev-2.8, and now it works. I have only keyboard-issued orientation working, but it’s a start. The keyboard program is simple:

#!/bin/bash
ARG=$1
CMD=${ARG:0:1}
case $CMD in
    r) 
        xrandr --screen 0 -o right
        xinput set-prop --type=int --format=8 "ELAN Touchscreen" "Evdev Axes Swap" 1
        xinput set-prop --type=int --format=8 "ELAN Touchscreen" "Evdev Axis Inversion" 0 1
        ;;
    l)
        xrandr --screen 0 -o left
        xinput set-prop --type=int --format=8 "ELAN Touchscreen" "Evdev Axes Swap" 1
        xinput set-prop --type=int --format=8 "ELAN Touchscreen" "Evdev Axis Inversion" 1 0
        ;;
    i)
        xrandr --screen 0 -o inverted
        xinput set-prop --type=int --format=8 "ELAN Touchscreen" "Evdev Axes Swap" 0
        xinput set-prop --type=int --format=8 "ELAN Touchscreen" "Evdev Axis Inversion" 1 1
        ;;
    *)
        xrandr --screen 0 -o normal 
        xinput set-prop --type=int --format=8 "ELAN Touchscreen" "Evdev Axes Swap" 0
        xinput set-prop --type=int --format=8 "ELAN Touchscreen" "Evdev Axis Inversion" 0 0
        ;;
esac

And now, yoga r rotates the screen to portrait mode “right”, yoga l is portrait mode “left”, yoga i for “inverted” (upside down, “media mode”), and anything else, even just plain yoga, will put everything back to laptop mode, keyboard and all.

“Not for long,” I told him.

I got it home and decided that I was going to forgo the Gentoo song & dance.  I will miss Gentoo, but only a little.  Instead, I installed the “user-friendliest” of all Linux distro, Linux Mint.  Mint is even better than Ubuntu in a lot of ways, with its nice mix of high-end, “your mom can use it” window managers as well as the Gnome 2.0 maintenance branch known as Mate.  I installed Mate, then rehacked the core manager to give me wrap-around keyboard navigation of the multi-desktop handler.  (This is a feature that I first encountered in Solaris Motif, and am absolutely confuzzled as to why Metacity, the Gnome/Mate window manager, doesn’t support it.)

Installing Mint was a bit of a dance: first, you have to tell Windows you want to use “Legacy mode.”  This unlocks the security features designed to prevent you from installing anything over the base OS.  That’s painful enough, but then I put in the Mint USB stick and everything went smoothly– until I needed wireless.  The wireless for this laptop isn’t stable yet; I had to install a third-party driver from Github, but there was enough of an environment on the USB stick to support doing so, and then I had wireless.

Mint Just Works.  Suspend to disk, automatic media detection, the camera, the sound.  There are a few bugs related to the laptop being so new– brightness DOWN works from the keyboard, brightness UP doesn’t.  There’s a menu option to control brightness, so it’s not a fatal problem.   Most of the problems relate to Yoga’s famous “tablet mode”: The touchscreen isn’t in Grails yet so gesture support isn’t there.  I can find in the kernel the endpoints for the motion sensors and gyroscope, but there are no drivers to interpret the signals coming from those hardware components yet, so they don’t emit anything.  Screen rotation 180° works fine for movie-watching mode, but 90° in either direction and the touchscreen doesn’t track correctly with finger touches.  The system emits the same signal for “transition to/from tablet mode”, so it’s not possible (yet) to detect which mode you’re in.  The Windows key in tablet mode is mapped to the same one on the keyboard, so remapping it for other purposes in tablet mode depends on my being able to detect tablet mode.

But those are challenges, not merely problems.

Still, choosing Mint over Gentoo feels like I’m giving up something.  Some deep understanding of how all the parts are put together.  I found myself chasing down new runlevel operations and other things most people aren’t concerned with, and solving them, so I’m not hopelessly lost.  But if I’m deep-diving into writing a new Linux kernel driver to get at the gyroscope and new Grails packages for the touchscreen, pushing aside my understanding of all those other components feels a little disloyal to the tradition of difficult programming from which I come.

Still, as a writing tool, this machine is unbeatable.  Instant-on and six hours of battery life are nothing to complain about.

Two months ago I was complaining that I didn’t have a job, and then suddenly I had one. I work for Splunk now, an enterprise-level machine data analysis company. Their core product is pretty magical: you can shove just about any logfile, real-time stream, script-generated datasource, or anything else at it, and then interrogate the data to understand and monitor your networks, server farms, client connections, whatever. It does an amazing job of correlating data through little more than keyword observation. There’s even a free version, which, while limited to a half-gig of data, is a good way to start off taking apart your weblog files.

They have me niftying up the third-party web framework, the thing large companies use to integrate the data we collect with their own network intelligence dashboards, visualization systems, whatever. It’s all my kind of thing: Python back-end, Javascript/Backbone/jQuery front-end stuff, lots of clever closures and event handling.

But I decided, for entertainment purposes only, to learn more about a part of the system I know almost nothing about: the Java SDK. And it wasn’t enough to work in a language I don’t know (since I don’t know Java) with SDK’s I’ve never seen before. I had to go make it work with a language that, to the best of my knowledge, no one has ever demonstrated compatibility with Splunk before. I had to make it work in Clojure.

Yes, I’m a Hipster Hacker. I mean, come on, if you’re not making it difficult for yourself, what is education for, anyway?

The complete source code for this project is available at Github.

Disclaimer

I am a newbie to all of this. Clojure, Splunk, even Java. This is entirely raw and beginner level stuff. I did this for my own edification. This code in no way represents the state of the art at Splunk. It’s definitely not warranted in any way by me or anyone else, and it’s licensed under the Apache Public License, V2. Use at your own risk. Don’t make me go ALL CAPS on you.

Literate Program

A note: this article was written with the Literate Programming toolkit Noweb. Where you see something that looks like this , it’s a placeholder for code described elsewhere in the document. Placeholders with an equal sign at the end of them indicate the place where that code is defined. The link (U->) indicates that the code you’re seeing is used later in the document, and (<-U) indicates it was used earlier but is being defined here.

Install everything

First, install Clojure, Leiningen, the Splunk free server, and the Splunk Java SDK. Start up the free server and give it some data. I gave it the dataset from earthquake.usgs.gov– “earthquakes greater than Richter 2.5 in magnitude over the past 30 days.”) but you can give it whatever you want.

Completing the installation is a bit tricky because this example relies on both the Splunk Java SDK, which is not in any Clojure or Maven repos, and the Splunk Java SDK’s Command utility class, which Splunk has provided as a helpful tool for unpacking the command line and for understanding the Splunk command file (.splunkrc on most Linux and Mac boxes). You’ll have to download the Splunk Java SDK yourself and install the Splunk JAR file. You’ll also have to create, from the root of this project directory, resources/com/splunk, and deposit Command.class (which can be found in the SDK’s tree somewhere) in the newly created directory.

I used lein localrepo to install the Splunk JAR file. It seems to have worked for me.

Create the project

In your workspace directory, whatever you call it, build a new project and call it splunk.

lein new splunk

Open the new directory and create a new directory, lib. Find the Splunk jar file in the SDK, and copy it into lib. Edit project.clj so it looks something like this:

<project cli rev 1>=
(defproject splunksearch "0.1.0"
  :description "SplunkSearch: An implementation of the Splunk Search example program in Clojure"
  :url "https://github.com/elfsternberg/splunk-search"
  :license {:name "Apache Public Licence 2.0"
            :url "http://www.apache.org/licenses/LICENSE-2.0"}
  :dependencies [
                 [org.clojure/clojure "1.4.0"] 
                 [commons-cli "1.2"] 
                 [gson "2.1"] 
                 [opencsv "2.3"]
  :plugins [[lein-localrepo "0.4.1"]]
  :main splunksearch.core)

Run lein deps to install your dependencies.

Then, use the localrepo command to install the splunk jar in your Maven repository:

lein localrepo coords splunk-1.1.jar | xargs lein localrepo install

Now, edit the project file to show your new dependency:

<project cli rev 2>=
(defproject splunksearch "0.1.0"
  :description "SplunkSearch: An implementation of the Splunk Search example program in Clojure"
  :url "https://github.com/elfsternberg/splunk-search"
  :license {:name "Apache Public Licence 2.0"
            :url "http://www.apache.org/licenses/LICENSE-2.0"}
  :dependencies [
                 [org.clojure/clojure "1.4.0"] 
                 [commons-cli "1.2"] 
                 [gson "2.1"] 
                 [opencsv "2.3"]
                 [splunk "1.1"]]
  :plugins [[lein-localrepo "0.4.1"]]
  :main splunksearch.core)

Now you’re ready to begin. Fun, huh?

[Edit: I've gotten at least one email telling me that localrepo only created the entry in your local Maven repository, but didn't copy the jar file there.  You might have to do that yourself.  For most of us it's in $HOME/.m2/repository/splunk/splunk/1.1 Welcome to the cutting edge.]

Create a simple query

The Splunk Java SDK has its own custom methods for dealing with arguments from the command line, from your run commands file (all those files in your home directory that end in “rc” on Linux and sometims Mac OS), and arguments to send to the remote Splunk server. We need two sets of arguments, one to define access to the service, and one to define the output mode of the data we expect to get back.

First, we have to create the namespace and import everything we’re going to need:

<create the namespace>= (U->)
(ns splunksearch.core
  (:require [clojure.java.io :refer :all])
  (:import (com.splunk Service ServiceArgs Args ResultsReaderJson ResultsReaderCsv ResultsReaderXml Event))
  (:import (com.splunk Command))
  (:import (java.io InputStreamReader OutputStreamWriter)))

I pretty much followed the code in order, a list of procedures. Remember, this document serves mostly as a stream-of-consciousness “this is what I learned about Clojure while wrestling with Splunk” history of my project.

The Splunk Command class requires a series of definitions added, and then it attempts to parse the command line. Command instantiates via a static factory method, so that’s accessed with a slash; the doto macro allows me to access the created object repeatedly, passing methods and arguments, and guarantees the object created is returned regardless of the return object of the last method. Clojure args have to be coerced via into-array into a Java array for Java-based args parsers to make sense of them.

<parse command arguments>= (U->)
(defn build-splunk-command [args]
  (let [command 
        (doto (Command/splunk "search")
          (.addRule "count" Integer 
                    "The Maximum Number of results to return (default: 100)")
          (.addRule "earliest_time" String
                    "Search earliest time")
          (.addRule "field_list" String
                    "A comma-separated list of the fields to return")
          (.addRule "latest_time" String
                    "Search latest time")
d          (.addRule "offset" Integer
                    "The first result (inclusive) from which to begin returning data. (default: 0)")
          (.addRule "output" String
                    "Which search results to output {events, results, preview, searchlog, summary, timeline} (default: results)")
          (.addRule "output_mode" String
                    "Search output format {csv, raw, json, xml} (default: xml)")
          (.addRule "reader" 
                    "Use ResultsReader")
          (.addRule "status_buckets" Integer
                    "Number of status buckets to use for search (default: 0)")
          (.addRule "verbose"  
                    "Display search progress")
          (.parse (into-array String args)))]
    (if (not= (count (.args command)) 1)
      (Command/error "Search expression required" nil))
    command))

My example code was taken from the Search/Program.java file, provided with the SDK. That program had a ton of local variables to control search generation, stream configuration, reader and output generation. I decided that all had to go into a simple map, which I could then refer to at any time.

Yes, the names above are repeated here. That’s a bit of a code smell, I think.

<build the arguments map>=

(defn build-argument-map [command]
  (let [opts (.opts command)
        ruleset [["count" 100] 
                 ["earliest_time" nil ]
                 ["reader" false]
                 ["verbose" false]
                 ["field_list" nil ] 
                 ["latest_time" nil ]
                 ["offset" 0]
                 ["output" "results"]
                 ["output_mode" "xml"]]]
    doall (into {} (for [[k v] ruleset] [k (if (.containsKey opts k) (.get opts k) v)]))))

Now that I have my argument map, I need to process it into Args objects understood by the Splunk Service class. I don’t know about you, but this all feels just a bit fiddly:

<build the queryargs object>= (U->)
(defn build-splunk-queryargs [argument-map]
  (let [rulelist ["earliest_time" "field_list" "latest_time" "status_buckets"]
        qa (Args.)]
    (doseq [fieldname rulelist]
      (if (argument-map fieldname)
        (.put args fieldname (argument-map fieldname))))
    args))

Later, I have to build the output Args object, which the Splunk Service uses to to configure the output. Obviously. It’s the exact same code, and it wasn’t cut and paste. I’m thinking this needs an abstraction.

<build the output args object>= (U->)

(defn build-splunk-output-args [argument-map]
  (let [rulelist ["count" "offset" "output_mode"]
        args (Args.)]
    (doseq [fieldname rulelist]
      (if (argument-map fieldname)
        (.put args fieldname (argument-map fieldname))))
    args))

Now that we have everything, le sigh, it’s time to pass it all to the server. I don’t even care much about the Service object once I’ve instantiated it. I’m using it for a single query, a single Job on the server. And it’s a bit redundant to pass both the command and the argument map, but I’m using the map to configure other program behaviors, so it stays as a separate copy of the command content. Between the double-dot argument and the chain operators, I’m seeing a lot of Haskellian inspiration in Clojure.

<send the query to the server>= (U->)
(defn build-splunk-job [command argument-map] 
  (let [queryargs (build-splunk-queryargs argument-map)
        service (Service/connect (.opts command))
        job (.. service (getJobs) (create (first (.args command)) queryargs))]
    (while (not (.isDone job)) 
      (if (argument-map "verbose")
        (println (format "\n%03.1f%% done -- %d scanned -- %d matched -- %d results"
                         (* (.getDoneProgress job) 100.0)
                         (.getScanCount job) 
                         (.getEventCount job) 
                         (.getResultCount job))))
      (Thread/sleep 1000))
    job))

There are a number of different things we can get from the server. The correct setting is often “preview”, meaning “show me any valid data you’ve collected, even if it’s not complete.” Preview will return everything even if the job is complete, so it’s safe to use at all times. In this case, it’s a one-shot, so I’ll use “results” instead.  But you can see all of the options below.  Even I don’t know what they all mean.

Here, using the argument map and the instructions to build the output args object, I ask for a stream (an actual Java InputStream) from the server of the data I want:

<request a stream from the server>= (U->)
(defn get-splunk-stream [job argument-map]
  (let [outputargs (build-splunk-output-args argument-map)
        output (argument-map "output")]
    (case output
      "results" (.getResults job outputargs)
      "preview" (.getPreview job outputargs)
      "searchlog" (.getSearchLog job outputargs)
      "summary" (.getSummary job outputargs)
      "timeline" (.getTimeline job outputargs)
      )))

There are two kinds of read operations: we could use the Splunk ResultsReader, which parses the content for you and turns it into a hashmap of keys and values, or you can just get the raw data. I’m going to deal with the readers first.

Instantiating a reader irked me. I could not figure out how to make Java constructors act like first-class objects; I wanted to be able to pick a class and pass it back to the calling function, which could instantiate it on the fly after dereferencing it. I’m sure there are Clojure gurus who can help with that:

<construct a reader of the appropriate type>= (U->)

(defn construct-reader [stream output-mode]
  (case output-mode
    "xml" (ResultsReaderXml. stream)
    "json" (ResultsReaderJson. stream)
    "csv" (ResultsReaderCsv. stream)))

This returns a reader-ready function to print out content based upon the output mode. The when macro for handling loops was a real eye-opener.

<make a stream reader with a mode parser>= (U->)

(defn get-streamtype-reader [output-mode]
  (fn [stream] 
    (with-open [reader (construct-reader stream output-mode)]
      (loop [e (.getNextEvent reader)]
        (when (not= e nil)
          (println "EVENT:********")
          (doseq [k (seq (.keySet e))]
            (printf "%s ---> %s\n" k (.get e k)))
          (recur (.getNextEvent reader)))))))

This returns a reader-ready function to print out the raw content.

<make a generic stream reader>= (U->)

(defn generic-reader []
  (fn [stream]
    (with-open [reader (InputStreamReader. stream "UTF8")
                writer (OutputStreamWriter. System/out)]
      (try
        (let [buffer (char-array 1024)]
          (while true 
            (let [count (.read reader buffer)]
              (if (== count -1) (throw (Exception. "EOF")))
              (.write writer buffer 0 count))))
        (catch Exception e nil)))))

I’m completely sure that there’s a better way to acheive the symmetry I wanted, and that the function that takes no argument above in “make a generic stream reader” is completely gratuitous, but I kinda liked the way this one came out.

<get the stream reader>= (U->)

(defn get-reader [command argument-map]
  (let [use-reader (.. command opts (containsKey "reader"))]
    (if use-reader
      (get-streamtype-reader (argument-map "output_mode"))
      (generic-reader))))

This has to reveal just how bleedingly new I am to Clojure, because I’ve never seen a “let cascade” like this before in anyone else’s code. But it works!

<main>= (U->)
(defn -main[& args]
  (let [command (build-splunk-command args)]
    (let [argument-map (build-argument-map command)]
      (let [job (build-splunk-job command argument-map)]
        (let [stream (get-splunk-stream job argument-map)]
          (let [reader (get-reader command argument-map)]
            (reader stream)))))))

And the entire program ends up looking like

<core.clj>=
<create the namespace>

<parse command arguments>

<build the argument map>

<build the queryargs object>

<build the output args object>

<send the query to the server>

<request a stream from the server>

<construct a reader of the appropriate type>

<make a stream reader with a mode parser>

<make a generic stream reader>

<get the stream reader>

<main>

And that’s it.

You can now run the program with lein:

<command_line>=
lein run --output_mode csv "search magnitude > 4.5"

Assuming you have something in your Splunk database with a “magnitude,” you should get a CSV dump of all the fields related to it.

Now you know how to talk to the basic Splunk service using Clojure. I’m sure if you’re comfortable with Clojure none of the Java access weirdness surprised you at all, but for me this was a pretty good exercise. At times I felt like I was writing in a “fantasy LISP”, a Lisp that actually, you know, had real world applicability, and that should have done the things I would have expected of a LISP. That fantasy LISP was pretty close to the real deal; it only took me a few hours of lein run sessions to knock out all the bugs.

Index

• <build the argument map>: U1
• <build the arguments map>: D1
• <build the output args object>: D1, U2
• <build the queryargs object>: D1, U2
• <command_line>: D1
• <construct a reader of the appropriate type>: D1, U2
• <core.clj>: D1
• <create the namespace>: D1, U2
• <get the stream reader>: D1, U2
• <main>: D1, U2
• <make a generic stream reader>: D1, U2
• <make a stream reader with a mode parser>: D1, U2
• <parse command arguments>: D1, U2
• <project cli rev 1>: D1
• <project cli rev 2>: D1
• <request a stream from the server>: D1, U2
• <send the query to the server>: D1, U2

But there are two things that, well… one, it’s javascript. My Coffee and Haskell skills are dying on the vine. And two, it’s older javascript. I’m not allowed to do anything wild. No Bacon. No FRP. No Node. This thing has to be solid and working and be usable by people who don’t have time to master the intricacies of the cutting edge, people who want answers not fanciers (okay, that’s a stretch), and whose Javascript experience might just be a couple of O’Reilly books and an API.

Last week I released Tumble, a little hack for parsing Tumblr-like templates and rendering their contents. There’s a line in the source code that reads “This code stinks!” And it did. It was the same logic repeated three times: Once in the parser when it was found, once in the parser/contexter API, and a third time in the contexter itself.

My first order of business was to reduce the parser to a single rule. There are two kinds of objects in Tumbler: rendered objects and blocks. A rendered objects looks like {this} and tells the rendered “replace this with whatever this name refers to.” Usually, that’s just a string from the database; sometimes, though, it can be the result of a function. A block looks like {key:something} … a bunch of stuff … {/key:something}. Blocks have three different rules: They can be conditional (render this block if this data is true), descendent (render this block with this other source of data), or iterative (render this block many times, once for each item in this list of data). I stared out with rules like “just text”, “variable”, “ifblock”, “descendblock”, “manyblock”, and corresponding rules to find {many:foo}{/many:foo}, etc. things.

That turned out to stink. So I reduced the rule to a simple {blocktype:blockname}{/blocktype:blockname} finder rule.

The Contexter would do the right thing with each kind of blocktype, using data refernced by blockname.

The problem was the glue between contexter and parser. I was using the parser to derive the blocktype, then propagating that up to the contexter through a very crude set of manually written code paths. There had to be a better way.

Since I’d thrown out the various block types from the parser, the javascript for initial handling had also been thrown out. I found it in git. It looked like this:

    // TODO: Yeah, this code stinks.

    conditional = function(t, ps) {
        return function(content) {
            return content.if(t, function(c) {
                return sections(ps, content);
            });
        }
    };
    
    descendant = function(t, ps) {
        return function(content) {
            return content.descend(t, function(c) {
                return sections(ps, content);
            });
        }
    };

You can tell. It’s the same damn thing, over and over. And there’s a lot of visual clutter there.

I decided to re-write the glue in coffeescript. Here’s the whole of it now:

module.exports = (ast, data) ->
    context = new Contexter(data)
    cmd = (o) ->
        switch o.unit
            when 'variable' then  (context) -> context.get(o.name)
            when 'text'     then  (context) -> o.content
            when 'block'    then  (context) -> context[o.type] o.name, (context) ->
                (cmd(p)(context) for p in o.content).join("")
    (cmd(o)(context) for o in ast.content).join("")

This is way smaller. And you can see where my Haskell learning has paid off. I’m actually reasoning about the kinds of data, the kinds of closures I want the interpreter to work on as it descends the parse tree. The last line is pure magic. It says “For the current point in the AST for our template, get a function that can handle the current context for our data.” And it does. Perfectly. Even more beautiful is the way the ‘block’ handler automagically repeats that line in order to descend further down the AST tree. That last ‘context’ in that ‘block’ handler isn’t the current context– it’s whatever context the internal corresponding rule requires, and I don’t have to care about what that rule is, because this is just code-as-data. Sweet.

It’s been years since I’ve worked with lex and yacc. Hell, I even worked in Antlr briefly, at F5 Networks. But Haskell, Coffeescript, and PEG have made writing something like this blazingly easy.

I pushed the latest rev up to github at ElfSternberg/Tumble. It’s proving to be an interesting experiment.

Take a close look at the WordPress Database Schema diagram and tell me what’s… off… about it.  It should leap right out at you.  Absolutely everything in the WordPress Database Schema is related– posts, users, and comments are all interrelated to one another.  And off in the lower right-hand corner, there’s Options, all by itself, unrelated to anything else.

And then I realized: Everything on that page is about the content.  It describes the Content Layer.  Except Options.  Options isn’t about the Content at all.  Options in WordPress controls a completely independent program hiding inside WordPress: The Presentation Layer.  Actually, Options is a grab-bag of controls over everything: settings for the authentication layer, settings for the presentation layer, settings for the administration layer, settings for presenting the administration editor, etc., etc.  The one thing Options does absolutely nothing, and I mean nothing, for, is the Content layer.

Huh.

Maybe French Press is the next project after all.

This week, I finally wrapped my head around Tumble, the basic parser engine I’ve been working my way up to since about January or so.  It really shouldn’t have taken so long, but I had a lot to learn, a lot of false starts, a change of job, and a generally busy life.

Tumble is one part of a project that I’ve been working on for the past year.  The other parts are named Crest, Dragon, Candy, and French Press– they’re all part of a decoupled CMS written in NodeJS mostly meant for fiction writers.  Candy is a meta-system on top of Tumble for theme management.  French Press is a back-end.  Crest is an intake engine and Content-Editable aware piece for fixing stories after they’re done.  Dragon is a template package for Tumble that spits out the stories in EPUB and LaTeX formats for alternative distribution channels.

I should work on any one of those.  Probably French Press next, as it would help get the back-end toolkit rolling, and it must be working before Crest or any of the other toolkits make sense.  But I don’t wanna.  I’m not even particularly unhappy with Tumble; it’s a good piece of software, all told, and it has alternative output mechanisms that make it suitable to Candy or Dragon, but it’s… I’m just glad it’s done, and it’ll be a few days before I want to get back into that sort of thing.

Do you ever have post-project blues?

It’s fairly simple.  This is just version 0.0.1, so don’t expect it to be safe like liquid (although that’s ultimately a goal), and I’ve already had an idea for refining it further.  Check it out:

ElfSternberg/Tumble

I went to the SeattleJS meetup this evening.  The presentation format is two hours long: a little schmoozing at the front, an introduction, then 20 minutes of lightning talks from the audience, followed by two 20 minute presentations.  The first was on Meteor, the second on Arduino and NodeJS, and then a little more schmoozing at the end.  The introduction had high points; he asked “What do people want to talk about?”  There was a desultory “jQuery,” from the audience, then “Backbone.”  I threw out, in quick succession, “Node,” “Coffeescript,” and finally, “Functional Reactive Javascript.”   I didn’t mention Mocha or Grunt.  He asked how many people were here looking for work.  Hands went up.  How many recruiters?  Just about as many hands.  ”After the presentations, recruiters and job seekers, find each other.  I recommend you move over to that side of the room to make it easy.”

There were lightning presentations.  The first was mindblowing, a demonstration of visual recognition in Javascript.  He could wave his hands at the computer’s camera and control the paddle in a pong game.  The next were more pedestrian, but no less professional and amazing: a bible study website (“It’s hard selling bible study in the Pacific Northwest, believe me”), a socket.io demo.

Completely on impulse, and because there was a lull, I stood up and said, “Hi, I’m Elf Sternberg, and I got laid off a month ago.  That’s important to what I’m about to show you.”  I showed them FridgeMagnets, and went into the details of its construction: Coffeescript, Cake, Haml, Less, the separation of axes theorem, as well as I why I wanted to learn it: the animation, unicode, and audio APIs.  And then I said, “As straightforward a toy as this is, it’s important.  I was sending out resumes every day for three weeks without getting a return call.  Then, on the advice of a friend, I put the address for this toy, and a few like it, and my GitHub with the source code for it, on my resume, and re-sent it.  The next day I got 18 phone calls.  That weekend I had three offers.  Next Monday I start my new job.”  That got applause.  ”So let me encourage you, please, since Ryan showed us how many people here are looking for work,  get a GitHub account, get a StackOverflow account, get your own blog on WordPress or Tumblr or Livejournal, and let people know what you’re passionate about.  It doesn’t have to be big.  This is a toy.  But it shows people what I do, and how I do it.”  That got more applause.  ”Thank you.”

I have never been more terrified in my life.  I talked too fast, and I could have pissed myself up there.  What the hell was I thinking, shooting off like that in front of a hundred people?  I should do it again.

After the other two presentations– one of which, I helped the presenter figure out how to determine his own IP address, since local DNS wasn’t resolving correctly– I had recruiters coming up to me and asking me to explain to them how this GitHub thing worked.  They knew, in an abstract sense, that open source meant that some programs were available in some arcane “raw” source, freely available, somewhere on the Internet, but they were only now hearing about GitHub and StackOverflow.  I explained it as best I could, and some of them nodded and others just looked puzzled.

So let me re-iterate: if you want a job as a programmer, get yourself a GitHub and a StackOverflow account.  Especially if you’re a woman.  The head of hiring at Etsy said that whiteboarding, the common practice in interviews of making you write code on a whiteboard to prove you’re “smart… and quick!” is stupid.  Women are generally more deliberate and careful than men, less likely to shoot from the hip, more likely to be complete and correct.  Very few jobs in programming require the “quick” part.  Not in the sense of knowing the answer up front and immediately.  A GitHub or StackOverflow account shows what you’re capable of, and the interview should be about personality and cultural fit, and it’s a much better, and gender-neutral proving ground, than the hyper-aggressive “Skate… or Die!” of whiteboarding.

But, to make it fun, I decided to hack the recent posts scripts to use as much functional programming as possible. I couldn’t do it with WordPress’s theLoop(), because that doesn’t convert nicely into an iterator, but I could down in the category form, which does have an array of categories.

The first part was to get the list of categories, and to pull out the base for each input field:

    $cats_instance = $instance['cats'];
    $option_base ='<input type="checkbox" id="'. $this->get_field_id('cats') .'[]" name="'. $this->get_field_name('cats') .'[]"';

The second part takes any given category we want to show and know if it’s checked. This is a pure function; we want to transform a category into a boolean which asserts that it’s in the list of categories picked for this instance. We need to pass the instance into the main function, and PHP provides the use keyword to enclose variable in an anonymous function. Then, when this function is called, we need to enclose the passed-in variable to our is_a_cat function so we can scan it against cats_instance (which, now that I think about it, should really be named instance_cats, but whatever):

    $isChecked = function($c) use ($cats_instance) {
      $is_a_cat = function($ic) use ($c) { return ($ic == $c->term_id); };
      return (count(array_filter($cats_instance, $is_a_cat)) > 0);
    };

And finally, we’re going to reduce the list of all categories to an HTML string representing the form. We need to pass in $isChecked and $option_base.  Because of a bug in array_reduce, the container type declaration only ever takes a NULL or an integer, never a string or array, so the if statement there works around that.  Sadly, it’s a work-around, not a proper guard condition.

    $reduce_categories = function($result, $c) use ($isChecked, $option_base) {
      $checked = ($isChecked($c)) ? ' checked="checked"' : '';
      if ($result == NULL) {
        return $option_base . $checked . ' value="' . $c->term_id.'" /> ' . $c->cat_name . '<br />';
      }
      return $result . $option_base . $checked . ' value="' . $c->term_id.'" /&t; ' . $c->cat_name . '<br />';
    };

This is echoed immediately afterward to the screen.  Note the almost complete lack of variables.  Even $reduce_categories() is effectively invariant for this call.

<?php echo array_reduce(get_categories('hide_empty=0'), $reduce_categories, NULL); ?>

In all, this is a nicely functional routine in which we describe how the result is built, isolate exactly those elements necessary to build the result, and describe with some precision the enclosures necessary for each function to make sense.

I don’t know PHP at all.  PHP5 looks like some unholy archaic Perl crossover with some weird modern additions tacked on.  There’s still too many variables and references in this code and whether or not PHP is pass-by-reference or pass-by-value, or the oddly intuitive hybrid that lives in Java, Python and Javascript, still escapes me.  There’s no equivalent of the in_array() function that takes a callback as its truth function, so isChecked() scans the entire array on every iteration.

But this was fun. And it works on PHP 5.3 and above. And I can claim, with some truthiness, that it’s both functional and less error-prone. And that makes me happy.

TOXIC

“Terabytes Of XML, Indexed & Compressed.” I almost got hired for this idea alone, once upon a time, but sadly I was working already and had no desire to move to Boston. TOXIC is a simple idea: Very large stores of text, like those used for publishing corporations, libraries, governments and the like, live in gigantic SGML and XML databases.  There exists a known set of tools for indexing large corpora of texts, and the most common is to use Huffman compression on both an index of terms and their location in the library.  The idea is that pulling the index off-disk and decompressing it, caching the result as needed, is faster than pulling an already uncompressed index off disk.  Likewise, because Huffman encoding can be resumed at arbitrary points in a compressed library, it’s faster to seek to a known compression point, uncompressing the block there, and seeking for the indexed term.  What I wanted was to sandwich a second index between the first and the corpus, with an equally compressed representation of the XPaths to desired sections of the corpora.  It’s such an obvious schtick that I’m surprised no one has done it for an open-source solution.

A C-Robots/Ogre-GEV world-conquering mashup.

I was a big Ogre-GEV fan way back in the day, and I enjoyed the Rivets parody as well.  What I wanted was a web-capable version of the game that could be played multi-player, but could also be run automatically, C-Robots style, with the programming language being something sensible and accessible, like Python or Javascript.  Sadly, when I concocted this cockamamie plan, it was only the late 90′s, and nobody had the bandwidth or browser capabilities to make it happen.  Since then, the browser has the power, but I don’t.

A painfully cute first-person shooter

One of my stories involves a sentient doll, only 30cms tall, who starts out as the companion to a little girl on a planet where the infrastructure experiences a massive and complete systemic breakdown. The doll, the last artificial intelligence on the planet, has to fight her way through bugs, mice, rats and nastier things in her goal of saving her person and restarting her planet’s infrastructure. I wanted to write a first-person shooter on the same theme, but lacked the skills or time at the time. I wanted her to have the squeaky voice of an anime character, to be dressed in pink and frills, and to show how that costume became more and more ragged and dirty, as she grew into her role. There’s a similar PS/2 game, although it’s third-person and poorly done, Hoi-Hoi San, and it was never released in the US. I wanted something good.

A Seam-Carving tool for PAM

Seam-Carving, also known as “content-aware resizing,” is the algorithmic reduction of an image by creating maps, either horizontal or vertical, describing paths from top-to-bottom or left-to-right, in which, for each path, the neighboring pixel differential is added up into an overlain entropy map, and then the path with the lowest entropy is removed. This is done systematically until the image has been cropped down to the desired size, with the pixels that represent the least amount of information loss being removed and the seam smoothed over. While it’s not perfect, as you can see in the wikipedia example, it might be useful in a number of settings. I still use the Portable Anymap programs on a regular basis, and adding seam-carving to the manipulations would be a useful addition.