There have always been attitudes that I don’t understand. James Hague’s recent Death of a Language Dilettante is one of them. Now, most people know that I’m a language dilettante; that I love everything new and shiny coming out of the programming language space, even if it does seem that in some ways that space has been tapped out, with languages becoming ever more blunt on one side with Go and Python, and ever more esoteric on the other with Agda and Idris.

James issues a challenge: “Give a language with a poor reputation (JavaScript, Perl) to someone who knows it passably well and–this is the key–has a strong work ethic. Let the language dilettante use whatever he or she wants, something with the best type system, hygenic macros, you name it. Give them both a real-world task to accomplish. My money is on the first person by a wide margin.”

Well, sure. Tooling matters. Experience matters. In the ${DAY_JOB} I write a ton of Javascript, and I write it more than passably well. I also write a ton of Python, again more than passably well. Those are two languages at which I consider myself working at an expert level.

On the other hand, I also love esoteric languages and, more importantly, every time I’ve worked in one, it’s made me a better programmer in the languages I use professionally. Learning Haskell made me really appreciate your responsibility for separating TheWorld from the program, and taught me the power of higher-order functions; learning Lisp made me really appreciate the power of expressions qua expressions in a way Haskell tried hard to make so simple you didn’t notice them. When you work in a “real world” language, those matters are present in everyday details, and knowing how to work with them is a gift, not a disease.

Right now, my “esoteric” language is Hy. Hy is an implementation of Lisp that produces a Python AST and runs on the Python VM. Hy implements its Python importlib layer incorrectly, and in a way that means certain Django features are unavailable to Hy developers, and I intend to fix it.

Unfortunately, Python’s importlib is pretty esoteric in its own right, and implementing a new pather/finder/loader has been a beast. But I’m closing in on a solution, and I intend to have it working soon.

And then I can do what I set out to do, which is write a Django app. Only I can do it in Hy.

And along the way, maybe get a new Django command, newhyproject, out of it, as well as newhyapp and maybe newhymigrations. But those are stretch goals. Right now, all I want is for Hy to interoperate with Django correctly, and then I can think about future projects.

So I’ll take James’s challenge, but I’ll have to say: “Both is good.”

There’s a comment I get from my peers from time to time that boils down to: “Your code is tight and expressive, and the names are well-chosen, but it’s ugly and it makes reviewers uncomfortable.” And that’s when I realized that functional programming, by its very nature, causes us to write visually unattractive code.

The human eye likes rhythm. Aesthetically pleasing visual arrangements often involve creating a sense of flow, a sense of time, and a sense of transition from one scene to the next. Repetition is a key element of beauty. Artists and musicians come up with motifs and set up the expectation of repetition, and then maintain, modify, or diverge from the repetition into a new landscape where a new repetitious motif must be established and maintained in order to sustain the impression of beauty.

This is utterly in contrast to DRY: Don’t Repeat Yourself.

Well-constructed, highly functional source code written in a traditional enterprise language like Javascript, Python or C++ has very little repetition. It’s not supposed to. The comforting rhythm of a long if/else tree is broken out into a lookup table. The comforting rhythm of long transformations is itself transformed into a brutally compact map/reduce. As we try to make the syntax of the language itself more succinct (the Javascript keyword function has been brutally shortened to =>, for one very recent example), familiar “hooks” onto which the eye might rest and understand become harder to find.

It’s not all this bad. Well-written Haskell is actually very pretty to look at, but it’s also daunting to realize that every word on the page is dense with meaning.  It’s also significant that Haskell’s organization within a source file isn’t sequential; the compiler hooks up all the definitions in the correct order regardless of their location.  This gives the developer greater freedom to organize code into meaningful units, but it can also mean that understanding the file isn’t as simple as reading it in order from top to bottom.

That sense of intimidation is even greater when you apply it to more traditional languages that can support a functional style. A language like Javascript can support strong functional idioms, but in doing so, you end up writing the code equivalent of Shostakovitch and the “difficult listening hour.”


Analysis Paralysis

Posted by Elf Sternberg as chat, personal, programming

There was an article highlighted in a variety of nerdy news sites called “I’m A Good Engineer But I Suck At Building Stuff,” and while I was reading it, I felt the writer Lionel Barrow’s pain.

Because I’m a better engineer, but I still suck at building (some) stuff. When Barrow writes: “I find myself instantly criticizing my technique, to the point of paralysis. This function is hard to test; this object’s dependencies need to be injected rather than initialized internally; that module needs an integration test; and so on and so forth,” I don’t sympathize. Not only do I know how to do all those things, I know how to avoid them. A hard-to-test function should be broken down into public components; injected dependencies shouldn’t be a difficulty, they should absolutely be the starting point of your composition; integration tests are mastered by giving your interface the smallest possible surface area. When Barrow writes, “Even when writing spike or proof of concept code, I find myself revisiting the same lines over and over again, looking for the best, most natural expression of the ideas it contains — obsessing over my own construct, rather than on the thing my code does,” I remind my self that the very first thing my code does is communicate with other developers. Code is meant to be read first and compiled second. Yeah, maybe if you’re doing a prototype you can cut that corner once or twice, but get it out of your habit.

Where I work, my fellow engineers have a joke about “Elf’s de-cleverizing pass.” I work in Javascript but am a Lisp programmer at heart, and I start by writing everything in terms of transformers and partials wrapped inside and around map/filter/reduce, and then at the end I will remove the partials and break out the transformers into functions and make the code look like something my peers expect. Most modules I write end up as long, easily-followable lists of var expressions that end with a single side effect, because that’s what all code is: an expression that ultimately has some real-word consequence.

On the other hand, I feel Barrow’s pain because I, too, suck at building stuff. Modifying other people’s code is easy. Even if I’ve been tasked with building “something entirely new” for my employer, the skeleton of every application we’ve ever built is in our repository, and if I can find something that does 51% of what my new task calls for, the rest is more or less fait accompli.

But when it comes to my own projects, I get analysis paralysis. I wanted to write a simple compiler with a few ideas of my own, something that would make my life as a developer easier, at least for a few follow-on projects, and …

There’s way too much to know about compilers. Stephen Diehl’s wonderful (if woefully incomplete) articles on “Write You A Haskell,” James Longster’s expedition into his Outlet project, all the effort I put into Lisp In Small Pieces, parsing, hygiene and scope, type systems (oy! Type systems!), compiling with continuations… My eyes start to glaze over and I get this terror that I’m never going to be smart enough or have time enough to do anything at all. The size of my ambitions overwhelms me. I lose track of what I’m trying to do as yet another article on CPS Transforms and Nanopass Compilation comes to my attention.

The twin impulses of wanting something useful now and wanting to know everything now are, rather than helping each other, constantly at war with time itself. I’m sure it’s an ADHD problem that no amount of Adderall is going to handle, because it’s not about attention, it’s about desire. I never have this problem professionally, because when it comes to money both what must be done and what I desire to be done are more or less well-aligned. It’s about the projects I’m really passionate about; I don’t want to screw them up, and there’s no pressure to get them out the door. I really don’t know how to make these impulses play well together.

While I can’t comment too strongly on the performance aspects of this article on Green Thread performance, there’s a paragraph at the beginning that caught my attention:

If one looks closely enough at green threads, such as Go’s ‘go’ statement, it’s not hard to realise that they are really a flow control mechanism similar to ‘if’, ‘for’ or ‘while’ statements. Where ‘if’ allows you to skip a block of code, green threads give you a way to easily switch between different points of execution. Very much like ‘if’ or ‘while’ they are glorified jump statements.

To which my initial reaction was, “Yes, that was covered in Chapter 3 of Lisp In Small Pieces.”

In fact, everything you need to know about control flow is covered in Chapter 3 of Christian Queinnec’s amazing, now 20-year-old book. In terms of control flow mechanisms for programs, we haven’t invented anything new since that book came out. If you’ve read Lisp in Small Pieces and Why Functional Programming Matters, you’ve read everything you need to create modern, composable, performant software. The biggest insight in LiSP is that all of these mechanisms for code parallelization, threads, communication modes, concurrency, and so forth are exactly the same as “if”; everything else is (important) details, like performance, but the underlying nature of programming is unchanged: there is one and only one instruction that matters: conditional jump. All else is commentary.

Necessary commentary. Abstractions necessary to constraining the development process in order to eliminate error. But still, commentary; when someone expresses “surprise” that Go’s go is semantically similar to if, it tells me that they’ve never taken, or somehow missed, the fundamentals taught in their CS program.


Javascript Misunderstandings

Posted by Elf Sternberg as Uncategorized

In a recent post, Zisis Maras recently talked about some interesting work with overriding the prototypes for Storage/localStorage to provide key/value space for different sessions running in the same browser. For the longest time, the advice was “Don’t ever overrride the native functions or methods,” but a lot of people seem to be ignoring that advice in the post-IE 11/Chromium/Firefox era.

But one thing he said bugged me. He talked about “cloning” functions, and then said his “favorite trick” for it was _func = func.bind({}).

This is such a terrible description (and comment) that it needs a clarification. Javascript’s context management is fundamentally sound: it’s call-by-reference with strong closure guarantees. It has one problem, though: the this special variable.

When the Javascript interpreter encounters the function keyword, it builds a new function. Variables within the function can be bound, which means that they’re meaningful only within the scope of the function, and free, in which case they were meaningful when the function definition was encountered. When the function is called, those free variable retain the value they had at the moment the function was constructed1. We call the free variables the function had at its construction its context.

When Javascript was new and poorly implemented, the this variable was introduced. It was meant to provide additional context in browser-based environments; specifically, it addressed how event information was made available to the HTML-3 “onclick” attributes, which pointed to a function named in a string argument, as strings are the only thing you’re allowed in HTML. The this variable was then redefined by successive Javascript versions to describe a number of different features, including (a) the event context, (b) the global environment, (c) a local environment to which it is bound (this word is important, remember it) with the new operator. The last is how Javascript does object oriented programming.

When a function is called with new and if the object it returns has functions bound to a .prototype attribute, it is called a constructor and the bound functions are called methods, and when that object.method() call happens, .method() gets its parent object in its this variable. (If the constructor doesn’t return an object, this is returned automatically.)

In callbacks and other asynchronous operations, this can get lost. It’s not unusual in modern Javascript to define an object that has asynchronous needs (fetch data, write data, listen to DOM objects), and an asynchronous handler within the object will need access to two different instances of this: The event’s context, and the object the event is meant to modify.

Traditionally, we used to write var that = this; before the callback was defined, making that a free instance of the object, so inside the callback that became our handle to the object, while this remained the handle on the event context.

In Javascript 5, the Function.bind() function takes a function and an object, and returns a function in whith this is locked down so that if the interpreter attempts to provide an alternative within that function, it’s prevented from doing so. Since most modern event handler protocols pass the event context in as an argument to the event handler, we no longer need the event context this, and .bind() exists to make it go away (while still supporting applications written in really ancient Javascript).

This is a long-winded way of saying you never “clone” a function in Javascript, the way one might deep copy an array or an object. Instead, all you’re doing is assuring that the function has a set and unchanging1 instance of this.

Understanding this is, depending upon your point of view, either absolutely essential (as it is in my day job) or absolutely unnecessary (as it is in my personal software projects, where I use closures and hidden contexts instead). I would go with essential if you want to write Javascript professionally. But Maras’ contribution is unfortunately only going to make that understanding harder.

1This is strictly true, but since Javascript is call-by-reference, this can lead to surprises for new developers. If a free variable was an array or an object (and objects are always objects), if the contents of the object or array are changed outside the function, they’re changed inside the function as well! The free variable refers to the object or the array, and not their contents. When we say “unchanging,” we mean that the thing referring to is unchanged; not the thing referred to.

I have less than ten hours to teach a dozen surly high schoolers to go from opening their notepads to writing Tetris. This is the objective: to give them enough HTML, CSS, and Javascript to be able to write a very simple video game.

Here’s what I told them:

A high school programming class and a high school cooking class are more or less the same. At the end of a cooking elective, if you followed all the steps you were required in class, you go home with two things: a cake, and a recipe for cake. All of you are going to eat the cake. Some of you might try to make the cake a second time. A few of you will wonder, “If I change this chocolate to strawberry, and make the cake again, will it work?” And one of you might go on to be a professional baker.

The same thing is true of this class. At the end of a class, you’re going home with a game written in HTML, and the source code to that game. The game will be fully playable. You’ll even have the source code, so you could try transferring the game to other computers to see how it works there. A few of you will look at it and wonder, “If I re-arrange these things, can I turn Tetris into Pac-Man?” (The answer is “Yes,” by the way.) And one of you might go on to be a professional software developer.

Oddly, the teacher who organized the elective tells me that that’s the best and most succinct description of what an elective like this is trying to accomplish. So I guess I’m doing something right.

Of course, there’s already that one guy who has a three-d game written in Unity, and took this class so he could learn how to put up a website about it…

As most people know, I’m a Lisp fanboy. Which is somewhat surprising as I tend to think that it’s one of my weakest languages, and whenever I’m reminded that Common Lisp is a Lisp-2 I get the heebie-jeebies; I really like single namespace languages. (I tend not to think of C as a namespace language; it’s really just assembly language with a nice syntax, and C++ is C with a ridiculous preprocessor.)

One of lisp’s classic problems is that it’s utterly reprogrammable. With the exception of a scant handful of absolutely essential core syntax terms (classically: cons, car, cdr, cond, eq, atom?, lambda, set!, quote) everything in the Lisp language is accessible and re-definable by the programmer. With macros you can add syntax that re-writes, mangles, and just plain defies scope and closure, that reifies complex procedures into one-word functions, and just plain lets you make one language into a different language. Lisp would be cool, except every Lisper’s Lisp is a little bit different from every other Lisper’s Lisp. It’s like the C++ problem of generics versus templates versus the STL. C++ has those problems because each of those topics is so huge someone who’s mastered one has little mental bandwidth left to master the others; in Lisp, those kinds of issues arrive because the developer wants them to.

So imagine my horror when I discovered that Perl 6 not only has the same capability, but allows you to define these new operators in any position in the syntax. Just look at this example:

sub infix:<¯\(°_o)/¯> { ($^a, $^b).pick }
say 'Coke' ¯\(°_o)/¯ 'Pepsi';

See that ‘infix’ thing there? That tells the Perl parser that you’re about to define a new kind of syntax. Your choices are: infix, prefix, postfix, circumfix, and postcircumfix. If your use case mixes your syntax with other syntax and the precedence isn’t correct, you add “is looser than” or “is tighter than (other syntax)” to the subroutine’s interface definition!

I loved Perl 4. Perl 4 put food on my family for six solid years. Perl 5 seemed like a pretty good idea until Moose and Catalyst made it feel like Haskell without the safety rails. I like Lisp, and if you can tinker with advanced grammars at least you know there’s only one canonical function position and absolutely no precedence wrangling needed at all.

There’s an old joke that Perl is a write-only language; six months after writing it, even the original developer has little idea how to understand what he wrote. With the capability to arbitrarily re-define precedence order, syntax positioning, and even Perl’s basic grammar for your precious, delicate snowflake of a use-case, it seems that Perl 6 really intends to make that ancient joke a reality.


JSFiddle as Teaching Tool

Posted by Elf Sternberg as Uncategorized

The last time I taught the high school "web development" class, I struggled with the resources available to me. The school didn’t have space to host student work; the Chromebooks available didn’t come with a text editor (a goddamned text editor! The most basic editor on the system is RTF!); the Chromebooks had also had Page Inspector disabled, which has to be the silliest administrative decision ever made. There is literally nothing malicious you can do with the Page Inspector. I can only imagine that it was diabled to prevent kids from learning how the web works.

Late in the quaterly cycle I started to teach a little Javascript, and for that I remembered JS Fiddle. Since it has working HTML and CSS panes, this time I started with it.

That made a huge difference. By the end of the very first class most of my students were monkeying around with CSS and HTML, moving things around, changing color, scaling text. I’m going to have to up my game if I’m going to have enough material for these kids even with one week less than last quarter.

It’s still very limited as a debugging tool; without Page Inspector we’re either going to have create our own debugging windows or spew message to alert(), which is never any fun at all. That’s old school debugging. But it’s really better than nothing, and it makes teaching programming possible even at the high school level.

Akin’s Laws of Engineering apply strictly to physical systems. Dave Akin was an engineer at NASA who specialized in designing launch vehicles, and his laws apply to build things that go into space. Many of his laws he attributes to other people, but one that is his own is

Any run-of-the-mill engineer can design something which is elegant. A good engineer designs systems to be efficient. A great engineer designs them to be effective.

His example is: an ordinary city has an elegant water system; New York City has an efficient water system; Rome has an effective water system (parts of it date back to Julius Caesar and are still in use!).

I’m quite certain that the designers of the Roman and New York water systems wanted a system that worked and engineered their way into elegance along the way.

The thing that gets me about Akin’s Law of Elegance is that it seems to me to be exactly backward, on the one hand, and to be an example of why Go is popular and Haskell is still struggling on the other. Javascript’s map/reduce/filter are examples of things that are both elegant and effective: they reduce the messy and chronically off-by-one for loop filled with expressions and allocators into something much more readable: (expression, and things to express on). They trade those for efficiency, but software has a lot more ‘give’ than launch vehicles, even in something as small as a watch. If you want all three, then I have to suggest Lisp or Scheme: you can extract the sequencing and allocation out of the passes into a single, transduced pass without function calls, saving yourself a lot of memory and CPU.

Given that fact that so few developers care about elegance, I really wonder if we even have systems that are efficient and effective. Go is “effective” in that it trades developer cycles for CPU cycles, but the things written in it have no long-term guarantees of maintainability; Go creates the illusion of this with a hard style guide, but style and elegance are still too different things: you can hire someone to buy you stylish clothes, but if you don’t walk in them well elegance will be beyond you.

Then again, most developers I know still wear a t-shirt and jeans to work, so I don’t expect elegance to be a trend anytime soon.

Pandastrike has a really good article called Facebook Relay: An Evil And/Or Incompetent Attack On REST, in which the author basically takes Facebook to the woodshed for not understanding REST, trying to break REST, and generally being your classic embrace / extend / extinguish (or etouderie, a beautiful word that has sadly fallen out of the English lexicon) big company imposing its will on everyone else.  As a graphQL fan, I wanted to like Relay, but every time I played with it my principle reaction was, “Okay, what is this really for?”  Pandastrike goes on to say that it’s good for only one thing, namely social networking data at the massive scale Facebook faces.

But, Pandastrike make one really terrible faux pax of their own in the article.  They make a point of quoting Ray Fielding, but then in the section on REST endpoints and type safety, write:

Although JSON Schema is not part of HTTP proper… if you use content types correctly, and also use JSON Schema with JSCK, you get strong typing over HTTP.

This is true, as far as it goes.  But it misses two incredibly important parts of Ray Fielding’s work, and makes me suspect their intentions.  JSCK, you see, is a product produced by Pandastrike.  And Fielding himself has said that doing REST with JSON is incredibly hard.  So hard, in fact, that the original work in REST mentioned that the transfer of representational state automatically implied hypertext as the representative of state transfer.  JSON is a terrible tool for hypertext.  You know what’s a great tool?  HTML: Hypertext Markup Language.  It’s not just for the browser and the page itself, it’s for every transaction that you commit between the browser and the server, and it carries with it all the metadata needed to make sense of the content.  Even better, unlike JSON, HTML has its own type checking mechanism that is part of the HTTP/HTML dual standard: its DTD, or Document Type Definition.  You’re not required to use the whole HTML standard, and you can even use XML with a stripped-down version that still comes with a DTD.

Pandstrike goes on about Facebook’s raking attack on a behavior scheme that’s been around for, oh, call it ten years.  But HTML and DTDs have been around for twenty years.

I’ll be fair: Working with HTML and XML on the browser is painful compared to JSON.  It uses more CPU to render and it takes more tooling to program correctly.  Nobody does it that way.  But to ignore it and imply you have a magic solution to an unsolved problem is to be as deceitful as the people you’re criticizing.



May 2016
« Apr