On Monday, work announced that we would be adopting Google Go as a third official internal language. Most software houses have two official languages: a highly demanding systems language with strong performance and memory guarantees, and an easy-to-write scripting languages with mediocre performance and few memory guarantees, so a third language was a bit unusual. Work’s language pair was C++ and Python, a highly conventional and conservative choice1.

My commute takes me past a bookstore so on the walk home Monday I bought Introducing Go and read it on the train, and the next morning read the rest. It’s a tiny book. We’re at the end of a project at work so I had spare time. I chose a little toy project.

I’ve been trying learn Rust, and I have a little server project called Monologued that I’ve been hacking on since early December. I haven’t gotten it working. I already had some major components working, and I decided to try at write Monologued in Go.

It took about six hours.

To be fair, I had work’s support at learning Go, and I don’t have it for learning Rust, so I could use office time to write the Go version, whereas my Rust time has to compete with my writing time, which is about 35 minutes twice a day, in a crowded transit train.

Here’s what I discovered:

Go is a good language for writing middling-performance servers. It’s concurrency model is brilliant and easy to use, and the garbage collection makes it easy to forget about memory management. The error handling is old-school but clear and fairly concise. The compiler is clearly optimized for the cognitive support of modern, "smart" IDEs, but I was just using Emacs and still managed to write the server in a short period of time.


I hated it.

Have you ever watched one of those music videos where very talented musicians take children’s musical instruments and do something utterly and mind-bogglingly amazing with them? There is something quite astonishing about watching someone take all the skill they’ve spent thousands of hours mastering only to wring lovely music out of toy pianos, plastic recorders, kindergarten bongos, and little wooden xylophones. It’s only when you step back for a moment and realize that if they sound that amazing on toys, they must be even more amazing when playing on instruments meant for adult hands.

That’s how I feel writing in Go. I didn’t learn anything writing in Go. It’s a language with guard rails, a language of Lego pieces. It’s "batteries included" library is so complete it has all the parts needed to build web servers instantly. Go is such a simple language that if you’re coming from Python the syntax enforcement will drive you crazy for a while but then, like Python’s whitespace, you’ll get used to it, and you’ll like the lack of the global interpreter lock. If you like Python 3, Go’s Unicode management is lousy and you won’t enjoy it, but it’s workable. If you’re coming from C, the tuple return type and memory management will make you very happy, and otherwise you’ll be completely in your element. All the headaches of POSIX-style networking are hidden behind a pretty API.

No one will ever write great software in Go.

Look at the great programs of the last 25 years. Look at the LAMP stack: Linux, Apache, MySQL, PHP— all written in C. The alternatives: FreeBSD, Microsoft Windows; Nginx or Lightspeed; Maria, Postgres, or Mongo; Perl, Python, Ruby— all written in C or C++. The MS Suite of Word, Excel, Access, Powerpoint, OneNote, Publisher— all written in C++. Almost all the great software of the last 25 years was written in C or C++.

This is not to say that I think C and C++ are perfect languages. They’re not; their closeness to the operating system and the hardware, the features that give them their immense power, also make them notoriously hard to master and exceptionally difficult to get right. D attempts to correct C++’s issues but hasn’t gotten much traction. But Go gets something right: We need something faster and smarter than Python without the nightmares of C and C++.

Rust takes a different tack: it learns from C and C++’s mistakes without hiding from your awareness the underlying issues of talking to an OS. Its rigid memory model takes a lot of effort to master, but it’s completely worth it. When you master Rust, you’ll have mastery over the bare metal, the machine itself. Go is the dull knife of programming.

I fully acknowledge that Go is a very good language for what it does. It meets a very important business need: it makes developers productive, and it lets them get product out the door in very short periods of time, and it’s stupid simple. It makes shareholders happy. It takes all the lessons learned since Pascal and reifies them into a usable language; it takes all the lessons learned since Lisp and ignores them completely.

motivation-001 Working in Go does not suck. The problem is that if you’re not working in something that sucks, you are not growing. You’re stuck. Rob Pike, the co-inventor of Go (and Unix, and UTF-8, and a ton of other things) once said

The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt. – Rob Pike

And that’s my problem with Go. It doesn’t encourage growth. A developer working in Go and only Go will help his business’s bottom line, and the shareholders will be happy, but he will always be a mediocre programmer which is why mediocre coders are so fond of Go. That quote above is on the wall in my office, and I try to live by it. You should too.

The alternative is that you will not be Rob Pike, a guy who’s still making changes in the industry even though he’s ten years older than I am. You’ll be writing the same thing for twenty years and then, when you’re forty, you’ll know nothing else… and then what will you do?

1In almost every job I’ve head, the pair has always been C or C++, Python or Perl. The oldest job I ever took had just COBOL; the oddest pair was Java and Perl in 2010, but that can be explained by the fact that it was a publishing toolchain and there’s a ton of Perl for managing LaTeX. Isilon’s choice of Python was a bit idiosyncratic; in 2000 chosing Python was considered radical, but 18 years later Python has completely overwhelmed Perl as the scripting language of choice, so the choice now seems prophetic. I’m pleased that I’m the one who made that choice.


Logbook: Working on Monologued this week.

Posted by Elf Sternberg as Uncategorized

Problem: I’ve been trying to get a simple, single-threaded server working under Rust, using the MIO (Metal I/O) abstraction over select/poll/epoll/kqueue since that’s what most closely tracks with my experience as a server writer. Unfortunately "my experience as a server writer" more or less came to halt with the collapse of Sprynet back in 1998. The objective here is to write a framework supporting older line-based protocols such as NNTP, POP3, and Gopher.

Attempted Solution: I’ve tried to abstract the sever out into several different layers: the server itself, the connection the server has to any given client, and the protocol. The connection marshals the input into lines, which are then placed in the protocol’s queue. The protocol generates responses, which are then placed in the outbound queue. The connection sets itself writeable if the protocol says there’s content to be delivered, and read-only if not.

Attempted Process: It looks more or less like any mid-grade developer’s server, with ridiculous craptons of guard statements, automatic panics, and at least one weird construct to "get around" the borrower. Inside the event loop I had to use an indexed get() into the array because the borrower didn’t like me using an iterator, claiming I’d borrowed a mutable value twice. I’m still waiting for the Rust "borrower insight" everyone claims is coming but I have yet to see.

Result: Well, it works. The only protocol generated so far is an embedded line-based Echo protocol. It’s still ugly. The number of additional libraries dragged in (net2, bytes, log, env_logger) balloon the server to almost 12MB in size; the equivalent C program, without all the niceties, comes out to about 18KB. Something is wrong here, but I’m not sure what.

Future work: Clean this thing the hell up. It’s neither functional nor SOLID, which means it’s just a heck of a mess. It’s a manageable mess, I suppose, but still a mess. Investigate if the executable size is the result of static fat binaries, to which I don’t object too strongly. Unembed the protocol.

New work: autorotate. As I mentioned earlier, I managed to score a brand new Microsoft Surface Pro 3 for $230 (not including the additional cost of a new keyboard, stylus, and power supply… sigh) from a woman who much preferred her MacBook and was moving and had to get rid of everything she didn’t want to carry. I even checked with Microsoft to see if it was registered as stolen, and bought it on the spot.

If you see my earlier post about getting it working, you can see that I liked Ayko Poel’s autorotate script. And it’s true, I did… until I bought a stylus and tried to draw on the thing in portrait mode. Poel’s scripts broke. There’s a bit of deep, almost occult esoterica about mapping screen touches to drawing coordinates, especially when rotating the screen, and Poel’s scripts failed at the intersection of advanced touch devices (the pen and the eraser) and coordinate transformation.

So I fixed it.

Along the way, I removed the huge horkin’ if statement and replaced it with a transformation matrix which encapsulates all the rules quite nicely. It actually doesn’t change the line count (sadly): Detecting if you’re in landscape or portrait is two expressions, and which landscape or portrait is two more expressions each, followed by a four-line table that maps these rules to transformations and labels. But I think it’s a hell of a lot more maintainable. I also fixed the palm rejection code (which is now a heck of a lot shorter) to understand that the pen and eraser both require palm rejection.

I changed device recognition to be a heck of a lot more robust, and I used Python’s namedtuple to make this all much more readable. I used a lot of regular expressions— but then, I’ve been good at regular expressions since 1993— and that lets me recognize the devices, the drivers, and everything else without any fiddling or meddling.


What I’m Doing Right Now: Monologued.

Posted by Elf Sternberg as Uncategorized

For the past couple of weeks, I’ve been learning a lot of Rust. As is my wont, my usual way of learning a language is to decide to write something in it, and see what comes from doing it. A while ago, I learned that John Carmack still regularly updates his .plan file, which is a text file that was once available through the Remote User Identifier Protocol documented in RCF-1288, popularly known as the finger protocol. The project is named Monologued.

I decided I wanted to write a finger server. The version I’ve decided to write is very stripped down. It doesn’t support user lists by default, it doesn’t support host-hopping by default, and it doesn’t even support user presence at all, which was the original intent of the protocol. What it does support is returning the user’s .plan file. It’s fairly paranoid, honoring .noplan over .plan (if the former was present, finger wouldn’t even acknowledge that the user requested existed), taking care not to leak usernames, and prioritizing the /etc/finger.conf exclude list over the include list.

I have learned a lot about the finger protocol and it’s threadbare implementation during my research. For example, there’s a Unix program, who, that will tell you who’s currently logged in to a given host. The fingerd server doesn’t even run on its own; it’s launched by inetd, a helper program for small network utilities not much in use these days, which hooks up finger to the internet via stdin/stdout. fingerd in turn runs the local finger program, which doesn’t even do any work– instead, it figures out who you asked for, runs who to spew the user’s identity and presence, and then runs cat on the .plan file if it exists. That seems like a bit of a security nightmare.

My object is to learn Rust, and to learn MIO (Metal I/O, the lowest-level Rust library for networking that’s not just wrapping the C network library yourself). Everything will be contained in a single executable. The progression I’ve planned is straightforward:

  1. Write a simple echo server, as shown in the MIO tutorials.
  2. Separate the Connection object from the Server object.
  3. Write a parser so that instead of returning an echo, we return the command found.
  4. Implement the command found.
  5. Secure the command found with the rules described above.
  6. Implement the rules as a configuration file.
  7. Implement the rules as command-line overrides.

Stretch goals:

  1. Implement a cache with arbitrary Sized<>-based ejection rules for smaller .plan files.
  2. Implement an fstat-based layer to eject .plan files that have expired.
  3. Implement an Inotify-based layer for MIO to eject .plan files that have expired.

Super Stretch goals:

  1. Implement opportunistic encryption (STARTTLS) for the Fingerd protocl.
  2. Modify the configuration so opportunistic encryption can be required.
  3. Write a client to demonstrate opportunistic encryption of RUIP.
  4. Document and submit an RFC for "Using Transport Layer Security (TLS) with the Remote User Identity Protocol (RUIP / Finger)."

Right now I’m working on 4, but I may have to backtrack as I think I’ve become overly entangled via 2, and may need to think harder about separating the read/write phases.

An acquaintance of mine claims that the last one is an “epic troll,” but I don’t know; not all of those old protocols are worthless.

One thing I forgot: in the book Ansible Up & Running, in chapter 3, there’s a Vagrantfile describing three containers with consecutive ports exposed and mapped to the host OS.  Each VM is defined individually, by hand.  I had read that a Vagrantfile is just Ruby, so I thought, “Screw that.  I’m gonna use a loop.”

It turns out you can’t just use a loop.  Because the VMs are booted asynchronously, they need closures.  You can replace the entirety of the Vagrantfile in chapter three with this:

config.ssh.insert_key = false

(1..3).each do |host|
  config.vm.define "vagrant#{host}" do |vagrant|
    vagrant.vm.box = "maier/alpine-3.4-x86_64"
    vagrant.vm.network "forwarded_port", guest: 79 + host, host: 8079 + host
    vagrant.vm.network "forwarded_port", guest: 442 + host, host: 8442 + host
    vagrant.vm.synced_folder ".", "/vagrant", disabled: true

The do |host| syntax creates a closure that ensures that the host variable will be different for each VM, run in a separate scope. Using a for loop, the host variable would be “3” at the end and when vagrant picks up the entry, all three VMs will try to allocate the same port, resulting in conflict and crash.  Again note that I’m using Alpine here.  If you’re using Ubuntu as the book recommends, make sure you change the container name.

For professional reasons, I’m working my way through the O’Reilly book Ansible: Up & Running. The book is pretty good, explaining the basics of Ansible so far as well as running me through a couple of exercises. The book uses Vagrant as its target container, which is fine by me. As is my wont, I made my life harder by changing things up: rather than using Ubuntu as my target OS, I chose to use Alpine, for which the base Vagrant image has no instance of Python. I also chose to automate my generation of Ansible’s hosts file.

The only secret to using Alpine over Ubuntu is simple: Alpine doesn’t ship with Python, and Ansible requires Python. Ansible has a work-around, though: RAW mode.

#!/usr/bin/env ansible-playbook

- name: Install Python on Alpine
  hosts: servers
  become: True
  gather_facts: False
    - name: update apk
      raw: apk update
    - name: install python
      raw: apk add python

Commands delivered raw will be fed directly to the shell, and so don’t require Python. They’re treated almost, but not quite, like -m command style commands. The latter still run through Python, which still seems odd to me, but Ansible is an okay tool and I’m not about to try and re-write it to my satisfaction.  become is still necessary, as installation commands still need to be run through sudo, and gather_facts must be False for this pass as fact-gathering is performed with Python.

Automating the translation of Vagrant’s auto-generated SSH configuration to something resembling Ansible’s host file output was trickier. My first thought was to use Perl, but then I remembered that when it comes to multi-line patterns, Awk is still the master of the game. I use Gawk here, but that shouldn’t make a difference.

vagrant ssh-config | \
    gawk 'BEGIN { RS="\n\n"; FS=" " } \
          /Host / { print $2 " ansible_host=" $4 " ansible_port=" $8 }'

Alpine’s layout of nginx, which is the server used in the tutorials, is different from that of Ubuntu. I’ll leave it up to the reader to figure out those differences. That shouldn’t be too hard.

I was reminded this week of Steve Yegge’s Programming Dirtiest Little Secret, in which Yegge claims that touch typing is the most significant skill a programmer can have, because the very mechanics of hunt-and-peck typing use up brain power that the professional programmer cannot afford to waste. Yegge says that the important stuff of programming, you have to have completely mastery of the device of expression; your computer must become an extension of thoughts to be expressed, not a tool to be managed.

I’m a touch typist. I wrote 2400 words this morning in about 90 minutes, without looking down once at the keyboard. I can’t look down at the keyboard; it won’t help, as the keys still read QWERTY and the mapping is set to Dvorak.

I am also, by any measure, an exceptionally good Python programmer. When I need to do any kind of data processing, web programming, scientific visualization, I reach for Python. It has all the libraries I could possibly want, it has a syntax that is reachable and easy to grasp, and it’s just plain hard to get wrong. I know the inner workings of the Cython VM to a ridiculous degree, have a passing knowledge of how Stackless and PyPy work inside, and can go on at absurd length about the brilliance and historical brokenness of Python’s import function. My Stack Overflow reputation is over 12 thousand, and half of that comes from answering Python questions.

The other day I wrote a small utility to analyze my personal projects directory and spit out a one-line description of each project, or “No description found” if it couldn’t figure out the description. It took about an hour. During that hour I:

  • looked up the documentation for os.path
  • looked up the syntax for Python’s with/open
  • searched for how to find the HOME directory in Python
  • searched for how to set maxdepth on os.walk

That is, during that hour, I knew that each of these features existed, but I didn’t know how to use them. I have places in my head where the vague knowledge that these capabilities exist resides, but no real library that lets me express myself fully without having to pause and go read some documentation or blog entry an how to do such-and-such.

If you read any advice on how to win customers, the advice is always: details, details, details. In this industry, though, there’s almost never time or money for the details. Things move too fast. We develop the minimum viable expertise necessary to create the prototype, at which point the sales department starts shouting, “Ship it!” We never have enough time to learn how to express ourselves in a given development environment before everything changes again and we start the learning process all over again from the beginning.

It is this ability to express our intent that we need to fully develop. We need to slow down and master our development environments before we can earn the time needed to attend to the details. Otherwise, we’re simply at the mercy of market forces to go faster and faster, and we’re all going to lose that race in the end.


Shiny New Linux Surface Pro 3!

Posted by Elf Sternberg as Uncategorized

So, after a heartbreaking month of having a Surface Pro 3 that was utterly unstable and unusable, I now seem (emphasis on seem) to have it completely stable and successfully running an instance of Linux Mint 18.2.

Required Equipment

Here’s all the things I did, probably in the order I did them. First: buy some stuff: A powered hub with at least four ports, a USB-to-Ethernet connection, and two 16GB USB memory sticks. If you want to preserve the installed Windows instance, you’ll need a second 32GB USB memory stick. (You’ll have to go on-line to find instructions on how to back up your Windows instance.)

I assume you have a USB-capable keyboard and mouse. Download the latest version of Linux Mint. For Ubuntu-based distributions, you must also go and get more up-to-date kernels. As of this writing, I used the Linux Kernel 4.12.8, amd64 versions. There are five files in that collection; the three you want are the headers, the generic headers, and the image; skip the ones label "lowlatency".

With another Linux computer, use unetbootin to install the Linux Mint ISO on one of the 16GB sticks. Do not reformat the drive to a Linux format; let unetbootin install the contents of the ISO into the FAT32-formatted stick.

The other 16GB stick can be any format Linux recognizes, and Linux recognizes FAT32. Copy the linux kernel files you downloaded onto it. You will also want to copy the latest versions of the MRVL firmware onto the stick to update the Surface Pro’s wifi firmware.

Installation of Linux Mint 18.2

Plug the hub into your Surface Pro. Into the HUB plug: the ethernet cable, hooked to a live network; a keyboard; a mouse; the 16GB stick with the bootable ISO on it.

Turn on the Surface Pro while holding down the "volume up" button; this will give you access to the BIOS. Disable "Secure Boot" and "Trusted Platform," and changed the boot order to USB → SSD. Exit and let the boot process proceed. If you’ve done it right, you’ll get the "Welcome to Linux Mint!" page. Do your thing: Install Linux Mint. I just blew away my Windows install; I didn’t care to keep it. That’s not the purpose of this exercise anyway.

When I tried to reboot, nothing worked. It kept insisting that there was no bootable medium! I found this solution, but I had to change hd2 to hd1 as I am not dual-booting. Once I was in to my instance of Linux, I was able to see via efibootmgr that the master boot record was corrupted; I installed and ran YannUbuntu’s boot-repair, and to my pleasure it actually fixed the problem the first time.

I rebooted back into the BIOS and switched back to "Secure Mode" for boot, and continued, and it worked fine. Yay!

Upgrade to the latest kernel

My hopes were dashed, though, when the network proved to be just as flaky as it had been earlier. But I had bigger problems: the keyboard and touchpad didn’t work. To fix that, put in the memory stick with the linux kernel image on it and type:

dpkg -i linux-*.deb

(If you downloaded more than one kernel, you’ll have to find a way to distinguish them.) Now reboot. If you have a kernel past 4.11, your keyboard, touchpad, and touchscreen should now all work. Yay!

Hibernate and Wifi

I did get the wireless working, too.

  1. I updated the firmware to the latest versions by copying the files in that directory to /lib/firmware/mrvl
  2. I added to /etc/NetworkManager/NetworkManager.conf:

    [device] wifi.scan-rand-mac-address=no

  3. I edited /etc/NetworkManager/conf.d/default-wifi-powersave-on.conf and changed "wifi.powersave=3" to "wifi.powersave=2".

The one thing I did not do, which I did last time, was blindly drop my old .config directory onto the new machine. The last time I tried this, I built the box, then restored from backup my personal account from my old Lenovo Yoga. This included all the keys, passwords, configuration details and some cached data (because idiots keep using .config as a cache directory; I’m looking at you, Google Chrome!) from the old NetworkManager, and it appears that some of those changes were what broke the old one.

The next step is to disable "sleep mode"; the Surface Pro doesn’t have a sleep mode! Instead, it’s either "off," "on," or "connected," the last of which is much like your phone when the screen is off: still processing network events and handling notifications. The Linux kernel isn’t up-to-date on this state, and almost no applications understand connected mode, so it’s useless. The best we can do is hibernate. This isn’t that bad because the Surface Pro has an SSD; booting to a hibernated screen is about 15 seconds, not terribly shabby. Not the ~3 seconds of my Yoga, but livable.

On Linux Mint 18.2, these commands were:

cd /lib/systemd/system
sudo mv suspend.target suspend.target.orig
sudo ln -s hibernate.target suspend.target

This links the systemd (gag) suspend target to hibernate, so that commands to suspend trigger hibernation instead. It’s actually pretty functional. Unfortunately, the mwifiex driver is still flaky, so you’ll have to handle it by hand. The following script goes into /lib/systemd/system-sleep/mwifiex, and has to be chown root mwifiex; chmod 755 mwifiex in order to run correctly.

set -e

if [ "$2" = "hibernate" ]; then
    case "$1" in
        pre) modprobe -r mwifiex_pcie mwifiex ;;
        post) modprobe mwifiex_pcie ;;

Now, theoretically, you have everything. Hibernate works and recovers; wifi comes back.


Bluetooth works out of the box with the 4.13.0 kernel, but the LE features only work with the very latest version of bluez, and even then you have to modify /lib/systemd/system/bluetooth.service, changing the Exec line to this:

ExecStart=/usr/libexec/bluetooth/bluetoothd --experimental

Adding the "experimental" line will let things like Bose products work with bluetooth.

Also, if you’re a pulseaudio user (and if you’re on Ubuntu or Mint, you are), add this line to the end of /etc/pulse/default.pa, and when your headphones connect Pulseaudio will automatically re-route the audio, rather than your having to do it manually.

load-module module-switch-on-connect


The cameras worked out of the box. I played with Cheese and it liked both cameras.


The best autorotate script I’ve found yet is Ayko Poel’s SurfacePro3 Scripts. It does automapping of the pen for you, which is really nice.


I really like Touchegg. I use QComicbook to read hentai comics, and by configuring both autorotate and touchegg’s configuration, I get an easy, pagable interface that I can use to flip through the pages as fast as I can.

Gimp and Krita

Both worked out of the box. Pen pressure worked out of the box, but getting the eraser and the button required a little configuration. Now that they’re working, though, I have a pretty good casual art pad as well as a decent writing machine and even a competent coding machine.

Next steps

I’d like to get autobrightness working. The bluetooth configuration is still a little flaky, but better than it used to be. One thing I’d really like to get is to map the accelerometer and/or the gyroscope devices to a joystick device, so that you could use the tilt of the surface pro as a kind of joystick. I once wrote a lightsaber toy that used the old Thinkpad HDAPS sensor as a joystick, and porting that to the Surface Pro (and maybe to Rust) would be a pretty good exercise.


“What Motivates You?”

Posted by Elf Sternberg as Uncategorized

I won’t link to the Orange Website for Hacker Bros, but a question asked yesterday has collided with a conversation I had had ealier this week. The question was: “What motivates you to do what you do?” I had a simple answer. These guys:


If you’re not familiar with the briliant Pixar film Inside Out, the film depicts an adolescent girl named Riley, and her internal struggle and emotions as she comes to grips with her parents’ decision to move to the big city. Those are three of  Riley’s five primary emotions, from left to right: Disgust, Anger, and Fear. (Not shown: Sadness and Joy.)

Almost everything I’ve ever done has been due to an emotional reaction, usually one of the three above. While I find my fiction writing to be a joy and a pleasure these days, three million words started with disgust: “Good grief, people, why are you liking that horrible Brady Bunch fanfic so much? It’s terrible! Anyone can do better than that. Let me show you.”

A lot of my smaller contributions to other open source projects have been due to anger. I was angry when my joystick didn’t work after getting Freespace compiled on Linux, and thus an obscure driver was born; I was angry when I couldn’t get my porn off Usenet, and thus the bugfixes to Python; I was angry when Evernote changed its policies, and thus my Enex ripper; I was angry when Delicious shut down, and thus my Delicious ripper. And so on.

The biggest emotional reaction I have, though, is fear. Fear that I’ll be obsolete someday, fear that as I get older, I won’t be so good they can’t ignore me, no matter how many grey hairs I’ve got, fear that my skillset will become irretrievably outdated if I stay in one place for too long.

I actively envy a lot of developers who find cool stuff to work on and really seem to get a thrill out of the long-term chase. I wish I did. I do get small frissons here and there, and there have been times where I have glimpsed something rare and wonderful, chased it down and admired it for what it was. I love learning new things.

Mostly I love how learning new things keeps the fear at bay. Which is probably not healthy. But it is what I’ve I got.

I’ve been struggling with the notion of theories and provers in the context of computer programming for a long time now. I was reading Peter Naur’s general concept of Programming As Theory Building and I don’t see how it connects to Type Systems at all.

In Naur’s thesis, there are three kinds of objects: (1) The world of physical things and events, (2) the world of mental objects and events (qualia?), and (3) the actual products of thought such as theories, stories, myths, tools, and social institutions. Programming, Naur says, is about the creation of objects of the third kind: the reification of thought products into processes. It isn’t enough to be intelligent; one can intelligently and skillfully perform all sorts of actions such as driving, typing, dancing and so forth; one must also have a theory, a context backing your actions such that you can meaningfully and coherently explain your theory, answer questions about it, argue on behalf of it.

In Naur’s Theory Building View of Programming (1985), the act of programming is the construction of a theory about how human processes can best be supported by one or more computer programs. This sorta coincides with Yourdon’s premise (1989) that one can’t begin to cybernize (Yourdon’s word) a process until one clearly understands the process he is seeking to enhance or replace. Prior to the Age of the Internet, Systems Analysis and Design was a serious discipline (I’m a Trained Cyberneticist™, kids, don’t try this without professional help!) in which the first step was to completely document the end-to-end product of some paper-pushing organization, and then try to figure out how to best enhance or replace that organization’s paper-pushing with a more efficient computerized version. It wasn’t enough to replace notebooks and carbons with keyboards and screens; sometimes, you had to replace whole systems with streamlined or adapted versions. We were taught that Analysis and Design was a holistic discipline, not just a programmatic one.

One place where I think Naur breaks down, and this may be due to the tools and processes available to him at the time, is where he says that he believes it is impossible to embed enough of the Theory (the outcome of the original team’s analysis and design) in any given program for it to be clear to other programmers without further input from the original team. The Theory is clear only in the team’s collective intelligence as they develop the program, and the program only articulates some mechanistic processes that support the theory.

I also believe that Naur falls down in the face of modern development because the disciplines of programming simply weren’t all that great in the 80s. Too much of it was dedicated to trickery to get more performance out of the system; the Theory embodied not just what the program had to do, but specialized assumptions about the hardware that would support it. Finally, programs today are built from tens, perhaps dozens, or even hundreds of libraries; I don’t believe we actually need a Theory for each library to competently use them in our productions.

I do believe we need a Theory for our own production, to explain why we need so goddamned many libraries, however.

Naur claims that few software development houses work according to a Theory. He may be right. There are tens of thousands of "IT shops" in this country, and most of them don’t really care. They just want to develop Yet Another App, their knowledge of the underlying technologies, be they HTTP, SSL, virtual machines, containerization services, and databases, is so tenuous as to amount to empirical daemonology: poke it and see if it pokes back. But that doesn’t matter, as CPU power is cheap and there’s enough scaffolding holding the whole leaking, clanking, steaming, whistling contraption to lurch forward like Howl’s Moving Castle, such that they don’t have to care. There are days when I feel like I’m working with a theory that is, at best, vague and hard to define.

As near as I can tell, the terminology of "theory" in Type Systems has some overlap to Naur’s: The programmer uses Types to constrain what the program does to fit within the Theory in the programmer’s head, and uses the Type Checker to assert that the Theory is complete and comprehensive, but I don’t think that the notion of "provers" in Type Systems has anything to do with Naur’s thesis.

Naur’s Programming as Theory Building is still relevant 30 years after its publication (and the year I started my CS degree program), but it describes challenges most of us no longer face, and was written at a time when the cognitive support tools were a desert wasteland compared to what we have now. It’s worth reading, and it’s worth supporting, because if you write software and you don’t have a clear theory about what the program will be and what it will do, you can’t write a good program.

Subscribe to Feed



January 2018
« Nov