paulsmith/superpowers - jj-ified fork

Superpowers is a consolidation of effective patterns for developing software with coding agents. It's been refined over many iterations to get more reliable performance from agents, especially for tasks too complicated to do without planning.

Like most agent tooling, superpowers assumes Git and bakes in workflows around it. My patches replace Git for jj and translate all Git-related concepts to their equivalents in idiomatic jj. For example, it instructs on the use of jj workspaces instead of Git worktrees.

If you use jj for your repositories, give it a try.

Keeping the patchset maintained is a straightforward task. After fetching from the upstream, I make a new revision on the main@upstream bookmark (tracking the remote branch) and rebase the patches onto it. This usually yields conflicts, but they are easy to fix. Then I will scan for any new untranslated Git-related changes since the last update, and incorporate them if necessary. I eyeball all the patches. Finally, I advance the jjify bookmark and push to GitHub. I have jjify set as the default branch in GitHub, so it appears at the repo's main URL.

I've done this patchset maintenance enough times that it was time to make a skill. I've copied the skill to this gist.

jjq will automatically test to see if the change can be cleanly rebased or merged on the trunk commit, and run a user-configurable check command (think: a command you might run in CI, like make test, cargo clippy) on the candidate new trunk revision. If the check passes, the new revision becomes the new trunk, and the bookmark (usually main, etc.) is moved to it.

paulsmith/jjq - local merge queue for jj

jjq helps with the problem of having many agents working on new features, bug fixes, and refactorings in parallel, and attempting to land their changes concurrently on the trunk.

We assume that the trunk is the head of a timeline in the repo that is "always green" or "blessed", where the tests always pass and there are no lints to fix.

If we want to keep trunk that way and reduce the potential chaos, but also maximize the amount of work that can be done on our repository at the same time, we could use a regular merge queue like the ones that hosted forges provide.

Merge queues help prevent PRs from becoming too stale to merge, because the trunk has moved since it was submitted. There are likely merge conflicts and/or failing tests that would result were the PR to be simply merged or rebased onto the trunk.

We could use the forge merge queues, but there are reasons we might not want to. The main reason is that in an environment with a lot of parallel development happening at once, the chance of a failed merge (either due to conflicts or failing tests) is higher. We want to quickly resolve this situation, ideally by reusing the context window of the agent responsible for the change.

You may have reasons for not using a centralized or hosted forge, or be using one without a merge queue feature. In any case, jjq can help increase velocity of agents on a repository locally.

One nice thing about Git is its support for pre-commit hooks. jj doesn't support pre-commit hooks. You could think of jjq as an advisory pre-submit hook.

How to use jjq

Get jjq via Homebrew brew install paulsmith/tap/jjq or from a GitHub release.

To use jjq in a jj repository, you initialize it one time with jjq init. This will prompt you for a trunk bookmark name (defaults to main), your check command (eg., make test), and optionally a landing strategy, whether you prefer rebasing (the default) or merging.

When a change is ready to be submitted, use jjq push <REVSET>.

$ jj log
@  nylxuwxp paulsmith@pobox.com 2026-02-24 11:22:43 5cf2bdd8
│  file b
│ ○  oqrsoyvx paulsmith@pobox.com 2026-02-24 11:22:21 e5fcf50c
├─╯  file a
○  mtswnruy paulsmith@pobox.com 2026-02-24 11:21:31 main 28f2b146
│  init
~
$ jjq push oqrsoyvx
jjq: revision 'oqrsoyvx' queued at 1 (trunk: main in /private/tmp/foo.LWAvuB)

You can inspect the state of the merge queue with jjq status.

$ jjq status
jjq: Queued:
  1: oqrsoyvxpptw file a

When you are ready to process the queue, use jjq run.

$ jjq run
jjq: processing queue item 1 (rebase strategy)
jjq: rebased 1 to main (now at oqrsoyvxpptw)
$ jj log
@  nylxuwxp paulsmith@pobox.com 2026-02-24 11:22:43 5cf2bdd8
│  file b
│ ○  oqrsoyvx paulsmith@pobox.com 2026-02-24 11:30:28 main dbf49b34
├─╯  file a
○  mtswnruy paulsmith@pobox.com 2026-02-24 11:21:31 28f2b146
│  init
~

Notice that the main bookmark has been moved.

For more on using jjq, type jjq --help, or jjq quickstart (the latter is usually enough for agents to use it effectively).

jj

I've long felt that jj is a nice fit with agentic coding patterns. jj's core model, different from Git, that the working copy is always committed, and revisions are mutable and easily reordered, rebased, and modified, aligns well with the slightly chaotic world of agents making spikes, refactoring, writing plans, trying things that wind up in dead ends and backtracking, etc. I never worry that an agent is going to make a hash of my repo, because I (or it) can always undo or inspect the op log and restore the repo to a known-good state.

jj workspaces are especially convenient for agents. Any task they do, I have them perform in a new workspace. Workspaces, which are isolated working copies in separate directories, inherit all the advantages of jj, avoiding the frustrations and limitation of Git worktrees.

A productive workflow for me is to have agents going in their own workspaces and attempting to land their changes onto the trunk via jjq. Then either I or a supervisor agent can run the queue, periodically or continually, and feel confident that if the change lands it won't break the build. If it fails, we can quickly reassign the agent to address the cause and resubmit.

Origins

jjq has become a solid building block in my engineering when using agents. It started life as a Bash script, driving the jj CLI and parsing its output.

(Incidentally, jj's templating feature for CLI output essentially provides you with your own custom stable API, emitting just the output your tool needs for its job. This is a nice alternative to using the jj library, which is not an option for non-Rust programs, and even for Rust programs, since the CLI handles some things on your behalf like reading user configuration.)

After using jjq as a script for a while and refining the interface (mostly by paving the cow paths that the agents made), I had Claude Code port the script to Rust.

I did an intermediate step, which was writing a RFC-style specification document, treating jjq like an abstract interface, spelling out what features and behaviors conforming implementations may or must support. The spec is still in the repo history. I'm not sure, not having a control group, whether this made the port more or less successful, or faster to implement. (There was the cost of writing the spec in the first place.)

Given how good agents are at porting from one language to another, it perhaps was a waste of time to write up an RFC-style spec. But I've wanted to explore trying some evals with the idea on future projects.

With the current release as of this writing, when you add a workspace in a jj repo, an empty commit with no description is created.

$ jj workspace add /tmp/example
Created workspace in "/tmp/example"
Working copy  (@) now at: xouyultn fbf52c12 (empty) (no description set)
Parent commit (@-)      : vvyvvnsr d55579fd Initial
Added 1 file, modified 0 files, removed 0 files

It's simple enough to add a message after the fact with jj describe. (You can do this without changing directories to the workspace by using the workspace's name suffixed with the @ symbol as the optional revset argument, eg., jj describe -r 'example@' -m "...".) But I'm using jj more and more in automation contexts where it's nice to be able to do as many atomic operations as possible to minimize race conditions (and jj has transaction semantics for many of its commands, I'm not an expert on it though) and fewer multi-step operations to keep things simple.

With the change at c64d42, you can create a new workspace with a commit message:

$ jj workspace add -m 'a new workspace' /tmp/example
Working copy  (@) now at: ontxouwv b6236a1e (empty) a new workspace
Parent commit (@-)      : vvyvvnsr d55579fd Initial
Added 1 file, modified 0 files, removed 0 files

This was my first contribution to the jj project and I felt it went well.

It was easy to get a local development environment set up, because jj has a Nix flake for this purpose. That alone made this so approachable. I'm not a Rust expert, and a complex Rust development environment can sometimes take the wind out of my sails. Lately, I do see more and more OSS projects maintaining a flake.nix. It's one of the key things I look for when making that not-always-rational decision of whether to try to contribute.

jj has a clear and thorough document for contributors to refer to. I understood what they were looking for in terms of the PR, the commit message, commit history hygiene, etc. After I pushed a change in response to some review comments, some automated tooling informed me my commit message didn't meet formatting specs, which clued me to the fact that they want a squashed commit in any case.

After I resolved the review feedback, my change was approved, and I could click the merge button in GitHub myself. I like when projects do this; it's a small thing, but it feels good to be the one to land the change.

However, the merge queue check failed. At first glance, it seemed like it was due to a transient error with the build environment on the GitHub side. So I requeued it, and it failed again. I'm embarrassed to say that I didn't look closely at the reason why either run had failed, but after some kind, clear, and direct feedback from the maintainer Martin, I quickly fixed up the problem (a failing snapshot test that I had missed when, I think, my change was parented to the trunk), and everything was green.

The use of snapshot testing is another nice aspect of the jj project: the Rust crate insta makes it easy to prevent regressions based on the textual output of programs, including the lifecycle part of managing snapshots when the thing they are testing inevitably change. All of which is a convenient way to test a CLI tool, especially with jj's templates that let you customize the output.

I was also added to the GitHub jj-vcs/contributors organization. This is just a really well-run FOSS project, kudos to the core maintainers.

Now that I've made this change, I'm motivated to contribute some more. I've got a list of small improvements that would make my daily work a bit better. I've been using jj heavily for a few years now. It's completely replaced Git for me. I'm relying on jj workspaces in parallel agent development scenarios; I've found they are a great fit for this task. I feel jj is a great fit with agentic development in general, as its design removes a lot of stumbling blocks for agents in common VCS tasks.

TL;DR: I made a website for the original classic Mac patterns

I was working on something and thought it would be fun to use one of the classic Mac black-and-white patterns in the project. I'm talking about the original 8×8-pixel ones that were in the original Control Panel for setting the desktop background and in MacPaint as fill patterns.

Screenshots via to Marcin's awesome interactive history

I figured there'd must be clean, pixel-perfect GIFs or PNGs of them somewhere on the web. And perhaps there are, but after poking around a bit, I ran out of energy for that, but by then had a head of steam for extracting the patterns en masse from the original source, somehow. Then I could produce whatever format I needed for them.

There are 38 patterns, introduced in the original System 1.0 in the 1984 debut of the Macintosh. They were unchanged in later versions, so I decided to get them from a System 6 disk, since that's a little easier with access to utility programs.

Preparation

• Download Mini vMac.
• Acquire "old world" Mac ROMs.
• Download a System 6 startup disk image.
• Download ExportFl disk image.
• Download sitPack disk image.
• Install "The Unarchiver" (brew install --cask the-unarchiver)
• Install the Xcode command-line tools.

Extraction process

Start System 6 (drag the ROM onto the Mini vMac icon, then drag the System 6 disk onto the window when you see the flashing floppy disk). Mount the ExportFl and sitPack disks by dragging their files and dropping on the classic Mac desktop.

In emulation

Double-click sitPack to launch the program. Command-O to open, then navigate to the startup disk by clicking "Drive". Scroll to find "System Folder" and double-click on it. Scroll to the bottom, select "System" and click "Open". Save the output file as "System.sit" in the top-level of the startup disk. Quit sitPack back to the Finder.

Start the ExportFl program. Command-O or pick "Open" from the "File" menu. Find the "System.sit" created in the last step and click "Open". A regular file save dialog will appear on the modern Mac, pick a location and save the file.

On the modern Mac

Drag the "System.sit" file onto The Unarchiver, or open the file from within it. This will produce a file called "System" (with no extension).

Run DeRez (part of the Xcode developer command-line tools) on the System file. I first added /Library/Developer/CommandLineTools/usr/bin to my $PATH, then ran:

$ DeRez -only PAT\# System > patterns.r

This produces a text representation of the PAT# resource in the System file. It's a series of bytes that comprise 38 8×8 patterns meant for QuickDraw commands. There's a leading big-endian unsigned 16-bit number (0026) to indicate the number of 8-byte patterns to follow.

data 'PAT#' (0, purgeable) {
	$"0026 FFFF FFFF FFFF FFFF DDFF 77FF DDFF"
	$"77FF DD77 DD77 DD77 DD77 AA55 AA55 AA55"
	$"AA55 55FF 55FF 55FF 55FF AAAA AAAA AAAA"
	$"AAAA EEDD BB77 EEDD BB77 8888 8888 8888"
	$"8888 B130 031B D8C0 0C8D 8010 0220 0108"
	$"4004 FF88 8888 FF88 8888 FF80 8080 FF08"
	$"0808 8000 0000 0000 0000 8040 2000 0204"
	$"0800 8244 3944 8201 0101 F874 2247 8F17"
	$"2271 55A0 4040 550A 0404 2050 8888 8888"
	$"0502 BF00 BFBF B0B0 B0B0 0000 0000 0000"
	$"0000 8000 0800 8000 0800 8800 2200 8800"
	$"2200 8822 8822 8822 8822 AA00 AA00 AA00"
	$"AA00 FF00 FF00 FF00 FF00 1122 4488 1122"
	$"4488 FF00 0000 FF00 0000 0102 0408 1020"
	$"4080 AA00 8000 8800 8000 FF80 8080 8080"
	$"8080 081C 22C1 8001 0204 8814 2241 8800"
	$"AA00 40A0 0000 040A 0000 0384 4830 0C02"
	$"0101 8080 413E 0808 14E3 1020 54AA FF02"
	$"0408 7789 8F8F 7798 F8F8 0008 142A 552A"
	$"1408"
};

It would have been simple enough to parse this text, but I had Claude quickly make a Python program to do so and output them in .pbm format, which is part of the Netpbm image format class. This is a simple image format that is text-based, a '1' or a '0' indicating a black or white pixel in a row and column.

For example, this subway tile pattern is represented like this in .pbm:

P1
8 8
1 1 1 1 1 1 1 1
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 1 1 1 1 1 1 1
0 0 0 0 1 0 0 0
0 0 0 0 1 0 0 0
0 0 0 0 1 0 0 0

From here, I can generate image files for the patterns in any format and resolution I want, using ImageMagick or similar. It's important when scaling the patterns to use -filter point, so that ImageMagick doesn't try to interpolate the pixels it needs to fill in, which would lead to blurry results.

Why do all this?

It's nostalgic, I have a fondness for these old patterns and the original B&W Mac aesthetic, it reminds me of playing games like Dark Castle and Glider, messing around with HyperCard, and using Tex-Edit and hoarding early shareware programs.

The whole point of the above is to get a copy of the System file out with the resource fork intact, that's where the desktop patterns live.

According to old classic Mac manuals, the patterns were QuickDraw bit-pattern resources, a simple bitmap of 8 bits per row packed into 8 bytes (columns). It was fast for QuickDraw to copy them over an area of the screen. For example the following pattern was used for the default gray desktop pattern on black-and-white Mac screens.

I could have extracted all 38 patterns other ways: I could have screenshotted each one, I could have looked at each one and hand-written .pbm files, both of which would have been tedious and error-prone.

Ultimately, I wanted to extract the exact original data from the source (or close enough copy thereof) and have the patterns in a format I considered archival for this limited purpose (.pbm files are trivial to parse and manipulate).

Head over to my pattern site to get the patterns for yourself.

(Credit for replica Geneva 9pt and Chicago 12pt fonts)

Here's everything the man page and jj website has to say about jj absorb:

$ jj help absorb
Move changes from a revision into the stack of mutable revisions

This command splits changes in the source revision and moves each change to the
closest mutable ancestor where the corresponding lines were modified last. If
the destination revision cannot be determined unambiguously, the change will be
left in the source revision.

The source revision will be abandoned if all changes are absorbed into the
destination revisions, and if the source revision has no description.

The modification made by `jj absorb` can be reviewed by `jj op show -p`.

Usage: jj absorb [OPTIONS] [FILESETS]...

Arguments:
  [FILESETS]...
          Move only changes to these paths (instead of all paths)

Options:
  -f, --from <REVSET>
          Source revision to absorb from

          [default: @]

  -t, --into <REVSETS>
          Destination revisions to absorb into

          Only ancestors of the source revision will be considered.

          [default: mutable()]
          [aliases: to]

From this I gather that it's a little like jj squash: it combines changes with earlier ones, in keeping with jj's overall spirit of stable but mutable changes. But this piece is a little unclear: "moves each change to the closest mutable ancestor where the corresponding lines were modified last." In other words, it will distribute diffs from multiple files back in time to previous revisions where those files were created or edited. Also, unlike a squash which updates a single previous revision, absorb will potentially update many revisions. Further, since it works at the level of the line, it may split changes in a single file across multiple revisions. Let's test our our understanding by trying it out.

$ jj git init

I'll create a few files with different kinds of content in a series of commits.

$ echo $'a b c\nA B C\n1 2 3' > a && jj commit -m "Add file a"
$ echo "The time is now $(date)" > b && jj commit -m "Add file b"
$ echo $'- red\n- green\n- blue' > c && jj commit -m "Add file c"

Our history at this point looks like:

@  zmmkxkyk paulsmith@pobox.com 2025-08-20 10:45:48 d5483ce5
│  (empty) (no description set)
○  qnwlwkrk paulsmith@pobox.com 2025-08-20 10:45:48 da79bd35
│  Add file c
○  rpxrzqqq paulsmith@pobox.com 2025-08-20 10:45:48 df1c52d0
│  Add file b
○  qozqypuz paulsmith@pobox.com 2025-08-20 10:45:48 a0308780
│  Add file a
◆  zzzzzzzz root() 00000000

Now I'm going to edit each file, making a small change to each.

$ vim a b c
$ jj diff --git

Our change has this diff:

diff --git a/a b/a
index 31de6f3761..ce7afc22d7 100644
--- a/a
+++ b/a
@@ -1,3 +1,4 @@
-a b c
-A B C
+a b c d
+x y z
+B C
 1 2 3
diff --git a/b b/b
index d904e394e9..d38c71e8a0 100644
--- a/b
+++ b/b
@@ -1,1 +1,2 @@
-The time is now Wed Aug 20 10:45:48 CDT 2025
+The time is now:
+Wed Aug 20 10:45:48 CDT 2025
diff --git a/c b/c
index d7b1eb9e88..02a89b7084 100644
--- a/c
+++ b/c
@@ -1,3 +1,7 @@
 - red
+- orange
+- yellow
 - green
 - blue
+- indigo
+- violet

Let's absorb this change now. Our expectation, according to that help text, is that jj will spread these individual files diffs across the previous revisions it finds where those individual files were last changed.

$ jj absorb
Absorbed changes into 3 revisions:
  qnwlwkrk bb89dd81 Add file c
  rpxrzqqq 7981140d Add file b
  qozqypuz fc76ab5e Add file a
Working copy  (@) now at: qsmyrsyu b0f4d79f (empty) (no description set)
Parent commit (@-)      : qnwlwkrk bb89dd81 Add file c

Indeed, "absorbed changed into 3 revisions". And we can see that our working copy revision is also now empty, having been abandoned after all of its changes were moved out.

What jj did was, for each change in the working copy, find in the history the last (mutable) revision that affected the same lines as the change, and apply the diff as a patch, which produced a new commit (in the Git sense) for that change (in the jj sense, a stable ID across potentially many modifications to a particular revision).

The way I've found it to be very very useful are those occasions when I'm in the middle of editing some files, and realize I need to make an edit to one or more files unrelated to those other in-progress ones - maybe a typo, maybe a tweak that provides a more stable foundation for the work in was in the middle of. I could jj split, but then I'd have a minor commit in the history, which is sometimes fine, or jj squash, but then I'd have to find the right previous revision to squash into. And in either case I'd have to take a little care to get it just right so as leave the original working copy files alone. jj absorb is a way of saying, this edit is out of time with the rest of this working copy, and should be with the revision that those files/lines are connected with, and jj will figure that out. It tends to produce tidier histories with more logically consolidated revisions.

There's a final bit in the help text about using jj op show to see what absorb did in detail:

$ jj op show -p --git
bacc682f6f3c paul@oberon 1 minute ago, lasted 26 milliseconds
absorb changes into 3 commits
args: jj absorb

Changed commits:
○  + qsmyrsyu b0f4d79f (empty) (no description set)
│ ○  - zmmkxkyk hidden 061c0eb3 (no description set)
├─╯  diff --git a/a b/a
│    index 31de6f3761..ce7afc22d7 100644
│    --- a/a
│    +++ b/a
│    @@ -1,3 +1,4 @@
│    -a b c
│    -A B C
│    +a b c d
│    +x y z
│    +B C
│     1 2 3
│    diff --git a/b b/b
│    index d904e394e9..d38c71e8a0 100644
│    --- a/b
│    +++ b/b
│    @@ -1,1 +1,2 @@
│    -The time is now Wed Aug 20 10:45:48 CDT 2025
│    +The time is now:
│    +Wed Aug 20 10:45:48 CDT 2025
│    diff --git a/c b/c
│    index d7b1eb9e88..02a89b7084 100644
│    --- a/c
│    +++ b/c
│    @@ -1,3 +1,7 @@
│     - red
│    +- orange
│    +- yellow
│     - green
│     - blue
│    +- indigo
│    +- violet
○  + qnwlwkrk bb89dd81 Add file c
│  - qnwlwkrk hidden da79bd35 Add file c
│  diff --git a/c b/c
│  index d7b1eb9e88..02a89b7084 100644
│  --- a/c
│  +++ b/c
│  @@ -1,3 +1,7 @@
│   - red
│  +- orange
│  +- yellow
│   - green
│   - blue
│  +- indigo
│  +- violet
○  + rpxrzqqq 7981140d Add file b
│  - rpxrzqqq hidden df1c52d0 Add file b
│  diff --git a/b b/b
│  index d904e394e9..d38c71e8a0 100644
│  --- a/b
│  +++ b/b
│  @@ -1,1 +1,2 @@
│  -The time is now Wed Aug 20 10:45:48 CDT 2025
│  +The time is now:
│  +Wed Aug 20 10:45:48 CDT 2025
○  + qozqypuz fc76ab5e Add file a
   - qozqypuz hidden a0308780 Add file a
   diff --git a/a b/a
   index 31de6f3761..ce7afc22d7 100644
   --- a/a
   +++ b/a
   @@ -1,3 +1,4 @@
   -a b c
   -A B C
   +a b c d
   +x y z
   +B C
    1 2 3

Changed working copy default@:
+ qsmyrsyu b0f4d79f (empty) (no description set)
- zmmkxkyk hidden 061c0eb3 (no description set)

Which also reminds that every action in jj that changes the state of the repo can be trivially rolled back:

$ jj undo
Undid operation: bacc682f6f3c (2025-08-20 11:04:55) absorb changes into 3 commits
Working copy  (@) now at: zmmkxkyk 061c0eb3 (no description set)
Parent commit (@-)      : qnwlwkrk da79bd35 Add file c

Click here for the cheatsheet

If you're like me, then lately you've been using the new VCS kid on the block, jj. And if you're really like me, then you have encountered the built-in diff editor/merge tool when doing a revision split (jj split), when doing the so-called squash workflow (interactively via jj squash -i), or when resolving a conflict (jj resolve). The tool is bundled with jj. It comes from a project named scm-record. The binary executable you're invoking is named scm-diff-editor, and you might see reference to that name in jj docs, but you don't need to know that if you just leave jj configured by default to use it as the built-in diff/merge tool.

And if you're really, really like me, then when you were presented with the built-in diff/merge tool in your terminal upon executing one of those commands, you were frustrated and felt dumb as you mashed keys and tried to understand what you were supposed to do and how. It's a spartan interface, and as far as I can tell, and I certainly tried to tell, because I queried the usual search indices and other knowledge resources, and combed the jj docs, and noted the apparent complete lack of scm-record docs, there is not much yet out there in the public to help you through editing a diff with it and ultimately amending your jj commit. I don't fault those involved, it's a young and maturing project, and the editor itself is a separate effort from jj, but seeing as how it is a key step in the recommended way of using jj, I needed more help. (Incidentally, Steve's doc linked there was the only documentation I was aware of when I trying to figure this out, so I appreciate him for providing a practical example).

The members of our group were selected by Todd, working with Greg and Ryan and others behind the scenes to identify people who could help because they had that kind of technology experience. HealthCare.gov, having launched days earlier on October 1, 2013, wasn't working. From the vantage from the top of the political leadership in the country, it was clear outside help was needed. Todd, the CTO of the United States at the time, which is a position in the White House, was tapped to help fix it. His plan was to provide reinforcements to complement the team of government employees and contractors that had built HealthCare.gov and were in the midst of operating it. We were to be a small team, very discreet. Todd was our leader. It was already a high-pressure, stressful situation, so insertion into that context meant melding in, not blowing things up. It was to be a low-key mission of information gathering and assessment, not the cavalry storming in. Todd told us the next 30 days would be critical. The goals were to enroll 7 million people by March 31, 2014, the end of the period known as open enrollment. When the media was eventually informed of our existence by the White House, we were referred to as "the tech surge".

When Todd called me two days prior on October 16 to ask me if I would join the effort, he didn't have to explain the stakes. I understood what it meant for the website to work. I immediately agreed, and put on hold what I was doing, which happened to be raising money for a startup I had founded. I took Todd's call while walking the grounds of the Palace of Fine Arts in San Francisco, having met with VCs earlier in the day. I was living in Baltimore at the time with my wife and toddler daughter. I flew back home right away, and before I knew it I was taking the earliest morning train I could from Camden Yards to DC's Union Station. I thought I had timed it right, but still wound up running across Pennsylvania Avenue so as not to be late.

We couldn't have started any sooner even if we had wanted to. The federal government had shut down on October 1, the same day HealthCare.gov had launched. The shutdown prevented anyone from outside the main team working on HealthCare.gov from coming in to help. So while days passed with the news dominated by the twin stories of the shutdown and the slow-moving catastrophe of the launch, a vacuum of information formed, as well as a surplus of speculation and worry. The White House couldn't figure out what was wrong with it, and the implications of it failing were troubling. The Affordable Care Act, the signature domestic policy achievement of President Obama's tenure, had gone into effect, and the website was to be the main vehicle for delivering the benefits of the law to millions of people. If they didn't know what was wrong with HealthCare.gov, other than that it was manifestly not working, plain for everyone to see, and therefore they might not be able to fix it, what would that mean for the the fate of health care reform? Fortunately, the shutdown ended on October 17, which meant we could get to work and finally understand what was going on.

Some of us already knew each other, but everyone was new to someone else. We introduced ourselves, headed inside, and after breakfast in the Navy Mess, headed upstairs to the Chief of Staff's office. Denis McDonough shook our hands as we somewhat awkwardly stood in a line. He asked us directly, "can you fix it"? In our nervous energy, I remember some of us blurting out, "yes". We had confidence, but we also were eager to dive in, learn as much as we could, and get going.

A van was procured, along with a driver. We piled in and headed across the Mall to the Hubert H. Humphrey Building, headquarters of the US Department of Health & Human Services (HHS). Entering the lobby, we passed Secretary Kathleen Sebelius. She wasn't there for us, she was welcoming federal employees back to work after the shutdown. Our meeting there was with Marilyn Tavenner and her staff. Tavenner was the Administrator of the Centers for Medicare & Medicaid Services (CMS), the largest organization in HHS, and the owner of HealthCare.gov.

Our meeting with Tavenner and her team yielded our first details of the system that was HealthCare.gov. We learned how it was structured and what its main functional components were. We heard first hand what CMS leadership knew about what was wrong, or at least where in the system they could see that things were not working. It was our first sense of the size and complexity of the site, both in terms of functionality, but also in terms of the number of contractors, sub-components, and business rules such as for eligibility. But remember, these were not technical people - they were health policy experts and administrators. Most of the things they were reporting to us were business metrics about the site, descriptions of high-level performance. It was helpful to hear this perspective, and indeed many of these metrics would drive our later work. But this would not yet be the time to learn about the technical challenges the team was facing.

Our morning continued back in the van, leaving DC for CMS headquarters in Woodlawn, Maryland, just outside Baltimore, a 45-minute drive. Here we met with the leadership of HealthCare.gov itself, a group of people including Michelle Snyder, CMS's COO, Henry Chao, one of its main architects, and Dave Nelson, a director with a telecom background, who was being elevated and would oversee much of the rescue work from CMS's perspective. The sketchy picture of HealthCare.gov we had was coming into greater relief. We learned about deployment challenges and bottlenecks, more about where specifically users were getting "stuck" using the site, and we started to hear bits and pieces about the particular technologies being used, including something called MarkLogic, an XML database, which was new to us. We even started to get some details about the deployment architecture and the types of servers involved. Again, the theme of complexity stood out. But we were also still talking at a fairly high level. To really understand what was wrong with HealthCare.gov, we'd have to move on.

The afternoon was spent a few miles down the road back toward DC in Columbia, Maryland, at something called the XOC, or Exchange Operations Center. This was to be the mission-control-style hub of operations for HealthCare.gov. We found a room staffed with a few contractors and some CMS employees. (The XOC eventually would be the site of so much activity during the rescue that a scheme to keep people out was needed.) Here was where, away from the leadership-filled rooms, we finally heard from technologists who were directly working on the site. What we heard was troubling. There was a lack of confidence in making changes to the source code. A complex code generation process governed much of the site and produced huge volumes of code, requiring careful coordination between teams. Testing was largely a manual process. Releases and deployment required lengthy overnight downtimes and often failed, required equally-lengthy rollbacks. The core data layer lacked a schema. Provisioning hosting resources was slow and not automated. Critically, there was a pervasive lack of monitoring of the site. No APM, no bog-standard metrics about CPU, memory, disk, network, etc., that could be viewed by anyone on the team. By this point, the looks we exchanged amongst ourselves betrayed our fears that the situation was much worse than we initially expected. I carried with me that day a Moleskin notebook that I furiously scribbled in so as not to forget what I was hearing. You can see as the day worn on my panic start to rise reflected in my writing.

With that dose of reality and the evening setting in, we collected ourselves back in the van and set off for our final field trip of the day, to suburban Virginia for the offices of CGI, the prime contractor of HealthCare.gov. In Herndon, we were greeted by many people on their leadership team and the technical teams who had built the site, even though it was getting on past business hours at this point. This was as much an interview of us by them as it was a chance for us to ask questions. We had a brief opportunity to ingratiate ourselves and win their trust. We did that in part by showing our eagerness to dig in on some of the things we had learned earlier in the day, proving our engineering bona fides (specifically with high-traffic websites), and emphasizing that we were not there for any other reason than to help. This was a team on edge, under the gun and exhausted. We needed them to succeed, and they weren't going to work with us if they thought we were there to cast blame.

We asked many questions, but they mostly boiled down to, what's wrong with the site, and how do you know what's wrong? Show us where in the system this or that component isn't performing the way you expected. And they and CMS mostly couldn't do that. They had daily reporting and analytics that produced those high level business metrics. But again there was that lack of monitoring of the system itself, real-time under load. So we focused on that.

It was getting late. Could we throw a Hail Mary? Walk out of that building leaving behind something of tangible value, something promising they could build on? We said, there's lots of APM-style monitoring services, but we're familiar with New Relic. Could we install it on a portion of the servers? Yes, we know there's a complicated and fragile release process, but if we bypassed that and just directly connected to some of the machines, we could install the agent on them and be receiving metrics almost immediately. Glances were exchanged. A CMS leader in the room made a call on their cell phone - we actually have some New Relic licenses already, we can use those. There was also a hestitancy to see if this kind of extraordinary, out-of-the-norm request would be approved - clearly, even during this period of turmoil, all the stakeholders stuck to the regular release script. The CMS leader nodded their approval. A small group assembled to marshall the change through. Many folks had stuck around, even though it was nearing midnight. Then on a flatscreen in the conference room, we pulled up the New Relic admin, and within moments, the "red shard" (the subset of servers we had chosen for this test) was reporting in. And there it was - we could see clearly the spike in request latency, even at this late hour, that indicated a struggling website, along with error rate, requests per minute, and other critical details that had basically been invisible to the team to that point. Imagine a hospital ICU ward without an EKG monitor. Now that they knew exactly in what ways it was bad, they could start to correlate them with the business metrics and other aspects of the site that needed improving. They could then prioritize the fixes and actions that would yield the biggest improvements.

That would come later. For now, exhausted, well past midnight, we left Herndon and rode the van back to DC. At roughly 2 A.M. we reconvened in the Roosevelt Room in a completely silent White House for a quick debrief. As we reflected on what we had just experienced, another thought was settling in over the group - that this was obviously not over by any stretch, that none of us were going home any time soon, that the challenge was much larger in scope than we had imagined, that any notion we may have had at the outset of possibly just offering some suggested fixes and moving on was in retrospect hilariously naive, that this was all we were going to be doing for the foreseeable future until the site was turned around. Indeed, we all managed to find a few hours of sleep in a nearby hotel and then were right back at it in the morning, heading straight out to Herndon.

This was just day one, a roughly 18-hour day, and it certainly wasn't the last such marathon. Over the next two-and-a-half months until the end of December, the tech surge expanded and took on new team members, experienced many remarkable events and surprises, and ultimately, successfully helped turn HealthCare.gov around. Millions enrolled in health care coverage that year, many for the first time. I hope to tell more stories of how that happened over the next few weeks and months.

The central question “Oppenheimer” asks is, to whom or what is J. Robert Oppenheimer bound? Director Christopher Nolan spares us a hoary answer like “the truth!”, but early on we are told it is to theory and mathematics, wherever they may lead. They first lead Oppenheimer out of the lab and into the arms of pre-war continental quantum physicists, who are forging a nascent field of inquiry that our hero immediately grasps and excels at. He becomes friends with physicist Isidor Rabi while in Germany, bonding over their shared New York Jewish backgrounds. We see Rabi nurture Oppenheimer several times, offering food from his pocket and quiet counsel, evincing an almost maternal protective quality. Oppenheimer, back in the States, forms allegiances with students and organizers of various labor movements, including the Communist Party, but weakly, always as or via a proxy (using the party as a channel to fund republicans of the Spanish Civil War; as a means to start his relationship with Jean Tatlock). Several scenes hint at deeper connections, but cut away before he does anything incriminating (“but that would be treason …”). Of course his “true” allegiance or lack of to the Communist Party and to the Soviet Union hangs over the balance of the post-test movie, which seems content to leave it unanswered, or perhaps, answered sufficiently by his other deeds, which several characters make voice of.

What of Oppenheimer’s non-professional bonds? He’s prepared to boil off his child like so many neutrons in a fission reaction, driving the colicky baby to a friend’s house in hopes of being relieved of parental duties. In spite of his affairs and scarce evidence of marital happiness, his relationship with his wife Kitty is shown to endure (“we’re adults, we’ve been through fire together”, their own form of fusion), surviving at least to his public image rehabilitation late in life. Frank Oppenheimer, kept to the periphery early on, becomes essential to the triumph at Los Alamos, reuniting the brothers on the same mesa where they forged a connection to the land, a feel for the weather of the desert. He pursues Jean Tatlock with flowers and is repelled; she later makes her own pursuit, reminding him of his off-hand oath, “you said you’d always answer” — a promise he’s now incapable of keeping, acted on by multiple forces much stronger than she.

Oppenheimer becomes an attractive force himself to build Los Alamos, cajoling scientists and convincing the US military, overcoming each group’s respective reservations, the former about the endeavor itself, the latter about him. He’s the nucleus of the most important thing that’s ever happened, to quote Gen. Groves, but we see him often distracted, gazing into the middle distance, drifting off to Chicago or San Francisco, giving misleading testimony to the army’s quietly menacing interrogator. Still, it’s Oppenheimer keeping the energetic particles of scientists around him from flying off or creating ruinous inter-personal explosions. They eke out just enough collaboration and luck to blow up the gadget before Potsdam — immediately, the military dissolves their bond to the man who secured their supremacy (“we got it from here”). We see Oppenheimer unmoored, isolated, radiating out his misgivings and the horror of his revelations.

In a pivotal scene, Oppenheimer dismisses a report of a nuclear chain reaction, stating that theory proves it can’t be so. When his neighbor colleague reproduces the experiment, he immediately forms a new theoretical understanding from it — the bond to pure theory is broken, and a new one connecting theory and practice is made. It takes him no time again to accelerate to the logical end of the implications, and this time, theory must wait for practice to catch up. When the new experiment is finished, first at Trinity and then at Hiroshima and Nagasaki, it’s no longer about what the physics demonstrates, but what it means for the notion of humanity and civilization: his revulsion at the reception of the bomb among his peers and the public is shown itself as a terrible blinding fire, one that now lives within him.

Stray observations

• “Oppenheimer” centers language throughout, and positions Oppenheimer as a language savant in technical ways, but deficient in others. He learns enough Dutch to teach physics in Europe. He reads Marx in the original German. He quotes Sanskrit to his lover. He’s also a translator, bringing the foreign language of quantum mechanics to the United States, and bridging the gap between academics and the military. He fails to learn the language of Washington, and his character is assassinated as a result. Kitty is bedeviled, who can’t understand how a man can be so proficient in one domain and so passive when it comes to himself and his family.
• There’s a rich symbolic history to the apple, and one figures prominently in early scenes. First as an impulsive attempt on his professor’s life. We see Oppenheimer stab the apple, which is a pre-quantum apple, Newton’s apple; the needle with which he injects the cyanide is a dagger in Newtonian physics, which Einstein, whom we encounter multiple times including in despair in the final scene, inflicted the first mortal wound, but which quantum mechanics and the bomb finally killed. And then as the poison fruit of knowledge carried by Niels Bohr, who introduces him to quantum mechanics, which leads to our expulsion from Eden when the atomic weapon is used.
• The circular badges that the scientists wore at Los Alamos, which had labels like “K-16” and “C-43”, which I’m sure were for some organizational purpose the film doesn't explain (as far as I remember), made me think of them like personified isotopes from the periodic table of the elements.
• The act III Strauss plot was less successful to me than the rest of the film. The revelation that he was humiliated and vindictive and then used the apparatus of official power to seek his revenge didn’t land for me in the way I think was intended, perhaps because, while certainly despicable, it’s not shocking nor particularly novel, even discounting our recent experiences with vengeful politicians. Setting that aside, from a storytelling point of view, as an extended denouement after the wallop of the Trinity event, it has to work extra hard to sustain a clear narrative focus, and I felt the film suffered overall for it.
• Kudos to Jennifer Lame, who edited the film, for making scene after scene of extensive dialogue so compelling and propulsive. And to Hoyte van Hoytema, the director of photography — it’s tonally gorgeous and lightly under-saturated in a way that serves the somber mood. What more can be said about 70mm film? It just glows. It’s very much worth seeing in a theater.

The reason for this is a phenomenon called "bufferbloat". I'm not going to explain it in detail, there are plenty of good resources to read about it, including the eponymous Bufferbloat.net. Bufferbloat is the result of complex interactions between the software and hardware systems routing traffic around on the Internet. It causes higher latency in networks, even ones with plenty of bandwidth. In a nutshell, software queues in our routers are not letting certain packets through fast enough to ensure that things feel interactive and responsive. Pings, TCP ACKs, SSH connections, are all being held up behind a long line of packets that may not need to be delivered with the same urgency. There's enough bandwidth to process the queue, the trick is to do it more quickly and more fairly.

Fortunately, because bufferbloat is in part a function of how we configure our routers, it's within our ability to solve the problem. But first, we have to diagnose it, and establish a concrete baseline to improve from. The speed test at dslreports.com tests for bufferbloat in addition download and upload speeds, so we'll use that tool to see how we're doing.

First, I run the speed test, and get the following results:

Here you can see the issue starkly: 120 Mbps down and 12 Mbps up yields an "A+" grade (debatable), but we get an "F" for bufferbloat.

We define bufferbloat here as the increased latency of a standard ping while downloading or uploading a large file over ping times while otherwise quiescent.

In our case, our idle latency is 12ms average, a download bloat of about 660ms, and an upload bloat of about 280ms, on average.

The fix is to apply a queue management strategy to our router. Ordinarily, I'd be wary of this. In my experience, QoS administration tends to be fussy and full of unintended consequences. I always felt as if I had cast too broad a net, inadvertantly degrading overall network performance to get slightly better results from one application. And I wasn't sure around what fixed-point I was optimizing. In this case, bufferbloat gives us the measurable target. Administration is made much easier by the appearance of a new algorithm that's easy to apply to network interfaces. It doesn't require much tuning, and you don't need to futz with individual ports or percentages.

Details vary widely by router operating system and administrative UIs. In our case, the router is running OpenBSD. (And if yours isn't, why not? Get a PC Engines board, throw obsd on it, and you have an inexpensive solution with world-class security, efficiency, and performance, that's simple to operate and well-documented.) The OpenBSD way of being a router is through its pf system, which is analogous to Linux's iptables, but much more capable and efficient. Since 6.2, pf has implemented something called "FQ-CoDel", which is an algorithm for scheduling packets fairly and is designed to prevent bufferbloat. It is exposed via the flows option on a queue rule. In principle, all we need to do is add two rules, one to fix uplink bufferbloat and one to fix downlink. Let's see how this goes.

In our /etc/pf.conf, we first add a single line to handle the uplink. This will apply a FQ-CoDel queue to the network interface attached to our WAN link, or the cable modem in our case. The way to think about it is, FQ-CoDel is strategy applied to outbound packets only, as they exit the interface, so even though the WAN interface is duplex up and down, in order to handle the downlink part we'll apply it to the network interface connected to our LAN, which we'll do next.

An important detail. In order for the queue algorithm to do its thing, it needs to know the bandwidth of the outbound link. According to Mike Belopuhov, the implementor of FQ-CoDel in OpenBSD, we need to specify 90-95% of the available bandwidth. Fortunately, we've just measured it.

The line to add to pf.conf to fix bufferbloat on the uplink is (assuming em0 for the WAN interface):

queue outq on em0 flows 1024 bandwidth 11M max 11M qlimit 1024 default

A couple of notes. outq is a label we give, but it's an opaque string to pf. 11M means 11 megabits per second (92% of the uplink bandwidth). qlimit is also specified explicitly, because its default value of 50 is too low for FQ-CoDel. The default keyword is required.

And that's it: we don't need to alter our filtering rules to assign packets to a queue: all outbound packets on this interface are assigned to our new queue.

Now let's reload pf with the config change, and re-run the speed test.

$ doas pfctl -n -f /etc/pf.conf && doas pfctl -f /etc/pf.conf

Uplink latency under load is now down to 17ms on average, from 280ms. That's a mere 5ms worse than idle.

(I discount the apparent decrease in uplink bandwidth from this test result. In my experience, dslreports.com could vary by 10-15% in reported bandwidth run-to-run, but over time it converged on 12 Mbps.)

The downlink fix is nearly the same, we just adjust for the name of the interface (the LAN NIC is called em1) and for its 90-95% bandwidth upper bound, which is 110 Mbps.

queue inq on em1 flows 1024 bandwidth 110M max 110M qlimit 1024 default

Reload, re-run:

Always nice to get an A. Downlink latency under load is now 24ms, from 660ms.

I haven't elided much, I think that's a pretty decent result for two lines of config. If you want to go further, there's a quantum knob to turn (baseline is your NIC's MTU, but look at what OpenWRT does for guidance), but that's about it.

Post-fix, my observation is that things feel much snappier. Aside from the ping time improvements, I don't have other measurements to cite. But so far, FQ-CoDel seems to have fixed bufferbloat on my network and made for a substantially better experience.

After years of looking at the sewing machine in the closet and saying I should learn how to use it, I did something about it this year. Michelle got me a class at a local crafts store as a gift, and I made a pillow cover. It turned out pretty good, and I enjoyed doing it, so I kept at it. I didn't have a ton of time this year to devote to it, but by the end of the year I had made some blankets for friends and quillows for the girls, a pair of pajama pants for Michelle, and repaired the lining of a friend's handbag. I'd like to keep at it, maybe tackling Halloween costumes next year, and doing projects with Maxine, like adding circuits to fabric ala FLORA.

Running

I've run for exercise before, but never with any consistency. This year I set out to run at least 2 times a week, at least 20 minutes per run. Turns out, I liked it a lot. So much so that I was running almost every weekday by April. Unforunately, my knee didn't like it so much, and in the middle of a run on the Bloomingdale Trail, it suddenly seized up and I had to take an Uber home. Luckily, I hadn't done any damage, and an orthopedist said I was just overdoing it. I took about a month off and ramped back up slowly. By October, I peaked for the year, doing about 10 miles per week. In November, I ran in my first 10k, the Lincolnwood Turkey Trot. All told for 2016, I logged 156 miles. For 2017, I'd like to attempt a half-marathon, and double my yearly mileage. Here's my Strava profile.

Ad Hoc

The company I started with Greg in 2014 began the year with 7 people and ended with 41. The growth was thanks to winning our first contracts: we had been around long enough as a company and had enough "past performance", as they say in the industry, to compete and be awarded tasks on our own, instead of only working as subcontractors. The contracts were with CMS, to continue our work on HealthCare.gov, and with the Department of Veterans Affairs, to help build Vets.gov. We also earned spots on two highly sought-after contract vehicles, the ADELE BPA with CMS, and FLASH with the Department of Homeland Security, that have will have opportunities for us down the road to bid on. It was a great year for Ad Hoc, and I'm proud of what we've built: a great team, and useful software that is delivering actual benefits and services to real people. For example, this year, we took over the core shopping part of HealthCare.gov, known internally as Plan Compare. During open enrollment so far, which is still going on until the end of January, over 3.5 million households have enrolled for plans using our software. HealthCare.gov also saw its biggest single-day enrollment tally ever, on December 15th. I'm also proud that we're proving out the model of providing modern software engineering and design services to the government that are efficient, work well, cost less to build and operate, and are just better than the status quo. There's a lot of uncertainty ahead for the programs that we're working on. There's not much we can about that, other than continue to do the work we have in front of us until it changes, and look for additional opportunities in state, local, and maybe outside of government.

House

Michelle and I bought a house in south Evanston in January, and we've been renovating it since. It's an old Victorian-style home from the 1890s with a good stone foundation and timber frame. We're doing extensive changes to the interior, with a new layout and all new flooring, doors and windows, and systems like electrical and HVAC. The exterior is mostly unchanged from a framing perspective, but we're updating the siding, and we dormered out part of the sloping roof so we could have a master bedroom on the third floor. We're also converting the garage into a two-story garage/coach house combo: the plan is to have an office on the second floor for Michelle and me. We had hoped, at the beginning of the year, to be moved in by fall, but these things go the way these things go. As of this writing, we're about a month away from being able to pull up stakes here in Logan Square. We've been working with our friend and architect David Burns, which has been great. We spent time together with him talking about what we wanted, and he came up with a vision, drew up detailed plans, and has been managing the overall construction process. We hired Conrad Szajna of FormedSpace to be our general contractor, and he's hired a great team of subcontractors and tradespersons to do the work.

Family

The best part of my year was spending time with my family. Maxine started Kindergarten at Lincoln Elementary School in Evanston, and Veronica has grown into a full-fledged toddler. We didn't do as much travel as we would have liked this year, but we enjoyed biking together (we got a trailer this year for the girls), exploring our fair city, and working on projects, like the homemade arcade Maxine and I have been building together.

A few other things of note from the year:

• We participated in the Volkswagen settlement and chose to have them buy back our Jetta TDi. Good riddance. We bought a new Mazda CX-9 as a replacement.
• We volunteered for the Hillary Clinton campaign, including taking Maxine up to Kenosha, WI, to knock on doors for GOTV on election day. Well.
• I continue to feel grateful in so many different ways, for dear old friends and new ones we made this year, for our families immediate and extended, for our relative health and wealth, for our general dumb luck to be this fortunate and safe, recognizing just how contigent, random, and unlikely that is.

In addition to the performance benefits of the new SSA-based backend, there is a suite of new tools that allow a developer to interact with the SSA machinery. One such tool outputs the intermediate SSA statements, optimization passes, and resulting Go-flavored assembly. This is done by setting the environment GOSSAFUNC to the name of a function to disassemble when using the go tool, for example:

$ GOSSAFUNC=main go build

This invocation will output to the terminal, but the more interesting artifact is an HTML file, named ssa.html, written out to the current directory. Open the file in your web browser and you’ll see something like:

What you are looking at is a table with many columns extending to the right, each one except for the first and last representing an optimization pass over the preceding SSA form. (I counted 37 separate passes.) The first column is the the compiler’s initial, unoptimized SSA output, and the last column is the Go-flavored assembly that will be turned into machine code for the final compiled binary executable or shared library.

While this can look intimidating to the uninitiated, SSA is relatively simple by design -- each line represents a either a value being assigned the result of an instruction (i.e., one of the infinite number of registers), or a label of a "basic block" (a set of statements, aka, the things between curly braces in source code), or the exit of a basic block which jumps execution to a different basic block (eg., control flow like an if-statement or returning from a function call).

For example:

v4 = Const64 <int> [42]

Means assign the 64-bit integer constant value 42 to the register labeled v4.

b5: ← b4
  v15 = Copy <mem> v14
  v16 = StaticCall <mem> {runtime.printnl} v15
Call v16 → b6

Means b5 is the label for a basic block with two statements. It concludes with an exit Call instruction, taking program execution to another basic block, b6, when returning from the function call that produces the v16 value.

The tokens like Const64, Copy, and StaticCall are analogous to assembly instructions like MOV and LEA.

One special operation is Phi, or a "Phi node". Notice that a Phi node takes two arguments, which are two values. Also notice that a basic block with a Phi node has two basic block labels next to its own label, unlike every other basic block:

b3: ← b1 b2
   v20 = Phi <int> v4 v6
   ...

This is an interesting construct and it relates to program control flow. A basic block is defined by having a single entry and a single exit point, and having a set of statements that execute sequentially (i.e., no branching logic) in between. And "SSA" stands for "single static assignment", which means that each value is assigned one and only one time. But what do you do if you have a reference to a variable that could have different values depending on which branch of an if statement the program took? A Phi node is a way of resolving this apparent contradiction. Since each branch of an if statement by definition assigns to a unique value, a Phi node coalesces them into the final value depending on which branch was actually taken. So you can think of it as the run-time retrieval of a value based on some condition. This is why the block has two dependencies at the top rather than just one.

Let’s write a silly program to motivate some examples:

package main

func main() {
	x := 5

	if 1 < 0 {
		x = -42
	}

	println(x)
}

Let’s start with the initial basic block, b1:

b1:
  v1 = InitMem <mem>
  v2 = SP <uintptr>
  v3 = SB <uintptr>
  v4 = Const64 <int> [5]
  v5 = ConstBool <bool> [false]
  v6 = Const64 <int> [-42]
  v11 = OffPtr <*int64> [0] v2
If v5 → b2 b3

After some program initialization, v4 is the assignment to the local var x in our code of the constant 5. Go knows at compile-time that 1 < 0 is always false so it just assigns false to v5. v6 is the assignment of -42 to x that will happen during program execution.

At the end we have the basic block exit, If v5 → b2 b3. This tests the truth value of v5 to decide whether to jump program execution to either b2 (if true) or b3 (if false). This is similar to the following chunk of assembly:

    JNZ b2
b3:
  ...
b2:
  ...

One nice thing about the Go SSA HTML view is you can click on any token in the SSA form and it will highlight the references to and from that element.

We can see from the different colors how the control flow will go. You can visually connect the blocks of code that will execute and the assignments, function calls, and additional branching that will result.

Clicking on the Phi node and its dependencies, you can see from where the possible values come from in previous control flow.

Moving on, the function call prints out the integer value is in the following basic block:

b4: ← b3
  v9 = Copy <int> v20
  v10 = Copy <int64> v9
  v12 = Copy <mem> v8
  v13 = Store <mem> [8] v11 v10 v12
  v14 = StaticCall <mem> {runtime.printint} [8] v13
Call v14 → b5

The StaticCall instruction invokes the function from the Go runtime that is specialized to format integer values and print them to the terminal. One interesting thing to note is that the preamble to call sets some things up in memory, the location of which is fed to the printint function. If you notice, v11 refers back to the value set in b1, which is a pointer offset from v2, which was set from the stack pointer SP near the top of the program initialization. Which makes sense, because the generate assembly language needs concrete memory locations to address when invoking functions taking pointers.

There’s much more to investigate here, including the particular optimization passes, and tracing how individual instructions make their way through to the final assembly or are eliminated. But hopefully this has given you an introduction into SSA and how it maps to constructs in your applications.

I'll often have a slice that I want to filter down on, removing elements based on some test, and I would prefer to modify the slice in-place for whatever reason, either because I want to retain the reference to the original slice or I don't want to allocate a new slice as destination for the desired values.

You might think that modifying a slice in-place during iteration should not be done, because while you can modify elements of the slice during iteration if they are pointers or if you index into the slice, changing the slice itself by removing elements during iteration would be dangerous.

Here's a straightforward way to accomplish it. The idea is that, when you encounter an element you want to remove from the slice, take the beginning portion of the slice that has values that have passed the test up to that point, and remaining portion of the slice, i.e., after that element to the end, and copy them over the original slice. Then, assign a slice expression up to the number of values that passed the test to the original variable.

Here's an example. Let's say I have a slice of integers, and I only want to retain the even ones.

var x = []int{90, 15, 81, 87, 47, 59, 81, 18, 25, 40, 56, 8}

i := 0
l := len(x)
for i < l {
	if x[i] % 2 != 0 {
		x = append(x[:i], x[i+1:]...)
		l--
	} else {
		i++
	}
}
x = x[:i]
	
fmt.Println(x)
// [90 18 40 56 8]

The i variable is used to keep track of the number of even values found in the slice. When an element is odd, we create a temporary slice using append and two slice expressions on the original slice, skipping over the current element. The temporary smaller slice is copied over the existing, shifting down the remaining values. The l variable makes sure we make the right number of comparisons despite moving things around. It's important to note the memory location of the original slice is unchanged with the copy. No new heap allocations are performed, even with the temporary slice.

Update: A number of people, including here in comments and on the golang reddit, have pointed out that the method I outline here is pretty inefficient; it's doing a lot of extra work, due to the way I'm using append. A much better way to go about it is the following, which also happens to have already been pointed out in the official Go wiki:

y := x[:0]
for _, n := range x {
    if n % 2 != 0 {
        y = append(y, n)
    }
}

This also has the benefit of being simpler and shorter. Use it instead!

The idea is to use a counting semaphore, modeled with a buffered channel, to cause new clients to queue which arrive after the nth current client, where n is the size of the buffer.

Ideally, we wouldn't want to limit the amount of concurrency to our application, but practically, there are limits on underlying resources, and forcing clients to queue after a certain limit gives us control over that resource utilization.

Let's say we have a simple HTTP handler that requests access to some expensive resource, like a database or complex computation:

package main

import (
    "io"
    "log"
    "net/http"
)    

func main() {
     http.Handle("/", http.HandleFunc(func(w http.ResponseWriter, r *http.Request) {
         res := getExpensiveResource()
         io.WriteString(w, res.String())
     })

     log.Fatal(http.ListenAndServe(":8080", nil))
}

The handler can be requested by an unbounded number of clients, potentially exhausting our resources.

Let's add a counting semaphore that will gate entry into the handler:

func main() {
     const maxClients = 10
     sema := make(chan struct{}, maxClients)

     http.Handle("/", http.HandleFunc(func(w http.ResponseWriter, r *http.Request) {
         sema <- struct{}{}
         defer func() { <-sema }()

         res := getExpensiveResource()
         io.WriteString(w, res.String())
     })

We make a channel of type struct{}, because we are only interested in the send/receive semantics of the channel, not its value. The first statement of the handler is a send on the channel, which will succeed up to maxClients number of simulatenous requests. Think of the buffered channel as having empty slots, and being able to send on it means that you can fill a slot and proceed. If there are no empty slots, in other words, the length of the channel is equal to the buffer size, then the send will block, and will have to wait to proceed until a slot frees up. The next statement defers until after the handler has returned or panicked, and frees a slot by receiving from the channel.

If we have more than one handler to limit access to, we can move the semaphore into a middleware and wrap the original handler, leaving the body of it unchanged:

package main

import (
    "io"
    "log"
    "net/http"
)    

func maxClients(h http.Handler, n int) http.Handler {
     sema := make(chan struct{}, n)

     return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
         sema <- struct{}{}
         defer func() { <-sema }()

         h.ServeHTTP(w, r)
     })
}

func main() {
     handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
         res := getExpensiveResource()
         io.WriteString(w, res.String())
     })

     http.Handle("/", maxClients(handler, 10))

     log.Fatal(http.ListenAndServe(":8080", nil))
}

Note that this implementation will cause clients beyond the maximum number to queue without bound, until they hit the system limit of the listen(2) backlog.

This pattern can be used to control the amount of concurrency to any resource, not just net/http handlers.

It is this embeddedness that I believe will ultimately make The Bloomingdale Trail and the entire 606 system of parks a success. It's not a jewel, a thing to be admired, with its aesthetics upfront. It's a relentessly practical bit of new human-scale infrastructure in a vibrant residential area. It will materially improve the lives of its neighbors each day by enabling them to be active, to commute, to play, and to discover in a new and unique way. It's worth remembering that the project was funded largely by federal transportation dollars, earmarked for reducing traffic congestion and air pollution. People will wonder what this thing is, and the answer will be in its daily use.

I remember walking around the old Bloomingdale Line, a disused elevated railroad embankment, in 2002 with a group of work colleagues. We would sometimes take our lunch up top, ducking under a fence at Milwaukee and Leavitt to gain access. The germ of the Friends of the Bloomingdale Trail was planted there; the non-profit community organization officially formed a year later. The circumstances at the time were fortunate: the development of the High Line in New York provided a template and a healthy competitive jolt; the railroad company was looking to rid themselves of their responsibilities to the line; the City wanted to tear down the embankment and spanning viaducts, providing further impetus; and crucially, the rights-of-way were all contiguous and owned by the City: there need be no time-consuming negotiating with private owners to acquire the trail's property, as there was in New York. From there we held community meetings, trash pick-up days, festivals, goofy but earnest Valentine's Day events, led tours, pitched aldermen and city planners, documented the Trail as it existed, helped open a new neighborhood park next to the Trail, printed posters and brochures, hosted arts events, let David Schalliol do his magic, connected with open space plans, and started a partnership with the Trust for Public Land and the City of Chicago to design and build the Trail. In 2007 and 2008, we convened neighbors in a series of meetings and surveys to listen to, capture, and synthesize the community's vision for the project. The product of this effort, the Community Visioning Update, was perhaps our most important practical work as an organization: this document was incorporated into the City's official request for proposals for design and construction. To the best of our ability, we made sure the future Trail would be reflective of the community it came from and would serve.

It's time now to celebrate the opening of the Trail and begin a new phase in the life of FBT. The original goals of the organization were to:

• Preserve the elevated right of way
• Beautify the public space
• Create a new, mixed-use trail/linear park
• Establish a broad coalition that supports the proposed park
• Connect with neighborhood schools and institutions

Our new mission is to be the community stewards of the Trail, and to that end, we recently applied and have been approved to be a Chicago Park District Advisory Council, or PAC. As befits our unusual new park, we're breaking new ground as a PAC. We're unique in that our bylaws state there will be board representation from each of the 4 neighborhoods, and from each of the constituent park PACs (Julia de Burgos, Walsh, Churchill Field, and Kimball). Because no other park covers as much ground, cuts through as many neighborhoods, and links up as many adjacent smaller parks, governance and community orginizing around The Bloomingdale Trail will be a new experiment for all involved.

One last thought. There are very few good west-east routes in Chicago: most transporation infrastructure radiates from and to the Loop. The Bloomingdale Trail is a stroke across the spokes, and the physical, economic, and cultural circulation it promotes will be fascinating to watch. But there are bigger things at stake. Even before this new park was built, the Trail conspicuously ended at the north branch of the Chicago River. (Now it ends at Ashland, that street's bridge having been born-again over Western.) It's always been a dream and a goal of FBT and the 606 partners to extend the Trail across the river in a future phase. From there, on-street bicycle paths can be knit together, ultimately arriving at the lakefront. However, there's an even bigger dream to be dreamt. A few miles west of the western terminus of the Trail, the Illinois Prarie Path has its eastern endpoint. The IPP carries you out due west 60 miles past the outer suburbs. A network of rural trails beyond can be followed all the way to Iowa. So while we celebrate the opening of Chicago's next great park tomorrow, the notion of a bicycle trip that begins at the Mississippi River and ends at Lake Michigan, on bike paths the entire span, should stay in the back of our minds as a not-too-distant possibility.

Look up! It's The Bloomingdale Trail

tl;dr: I tried to make a map of Chicago election results, I found only out-of-date wards & precincts shapefiles, I had to FOIA the up-to-date versions, I got them, I republished them so anyone can download them, and finally made that map.

Read on for the full saga.

After this week’s municipal general elections in Chicago, I was looking for detailed results in the mayor’s race, which didn’t end Tuesday night but is headed for a run-off between Mayor Rahm Emanuel and challenger Cook County Commissioner Chuy Garcia on April 7. Specifically, I wanted to see where in the city the support for each candidate was, and at as granular a level as possible.

The Chicago Board of Elections posts vote tallies by precinct (50 wards in Chicago, with on average 40 precincts per ward). Precincts are the smallest unit of political geography—in Chicago, they are roughly a few square city blocks each. Given the neighborhoody nature of Chicago and the block-by-block affinities that exist (which leads politicians to produce carefully sculpted gerrymanders like the 2nd Ward in order to corral voters into favorable pens), a map showing the relative intensity of voting percentages per candidate by precinct would be a good tool for aiding detailed understanding of this election or any election, and a building block for many possible similar analyses in the future.

So I set out to make such a map. My plan was to gather the vote totals per precinct, shapefiles of the city ward and precinct boundaries, and join them together using tools like d3 to draw a choropleth or thematic map in a web browser. This is a straightforward plan and is well-trod ground. However, I naïvely assumed the official source material I gathered would be accurate and up-to-date.

After scraping the vote totals from the BOE site[1], I downloaded the wards and precincts shapefiles from the City of Chicago’s Data Portal site, which is a service that hosts many different types of data, from building permits to restaurant inspections. I did this by typing “wards” and “precincts” into the search box and downloading from the results pages the links titled “[Boundaries

• Wards]dataportalwards” and “Ward Precincts”. There was nothing to indicate that these files were out of date, nor anything else to indicate that they were not the current, authoritative source of these data sets.

I put together a first draft of the map and shared it with some colleagues who are experts in mapping and Chicago data. They quickly pointed out that the map appeared to be using the old wards and precincts.[2] In 2012, the city council approved a new set of ward boundaries, redrawing the city’s political map. They were to go into effect in 2015, and this week’s election, which included all 50 aldermanic races, were to be contested on this new geography. The conspicuously missing new 2nd Ward was the tip-off my map was wrong.

I searched for the updated boundaries, but came up with only unofficial sources, and only for wards at that. There was the WBEZ map from their original 2012 story, and the Tribune had created a side-by-side comparison of the old and new wards. But I couldn’t trust these for my own use, because of their uncertain provenance. And without matching updated precincts, I couldn’t join vote totals for use in a map in any case.

Taking a page from the people person at my old job, I made a phone call to the Board of Elections: maybe I could just ask for the data and they would give it to me? I stated my request very plainly and without explanation of motive, and was told to “hold please” a couple of times while I bounced between departments. A few moments later, I heard “Districts and Boundaries” on the line. Success! Here was, literally, the person who could help me, right then. Or so I thought. I repeated my request, and without a moment’s hesitation, the Districts and Boundaries voice said that I would need to contact the BOE’s FOIA officer, and here was their email address.[3]

It was hard to tell how much of this was bluffing, as in, let’s see you actually bother to make a FOIA request, but I went ahead and stubbornly wrote an email to the FOIA officer anyway. I was under no illusions that my request would be fulfilled quickly enough to make my post-election map still relevant.

I then took to Twitter to register my displeasure for this state of affairs—we just had a citywide election for our top local offices, operating on the assumption of the new city council-vouched districts, and yet, despite nearly a decade of the open data movement, despite official portaldom, the key base layers of the political strata were still available only to the learned monks—and moved on.

Lo, but was my request not answered but a few scant hours later! I can’t tell you how surprised I was to see this in my inbox:

I thanked the officer and downloaded the payload, which was a set of 50 folders, each corresponding to a ward and containing a shapefile of that ward’s precincts therein. I eyeballed the boundaries with QGIS and was satisfied that they appeared to be legit. (Again, the shape of the notorious 2nd Ward was the main clue.)

In the absence of official publication, I was determined to at least not have the next person who goes looking for wards and precincts to wind up in FOIA land. As relatively pain-free as this episode was, the fact that I had to engage with the FOIA plumbing in order to fulfill a minor data request is not good. And there is every reason to think that a typical FOIA request will take orders of magnitude longer to fulfill than my jackpot.

My approach was to self-publish the data, but to be clear about its source and my methodology for any transformations. While I’d much rather prefer this data appear on Data Portal, I’d also prefer not for our collective energies to be wasted in pursuits such as these.

Regarding those transformations, I had a set of precincts, but I also wanted the wards that derive from them (a ward is completely defined by its constituent precincts). I imported the precincts into a PostgreSQL database with the PostGIS extension. From there I created wards by grouping precincts by their ward number, and unioning their geometries (i.e., merging a bunch of small precinct polygons into one large ward polygon). Then I exported from the database into various geospatial data formats—Shapefile, TopoJSON, GeoJSON, KML, etc.

I made these exports available for download by anyone, hosted on GitHub.

I finally was able to make the map I wanted, at least, the first-order map, a basic voter preference density map. I hope to build on this data infrastructure with different overlays, result sets, future elections, and so on.

You can view the map here; choose between mayoral candidates in the drop-down selector to update the map with their vote percentages.

With several candidates, in can be useful to see them arrayed as small multiples for easier comparison[4]:

I’d like to see the left hand of the operators of the Chicago Data Portal talk with the right hand of the Chicago Board of Elections, and simply take down the pre-2015 ward and precinct boundaries (or better yet, rename them to something that won’t be mistaken for the most recent version and leave them up for historical research) and get the current shapefiles uploaded as soon as possible. In the meantime, I hope that interested parties will avail themselves of my hosted shapefiles.

More generally I’d like for stakeholders in the world of government data to reflect on the state of the open data movement, and consider examples such as these as the tiny abrasions that impede all sorts of productivity, beyond my modest map-making efforts. On one hand, we’ve made enormous progress; on the other, we’re still fighting the same 10-year-old battles.

And to the FOIA officer at the BOE who responded so promptly, many thanks!

1. It is 2015 and the third-largest U.S. city is still publishing official election results on a decade-old system that doesn’t lend itself to machine-readability without substantial friction, which violates #5 of the 8 Principles of Open Government Data. I wrote a Python program to extract the data from the particular formatting of the BOE site. ↩
2. In my defense, while I’ve lived in Chicago for more than 10 years, I only recently moved back after a 5-year hiatus, so my map intuitions are a little stale. ↩
3. Thus arguably in violation of #1, #3, #4, and #6 of the 8 Principles of Open Government Data. ↩
4. For this I just screenshotted and collaged them in an image editor. ↩

I’m going to show how to use the LLVM API to programmatically construct a function that you can invoke like any other and have it execute directly in the machine language of your platform.

In this example, I’m going to use the C API, because it is available in the LLVM distribution, along with a C++ API, and so is the simplest way to get started. There are bindings to the LLVM API in other languages — Python, OCaml, Go, Rust — but the concepts behind using LLVM to generate code are the same across the wrapper APIs.

This example sort of skips to the middle phase of compiler construction. Assume the frontend (lexer, parser, type-checker) has built an AST and we’re now walking it to emit the intermediate representation of the code for the backend to take and optimize and spit out machine code.

In this case, we’ll just type out the straight-line procedural code for a simple function that would normally be dynamically cobbled together in a AST walker function, calling the LLVM API when it encounters certain nodes in the tree.

For the example, we’ll build a simple adder function, which takes two integers as arguments and returns their sum, the equivalent of, in C:

int sum(int a, int b) {
    return a + b;
}

To be clear about what we are doing here: we are using LLVM to dynamically build an in-memory representation of this function, using its API to set up things like function entry and exit, return and parameter types, and the actual integer add instruction. Once this in-memory representation is complete, we can instruct LLVM to jump to it and execute it with arguments we supply, just as if it was a executable we had compiled from a language like C.

Click here to view the final code.

Modules

The first step is to create a module. A module is a collection of the global variables, functions, external references, and other data in LLVM. Modules aren’t quite like, say, modules in Python, in that they don’t provide separate namespaces. But they are the top-level container for all things built in LLVM, so we start by creating one.

LLVMModuleRef mod = LLVMModuleCreateWithName("my_module");

The string "my_module" passed to the module factory function is an identifier of your choosing.

Note that as you’re navigating the LLVM C API documentation, different aspects are grouped together under different header includes. Most of what I’m detailing here, such as modules and functions, is contained in the Core.h header, but I’ll include others as we move along.

Types

Next, I create the sum function and add it to the module. A function consists of:

• its type (return type),
• a vector of its parameter types, and
• a set of basic blocks.

I’ll get to basic blocks in a moment. First, we’ll handle the type and parameter types of the function — its prototype, in C terms — and add it to the module.

LLVMTypeRef param_types[] = { LLVMInt32Type(), LLVMInt32Type() };
LLVMTypeRef ret_type = LLVMFunctionType(LLVMInt32Type(), param_types, 2, 0);
LLVMValueRef sum = LLVMAddFunction(mod, "sum", ret_type);

LLVM types correspond to the types that are native to the platforms we’re targeting, such as integers and floats of fixed bit width, pointers, structs, and arrays. (There’s no platform-dependent int type like in C, where the actual size of the integer, 32- or 64-bit, depends on the underlying machine architecture.)

LLVM types have constructors, and follow the form "LLVMTYPEType()". In our example, both the arguments passed to the sum function and the function’s type itself are 32-bit integers, so we use LLVMInt32Type() for each.

The arguments to LLVMFunctionType() are, in order;

1. the function’s type (return type),
2. the function’s parameter type vector (the arity of the function should match the number of types in the array), and
3. the function’s arity, or parameter count,
4. a boolean whether the function is variadic, or accepts a variable number of arguments.

Notice that the function type constructor returns a type reference. This reinforces the notion that what we did here is the LLVM equivalent of declaring a function prototype in C.

The third line in here adds the function type to the module, and gives it the name sum. We get a value reference in return, which can be thought of as a concrete location in the code (ultimately, memory) upon which to add the function’s body, which we do below.

Basic blocks

The next step is to add a basic block to the function. Basic blocks are parts of code that only have one entry and exit point - in other words, there is no other way execution can go than by single stepping through a list of instructions. No if/else, while, loops, or jumps of any kind. Basic blocks are the key to modeling control flow and creating optimizations later on, so LLVM has first-class support for adding these to our in-progress module.

LLVMBasicBlockRef entry = LLVMAppendBasicBlock(sum, "entry");

Note the "append" in the name of the function: it’s helpful to think of what we’re doing as growing a running tally of chunks of code, and so our basic block is appended relative to the function we added to the module previously.

Instruction builders

This notion of a running tally fits with the instruction builder, which is how we add instructions to our function’s one and only basic block.

LLVMBuilderRef builder = LLVMCreateBuilder();
LLVMPositionBuilderAtEnd(builder, entry);

Similar to appending the basic block to the function, we’re positioning the builder to start writing instructions where we left off with the entry to the basic block.

LLVM IR

Sidebar: LLVM’s main stock-in-trade is the LLVM intermediate representation, or IR. I’ve seen it referred to as a midway point between assembly and C. The LLVM IR is a very strictly defined language that is meant to facilitate the optimizations and platform portability that LLVM is known for. If you look at IR, you can see how individual instructions can be translated into the loads, stores, and jumps of the ultimate assembly that will be generated. The IR has 3 representations:

• as an in-memory set of objects, which is what we’re using in this example,
• as a textual language like assembly,
• as a string of bytes in a compact binary encoding, called bitcode.

You may see clang or other tools emit LLVM IR as text or bitcode.

Back to our example. Now comes the crux of our function, the actual instructions to add the two integers passed in as arguments and return them to the caller.

LLVMValueRef tmp = LLVMBuildAdd(builder, LLVMGetParam(sum, 0), LLVMGetParam(sum, 1), "tmp");
LLVMBuildRet(builder, tmp);

LLVMBuildAdd() takes a reference to the builder, the two integers to add, and a name to give the result. (The name is required due to LLVM IR’s restriction that all instructions produce intermediate results. This can further be simplified or optimized away by LLVM later, but while generating IR, we follow its strictures.) Since the numbers we wish to add are the arguments that were supplied to the function by the caller, we can retrieve them in the form of the function’s parameters using LLVMGetParam(): the second argument to is the index of the parameter we seek from the function.

We call LLVMBuildRet() to generate the return statement and arrange for the temporary result of the add instruction to be the value returned.

Analysis & execution

That concludes the building instructions phase of creating our function; the module is now complete. The next phase of the example is setting it up for execution.

First, let’s verify the module. This will ensure that our module was correctly built and will abort if we missed or mixed up any steps.

char *error = NULL;
LLVMVerifyModule(mod, LLVMAbortProcessAction, &error);
LLVMDisposeMessage(error);

LLVM provides either a JIT or an interpreter to execute the IR we’ve built. It will create a JIT if it can for the target platform, and fall back to an interpreter otherwise. In any case, the thing that will run our code is called the execution engine.

LLVMExecutionEngineRef engine;
error = NULL;
LLVMLinkInJIT();
LLVMInitializeNativeTarget();
if (LLVMCreateExecutionEngineForModule(&engine, mod, &error) != 0) {
    fprintf(stderr, "failed to create execution engine\n");
    abort();
}
if (error) {
    fprintf(stderr, "error: %s\n", error);
    LLVMDisposeMessage(error);
    exit(EXIT_FAILURE);
}

We could hard-code some integers to be summed, but it’s easy enough to have our program receive them from the command line.

if (argc < 3) {
    fprintf(stderr, "usage: %s x y\n", argv[0]);
    exit(EXIT_FAILURE);
}
long long x = strtoll(argv[1], NULL, 10);
long long y = strtoll(argv[2], NULL, 10);

Now that we have two integers in the representation of our host language, we need to transform them into the analogous representation in LLVM. LLVM provides factory functions that convert values into the types we need to pass to our function:

LLVMGenericValueRef args[] = {
    LLVMCreateGenericValueOfInt(LLVMInt32Type(), x, 0),
    LLVMCreateGenericValueOfInt(LLVMInt32Type(), y, 0)
};

Now for the moment of truth: we can call our (JIT’d) function!

LLVMGenericValueRef res = LLVMRunFunction(engine, sum, 2, args);

We have a result, but it’s still in LLVM-land. We recover it to a C type, the reverse operation from above, and print the sum:

printf("%d\n", (int)LLVMGenericValueToInt(res, 0));

And there we have it. We’ve programmatically constructed a function from the ground up, and had it run directly in machine code native to our platform. There is much more to LLVM, including control flow (eg., implementing if/else) and optimization passes, but we’ve covered the basics that would be in any LLVM-IR-to-code program.

Compiling

In order to compile our program, we need to reference the LLVM includes and link its libraries. Even though we’ve written a C program, the linking step requires the C++ linker. (LLVM is a C++ project, and the C API is a wrapper thereof.)

$ cc `llvm-config --cflags` -c sum.c
$ c++ `llvm-config --cxxflags --ldflags --libs core executionengine jit interpreter analysis native bitwriter --system-libs` sum.o -o sum
$ ./sum 42 99
141

Bitcode

One final thing. I mentioned previously that LLVM IR has three representations, including bitcode. Once you have a completed module, you can emit bitcode and write it out to a file.

if (LLVMWriteBitcodeToFile(mod, "sum.bc") != 0) {
    fprintf(stderr, "error writing bitcode to file, skipping\n");
}

From there, you can use tools to manipulate it, like llvm-dis to disassemble the bitcode into the textual LLVM IR assembly language.

$ llvm-dis sum.bc
$ cat sum.ll
; ModuleID = 'sum.bc'
target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"

define i32 @sum(i32, i32) {
entry:
  %tmp = add i32 %0, %1
  ret i32 %tmp
}

Source code of example

Here is the complete source of the program from above:

/**
 * LLVM equivalent of:
 *
 * int sum(int a, int b) {
 *     return a + b;
 * }
 */

#include <llvm-c/Core.h>
#include <llvm-c/ExecutionEngine.h>
#include <llvm-c/Target.h>
#include <llvm-c/Analysis.h>
#include <llvm-c/BitWriter.h>

#include <inttypes.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char const *argv[]) {
    LLVMModuleRef mod = LLVMModuleCreateWithName("my_module");

    LLVMTypeRef param_types[] = { LLVMInt32Type(), LLVMInt32Type() };
    LLVMTypeRef ret_type = LLVMFunctionType(LLVMInt32Type(), param_types, 2, 0);
    LLVMValueRef sum = LLVMAddFunction(mod, "sum", ret_type);

    LLVMBasicBlockRef entry = LLVMAppendBasicBlock(sum, "entry");

    LLVMBuilderRef builder = LLVMCreateBuilder();
    LLVMPositionBuilderAtEnd(builder, entry);
    LLVMValueRef tmp = LLVMBuildAdd(builder, LLVMGetParam(sum, 0), LLVMGetParam(sum, 1), "tmp");
    LLVMBuildRet(builder, tmp);

    char *error = NULL;
    LLVMVerifyModule(mod, LLVMAbortProcessAction, &error);
    LLVMDisposeMessage(error);

    LLVMExecutionEngineRef engine;
    error = NULL;
    LLVMLinkInJIT();
    LLVMInitializeNativeTarget();
    if (LLVMCreateExecutionEngineForModule(&engine, mod, &error) != 0) {
        fprintf(stderr, "failed to create execution engine\n");
        abort();
    }
    if (error) {
        fprintf(stderr, "error: %s\n", error);
        LLVMDisposeMessage(error);
        exit(EXIT_FAILURE);
    }

    if (argc < 3) {
        fprintf(stderr, "usage: %s x y\n", argv[0]);
        exit(EXIT_FAILURE);
    }
    long long x = strtoll(argv[1], NULL, 10);
    long long y = strtoll(argv[2], NULL, 10);

    LLVMGenericValueRef args[] = {
        LLVMCreateGenericValueOfInt(LLVMInt32Type(), x, 0),
        LLVMCreateGenericValueOfInt(LLVMInt32Type(), y, 0)
    };
    LLVMGenericValueRef res = LLVMRunFunction(engine, sum, 2, args);
    printf("%d\n", (int)LLVMGenericValueToInt(res, 0));

    // Write out bitcode to file
    if (LLVMWriteBitcodeToFile(mod, "sum.bc") != 0) {
        fprintf(stderr, "error writing bitcode to file, skipping\n");
    }

    LLVMDisposeBuilder(builder);
    LLVMDisposeExecutionEngine(engine);
}

See the GitHub repo for the Makefile and details on how to build the example on your machine.

Texas "Tremodillo"

I stopped attending college after my freshman year, in part to go be the guitar tech for a band. This was 1996. Some friends of mine from high school had formed it, and a major label had signed them to produce a record. The band had me spend a week with César Díaz at his home and workshop in Pennsylvania. César had been the tech for guitarists like Stevie Ray Vaughn and Eric Clapton. Vaughn was, I think, César’s idol. He was a guitar player himself, and even had his guitar set up like Stevie’s, with heavy strings and high action. César had stopped touring, and made guitar amps and effects pedals instead. He also would teach new guitar techs the secrets of the profession. The Texas Tremodillo was one of two pedals he made. It had a tremolo effect, which is like a fast, regular wobble. We would spend the day in his workshop, tinkering on an amp that had blown a capacitor, or on a pedal that was buzzing. He was soft-spoken, reserved, and didn’t have a lot of patience for others. At night we’d get dinner at a local Indian place, or eat with his wife, whom I remember being kind, and their young son. One day we drove in to New York, somewhere in the Lower East Side or Village, to peck around at a used guitar show in an auditorium. We ran into Jimmy Vivino outside. He was wearing a pork pie hat and we walked around with him, looking at guitars—they were close friends. Later that night, Vivino and a bunch of other guys showed up at César’s workshop to jam. They played “The Weight” and many other classic rock songs. César soloed on an Stevie Ray Vaughn song. César was kind to me in the end. He wasn’t too annoyed when I would later call him to ask what to do about a failing pickup, or if I could use these tubes instead of those in this amp. I stopped working for the band after a year or so, and forgot about César, except when I would see the Tremodillo shirt in my closet. I rarely wore it—it was too big, and I felt like I should preserve it somehow, which I never made an attempt to do. Sometime later I read that he died in the early 2000s. He had been sick with liver failure not long after I visited him. At one point he had a transplant, but he only lived another couple of years. RIP, César.

Vote, F*cker

This is a shirt Ben Helphand gave me after a trip he took to Oregon. This would have been around 2004. The Bus Project had made the shirt. Ben and I became friends after we went to Minnesota in 2002, volunteering on the senate campaign of the late Paul Wellstone. After that, we would scheme up ways to try to improve small-d democratic participation. The Bus Project was a big part of the inspiration that led to the creation of the Election Day Advent Calendar in 2006. I’ll miss the Vote, F*cker shirt but many washes rendered it too shrunken and I looked like a sausage in it.

Pope Benedict’s Army

In 2005, I played on a co-ed softball team with my then-girlfriend, now-wife Michelle. We had been dating for a few months, and exploring a new group of friends in common together, some of whom were on the team. A conclave had elected Joseph Ratzinger pope on April 19. Three weeks later, the team organizer, John Pick, wrote the potential players an email:

I pulled the trigger on the league and put it on my credit card.

Belmont and California
6:15- 7:15
Thurs nights
starting June 2
80 per person.

we're called Pope Benedict's Army

IIRC, we did not do the pontiff proud. I think we might have won one game?

Molly Sircher, who was our ringer, conceived and produced the shirt. She had played softball at DePaul, and was by far the best player on our team. I wanted to hang on to the shirt, but the shape was a little boxy, and it had a musty smell it picked up probably during that summer and never got rid of.

PyCon US 2012

In 2012, I gave a talk at PyCon for the first and so far only time. The U.S. conference was in Santa Clara, CA, near San Jose. At the time, I was the deputy director of technology at the DNC. I was eager to attend tech conferences during my tenure there, to try to recruit software engineers to work with me on the campaign. This trip was a bust on that score, and I felt afterwards that my presentation had been lackluster. In retrospect, it was a stressed time. I felt like we didn’t have enough engineering help at the DNC to get through the campaign, and that it was difficult to attract any. I always liked Idan Gazit’s official PyCon snake logo that year and thought it was a handsome shirt. But it’s too small now, into the donation pile you go.

It’s much better to just let the OS assign an unused port to this quick webserver process, since you don’t really care where it goes. You can do this by passing 0 as the port argument, and that totally works: Python prints out the port it started the HTTP server on. There’s just one problem that trips me up: it prints out this new port number in such a way that you have to either mouse over, select, copy, open a new tab, paste, after typing in “localhost” or “0.0.0.0”, or you have to eyeball it and type it in with the new tab:

$ python -m SimpleHTTPServer 0
Serving HTTP on 0.0.0.0 port 61200 ...

See what I mean, you have to snag that 61200 somehow. I just want to start a webserver and have it immediately open to that address in my browser! That output should be clickable or hook into OS X’s open.

So this shell script does that.

$ ./quickserver
Serving HTTP on 0.0.0.0 port 61209 ...
http://0.0.0.0:61209/
127.0.0.1 - - [19/Jun/2014 17:21:56] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [19/Jun/2014 17:21:57] code 404, message File not found
127.0.0.1 - - [19/Jun/2014 17:21:57] "GET /favicon.ico HTTP/1.1" 404 -

Probably too small to deserve it’s own repo but I figured someone might want to make it work on Ubuntu or whatever. Here it is on GitHub.

• Immediate-Mode Graphical User Interfaces Immediate-mode GUI is a straight-forward way of rendering a UI. It’s so simple, in fact, that I had to watch the video twice and do a tutorial to understand it. (I did the tutorial using SDL on OS X, and then ported it to JavaScript and canvas. Incidentally, I also used the C preprocessor on my JavaScript file, following this, to get statically-generated IDs for the widgets, it worked well.) The short explanation of immediate-mode GUI is, in your render() function that’s called for each frame of your application (ala requestAnimationFrame), you call functions that handle everything needed to draw, handle events, change state, and trigger other events, for your UI’s widgets. Your code looks something like if (button(id, x, y)) buttonWasPressed();, and that’s the entirety of rendering a button widget to the screen and handling click events on it. (In most cases, the widget functions return a boolean of whether the button was pressed, text field was changed, etc.) There are no callbacks or separate bindings. You maintain a tiny bit of global state that helps coordinate all the action. The upside is you have total control over your UI’s appearance and behavior. The downside is, you have to implement all of your UI’s appearance and behavior yourself. My feeling so far is that it is not something you would do if you were just implementing a typical UI in a web browswer, because you have all the browser’s widgets already at your disposal (not to mention HTML and CSS layout). You’d be reinventing the wheel. But it seems an ideal approach for a game UI (which is where I believe the idea originated, in the game development world), on platforms where you don’t already have a core UI or widget library available, in a native mobile application where performance is paramount, or any kind of custom application, even on the web, where you want or need complete over the UI, because, for instance, the supplied browser form elements don’t suffice. For example, immediate-mode GUI would fit something like Soundslice’s custom interface perfectly. (via)

• Functional reactive programming This was an eye-opening talk for me. FRP could show us the way out of the fly bottle of complicated, callback-knotted async JavaScript UIs in the browser. The core idea is to treat events not as isolated occurances to be handled on a per-callback basic, but instead as collections, and once you do that, you have the power of higher-order functions like map, reduce, filter, and merge to describe complex behaviors as sort of a pipeline of collection processing. If you took Python’s list and generator comprehensions to browser events, you start to get the idea. RxJS is the tool highlighted in the talk, but bacon.js also seems to be a popular FRP library for JavaScript (haven’t tried it myself). There’s also a browser-based FRP tutorial to work through.

• Traceur Programming FRP in JavaScript becomes a lot more pleasant with the new anonymous function syntax ((x) => x + 1 instead of function(x) { return x + 1; }) coming in JavaScript 6, or ES6. Traceur compiles ES6 to JavaScript that will run in current browsers, so you can code and get the benefit of the new syntax and other upcoming language features now. I have it as a build step in a Makefile, alongside minification. Then presumably, barring language-breaking changes, you’d be able to remove the build step at some future date when ES6 has become widely adopted.

• Elm Elm is an entire language built around FRP which targets the browser. It is a Haskell or OCaml-like language that compiles down to HTML, CSS, and JavaScript. It seems to rely on the functor concept, which it calls ‘lift’, to convert browser events into something its built-in higher-order functions can process. It’s arguable that because it is a functional language like Haskell, it’s more naturally suited for dealing with the sorts of concurrency issues in UIs that libraries like RxJS were created to address in JavaScript. I’m still just in playground mode with it.

• GopherCon talks It says something about the Go community how uniformly excellent and entertaining these talks are. Interesting and dense with practical knowledge.

I also recently tried to teach myself Acme. You can certainly glimpse the power of a system like that. But ultimately I decided editing speed is more important to me, and I’m pretty fast in Vim, so I abandoned the effort.

CoreOS seems like it could become pretty important.

Programming a computer, still a fun thing to do.

I also appeared on All In with Chris Hayes on MSNBC on February 28, 2014 to talk about it and my time with the ad hoc team.