Paul Smith

Going to work for Democrats

2011-09-04T16:00:00Z

I have some exciting news: in a couple of weeks, I will be the new deputy director of technology at the Democratic National Committee. I’ll be helping to create software that helps get Democrats elected. It’s a great opportunity for me to do what I love for an organization that I passionately support. I hope to help make the technology department there top-notch in terms of software engineering practices, bringing what I’ve learned from having helped develop, deploy, and grow EveryBlock. And I’ll have some projects of my own, which will hopefully advance the state of campaign software somewhat. It’s especially exciting, too, to be able to collaborate with friends in Chicago working on that one campaign.

I have been volunteering on political campaigns—federal, state, and local races—for years, and have often lamented the state of campaign software. It’s partly understandable, because campaigns tend to be all-hands-on-deck, hair-on-fire affairs, where it’s hard to justify long-range planning and software development, even if it might make the lives of your staff, organizers and volunteers easier, since your organization may not even exist for more than a few months. And campaigns rarely have in-house software engineers, so opportunities to capture and encode knowledge in the form of software, and explore new technologies, are missed.

Obama For America gets this—that’s why they’ve hired like a start-up for this campaign cycle, recognizing that great software is a competitive advantage and no longer an afterthought you contract out for. And the DNC gets it, too, and that’s why I’m excited to join them. The chance to help re-elect this president, restore Democratic majorities in Congress, and also to help down-ballot Democrats across the country in this and future campaign cycles is one I couldn’t pass up.

I’ll be commuting to the DNC’s offices in Washington, D.C. from Baltimore on a regular basis, though I’ll still be working from home a couple of days each week, so that I won’t too miss much of this kind of stuff.

It’s a bittersweet development, because I’ll be leaving EveryBlock, which I helped found 4 years ago. With the success of the recent relaunch, though, I feel now is as good a time as any to step away. The site is in great hands, and the response from users to the new version has been enthusiastic. It couldn’t have a better home than msnbc.com, who have provided great guidance and resources. I’m thrilled with the success we’ve had and for how far it’s come, and I’m confident that it will continue to be the best place on the internet to help make your block a better place.

I am particularly grateful for having worked with my great EveryBlock colleagues. I’m humbled by them and their talents and work ethic. It was a privilege to learn from them and improve my craft, however modestly, by their examples.

For now, I’m focusing on winding down at EveryBlock, and getting prepared for a new commute (I’ll try to hack it with the MARC train and a bike) and a campaign season now fully engaged. It’s a thrilling opportunity, and I hope to make the most of it.

—@paulsmith

Maxine

2011-08-27T20:10:00Z

One month ago today, Maxine Mills Smith arrived in this world. Her mom gave birth to her at Mercy Medical Center in Baltimore, Maryland, at six in the morning. She was, and continues to be, a long and strong gal. She enjoys being carried about and looking up at the changing scenery. Like her father, she zonks out to the motion of a vehicle ride, be it stroller or car. She has healthy lungs, likes to exercise them, and is a vocal chirper. She has been respecting her mom and dad with 5 and 6-hour stretches of sleep at night, but likes to mix it up from time to time and throw her mom some napless curveballs. When she is awake, she is bright and alert. Given the choice, she’d rather have some light assistance and try to stand and monster march (I said she was strong) than squirm around on her belly like a beetle. Her binky is a frequent companion, and she is a champion eater of mom’s milk. The doctors and nurses at the pediatrician’s office think she is doing fine, and give her an extra wink. We think she’s is doing fine, too. Family love Maxine. Welcome home, my daughter.

Parsing and formatting date/time in Go

2011-05-20T02:00:00Z

Go takes an interesting approach to parsing strings to time objects, and formatting time objects as strings. Instead of using codes like most languages to represent component parts of a date/time string representation—like %Y for a 4-digit year like “2011” or %b for an abbreviated month name like “Feb”—Go uses a mnemonic device: there is a standard time, which is:

Mon Jan 2 15:04:05 MST 2006  (MST is GMT-0700)

Or put another way:

01/02 03:04:05PM '06 -0700

Instead of having to remember or lookup the traditional formatting codes for functions like strftime, you just count one-two-three-four and each place in the standard time corresponds to a component of a date/time object (the Time type in Go): one for day of the month, two for the month, three for the hour (in 12-hour time), four for the minutes, etc.

The way you put this into action is by putting together the parts of the standard time in a layout string that matches the format of either the string representation you want to parse into a Time object or the opposite direction, when you want to generate a string representation from an Time object.

Parsing:

package main

import (
    "fmt"
    "time"
)

func main() {
    value  := "Thu, 05/19/11, 10:47PM"
    // Writing down the way the standard time would look like formatted our way
    layout := "Mon, 01/02/06, 03:04PM"
    t, _ := time.Parse(layout, value)
    fmt.Println(t)
}

// => "Thu May 19 22:47:00 +0000 2011"

Formatting:

package main

import (
    "fmt"
    "time"
)

func main() {
    t := time.SecondsToLocalTime(1305861602)
    t.ZoneOffset = -4*60*60
    fmt.Println(t.Format("2006-01-02 15:04:05 -0700"))
}

// => "2011-05-20 03:20:02 -0400"

There are predefined constants in the time package for common formats such as RFCs 822 and 3339.

The use of a mnemonic over obscure formatting codes I think reflects the pragmatism of Go’s developers and their focus on creating a language that makes its users more productive. While it is a break with tradition to abandon strftime-style formatting, they probably recognized that most developers, no matter how experienced, reach for man strftime or similar documentation more often than not (hell, I have a mug with the codes printed on it), and each time they do, it is an expensive context switch, in terms of lost focus. Writing the standard time down the way yours looks may be quirky, but it's easy to recall, and it also happens to match the shape of your time string, syntatically.

It’s fascinating to see a language with the pedigree of Go—from the minds of long-time C and Unix developers—toss aside certain old and venerable ways of doing things. But it’s consistent with a language that’s relatively small (in the good way of being comprehensible and coherent), focused on efficiency, and careful in what it includes.

A partial list of metaphors, visual and otherwise, anti-skeuomorphism zealots have to tackle to reach utopia

2011-04-05T18:48:00Z

Motivated by irrational hatred and overstated claims of the continuing utility of a 3.5" floppy disk icon to mean “save”:

Handset icon, meaning “phone,” on every iPhone, Android
email
Website (implies a physical place)
Wrench or cog icon, meaning “settings” or “preferences”
Shopping cart icon
Clock with analog hands, meaning “time” or time-centric application
Incandescent light bulb, several meanings, including energy settings
Bound, paper book, meaning “book”

Of course there are many, many anachronistic interface elements and metaphors in the world, and we get by okay. Mainly, this is because new generations don’t suddenly appear next to us and start using our computers without any foreknowledge of the metaphorical items. They spend time learning with the old fogeys for whom information was sometimes stored inside square pieces of plastic and metal. This overlap is necessary in general because knowledge about tools is not encoded in our genetic material. All understanding of use is part of a multi-layered strata of culture, experience, and relationships.

The real problem anti-floppy-disk people have is explaining a harm, specifically, a harm that matters. Often when we think of user interface improvements that matter, we think of examples like improving medical devices to reduce user error, changes that literally save people’s lives. Or tweaks to software that improve user efficiency and productivity, saving money. It’s hard to conceive of what might be improved by finding a better metaphor for “save” other than some designers’ personal sensibilities.

More Redis internals: Tracing a GET & SET

2011-03-10T19:48:00Z

In my previous article, I took a superficial look at how Redis starts up and prepares itself to process commands. In this article, I'll follow a GET and a SET command as they move from client through the server and back. The GET will be for a key that doesn't exist, and the SET will set that key. Then I'll look quickly at a subsequent GET and how it differs.

As before, I'm exploring Redis with the source code open in my editor and indexed with a tags file, and the Redis server running under gdb in another terminal.

Caveat: this article was written against the codebase of Redis 2.2.1. With respect to my previous article, for a list of what has changed in Redis since I wrote it, see this comment on HN.

Edit: I made two minor changes based on feedback—Redis keys are not Redis objects, they are sds strings; and you don’t have to hack the Makefile to compile without optimizations.

GET

Let's execute the command get users:1234 on Redis.

Preparing

If you inspect certain variables under gdb, you might not get what you want. Instead you see a message like "." This is because the compiler has optimized the machine code in such a fashion that the portion of memory you wanted to look at that was never used, at least for the variable under inspection. To make stepping through the code and inspecting a little easier, we make sure that Redis is not compiled with optimizations. You can do this by either building Redis with the following invocation:

make noopt

or by setting an environment variable:

OPTIMIZATION= make

Sending the command from the client

If we look at the repl loop of redis-cli, we see that it uses the linenoise library to split the arguments of the command. It dispatches on the first argument. The loop checks for client commands that aren't processed as a command by the Redis server, like exit/quit, clear (to clear the screen), and connect (which is a way to connect to a specified Redis server on host port, instead of the default host of 127.0.0.1 and port of 6379.

Any other argument is considered the name of a command to send to the server. The remaining arguments are the arugments for that command.

We're trying to get the on-the-wire representation of the get users:1234 command. redis-cli uses a redisContext struct to encapsulate the state of the connection to the server, as well as the output buffer where the command in the form of the Redis protocol is written for sending to the server. We know from reading the source of hiredis (the Redis C client library that the redis-cli program is built on) that the obuf field is where the raw command is stored:

# hiredis.h:107
/* Context for a connection to Redis */
typedef struct redisContext {
    int fd;
    int flags;
    char *obuf; /* Write buffer */
    int err; /* Error flags, 0 when there is no error */
    char *errstr; /* String representation of error when applicable */

    /* Function set for reply buildup and reply reader */
    redisReplyObjectFunctions *fn;
    void *reader;
} redisContext;

If we set a breakpoint on cliReadReply, we can inspect the output buffer and see exactly how the command looks as a bytestring bound for the server.

client $ gdb src/redis-cli
(gdb) b cliReadReply
(gdb) run
Starting program: /home/paul/src/redis-2.2.0-RC2/src/redis-cli 
Reading symbols for shared libraries +. done
redis> get users:1234

Breakpoint 1, cliReadReply (output_raw_strings=0) at redis-cli.c:421
421         if (redisGetReply(context,&_reply) != REDIS_OK) {
(gdb) p context->obuf 
$1 = 0x100102428 "*2\r\n$3\r\nget\r\n$10\r\nusers:1234\r\n"

We see that the Redis command get users:1234 as entered in our client is translated to *2\r\n$3\r\nget\r\n$10\r\nusers:1234\r\n. Any Redis client is going to convert our command expressed in its respective syntax to the same on-the-wire format. So in Python:

>>> redis_client.get('users:1234')

will send the same *2\r\n$3\r\nget\r\n$10\r\nusers:1234\r\n bytestring to the server.

Let's print that bytestring to screen and render those `rn's as line feeds so we can see an expanded view and get a better look at the protocol.

*2
$3
get
$10
users:1234

The first bit is *2, which tells us that the arity of the command, including the command name, is 2. That is, the server should expect two arguments to follow.

The next bit is $3, which means that the length of the first argument in bytes is 3. The argument itself follows, our command name, get.

The next bit after that is $10, so the length in bytes of the second argument is 10. Our one and only argument to the command is next, users:1234, the key we are trying to get.

Receiving the command on the server

If you remember from the last article, readQueryFromClient is a good place to set a breakpoint in your debugger on the server side for inspecting an inbound client command.

server $ gdb src/redis-server
(gdb) b readQueryFromClient
Breakpoint 1 at 0x100011445: file networking.c, line 882.
(gdb) run redis.conf
Starting program: /home/paul/src/redis-2.2.0-RC2/src/redis-server redis.conf
Reading symbols for shared libraries +. done
[63700] 01 Mar 11:04:40 * Server started, Redis version 2.2.1
[63700] 01 Mar 11:04:40 * The server is now ready to accept connections on port 6379

Now back in the terminal with the client running in the debugger, continue to send the command to the server, which will stop at the breakpoint we set.

# client
(gdb) c
Continuing.

# server
Breakpoint 1, readQueryFromClient (el=0x100200000, fd=5, privdata=0x100804e00, mask=1) at networking.c:801
801     void readQueryFromClient(aeEventLoop *el, int fd, void *privdata, int mask) {

Let's step to the following line:

# src/networking.c:808
nread = read(fd, buf, REDIS_IOBUF_LEN);

If we step to here in our debugger, we see that the server read 30 bytes. If you count the number of bytes in our Redis protocol-encoded command, *2\r\n$3\r\nget\r\n$10\r\nusers:1234\r\n, you'll see it's 30. Just for good measure, let's look at the 30 bytes beginning at the memory location pointed to by buf:

(gdb) p nread
30
(gdb) x/30cb buf
0x7fff5fbfeaf0: 42 '*'  50 '2'  13 '\r' 10 '\n' 36 '$'  51 '3'  13 '\r' 10 '\n'
0x7fff5fbfeaf8: 71 'G'  69 'E'  84 'T'  13 '\r' 10 '\n' 36 '$'  49 '1'  48 '0'
0x7fff5fbfeb00: 13 '\r' 10 '\n' 117 'u' 115 's' 101 'e' 114 'r' 115 's' 58 ':'
0x7fff5fbfeb08: 49 '1'  50 '2'  51 '3'  52 '4'  13 '\r' 10 '\n'

And we can see our whole command bytestring is there, byte by byte.

The server has now read in the entirety of our command in one step. (Because we had a relatively short command, one that fits inside a kernel buffer, and we are the only client on a loopback network device, this is the case, but it need not be. Since Redis is event-driven, this function, readQueryFromClient, is called whenever there are bytes from buffers to be read. If our command was particularly long, or there was a lot of network contention, the command may take more than one I/O event before it is fully read. For this reason, Redis builds up a buffer per client and appends bytes to it on each call to this function. It only proceeds with processing the command when it has been fully read. But we don't need to consider this in our simple example, so we will proceed.)

We're going to elide the processing of the input buffer. This is the point where the server takes the Redis protocol-encoded bytestring of our request and unpacks it into arguments on the client struct object. If you are interested in the details of that parsing, examine the function processMultibulkBuffer in networking.c. All we are interested in at this point is that the argc member of the client object is the number of command arguments (counting the command name itself) and argv is a pointer to the list of arguments.

The bit of code we care at this point is processCommand. The first thing the server does is look up the command in its command table (see "Setting up command table" in the previous article, but note that this lookup is now O(1), see the HN thread linked above). Assuming the command is found (which our get will be), the server will double-check that the arity of the command as defined in the command table matches the number of arguments received from the client (c->argc).

# redis.c:998
cmd = lookupCommand(c->argv[0]->ptr);
if (!cmd) {
    addReplyErrorFormat(c,"unknown command '%s'",
        (char*)c->argv[0]->ptr);
    return REDIS_OK;
} else if ((cmd->arity > 0 && cmd->arity != c->argc) ||
           (c->argc < -cmd->arity)) {
    addReplyErrorFormat(c,"wrong number of arguments for '%s' command",
        cmd->name);
    return REDIS_OK;
}

Skip down to the end of processCommand. Because our humble get is not a "multi" command like mget, mset, etc., it doesn't require queue-like processing of the underlying multiple commands, so we go right to call, which is where our command is dispatched.

# redis.c:953
void call(redisClient *c, struct redisCommand *cmd) {
    long long dirty;

    dirty = server.dirty;
    cmd->proc(c);
    dirty = server.dirty-dirty;

    if (server.appendonly && dirty)
        feedAppendOnlyFile(cmd,c->db->id,c->argv,c->argc);
    if ((dirty || cmd->flags & REDIS_CMD_FORCE_REPLICATION) &&
        listLength(server.slaves))
        replicationFeedSlaves(server.slaves,c->db->id,c->argv,c->argc);
    if (listLength(server.monitors))
        replicationFeedMonitors(server.monitors,c->db->id,c->argv,c->argc);
    server.stat_numcommands++;
}

Let's focus on line 952, cmd->proc(c);. This is Redis's dynamic dispatching of command function calling. Redis makes this clean and simple by encapsulating commands and giving all the actual underlying command functions the same function signature, taking our client object, which carries the payload of our command's arguments. So we're interested in looking into the details of the Redis command object and the actual function that will handle our get.

# redis.h:504
struct redisCommand {
    char *name;
    redisCommandProc *proc;
    int arity;
    int flags;
    /* Use a function to determine which keys need to be loaded
     * in the background prior to executing this command. Takes precedence
     * over vm_firstkey and others, ignored when NULL */
    redisVmPreloadProc *vm_preload_proc;
    /* What keys should be loaded in background when calling this command? */
    int vm_firstkey; /* The first argument that's a key (0 = no keys) */
    int vm_lastkey;  /* THe last argument that's a key */
    int vm_keystep;  /* The step between first and last key */
};

If we pop up to the top of redis.c, we see the definition of the Redis command table, and our get is the first entry.

# redis.c:71
struct redisCommand readonlyCommandTable[] = {
    {"get",getCommand,2,0,NULL,1,1,1},

getCommand is the function that does the actual work for our command. It's a thin wrapper for getGenericCommand.

# t_string.c:62
int getGenericCommand(redisClient *c) {
    robj *o;

    if ((o = lookupKeyReadOrReply(c,c->argv[1],shared.nullbulk)) == NULL)
        return REDIS_OK;

    if (o->type != REDIS_STRING) {
        addReply(c,shared.wrongtypeerr);
        return REDIS_ERR;
    } else {
        addReplyBulk(c,o);
        return REDIS_OK;
    }
}

void getCommand(redisClient *c) {
    getGenericCommand(c);
}

The arguments to lookupKeyReadOrReply are the client object, the key users:1234 we're trying to look up, and an object, shared.nullbulk that will be the default reply to the client if the key is not found.

# db.c:58
robj *lookupKeyReadOrReply(redisClient *c, robj *key, robj *reply) {
    robj *o = lookupKeyRead(c->db, key);
    if (!o) addReply(c,reply);
    return o;
}

lookupKeyRead is a thin wrapper for lookupKey that handles removing keys that have been set to expire.

Now we get to the heart of the get command -- looking up the key in the database.

# db.c:9
robj *lookupKey(redisDb *db, robj *key) {
    dictEntry *de = dictFind(db->dict,key->ptr);
    if (de) {
        robj *val = dictGetEntryVal(de);

        /* Update the access time for the aging algorithm.
         * Don't do it if we have a saving child, as this will trigger
         * a copy on write madness. */
        if (server.bgsavechildpid == -1 && server.bgrewritechildpid == -1)
            val->lru = server.lruclock;

        if (server.vm_enabled) {
            if (val->storage == REDIS_VM_MEMORY ||
                val->storage == REDIS_VM_SWAPPING)
            {
                /* If we were swapping the object out, cancel the operation */
                if (val->storage == REDIS_VM_SWAPPING)
                    vmCancelThreadedIOJob(val);
            } else {
                int notify = (val->storage == REDIS_VM_LOADING);

                /* Our value was swapped on disk. Bring it at home. */
                redisAssert(val->type == REDIS_VMPOINTER);
                val = vmLoadObject(val);
                dictGetEntryVal(de) = val;

                /* Clients blocked by the VM subsystem may be waiting for
                 * this key... */
                if (notify) handleClientsBlockedOnSwappedKey(db,key);
            }
        }
        server.stat_keyspace_hits++;
        return val;
    } else {
        server.stat_keyspace_misses++;
        return NULL;
    }
}

Redis uses its own hash table implementation to store keys and their values in memory. Inside the db object, the field dict is a pointer to the hash value for the current Redis database (remember there can be up to 16 separate databases in a single Redis server instance, indexed by number).

First, Redis calls dictFind with the database's hash table and a pointer to the key's bytestring. dictFind looks up the hash of the key in the table, using a standard algorithm that should be familiar to anyone who's implemented a hash table (check out dict.c starting at line 391 if you're interested, the table is an array with linked lists for colliding hashes).

If the key is found in the table, dictFind returns a pointer to the entry in the table. Otherwise, it returns NULL. Back in lookupKey, if the entry is not null, Redis retrieves the value (i.e., the Redis object our key references) from the hash table via dictGetEntryVal and takes care of a bit of bookkeeping for expiry and VM, if the key was found, and stats in either case (hits and misses). If the entry was NULL, then lookupKey also returns NULL; we'll see how this is handled by Redis for a reply to the client when the key is not found, which is the case for us at this stage.

With the value of lookupKey, we'll go back up the stack to our callers. Back to lookupKeyReadOrReply, we look at line 60:

# t_string.c:60
        if (!o) addReply(c,reply);

Since we got NULL from lookupKey this time, we call addReply. The value of reply here comes from the call in getGenericCommand, and it is shared.nullbulk. This field in the global struct object shared is initialize thusly:

# redis.c:712
shared.nullbulk = createObject(REDIS_STRING,sdsnew("$-1\r\n"));

We can see that it is a Redis string object who's on-the-wire value is $-1\r\n, meaning a length of -1, Redis's way of indicating null to a client, according to the protocol.

addReply builds the reply to the client. It does this by first setting up a write event on the main event loop listener with _installWriteEvent. This makes sure that the reply is written out to the client connection when there are bytes present in the buffer. Next, Redis adds the reply to the client's buffer. If the reply object were an non-string value like an integer, or a list, or a set, Redis would first decode it to a bytestring that can be serialized to, for example, on a socket. Redis string objects are encoded "raw," or as-is. The nullbulk object is technically a string object, so no decoding is necessary in our case. In any case, the reply bytestring is copied to the client's reply buffer with _addReplyToBuffer, which for all intents and purposes completes the execution of our get command on the server.

The client will read the on-the-wire reply of $-1\r\n and know that it is a string reply of length -1, and therefore is the null (or "nil," in the context of redis-cli) object, and to convert that into the appropriate object for the language of the client. Back to our redis-cli client patiently waiting for a reply from our breakpointed server, which we continue from, that looks like:

(nil)
redis>

SET

The set command proceeds much the same way as the get, up to the point of command dispatching on the server.

# client
redis> set users:1234 "Paul Smith"

# server
Breakpoint 1, readQueryFromClient (el=0x100400000, fd=6, privdata=0x100805e00, mask=1) at networking.c:801
801     void readQueryFromClient(aeEventLoop *el, int fd, void *privdata, int mask) {
(gdb) n
802         redisClient *c = (redisClient*) privdata;
(gdb) n
808         nread = read(fd, buf, REDIS_IOBUF_LEN);
(gdb) n
809         if (nread == -1) {
(gdb) print nread
$1 = 47
(gdb) x/47cb buf
0x7fff5fbfeba0: 42 '*'  51 '3'  13 '\r' 10 '\n' 36 '$'  51 '3'  13 '\r' 10 '\n'
0x7fff5fbfeba8: 115 's' 101 'e' 116 't' 13 '\r' 10 '\n' 36 '$'  49 '1'  48 '0'
0x7fff5fbfebb0: 13 '\r' 10 '\n' 117 'u' 115 's' 101 'e' 114 'r' 115 's' 58 ':'
0x7fff5fbfebb8: 49 '1'  50 '2'  51 '3'  52 '4'  13 '\r' 10 '\n' 36 '$'  49 '1'
0x7fff5fbfebc0: 48 '0'  13 '\r' 10 '\n' 80 'P'  97 'a'  117 'u' 108 'l' 32 ' '
0x7fff5fbfebc8: 83 'S'  109 'm' 105 'i' 116 't' 104 'h' 13 '\r' 10 '\n'
(gdb) print (char *)buf
$2 = 0x7fff5fbfeba0 "*3\r\n$3\r\nset\r\n$10\r\nusers:1234\r\n$10\r\nPaul Smith\r\n"

This time, our protocol-encoded bytestring is 47 bytes long, owing to the extra argument "Paul Smith" and the length tag it requires. Also notice the leading *3 indicates there are three arguments: set, users:1234, Paul Smith.

Let's skip ahead now to the point in call in redis.c, after the command has been looked-up in the command table and the server is about ready to call the underlying proc function with the client object argument. The set command, in the form of a redisCommand struct, looks like this:

# redis.c:73
{"set",setCommand,3,REDIS_CMD_DENYOOM,NULL,0,0,0},

Notice that the arity of the command is 3, which includes the leading command name, plus key and value, and matches what we expect from the client. The set command has a flag that get did not: the constant REDIS_CMD_DENYOOM means that, in out-of-memory situations where Redis can't allocate any more memory, the execution of the command should be denied. (The absence of this flag means that Redis can continue to serve client "read" requests like get even when the server can no longer write any new data.)

I set a breakpoint on setCommand and let the server continue running until that point:

# server
(gdb) b setCommand
Breakpoint 2 at 0x10001a6e2: file t_string.c, line 48.
(gdb) c
Continuing.
Breakpoint 2, setCommand (c=0x100805e00) at t_string.c:48
48          c->argv[2] = tryObjectEncoding(c->argv[2]);

Incidentally, you can inspect the values of the client's command arguments at any time, with a simple gdb invocation. The arguments are of type robj, which has a field ptr that is a pointer to the actual value in memory. Since in our set case these are strings, we can inspect them by typecasting to char * like so:

(gdb) p (char *)c->argv[0]->ptr
$10 = 0x10032ae78 "set"
(gdb) p (char *)c->argv[1]->ptr
$11 = 0x10032b068 "users:1234"
(gdb) p (char *)c->argv[2]->ptr
$12 = 0x10032b098 "Paul Smith"

The first thing the server does in setCommand is encode the value being set with tryObjectEncoding. It will try to create an efficient encoding if the bytestring can be interpreted as an integer, for example. This can save space especially in the case where many numbers are being stored. Additionally, Redis will try to reuse shared integers as values instead of allocating resources for new ones -- see the previous article for more on the creation and use of shared integers.

Once the value being set has been encoded, setGenericCommand is called (set shares setGenericCommand with the setnx and setex commands). From here, dbAdd is called, with the client, key, and value as arguments. dbAdd will only add the value to the database's hash table if the key does not already exist. In our case, since the key users:1234 does not exist, the value of dictFind is null, and the function proceeds to add the value with dictAdd.

dictAdd takes the dictionary hash table, key, and value as arguments. It uses _dictKeyIndex to find the index of a free slot in the hash table for our new entry. See the implementation in dict.c and dict.h for the details of key hashing, and the structure of the dictionary and its component hash tables (each Redis dictionary contains two hash tables in order to provide incremental rehashing as the dictionary grows). dictAdd allocates memory for the new entry and stores it in the new index.

The server returns back up the stack from dictAdd and dbAdd to setGenericCommand, where it increments the reference count on our new value. Redis uses reference counting in order to free memory used by values that have been deleted or have expired. It then "touches" the key so that if any clients are watching the key, the next exec command will fail. It also increments the server's dirty flag, which it uses to determine when to write out the dump file to disk. Finally, it writes out the reply to the client, which is a shared object, shared.ok. This is special Redis string object in the protocol that consists of the bytestring "+OKrn". Clients will typically convert this into the equivalent "true" value for their language.

GET redux

Our key is now set, so we can try the get users:1234 command again and see how it differs for a found key.

# db.c:9
robj *lookupKey(redisDb *db, robj *key) {
    dictEntry *de = dictFind(db->dict,key->ptr);
    if (de) {
        robj *val = dictGetEntryVal(de);

        // ... skipping the lru & vm parts ...

        server.stat_keyspace_hits++;
        return val;

The point where a get on an existing key and a get on a non-existent key differ is line 11, where the de entry in the database's dictionary is found. dictGetEntryVal is a simple macro for accessing the field in the de struct that carries the value associated with the key. Redis updates its statistics to indicate a key hit and returns the value object.

Again, as with the key miss from above (remember the null value is a Redis object, too), the value is decoded into the Redis bytestring protocol. This is the response to the client, and we have concluded our GET/SET/GET dance.

The All-Blue-Collar League

2011-01-24T00:22:00Z

In honor of the all-union Super Bowl, I present the All-Blue-Collar League, a collection of U.S. professional sports franchises named after local unions, industry.

In solidarity:

NFL
Green Bay Packers
Pittsburgh Steelers
Dallas Cowboys
MLB
Houston Astros
Milwaukee Brewers
Seattle Mariners
Texas Rangers
NBA
Detroit Pistons
NHL
Edmonton Oilers

Thanks to @kostuch, @ikesmith, @joshandrews, @tcarmody, @lmsahistory, and @janieporche for suggestions.

Gold is the same as paper

2011-01-14T19:38:00Z

We won’t return to a gold standard because changing to a fiat currency was one of the great innovations of modern economics. You aren’t literally and figuratively weighed down by gold, and that allows central banks and governments to set policy that allows them to rebound from crises, spur growth, and lift people out of poverty and short lifespans. As Liaquat Ahamed points out, the gold standard is also a sure way to get yourself in a deflationary cycle.

But, wait! gold bugs will say. That paper isn’t worth the paper it’s printed on. Gold has real value, that’s why it was the world’s currency for many years. The problem with this is that money is a turtles-all-the-way-down concept. Gold is just as much a pointer to value as paper is, except it has many more downsides: it’s heavier, more cumbersome to exchange, and has a fixed supply.

Gold isn’t valuable in and of itself. You can’t live in gold. You can’t eat gold. You can’t clothe yourself in gold. A gold coin can’t paint your house, or build you a website. You could (hypothetically) use it to put your kids through college, so it that sense it’s valuable to you, but only in its exchange, because on the other end, the trustees and professors and janitors at the college need to live somewhere, eat something, wear something, and have websites built. And again they exchange it for those goods and services. So really the only thing that’s intrinsic to this little economy I’ve just described is the fact that you, the college trustees, the service providers, et al., all agree on the value or rate of exchange. It’s like an algebraic equation where constants cancel out — gold is the constant, so we can abstract it away. It may as well be “neighborhood fun bucks,” so long as everyone agrees on the rate of exchange for goods and services.

And there you have it. Economies have nothing to do with the particulars of their currencies, and everything to do with people and institutions trusting that, in exchange for some amount, I will receive something of perceived equal value. Trust, and perception. Currencies are just a convenient means by which we mitigate that exchange, an abstraction, a stand-in for a hypothetical exchange, ultimately, an exchange built on trust.

Interestingly, the only inherent value gold has is in its use as jewelry. Which, aside from any aesthetic concerns, couldn’t be more appropriate for our discussion, because people wear gold jewelry as a signifier of wealth. A pointer, a marker, an abstraction for the wealth itself, which, again, is just the potential for exchange.

It is this essential trust network at the core of economics that explains why there is so much noise about gold. It reflects a lack of trust on the part of gold advocates in our central banking and governmental institutions, which they feel pursue policies that aren’t in their interest, or are in the interest of undeserving populations. The switch to a gold standard, therefore, has nothing to do with inflation or any other “economic” concern, and everything to do with pursuing a political objective. The market of exchange in this little political economy is for ideas about reprioritizing our institutions. Which is a legitimate topic of national debate. But reverting to the gold standard, as a practical economic matter, would be bonkers.

Taking Go for a spin

2011-01-12T12:00:00Z

I was struck by the high-level abstractions Go provides for writing networked clients and servers, even by the standards of conventionally high-level languages like Python. Given Go’s identification as a systems programming language, this is initially surprising, because such languages tend to be low-level for performance reasons (Go does expose a lot of low-level interfaces through various packages like os and syscall), but of course there would be good support for these things because Go is a modern language designed by folks with many years of experience writing networked servers.

Specifically, the net package provides the Listener and Conn types that hide many of the details of setting up socket connections. A simple net.Listen("tcp", ":1234") is enough to get the equivalent of a listener, or server, socket. The bufio package provides buffered read methods, simplifying common tasks like reading lines from a socket.

Let’s dive in and take a look at one way to implement a classic echo server. I tried to use Go’s language features for concurrent communication, goroutines and channels, at crucial points. Where a different implementation in different language might fork a new process or spawn a new thread to handle a new client connection, a goroutine was started and the details of the connection communicated with a channel.

package main

import (
    "net"
    "bufio"
    "strconv"
    "fmt"
)

const PORT = 3540

All Go programs must have a package main. We declare a number of packages to be imported, net and bufio already mentioned, strconv for its conversion function Itoa() that converts an integer to a string, and fmt for printing strings to the console. Our echo server will listen on port 3540.

func main() {
    server, err := net.Listen("tcp", ":" + strconv.Itoa(PORT))
    if server == nil {
        panic("couldn't start listening: " + err.String())
    }
    conns := clientConns(server)
    for {
        go handleConn(<-conns)
    }
}

Every Go program must have a function main() in its package main. We start by declaring and initializing a new listener for our echo server. We choose the more generic net.Listen() over net.TCPListen(), because we can conveniently parameterize the type of the listener with a string to the first argument of Listen. (It is generally preferred in Go to deal with the most generic type, or interface, especially when writing function and method signatures, and either allow a specific type to be inferred by the compiler or specify it when declaring/initializing a variable.) ":" + strconv.Itoa(PORT) is a string concentation expression, and the port constant we defined earlier is converted to a string. (Strictly, a numeric constant is not an integer of a particular type, but the string conversion function works here because the compiler converts the constant into the concrete type to match the function signature.)

The idiom for Go when handling multiple return values where the last has an error type is to check the main object for nil, and then panic() with the stringified error object. This is generally preferred over printing to stderr and using os.Exit(), because it gives callers higher up the stack the chance to recover(), sort of a raise/catch exception flow.

Using Go’s compact syntax for simultaneous declaration and initialization, we set conns to the value of the function call clientConns(server). This is the channel we’ll use for getting new client connections.

The equivalent of a infinite loop like while True: or for (;;) in Go is for { ... }. Each time through the loop, we start a goroutine, calling handleConn() with the value of the receive operation on our client connections channel. The unary receive operator <- blocks until a value is available on the channel, in our case, a new client having connected.

func clientConns(listener net.Listener) chan net.Conn {
    ch := make(chan net.Conn)
    i := 0
    go func() {
        for {
            client, err := listener.Accept()
            if client == nil {
                fmt.Printf("couldn't accept: " + err.String())
                continue
            }
            i++
            fmt.Printf("%d: %v <-> %v\n", i, client.LocalAddr(),
                client.RemoteAddr())
            ch <- client
        }
    }()
    return ch
}

We create a new channel of type net.Conn, which corresponds to the type that we’ll get from calling Accept() on our listener connection object. We start off a new, anonymous goroutine which runs in an infinite loop, constantly accepting new connections. listener.Accept() blocks as long as there are no new clients to deal with, but since we’re running inside a goroutine and "detached" from the main program flow, other already-connected clients can continue to be handled without interruption (this is where a fork() or a new thread would happen in a typical server). Instead of panic()ing here if there is an error connecting with the client, we simply note it on the console of the server and move on. We also keep track of the number of clients we’ve seen with the counter i. fmt.Printf() works like you’d expect, though the %v format is not like anything in C’s printf() — it prints a value in a default format and works for any type, somewhat like repr() in Python.

The binary <- communication operator is used to send the client, of type net.Conn to the channel we created at the top. Go has lexical scope (and is garbage-collected), so ch is available inside the anonymous goroutine and after our function returns (because the infinite Accept()ing for-loop keeps the goroutine alive).

func handleConn(client net.Conn) {
    b := bufio.NewReader(client)
    for {
        line, err := b.ReadBytes('\n')
        if err != nil { // EOF, or worse
            break
        }
        client.Write(line)
    }
}

As we saw back in main(), handleConn() is invoked as a goroutine with each new client connection (the connection being received from the other side of the channel we just created). bufio.NewReader() wraps the client object with a nicer interface for reading lines of bytes from. If we decided to use the Read() method of the net.Conn object, our code would be more complex, having to check for the substring "\n" and testing for EOF. Instead, we can treat this as a line-oriented protocol, and simply get a line and write it back to the client, as long as there are lines to be read.

Here’s the whole thing.

The pattern of getting new connections from a channel and starting a goroutine to handle them is conceptually clean, and also happens to be a straightforward way to write a multiplexing, concurrent server.

Use Oplop for your web site passwords

2010-12-18T05:52:52Z

The Gawker breach made mainstream what’s been known for a while: we don’t use good passwords, and we use the same password over and over again on multiple sites, exposing us to a potential chain reaction of account intrusions if it is uncovered.

One way to address this problem is to use a password generator—the idea is that all you have to remember is a single, good, strong password, and the password generator will spit out a random string of characters to be used as a password on a particular site. This generated password is hard for an attacker to guess, and hard for them to “crack” with password-attacking software. You’re not supposed to remember or even care what the actual password is: you just remember the master password. And if one of the sites you have an account with is compromised, it may suck for you with respect to your sensitive information on that site, but because the generated password is unique to the site, you’re safe from having it used against you on other sites where you have accounts.

The password generator I use is called Oplop. It’s just a web site, so you visit it in your browser; there’s nothing to install. There are two fields to enter—a nickname, and the master password. The nickname is any name you want to identify the site or service for which you need a password. After you enter a nickname and your master password, you click the “Create” button, and an unguessable password is generated for you in the field below the button. Enter that password into the site that’s asking you for a password, and you’re all done. The next time you visit the site and need to enter your password, you repeat the process exactly to get the same unique password that’s just for that site.

There are a few tips and tricks to using Oplop successfully. The nickname is just a label that you remember to refer to the site you need a password for, but if it is different by even one character, an entirely different password will be generated. So it’s important to be consistent, and ideally use a simple convention so that you enter the label the same way every time. I’ve chosen a convention of using all-lowercase and no non-alphanumeric characters, and the domain name of the site, minus the top-level domain (the .com or .org, etc.). For example, if I’m on amazon.com, I’ll use “amazon” (without the quotes) for the nickname. If I was visiting a site with the domain name “Cat-Pics-123.net”, I’d use “catpics123” for the nickname. Whatever your convention for nicknames, it should be simple and you should use it consistently.

Your master password should be a good one—fairly long and hard to guess—since it is the key to the kingdom. Security researchers often praise the idea of “passphrase,” a sentence instead of a single word. These can often be easier to remember and are just as strong if not stronger than shorter, gobbledygook passwords, even if the passphrase is composed of simple words. A four or five word sentence, separated with spaces, should suffice, although longer passphrases are even better. In any case, use care when entering your master password into Oplop. By default, it doesn’t do any checking, so, like in the case of the nickname, a single character’s difference means a whole different generated password. You can optionally click on the checkbox “Validate master password?”, which adds an additional password field for you to re-enter your master password, providing a simple check that might catch a typo in the first field.

The final tip is to cut-and-paste the generated password out of its field instead of copy-and-paste. It’s not so much a tip per se because Oplop recommends that you do this right on the page, but it’s important enough to reiterate. Doing it this way prevents someone from snooping over your shoulder, or if you are careless and leave the Oplop page open untentionally on a shared computer.

You might have stopped short at the notion of a web-based password generator. Isn’t it insecure to trust a web site to generate a password for you? The answer is, you are only actually interacting with your browser—there is no network activity taking place when you generate a password with Oplop. That is because the entire Oplop program is just HTML and JavaScript loaded in your browser, and it is your browser on your local computer that takes the nickname and master password, performs the cryptographic magic on them, and outputs the generated password. In fact, Oplop works offline just as well. You can test it by visiting the site and bookmarking it on your iPhone or Android, deleting the browser window, putting the phone in airplane mode, then loading Oplop from the bookmark. It works just the same as if you were connected to the internet.

The browser-centric nature of Oplop gets to another key concept about it—there’s nothing to install. In fact, it is essentially already installed on any internet-enabled device with a JavaScript-enabled browser. There’s no central database of account names and generated passwords, like you might expect, and which password generators like 1Password and Password Gorilla are built around. These programs work similarly to Oplop in the end result, but their central database presents a user experience problem—it needs to be shared among the computers you want to use the program on. This can make it difficult to retrieve a password if you are on your friend’s laptop or a PC at an internet café. Not only do you have to install the password generator software itself, if you are allowed to by policy of the computer’s owner or manager, you have to synchronize the central database with the computer. By simply eschewing the notion of storing account names and passwords, Oplop provides a universally-accessible password management tool. Those unique generated passwords are just ephemera, the results of combining the secret you keep in your head (your master password), the site nicknames, and a cryptographic function in your computer. Even the notion of the “master password” is turned on its head—it’s not unlocking some database of passwords, it’s combined with the nickname to make a cryptographic hash, the bits of which make up the final generated password. You could start using a different master password at any time, which essentially would invalidate any extant passwords generated with the old one (you still would have to manually go in and update each password on each site, though).

There are other approaches to securing your passwords, including writing them down. In fact, if you have trouble remembering strong passwords or passphrases of sufficient length, you might consider writing down your Oplop master password on a slip of paper you keep in your wallet.

Oplop is not perfect—it outputs a fixed style of generated passwords that may not meet the password criteria for various sites and services, which require certain lengths, or a certain mix of characters. It also doesn't address the need of strong, crypotgraphically-secure storage of ad-hoc information such as credit card numbers and other sensitive account information, which are often stored alongside passwords and account names in more traditional central database-style password generator programs. But this is by design and not a failing of Oplop. I continue to use Password Gorilla for this scenario, especially for protecting financial information. Also, there are certain threats Oplop can’t protect against, And it’s worth understanding what they are before you start using it.

Oplop addresses the proliferation of passwords across multiple web site accounts, is simple to use, and works everywhere there’s a browser. So stop using “letmein” for all your web site accounts—you’d be much better off plugging that in to Oplop as your master password. But don’t. Use a better one.

del.icio.us is dead, long live del.icio.us

2010-12-17T05:55:03Z

It slipped today that Yahoo! will soon to shut down Delicious, née del.icio.us, the social bookmarking site. When it became obvious a few years ago care and feeding of the site had more or less stopped, I fled the service, like many other users, for the clones.

Delicious was important for two reasons. It was all about discovery—what your friends and the other interesting people you were following were reading. Predating Digg and reddit, it was the first service to do what Twitter and Facebook now do so well—point you at a URL you will probably be interested in, by virtue of having been vetted by your social network.

It was also important because it popularized "folksonomies," the now-ubiqitous act of tagging items as a lightweight, and therefore actually used and useful, alternative to categorization. You can thank del.icio.us for tag clouds and high-quality Flickr searches, for example.

Delicious was internet plumbing, but it never became essential internet plumbing. You could automatically update your blog from it with what you were reading (and so many people did that they stopped writing in long-form and just published from their Delicious accounts a link and some commentary, it resembled an early Twitter-esque microblog in that way). Lots of people offloaded their brains to it (especially reference material and how-to guides and programming tips, learning stuff they intended to come back to later). But unlike its big Yahoo! sister Flickr, which has become an essential pipeline for web images, Delicious never grew much beyond what it was in the early days, never caught fire as a social network in its own right despite millions of users. Whether this was due to neglect or mismanagement or cluelessness, in any case, it was a missed opportunity, to become essential.

People are wondering if Yahoo! understands what the shutdown means for their reputation, when it comes to safeguarding user data or being reliably available. It's a classic data liberation problem, and if Yahoo! doesn't make it easy for users to take their bookmarks out of Delicious (or their photos out of Flickr for that matter), folks might not trust a future Yahoo! product with their data.

Bookmarking lives on despite the fact that you can pretty much Google anything, replacing the offboard-brain reference index, and that social network-based discovery is now dominated by Twitter and Facebook—people still like being able to curate their own little slice of the web. pinboard.in is one post-Delicious site that looks particularly promising and vital; tellingly, it refers to itself as anti-social bookmarking.

Farewall, Delicious—for a few years at least, you were essential to me.