evan_tech

dns attack of doom

2008-07-22T15:57:31Z

If I've learned anything from the new Kaminsky DNS attack, it's that if you want to keep something a secret while disclosing to a trusted subset of vendors, you do not include publicity-hungry overeager bloggers in the list of people who can keep their mouths shut.

heavy protocols add up

2008-07-19T17:39:37Z

I found this discussion of Android's dropping XMPP interesting. (Disclosure: I have no insider knowledge about any of this.) In particular, this remark about compression in the context of the notoriously fat XMPP protocol:

Bandwidth used means radio transmissions sent, and overhead means more work done by the processor, both of which take battery power and reduce battery life. Meanwhile, compression turned out to not be very helpful. Since it's negotiated during connection startup, it doesn't help with startup overhead. It does help somewhat with steady-state bandwidth, but at the expense of additional CPU cycles. The result is that enabling compression actually reduced battery life in our tests -- it took more power for the CPU to do compression than we saved on radio power.

(I wonder though: perhaps they could've used a simpler form of compression? XML ought to be "easy" to compress. Maybe the spec doesn't allow it?)

You see a similar phenomenon with HTTP on a heavy page. Like CNN.com: Firebug says it took 135 HTTP requests to load the page. Many of them are to a CDN and only have ~400 bytes of request headers but all the ones to cnn.com (including the ads, apparently!) include cookies pushing them up to nearly a kilobyte. The net result is that the latency of the page starts getting affected by the end-user's upstream bandwidth, which is usually terrible. (But now, having typed that out, I wonder: does it really matter? Even those heavier requests fit within a packet anyway...)

dolt libtool wrapper

2008-06-10T23:05:51Z

dolt: double the build speed of libtool-dependent software. Examples of such software: kdelibs, gtk, libx11, libxml2, dbus.

So many CS problems are significantly improved by caching at the right place...

[hober put this on reddit. I had it first! :P]

utf-8 is hard

2008-05-25T21:02:02Z

Evite trying to do Mårtensson.

what you need to do in response to the openssl fiasco

2008-05-14T22:10:45Z

If you have used a Debian-based system to generate SSH keys in the past two years, your keys are likely no good. This document has instructions. In brief:

1) Delete your bad keys: .ssh/id_*. Fix all systems where you're trusting those keys (think .ssh/authorized_keys); someone has already published a table of all private keys, so it's just a matter of time before your system is brute-forced.

2) Update your systems. I see an "openssl-blacklist" package show up on both my Debian stable and my Ubuntu whateverletterthey'reon one. You'll get some debconf prompts about it clobbering stuff, including potentially your host keys, which means the next time you connect to the machine you'll get the "host keys have changed" message.

3) To make yourself feel less anxious, try running ssh-vulnkey to print an analysis of keys in standard paths on your system. (Run it as sudo ssh-vulnkey -a to check all users on your system.)

type-safe printf

2008-05-14T05:08:46Z

printf is a function with a complicated type. In C we used to just give up and tell the compiler "this function takes some other stuff that you shouldn't worry about" with the amusing "..." builtin. These days compilers have special support for annotating printf-like functions to provide type-checking. The other side of this is that an implementation of printf necessarily has a little tokenizer/parser for run-time processing of the format string, along with the associated performance penalty*.

Yet pretty much all programs that involve format strings ought to have the format strings known statically. Even a mini-language like printf turns out to have enough power to not be able to safely process untrusted input, as the "poke" instruction (named %n) demonstrated by creating a completely new class of security vulnerabilities. And without the compiler to help with type coercions, it's easy to write something invalid, especially when you're playing fast and loose with integer and pointer sizes across platforms.

Perl and Ruby neatly sidestep this problem by using string interpolation: at parse/compile time, the compiler scans the strings for bits to be expanded and just rewrites the "format" string to the equivalent concatenation of literal strings and variable values, which then uses the normal language's support for pasting strings together. (Prove it to yourself: perl -e 'use strict; print "$x";' aborts with a "compilation error".) But sometimes you really just want something like printf, and both those languages fall back on "figure it out at runtime" for that.

Supporting printf at all proves to be pretty difficult in more strict languages which generally require all types to be known. OCaml's compiler does some crazy hacks where sometimes a quoted string is interpreted as a format, a six-parameter type that, for example, needs its own concatenation operator. Haskell at least encodes it in the user-available language with some typeclass magic that gets you to more or less to feature parity with dynamic languages -- failure at runtime if the parameters don't properly line up.

But it turns out there's a nice paper that provides a type-safe encoding of printf that doesn't rely on any fancy language features. The paper is structured like this: (1) "wouldn't it be nice if printf worked like this?" (2) "oh wow check it out, here are the functions!" I've been staring at it for a week and though I can sorta see how it works, it's unclear to me how anyone would come up with it. Here's an overview from a person who lacks sufficient brain to say much smarter (me).

To start with, you don't use a string for the format string. This sorta seems like you've already given up, but you could imagine a macro expanding a format string into the proper expression here, much like how Perl/Ruby's interpolation works. Since this is functional code we're talking about, it ends up being an expression involving some functions. Format string concatenation ends up being encodable as composition, which means you end up with the same operator as Perl (.) for pasting them together.

The basic task then is that you need print (some magic here) to be able to give you a function of varying types, depending on what the magic is, so that print format1 3 "foo" can be type-checked that format expects an int and a string. So the type of printf must be (some magic type involving an a) -> a, where the polymorphic a is a function produced by the magic. And here's where the magic drops, painful in its simplicity:

lit :: String -> (String -> a) -> String -> a
lit text k s = k (s ++ text)

int :: (String -> a) -> String -> Int -> a
int k s val = k (s ++ show val)

printf :: ((String -> String) -> String -> a) -> a
printf fmt = fmt id ""

And that's it. lit is what converts a literal string into a format, while int is the placeholder for an int. So "my int is %d\n" would be expressed as lit "my int is " . int . lit "\n". If you drop this in to GHC you'll see that the type of printf (lit "my int is " . int . lit "\n") really is, as you'd expect, Int -> String -- it's waiting for you to give it an int so it can dump out the formatted string. The result of printf is just a plain function, so you do all the normal sorts of things you'd want, like partially apply it or pass it to map. The formatters are plain functions, too, so you can add your own formatter that, for example, can accept a list (as he does in the paper).

So how's it work?

Look at the two formatters, int and lit. The k parameter to the formatters is a continuation: it's where each formatter should pass the string constructed so far when the formatter is done. Then the s input parameter is the string as constructed so far. You can mentally expand printf (lit "foo" . int) as (lit "foo" (int (id))) "", where that empty string is your starting string and id is the innermost continuation which just gives you back the string that's been constructed.

You can also look at int like this, just with some added parens for clarity: (String -> a) -> (String -> (Int -> a)). It takes the continuation as a parameter, and the function it returns is sorta the same shape as the continuation but with Int -> a in place of a -- that's how it tacks on its need for an int to the greater formatting requirement.

But from there... uh, the types just work out. I don't know. It's pretty much magic.

[Edit: a commenter on reddit linked to this discussion of the general technique, which I haven't read yet but looks promising.]

* Though I'd argue you're in a pretty bad place if printf performance is your bottleneck.

2008-05-12T16:23:54Z

Dear Wordpress,

The links added to posts via your "related posts" feature are rarely (perhaps never?) actually "related" to the post you add the links from. This harmful in two ways: one, every time I click one of those links thinking the page author had more information for me to read (like Greg Linden's blog, which often has great related links), I find unuseful content and it frustrates me. Two, and maybe more worrying for you, you're training me to ignore those links; if you ever do improve the quality of the matching system later I'll never discover it.

concurrent editing

2008-05-10T22:29:43Z

While I'm on the subject of concurrent editing:

My reading group read Designing a commutative replicated data type a few months ago. The basic idea I've retained from the paper (is has been some months) is that one way to avoid conflicts is to design your data representation such that conflicts are impossible by making all operations commute, demonstrating the theory by presenting a design for a multiuser simultaneous editor. ("Like SubEthaEdit" is what I kept saying to people, but apparently few people I know have heard of it.) By representing positions within the buffer as adddresses related to characters you currently know about, and having a globally-defined resolution strategy for two edits to the same position, you can safely allow edits to come in from clients in any order and maintain consistent state.

(Commuting operaitons sounds like darcs, doesn't it? In fact, this fellow was discussing darcs's patch theory in connection to concurrent editing, though I suspect ultimately it's the wrong model...)

The paper's really pretty clever in a bunch of ways (like: how do you make a globally-consistent addressing scheme in the presence of simultaneous edits?) and a friend and I sat down to implement it as a web app with a bunch of Javascript. I had planned to target the release of App Engine but the lack of Comet (again, that bites me) sorta turned me off to the whole idea. And I have so many other projects I ought to be finishing first... He, however, powered on to get something that seems to mostly work but is a bit on the inefficient side.

PS If you're curious, the model Gobby uses (Gobby is another simultaneous editor) is described here. It's again the "hope we update often enough that there's no conflict" design I mentioned in the previous post.

more on bug tracking; distributed editing

2008-05-10T22:09:54Z

A few separate posts, all in the same area.

1) Most (all?) the distributed bug tracking software I've glanced at stores bugs in a directory, one file per bug. This seemed like poor design to me. I confirmed by showing Brad the output of ls on one; his full response was "doesn't scale" and turning back to what he was working on.

2) Having thought more about the relationship between code and bug state, I have concluded I was thinking about it the wrong way. Going back and reviewing your comments, I see a bunch of you figured this out before me. Here's the critical piece I was missing. Code has history, which is tracked by the graph of related versions. Bug state both refers to the code history and also has its own history, in that new bugs are opened and old ones are closed. Those two histories related to bugs are not the same: even when examining old code, you generally care about the newest bug state. (This is why most modern bug systems only let you use the newest bug state; making changes to it permanently clobbers the old state. However, note that most do care about showing you the history of modifications to a bug; the interesting view is the most recent copy of the bug's entire history.)

As Aristotle and Lee pointed out on my older post, connecting the code history graph and the bug state could be modeled as annotations pointing at commits. The state of bugs present in a given version is the collection of all bugs states that have been attached to an ancestor of that version. This means discovering a bug in a previous release "infects" (to use his term, which is a good one) all branches derived from that release, and a given branch is only fixed once it merges the code that fixes the bug. (Making that work efficiently is an exercise for the reader; I have some ideas that aren't worth sharing yet.)

3) Part of the reason I got thinking about all of this because I wanted a separate feature: a command-line interface to bug tracking. I hate using web apps both because web sites become inaccessible, get slow, or go down (a problem addressed by making it distributed) and just because I hate clicking around on web forms (a website can't, for example, query my current checkout for which branch I'm claiming the bug is fixed on). You could make a CLI-based interface to Trac -- maybe one exists already -- and it would at least address the second half of that.

At a superficial level, the command-line problem isn't really at all same thing as a distributed system. But there also is a connection at a deep level. I like to say (and here by "say" I mean "think" because nobody ever wants to listen to me jabber on about this stuff) that even centralized projects have distributed branches every time someone edits a file in their own checkout; it's just that the tools we have for those branches are weaker than the tools typically used for "real" branches. On most systems you typically can't record your changes until you've verified they would merge cleanly with the master branch (though monotone/mercurial/fossil fix this implicitly and git does if you're using the proper workflow); on most systems you can't examine what happened upstream in the same way you examine changes that happened locally.

This same problem -- that forks happen on every checkout -- is true with any web-based database; it's just that when you're using a website you tend to commit more frequently so the forks don't get a chance to conflict. And for that reason the tools for managing conflicts are usually pretty weak, as anyone who's encountered a conflict on a bug tracker has probably experienced (my impression is that most just say "click back and type in what you were saying again"). My favorite bad conflict-resolution system is probably Google Docs, which, upon a conflict, pops up a window saying something to the effect of, "This paragraph was edited while you were editing it. Here is your text; please copy and paste it back into the document and figure out how to fix it."

So back to the command line: you need to solve this conflict problem anyway for a distributed system to work, but you also probably need to at least improve it for a command-line-editable centralized system to work, because with a command-line app you don't really get a "back button". You could implement back-button-like functionality, but if you're going to implement new functionality to handle this case, perhaps you could implement a more sane model instead.

terminal emulation

2008-05-07T06:06:57Z

When you run script (see man 1 script if you're not familiar with it) it logs every keystroke you type, including backspacing over typos. Val asked today: how can you format that output so that it appears as the result? I found it interesting to think about. You'd imagine there'd be some way to run terminal emulation (?) and then somehow pipe that through cat. It's an interesting mixing of levels; I started running my mouth off about screen and vte but didn't come to a good conclusion.

Along those lines, it made me think of this interesting thread in which "cmd --color" produces escape codes while "cmd --color | cat" produces colored output. (Try to guess the circumstances before you read the other side of that link!)

your animation is too slow

2008-04-23T15:39:34Z

Animation in software has a real use: by showing intermediate states between two end states you let the user instinctually see how they're related. But as soon as your animation takes more than around 100ms it's gratuitous. Movement should be so fast I don't notice unless I'm looking at it, because otherwise it's happening so slow I'm waiting for it. Unfortunately, the difficulty of implementing animation tends to make programmers too proud of it (I personally have fallen for this) and you end up with these terrible 500ms fades so common in Flash or Javascript-heavy apps.

Here's a personal request from me to you: knock that shit off. It looks amateurish.

dvcs and offline

2008-04-22T16:41:07Z

I got a couple of comments on that previous post that betray a bit of a misunderstanding about how collaborative projects work in the presence of distributed version control.

Two possibilities: you're offline or online.

For circumstances when you really are offline (and again, I note that I really am without internet access for a surprising fraction of my hacking time) there is fundamentally no way a bug tracker that requires me to be online can work. For many, the answer to this problem is "wait until you're online" or "take notes in a text file", which is well and good if you're the sort to accept a sub-par situation as unsolvable. (I repeat from before: any time you're resorting to manually shuffling source code, your source control system is failing you*. You could make the same claim about any other tool.) So yes, bug tracking is importantly a communication tool, but if the situation is that I'm physically unable to communicate you can't cite non-communication as a negative of any proposed alternative.

Otherwise, even when you have internet access, no technology solves the communication problem if people aren't willing to communicate. "Private repositories" are a problem, but they're a problem no different from contributors who won't email or contributors who sit on a patch.

When I'm online collaborating with someone else, the workflows between an "offline" and "online" system are equivalent except for running the sync step after some operations. So I thought anon-commenter Daniel's point that email also has this offline-online hybrid behavior (where Outlook has the huge "Send / Receive" button on the toolbar) was right on -- I'd much rather use an offline email client that could sync than one that forced me to be online to use it**. And as he points out, email would be terrible for real-time communication if people only synced once every few days -- that's why you don't do that, with email or dvcs, if you intend to participate in a project. (Though in circumstances where I want to avoid email or hack on some code that doesn't yet compile, that workflow remains available.)

* By this metric the tools I use continue to fail me occasionally; they just fail less than the tools I used before.
** I remain hopeful such a technology will one day be invented. I gave up on mutt when I realized it can't scale beyond a few thousand messages and didn't provide any useful search. Maybe sup will get there.

distributed bug tracking

2008-04-21T00:39:41Z

Distributed bug tracking is the natural extension of distributed version control. Aside from the normal benefits of distributed version control, like being able to interact with bugs database while offline, there also seems to be a trend of making the interface to them work via the command line instead of annoying web-based systems. And, like with Trac, the integration of issue tracking with the source is pretty natural: when you've fixed a bug on a branch you can mark the bug as fixed in that branch, and when that branch lands on your "main" tree that tree's bug state is also merged as a natural consequence of how merges work.

It seems like there isn't dominant software for this yet. Here's my five-minute take on the software I can find:

Fossil is its own full version control system that integrates a bug tracker as well
bugs everywhere -- perhaps abandonware, last commit was July 2007
DITrack -- subversion only, "planning to be backend agnostic" (not sure how svn matches up with distributed, but ok)
DisTract -- only a web interface using Firefox-specific Javascript to write to disk(?), requires monotone, latest news August 2007
TicGit -- just learned about it five minutes ago so not sure yet; seems a bit janky to keep bugs in a separate branch
Ditz -- seems the most appealing to me except that it's all of three weeks old, has emacs integration*, last commit last week

For whatever reason these all seem to involve the most obscure technologies available: in the above list I see fossil, bazaar, monotone, and even Haskell.

From reading through these I find a surprising variety of models. Here's what seems to me to be the simplest and sanest model: the bugs live in a normal top-level directory in your tree alongside "src" or whatever other directories you have; each issue is in its own file; comments are modifications of the per-issue file.

But more generally, I'm not even sure if distributed is the appropriate model. The action of recording a new bug modifies the current version of a branch but the bug's existence usually is older than the most recent commit (for example, it often belongs in older branches that have branches off before the bug was added). So if a new bug is fixed in an older branch, there's no way to merge that new bug into the older branch without merging the entire state of the newer branch in. Is that sensible? I'm not sure. The alternatives all seem to involve tracking bugs separately from branches and trying to match them up after you commit (like when commit messages mention bug numbers) which always feels like a failure of technology.

The other issue that's I'm unsure about is how to integrate a sane web-based frontend for casual users who want to be able to query and report bugs without checking out the code. Some systems have web frontends but it seems to me conflicts could be hard to resolve. Maybe if you make sure a modification to an issue is always appends, and then add some smarts that auto-merges simultaneous adds by some textual timestamp included in the add.

Needs more thought. Sorry for the braindump.

* I'm a vim user, and don't really care about editor wars, so I mention it only to note that emacs integration isn't as useful for me.

google app engine limitations

2008-04-13T19:55:19Z

It's weird to see people talk about Google App Engine online because I think many people focus on minor details. Like, to make apps scale horizontally you do need a "shared-nothing" infrastructure, so that's not really novel. The BigTable aspects are sorta interesting except there's nothing there (in terms of application design) that you couldn't have gotten out of the paper, and the App Engine API is so high-level it's not that close to BigTable. It's more like any other high-level flat-address object database, like maybe CouchDB. The Python thing is also pretty irrelevant; they just picked a language they have experience with (having Guido around helped), and it's easier to launch supporting one API than n languages' worth. A good engineer just solves the problem with the tools available, and Python is a pretty good tool to start with.

As for evil plans to steal ideas or code, that's between you and your skepticism. Big companies are surprisingly good at doing shitty things, and Google is definitely big, but it's also true that within Google people really try to do the right thing. I was touched to see a privacy-concerned friend of mine start using Gmail after he was hired, saying that only after he saw how seriously they take privacy inside the company could he feel confident about using it. But I can't tell you anything that will change your mind about this subject.

I developed an internal application using Google App Engine on and off over a period of months (during its development I kept trying it out) and then finally rewriting a few weeks before launch (after the APIs had all settled).

Here are some real problems I've encountered:

1) All code runs only in response to HTTP fetches. So that means no cron jobs, and no persistent server-side processes. I know I just wrote above that you can't really have persistent jobs if you want to scale, but ultimately real apps do occasionally need these. For example, imagine a timed test app that needs a consistent view of time no matter which server (or datacenter!) the user hits. A time server becomes a single point of failure but when it's critical for your app it can be engineered around.

2) No long connections means no "comet" (server-push messaging).
My first thought on hearing about App Engine was to port lmnopuz but I can't.

3) Playing around with your data is hard. Since there's no way to perform operations on your data except by uploading code to the server, you're often left creating a new URL per operation you want to perform. Hacks like the shell helps with this, but a lot of the time I want to be able to just run a local script and see the output. (For my project I found a decent workaround: make a URL that accepts Python code as a POST and runs it. Then your scripts just need to know to serialize themselves into strings and send them over the wire.) But see the next point.

4) Slow table scans. My app had ~1200 rows that it performs various analyses on and produces graphs. I can appreciate that such a query is labor-intensive, and so I had written it to cache the results of the graph generation (the rows only change once a day). But I can't even seed the cache once because fetching 1200 rows is too slow to happen within a single query.

5) Bulk operations are hard. Say you want to delete all objects in a table (or class, I forget the App Engine term). The "delete" operation requires you fetch the object first, and then you're back into slow table scans land. The best you can do is batch up your processing into multiple smaller stages, each of which write their intermediate output into the data store: either make a page that auto-refreshes itself with Javascript and leave a browser pointed at it, or make a command-line script that repeatedly hits a URL on your app.

6) No arbitrary queries. (If you haven't read the docs in detail, you wouldn't know this, but any query that involves multiple attributes [columns, if you're still thinking SQL] of an object must have an index exactly matching the query. They make index creation and maintenance trivial, and even automatic in most cases.)
Though everyone's repeatedly shoehorned SQL underneath object-relational mappers, App Engine (and others) demonstrate that you can provide an object storage API and gain performance by not using SQL underneath. I argue the real utility of SQL is that it lets you quickly (in terms of programmer time, not machine time) perform queries that you haven't done before and won't do again. Say I learn about a bug where I built all of March's data with the word "none" in place of where a column should really be null (None in Python terms) -- that's a line of SQL to fix but it's a world of pain with App Engine due to the bulk operations thing.

With all that said, it's still pretty good. When I was looking to switch projects about a year ago, it came down to basically three projects and App Engine was one of them, because the guys who work on it are some of the best hackers I know at the company. All of the above bullet points (and minor stuff like the languages thing) aren't fundamental limitations of the design, they're temporary flaws that can be solved by good engineering and are surely being prioritized by the team. I'm pretty confident it'll improve rapidly.

the barbarians are at the gates

2008-04-08T18:29:35Z

I enjoyed this introduction to the tenth issue of "The Monad.Reader":

The barbarians are at the gates. Hordes of Java programmers are being exposed to generics and delegates; hundreds of packages have been uploaded to Hackage; the Haskell IRC channel has nearly hit 500 users; and it’s only a matter of time before Microsoft seals that multi-billion dollar bid for Hayoo.

The time has come to retreat and climb higher into our ivory tower: we need to design a language that is so devious, so confusing, and so bizarre, it will take donkey’s years for mainstream languages to catch up. Agda, Coq, and Epigram are some approximation of what functional programming might become, but why stop there? I want strict data, lazy codata, quotient types, and a wackier underlying type theory.

app engine invites

2008-04-08T17:35:42Z

If I know you and you want a cut-in-line invite to Google App Engine, drop me an email.

For an internal contest I wrote an app using this system, so I have a bunch of thoughts on it that I'll maybe write up later.

haskell syntax idea

2008-04-07T15:44:19Z

I often end up with nested parens, when I need binding on the right:
foo (bar (baz (etc x)))
That's ok in Lisp where the parens become invisible, but parens are infrequent enough in Haskell that they're ugly. So they provide an "apply" operator that just has different binding rules, and the above can be written:
foo $ bar $ baz $ etc x
but the dollar signs are huge! If it were colon, it could be written like this
foo: bar: baz: etc x
Where the colon instinctively means: "I'm done supplying arguments to this function; everything to the right of here is its final argument."

on my continuing love for dvcs

2008-04-07T15:34:19Z

I spent the weekend hacking on a project that involved a bunch of trying things out -- maybe this object needed to carry that field, or maybe it belonged over here, and each change required a cascade of changes throughout the code. I managed it all with local branches in git. I could write commit #1 with "this is the structural change I intended, but only the unit test compiles", commit #2 with "this other module was no good in the first place and should be another way", commit #3 with "restructure a third module to now use the new interface provided in #1 and 2". Then, when I decided it was no good, I could rewind the branch, start a new one, and transplant patches (like #2 above) from the dead branch and keep on. Right now I have eight branches in my local repo, and I've cleaned up some of the unused ones.

I was attracted to distributed version control initially because I really actually do a significant fraction of my programming offline: on planes, on the shuttle to work (which used to not have wifi). But since then I've come to see it in two new ways, the latter of which took me a while to settle into.

One (I think this is due to Ben) is that it's just locally cached version control: you can develop in the traditional centralized model if you like, where every commit immediately goes out to the server, but your local disk serves as a cache of data so any read-only commands are faster. (This was one of svn's innovations over CVS: they keep a copy of the tree in your .svn directory so "svn diff" is fast. The distributed model is the natural evolution of this.)

The other is that these systems are designed to manage your source code. Any time you're making a temporary backup of a file, or copying a directory into another place*, or making a patch file, or even holding off on checkpointing where you're at until you just get this one other thing working, your code management system is failing you. (I find I make liberal use of git rebase --interactive to squash crazy "this doesn't work yet" commits to make the tree make sense once I've decided the code is right.)

Last Friday on my project at work I ran into this sort of thing, like I'm perpetually running into it. I'm working on a kind of fundamental change to my project and, of course, one of the 120 high-level regression tests fails. In examining that code I find there's a bug in the testing code; the existing design masks the race condition. So now I'd like to fix that bug: what to do? I end up doing a second checkout and rebuilding (~30 minutes), fixing the bug there, rerunning all the tests to verify that bugfix didn't break anything. Then I manually create a patch file and apply it to my first tree. Now, after sending that bug fix for review, I learn that I should've done it a different way -- ok, fix tree #2, use patch -r to unpatch tree #1 and create a new patch to bring it back in ... yuck!

If I were more confident in my patch-management skills, you could imagine I could skip the second tree: I'd save my current work in a patch file, revert, fix the bug, test the fix, then reapply the patch. And at the code-review point I'd pop two patches off the stack to reshuffle them. (This is, in fact, exactly what Quilt manages.)

But this sort of problem has been solved already in a general way. What I should be able to do is: commit my work in progress in a local branch, rewind, write the bugfix, rebase on top of that. When the review comes, I can rewind and rebase again if necessary. And if I used two checkouts (see the footnote) I could just sync the branches between them, just as if I were making temporary commits server-side.

I still don't love git -- so clunky in so many ways! -- but using it makes me see how obviously useful this stuff is and how silly it will be if we're not all using it in a few years.

* Sometimes this makes sense in circumstances where you want to keep a separate copy of all the build output around. This is arguably the reason you want builddir != srcdir, though in my experience nobody cares about keeping that working.

tinymenu for firefox 3

2008-04-04T23:48:16Z

TinyMenu lets you put all your Firefox menus into a single button. I almost never use the menus so this lets me recover another row of screen real estate.

Here's a hacked-up .xpi I made that Works For Me in Firefox 3. Here's a repo where you can see the changes I made.

To install my non-secure extension (which will be refused by Firefox "because it does not provide secure updates"), you have to go to about:config and add a boolean pref extensions.checkUpdateSecurity set to False.

c-repl lives again

2008-03-30T05:28:24Z

Seeing my c-repl show up in Debian inspired me to rewrite and extend it.

See the new home page, with a more extended session. An extra especially cute feature this time around: #include-ing a header brings all of its functions into tab-complete scope.

It's not solid yet, so feedback welcome.

haskell trick #3: use the pretty-printer

2008-03-27T16:12:51Z

Here's an underappreciated Haskell standard library.

Text.PrettyPrint provides combinators for pretty-printing. It's kinda like a super-simplified HTML or TeX: you give it a bunch of boxes and how they relate to one another and it manages output. It's trivial to use.

A common case I have is printing something that's tree-like, where children are indented below their parents. The code for printing a node is straightforward: print this node's content, then map the print function over all its children with a bit more indentation. But the indentation bit is annoying! You have to remember to properly indent every line, and then that complicated wrapping, etc.

Instead, use the pretty-printer. A Doc is basically a box of text. text "foo" produces a Doc with that string. Then there are higher-level functions to compose Docs: nest indents a Doc, vcat puts a sequence of docs together, one above the other... and finally render converts a Doc to a string.

For my expenses chart I pretty-printed the parsed version of my expenses data.

showDays :: [DayLog] -> String
showDays days = render (vcat (map showDay days)) where
  showDay (DayLog day exps) = text (show day) <+> vcat (map showExp exps)
  showExp (Expense cost desc tags) = text (show cost) <+> text desc
                                 <+> showTags tags
  showTags tags = text "[" <> hsep (map text tags) <> text "]"

showDays outputs a sequence of days by vcatting them together.
showDay shows a single day by writing out the date on the left and then all the day's expenses vertically listed the right.
showExp and showTags just write out the fields in sequence, more or less.

The output looks like this:

2003-03-15 16 airport food [food]
2003-03-23 10 bart [transport]
           9 burger [food]
           10 ikea teapot [home]
           1 ikea water [drink]

All the expenses properly indented over on the right. (I didn't track expenses on my vacation, so there's a gap in the dates there...)

haskell trick #2: using list monad fail

2008-03-27T15:55:07Z

Here's a Haskell trick that I use infrequently but is still nice.

Sometimes you want to filter a list on something that can be pattern-matched on. For example, say you want to grab all the Justs out of a list of Maybes. (That's Data.Maybe.catMaybes, by the way. Or, if you want to be fancy, Control.Monad.msum, since Maybe's an instance of MonadPlus.) One way is something like:
filter (\x -> case x of Just _ -> True; _ -> False) xs
But that's ugly. It turns out that pattern match failure (only within a do block or comprehension) calls fail, which can be defined by the monad, and within the list monad it just skips over that item. So you could write the above as:
[x | Just x ← xs]
or even:
do Just x ← xs; return x

The latter sort of pattern comes in handy sometimes when your filtering expression is more complicated than a simple pattern-match, or something that can be tucked into a one-line comprehension.

haskell trick #1: using ErrorT

2008-03-27T15:47:54Z

Here's a Haskell trick that took me a while to figure out but that I use all the time.

First, the error monad: lets you sequence operations that may fail with error messages. There are a bunch of different types and type classes available so it's pretty general, but the one I always end up using is String as my Error type and Either as my MonadError instance. Here's a contrived example:

import Control.Monad.Error

-- suppose you have a function like:
parseRomanNumeral :: String → Either String Int
parseRomanNumeral input =
  -- code here; result in either (Left "parse error message") or (Right somenumber)
  -- equivalently, either (throwError "parse error message") or (return somenumber)

-- then you can use the monad like this:
addRoman :: String → String → Either String Int
addRoman a b = do
  inta ← parseRomanNumeral a
  intb ← parseRomanNumeral b
  return (a + b)

If either of those parses fail, then the result of addRoman is Left with the error. Otherwise, success is again Right. Hopefully you can see this composes easily.

Ok, more background: you'll often mix this with IO. For example:

loadConfigFile :: FilePath → IO (Either String Config)
loadConfigFile path = do
  contents ← readFile path
  -- parse parse parse.
  -- errors are:  return (Left err)
  -- success is:  return (Right ok)
  -- (again can use "throwError", but the success case then looks like
  --  return (return ok), which is sorta crazy)

-- now imagine loading two config files:
loadTwo :: IO (Either String (Config, Config))
loadTwo = do
  maybeconfig ← loadConfigFile "foo"
  case maybeconfig of
    Left err → return (Left err)
    Right foo → do
      maybeconfig ← loadConfigFile "bar"
      case maybeconfig of
        -- yuck! I won't even finish this

It'd be nice to again compose these like I did in addRoman. That's what the ErrorT monad transformer is for. Rewriting the above to use it, with the new bits underlined:

loadConfigFile' :: FilePath → ErrorT String IO Config
loadConfigFile' path = do
  contents ← liftIO $ readFile path
  -- plain "throwError" and "return ok" now work here.

loadTwo' :: IO (Either String (Config, Config))
loadTwo' = runErrorT $ do
  foo ← loadConfigFile' "foo"
  bar ← loadConfigFile' "bar"
  return (foo, bar)

This is much closer to what you're trying to express. The monad is letting you say: "first load the file foo, and stop here and return if there's an error. then, ...".

All that was background. Here's the trick.

I had to change the type of loadConfigFile so it would be fed into runErrorT. That was ok in the above example, maybe, because the code ended up clearer. But what about code that you don't control, that has the old type with an Either nested inside an IO? Simple: the ErrorT constructor exactly converts IO (Either String a) into ErrorT String IO a. So I could've used the original loadConfigFile and just written the second function like this:

loadTwo'' :: IO (Either String (Config, Config))
loadTwo'' = runErrorT $ do
  foo ← ErrorT $ loadConfigFile "foo"
  bar ← ErrorT $ loadConfigFile "bar"
  return (foo, bar)

In summary, the basic pattern is:

In circumstances where it makes sense, you can leave functions out of the ErrorT monad.
Then, when combining them, wrap plain IO calls with liftIO, the IO (Either ...) calls with ErrorT, and the whole block in runErrorT.

Final trick: you can even use non-IO-using plain Eithers here, too. Within loadTwo'':

  num <- ErrorT $ return $ parseRomanNumeral "iix"

The return brings the Either into IO (Either ...), then the ErrorT converts it into an ErrorT.

Whenever code gets this hairy, though, the real consideration is that you're composing layers at the wrong abstractions and you should restructure it.

dear ubuntu: wtf?

2008-03-24T15:41:26Z

$ apt-cache show ubufox | egrep '^(Desc| )'
Description: Ubuntu Firefox specific configuration defaults and apt support
 Extension package for Firefox provides ubuntu specific configuration defaults
 as well as apt support for firefox plugins/extensions.
 .
 You can uninstall this package if you prefer to use a pristine firefox
 install.
$ dpkg -L ubufox | grep searchplug
/usr/share/ubufox/searchplugins
/usr/share/ubufox/searchplugins/ask.xml

What does an Ask-specific search plugin have to do with Ubuntu-specific configuration?

(Before someone asks: I don't care about this from a Google-vs-the-world perspective -- please, go use Ask if you like it.)

deflate compression in googlebot

2008-03-11T21:58:53Z

Random bit of Google trivia: Googlebot recently added support for "deflate" compression, despite no real demand for it. Why? Go ahead and guess; the answer follows.

This allowed changing the Accept-Encoding header to gzip,deflate, which then makes it better match the Accept-Encoding sent by web browsers. This then allows proxies like Squid to share cache entries between requests from browsers and requests from Googlebot. Here's an interesting thread on the way the Accept-Encoding and Vary headers interplay to ruin things for proxies. (Here's a page I skimmed that goes into a bunch more detail.) You could argue the blame lies with Squid, which apparently(?) treats the Accept-Encoding header value as an opaque string rather than a list of encodings. On the other hand, doing something smarter depends on Squid magically knowing what circumstances caused a server to choose which Accept-Encoding combination.

[N.B.: I'm not an expert on any of this stuff, so feel free to correct me. Thankfully, I was also not involved in the implementation details, so I can excuse myself from blame!]

This was all prompted by an observation by one of the wikipedia folks, who emailed me about it. However, I asked him if this Google-side change helped and he never replied. :~(