evan_tech

stallman and free software

2009-07-11T07:44:07Z

Criticisms or calls for respect of Richard Stallman seem to miss the real importance of the man. People focus on gcc or emacs or the GNU stack, which have certainly been great works; or his more recent GNU/Linux posturing or interviews, which seem to me desperate and nearly pathetic; but these lose the greater context.

I once read the argument that Chomsky -- whether you agree or disagree with his ideas -- serves a useful purpose in delimiting a boundary of the debate. By being "radical", he allows for others to have positions that are more moderate versions of his without themselves getting pegged as radicals. Stallman served the analogous role for free software at a time when it was just not done (the GNU announcement was 1983; Wikipedia says the first nearly free BSD came out in 1991, and the Debian Manifesto was 1993). It'd be excessive to attribute Mozilla or the Free Culture movement just one person but he definitely planted the seed.

In part due to Stallman's influence, today we're not limited to arguing over the relatively minor difference of which megacorp (Microsoft? Apple? mine?) we'd like to license our computers from, but rather whether in a non-zero-sum game like software there are actually moral arguments to be had about sharing beyond simply applying capitalism.

Do I believe in or use the GPL? No, not anymore. But I do believe in free software, and still have a healthy respect for what came before.

pubsubhubbub

2009-07-10T01:51:50Z

I'd like to draw your attention to bradfitz and Brett's pubsubhubbub, a protocol for distributing real-time updates.

It feels very bradfitzy to me: it is simple, decentralized, seems obvious in retrospect, and pays careful attention to an incremental migration path from existing technology (here, people who provide static feeds with the expectation of clients polling).

The slides are brief and to the point. Give 'em a glance.

"operating system"

2009-07-08T06:32:03Z

Suddenly, developing a browser for my marginal operating system seems more important, huh?

(No, I can't answer any questions, except to note this future is different than the one I imagined.)

sweetcode's fate

2009-06-30T05:36:04Z

I finally cornered Dan -- the guy who ran sweetcode -- at a party and told him that I and many others mourn its passing. He seemed surprised.

He told me spent literally hours a day collecting the tidbits that made sweetcode what it was. Since the purpose was to bring things to you that weren't well-known, the normal avenues (hearing from a friend, seeing on a news site) weren't useful. He had to give it up once he got a real job.

We proceeded to discuss why current news aggregation approaches fail (what I'm interested in is good things I haven't heard about before, regardless of the date, while news uses "happened recently" as an heuristic for that), whether a platonic ideal of good exists for the sort of information sweetcode presented, and, y'know, Kaa's law and I got sad about sweetcode all over again.

using git with svnsync

2009-06-20T17:37:58Z

git.chromium.org is a git-svn mirror of the canonical SVN repository. It works like this: the SVN server pushes out, with svnsync, a mirror to another machine. That machine then has a cron job that runs git-svn against the local svnsync'd repo. (Part of this design was so this mirror machine doesn't have any access to the writable repo.)

By the way, there's info on our wiki about how to set up git-svn such that you can fetch with a fast git fetch from the mirror while still using the slow SVN server when it's time to commit.

This has been working fine for quite a while but I noticed that occasionally (rarely) it was getting the proper commit data but the author wrong.

$ echo $(git rev-list --author=chrome-bot origin | wc -l) $(git rev-list origin | wc -l) 86 14624
Half a percent of commits.

I asked around and the best guess is this surprising gotcha: SVN commits aren't atomic. :(
The author metadata is a separate property of a commit and so it's possible for my mirror to grab a commit before the author data has synced over.

What's the fix? svnsync puts a lock in the repo before syncing. Right now I check the lock. To be correct I'd need to grab the svnsync lock myself while I'm doing my copy. Another option is to rewind and try again whenever I see a bad commit get mirrored, but git-svn doesn't really like having history rewound without clobbering its metadata and I can't let it just rebuild its metadata from the commit history for complicated reasons outside the scope of this post.

In summary, now I have this git repo that has the wrong authors in some commits. Fixing it would require rebuilding history from the earliest instance of the problem, invalidating everyone else's copies. I haven't done it since I'm not convinced it's too important. Now that I look at the logs, it seems to have gotten much worse recently...

browser future

2009-06-14T23:37:41Z

Since I think about browsers all day here are some thoughts on the future. As always, I'm biased towards the Linux ecosystem, so in some sense this post could be considered a response to the question that came up recently in the Gnome world, which was roughly "What next? We've written all this GUI software but nobody really cares anymore."

Background: HTML as the new display protocol

X11 is a remote display protocol. It can do some pretty cool tricks but it is also terrible for latent (aka "anything but a short wire") connections, as nearly everything you do involves round trips. To fix this you want to be able to put more smarts into the low-latency piece of hardware you're locally interacting with. Many people have tried to improve this but they've mostly failed. (NX is pretty sweet but there's still not enough smarts client-side.)

In the meantime the world has evolved, for better or for worse, into a remote display protocol called HTML. (There are many interesting reasons for that, but they are beyond the scope of this post.) From the X11 perspective, HTML involves a loadable DSL for interface description, another for sandboxed execution, a well-understood and cacheable networking protocol, and a install base larger than anything before seen. It's not the system anyone would have designed if we were starting over but it's what we've got, and it continues to evolve rapidly. Any future platform that wants their own API will need to support the web one anyway -- witness the annoyingness of developing for Android/iPhone versus the the Palm phone as an example of embracing this.

Most of the software I use today runs in part on my machine and in part on other peoples' machines, and cloud computing hyperbole aside I expect this difference to only get blurrier. I used to scoff at the concept of "web applications" but consider that even relatively mundane websites like LiveJournal really are software -- without LJ's servers doing it for me it, the generation of my friends page would necessarily happen in code on my machine. One of the reasons I decided to work on Chrome was when I realized the only windowed apps I used anymore were terminals and browsers, and terminals more or less dead-ended evolutionarily with the original xterm (I have much more to say about that, actually, but it's for another time).

Software like GWT can be thought of as a toolkit (like GTK/Qt) in this world; existing apps are written to something analagous to xcb but stuff like Wave shows how things might work in the future. (I don't especially want to write Java, but conceptually GWT is in the right place.

Thesis

So here's a thesis: Chrome comes with a few almost orthogonal goals behind it -- tabs as processes, some novel* UI bits -- which are a good step but can't go far enough due to the ecosystem it runs in.

People still think of browsers as applications, but their modern use is more as a runtime coupled with a random assortment of retrofitted system-level services. In part this is because there is no other route for third-party application developers on the Other Platforms to "upload" themselves into system-level services, which is where I think the free software world has an opportunity.

Processes

The elevation (or you could say "recognition") of web applications as things your operating system can help schedule and protect from one another is an old idea, and surely many browsers in the future will do as we've done, but even in today's browsers the web processes are still driven by a fragile "browser process" that is the single point of failure. For example, the network stack all lives in the privileged frontend process, so any bug in the HTTP stack sidesteps the sandboxing and takes out all tabs simultaneously.

Why do networking bugs affect the UI? Why do downloads stop when when I close the last visible window? Ultimately it's because you have a hodgepodge of services (including high-privilege inputs like access to the mouse and keyboard mixed with untrusted network inputs) in a single place: the browser process. Researchers have even called it the "browser kernel", in fact. (One sad reason for the single monolithic process is so people don't complain more when they look at the process list in task manager. Another is that application vendors like Google aren't allowed to provide system services, while OS vendors like Microsoft aren't incented (or sometimes allowed) to integrate such services.)

HTTP, viewed as system-wide remote display protocol, shouldn't be implemented at the application level. Why don't other tools like wget know how to use my cookies implicitly? Or even simpler: my proxy settings? Cookies are credentials much like my user id; Unix apps have a builtin for the former (getuid()) but the web equivalent is considered an application-level problem (wget --load-cookies ~/.mozilla/firefox/mumble/etc).

MacOS and Windows have approached this problem by making HTTP fetching and proxy settings system libraries; however, by virtue of being owned by the OS distributor and closed-source it leaves you at the mercy of their bugs and limitations. (We measured an improvement in page-load performance across our (opted-in-to-measurements) user base when we switched off the Windows HTTP stack.) And the ease of integration is still too lacking; your favorite programming language of the month is still poorly reimplementing HTTP without integrating with my cookie store.

What I'd like: a network service that would offload HTTP interaction from applications. Applications could hand off jobs like "download this file" to the service, which would live beyond a given browser session and allow me to centrally control bandwidth, caching, proxies, limit/resue TCP connections to a particular host, etc.

Tabs

Why do we have tabs? Because window management has failed. As one commenter put it, on a modern machine we now have running native apps in a strip of buttons across the bottom of the screen and a list of running web apps in a strip of buttons across the top of a special browser window. On my laptop, with a tiling window manager, I have a tab for each X11 app I'm running and then one tab containing the Chrome tab strip within it.

One spiffy Chrome feature is that when you tear a tab off a window you can drag it to hot zones on the screen that will relayout your browser windows side by side. But these sorts of behaviors don't belong as application behavior; they're only forced there because the window management the system provides isn't adequate. (I believe Windows 7 introduces some features like this.)

What I'd like: It'd be a pretty small patch to Chrome to change the call to "open in new tab": rather than opening a new tab, it'd open a new window along with some extra hints (in the the window manager sense) about the relationship between this new window and the one that already exists. (Browsers have interesting logic related to whether new tabs appear rightmost or next to the current one, based on the interaction you've done.) My window manager is then responsible for what it's always done: managing windows, and I get to use the existing set of keys for swapping tabs, closing windows, etc.

Conclusions

I think the free software world is uniquely situated for these kinds of changes, because only there do we have enough control over our computers to be able to switch out fundamental pieces like these. (I tried and gave up on OS X due to frustrations over window management.)

On the other hand, it's wasted breath to post about this since I don't have time to implement any of it. And I've been sitting on this post for a month so I may as well publish it before I forget about it.

* Before anyone starts on it: very few ideas are "new"; selection, combination, and execution are relevant.

TODO

2009-06-14T22:58:49Z

I am pathetically behind on email -- I see I've failed to ack patches to at least four of my projects, and I noticed another has been forked (a good thing! though I should probably just give them ownership).

I suspect the fundamental problem is it's more fun to start something than finish it. If you have been trying to get a hold of me, please re-ping me if I don't respond in a day or two.

deb/rpm diffing tools

2009-05-01T16:39:21Z

Dear Linux hackers,

Chrome tends to push minor updates (often security) pretty frequently. We'd like to operate as a good member of the Linux ecosystem and just offer an apt/yum/etc. archive, but it seems unlikely to me that people will accept pulling a new 20mb binary package every few days.

One idea I've idly had is that we could provide deltas, ideally ones that would reconstruct the normal packages you'd get if you went through the slow path. For example, it could look at which package version you currently have installed, then combine the diff with the files you currently have installed to regenerate the files that would be in the newer package, and then construct the package from there and hand that off to dpkg/rpm. I've heard about such tools existing but my Googling is failing me. Do you have any links? Do you have any friends I could ask?

(For comparison, Firefox uses its own autoupdate system on Linux if you haven't installed it via a distributor, which is actually pretty reasonable when you consider the above. But this also requires that the running binary have write access over its own files, which more or less means it needs to be installed in your home directory. This is one of the reasons (aside from the fact Microsoft also recommends it) that Chrome on Windows installs into the Windows equivalent of your home directory.)

my very own flash vulnerability

2009-05-01T15:17:46Z

CVE-2009-0521 was actually me: I was poking around trying to make Flash not crash on Linux Chromium and noticed something wasn't quite right. I had just thought it odd; Tavis, the reporter, recognized it as a security problem.

(I really don't intend to post this much security-related stuff. I believe it's just coincidental that I've had a lot lately.)

jaunty upgrade

2009-04-29T16:59:44Z

Upgraded to Ubuntu Jaunty.

X hung with garbage on the screen after rebooting. Booted into "restore mode" (I forget the exact name), it gave me a text-based dialog where I could say X was broken. Picking that made it spew --help output to the console and redisplay the dialog, but the fix held.
Have new notifications display.
Volume buttons no longer work. (I'm on an X60.)
Turning my wireless on via the hardware switch again makes it auto-find wireless access points. This worked a few releases ago, then broke more recently, and now it's back to fixed.
Turning compositing on in X made everything very slow. (Despite this being an Intel chipset? Maybe related to above recovery problems?)
Suspend worked for me once or twice but it hasn't come back from suspend twice now so I guess it still doesn't work. It hasn't worked for so long that waiting another year to see if the next release helps is familiar territory.
I still can't figure out how to tell gnome what window manager I want. It's changed so many times over the years that none of the four methods I could find online worked.

I appreciate that releasing software this big is hard, and some of that is hardware issues, that part of the Linux social contract is that it's up to me to roll up my shirt sleeves and fix some of this stuff, etc. So I'll summarize by just saying that for me personally, the upgrade was a waste of time.

I'm considering doing a fresh reinstall in the hopes that I have cruft in /etc/acpi or whatever to help with the suspend thing. I was pretty excited the first time it came back from suspend.

(PS: We set up a Fedora 11 box at work for testing Chromium against different compiler/libraries/etc. They use ext4 by default and managed to get a nonbootably corrupt disk within a few days. I'd suggest avoiding ext4 for now -- it's been fine since we reinstalled with ext3.)

freetype bugs

2009-04-18T03:38:07Z

Since we were just talking about font parsing bugs: four more integer overflows in FreeType. Note the reporter (mentioned in the changelogs). Also note, consistent with what ajaxxx had remarked, that the exploits against FreeType historically have not been against the bytecode but simpler parts like the metrics.

presentation, teaching

2009-04-17T23:03:22Z

I gave a presentation at work today. I hope I didn't say too many things that were false.

I found I became much more confident speaking in front of a group of people by TAing CS classes in college. (The responsibility was typically to do about an hour a week before class as well as help with assignments.) Not only do you learn material really well when you have 30 people's worth of questions about it, I also had a good time of it. Some of my favorite student feedback:

in response to me telling my section that I do not require attendance and that I would fight requests to do any pop-quiz like behavior:
"I love you."
at the end of the semester, anonymous feedback:
"He didn't shave a lot of the time."

I'm still pretty nervous when talking before a group, but during this last talk I even managed to not talk fast.

I feel like I got lucky with the TAing thing; I wish it had been emphasized more in school. Knowing what I know now I definitely would've traded off taking one fewer class for teaching once.

security bugs

2009-04-16T19:51:14Z

http://seclists.org/fulldisclosure/2009/Apr/0129.html:

VIII. DISCLOSURE TIMELINE
06/28/2006 - Initial Contact
06/29/2006 - PoC Requested
06/29/2006 - PoC Sent
10/05/2006 - Vendor Status Update
01/24/2007 - Vendor Status Update
02/12/2008 - Vendor Status Update
03/31/2009 - CVE Assigned
04/14/2009 - Coordinated Public Disclosure

That's a loooong time to sit on a bug.

opentype condom

2009-04-10T01:28:57Z

http://github.com/agl/otc/tree/master:

The CSS font-face property is great for web typography. Having to use images in order to get the correct typeface is a great sadness; one should be able to use vectors.

However, the TrueType renderers on many platforms have never been part of the attack surface before and putting them on the front line is a scary proposition. Esp on platforms like Windows where it's a closed-source blob running with high privilege.

Thus, the OpenType Condom (OTC) is designed to parse and serialise OpenType files, validating them and sanitising them as it goes. Most significantly, it removes hinting bytes from the files as it processes.

rolling back

2009-04-08T17:38:07Z

git revert <HASH> undoes a commit: it commits the inverse of the change done by that commit. Concretely, it's just like a cherry-pick but with a backwards commit. (In git's implementation, the two are implemented by the same file.) How's it work?

The conceptually-simplest way to roll back is to apply an inverse patch, something like:
git diff HASH HASH^ | patch
but that does not work quite right in the presence of complicated operations, like deletes or adds. And if there are conflicts it'd be nice if all of that conflicting information known to git, so it could assist with running merge tools.*

Instead, consider a tree like this, where newer commits are on the right:
...---A---B---...---HEAD
Suppose B is the commit we don't want anymore.

Conceptually, you want to go from the state in B back to the state in A, but with all of the stuff in the middle up to HEAD still kept around. Imagine you had created a new branch off of B that contained the state of the world in A, like this:

A---B---...---HEAD
     \
      A

In that world, the rollback would just be merging that branch into HEAD.

To do this without creating branches, you can just directly run the merge command, which takes a base branch and a set of other branches to merge.
git merge-recursive B -- HEAD A
That says: "using B as the reference, merge as if HEAD and A were our interesting branches.".

More commonly, you've got the hash of a single interesting commit so it's instead:
git merge-recursive H -- HEAD H^

(Note: I'm not certain this is exactly correct; it depends on whether merge-recursive actually looks at the relationship between the branches for doing the merge. I wonder if it may be more correct to use git read-tree -m A B HEAD to read in all three trees into the index and then use git merge-index but both of those are plumbing commands I don't fully understand.)

* You could use git apply instead of patch to help with that, but that is not the subject of this post.

pre3d, a javascript 3d rendering engine

2009-03-27T13:46:33Z

If you looked at Chrome Experiments, you might have seen Dean's Monster demo. He's now published the source for pre3d, his 3d engine for JavaScript. It also does texture mapping, which I guess isn't used in any of the demos. Since he works with the v8 team it's likely it will be pretty zippy compared to other approaches.

startup time

2009-03-05T16:51:39Z

A: Knock knock.
B: Who's there?
.
.
.
.
.
.
.
.
A: Java!

(stolen from a reddit thread, though I don't remember the link)

using control.applicative

2009-02-28T22:35:27Z

A post in three sections, each a step of evolution beyond the last. Contrast with The Evolution of a Haskell Programmer.

Here's a situation that comes up decently frequently. Say you have some monadic operation that gives you back a basic type. Actually, let me make it concrete so it's easier to follow. Say you have a parsec parser that parses an integer:

p_int :: Parser Int

Deep elsewhere in your code, you have some data type that you'd like to stuff the result of that parse into. E.g.:

data Expr = EInt Int | EString String  -- etc
p_expr :: Parser Expr

So the question is: how do you use p_int within p_expr, converting the output? Let's start with the obvious way:

p_expr = do i <- p_int       -- parse an int
            return (EInt i)  -- wrap it in EInt

But it's so verbose! You can shorten by going point-free with the monad operators:

p_expr = p_int >>= return . EInt

But it always feels like golf to me when I see that in code; you may as well just do the do-expression in one line in only seven more characters:

p_expr = do x <- p_int; return (EInt x)

On the other hand, it does feel redundant to give the integer a name just to throw it away. The more experienced Haskell hacker knows about Control.Monad, which provides some operators to work with monads. Notably, there's liftM (parens added for clarity):

liftM :: (Monad m) => (a -> b) -> (m a -> m b)

That "lifts" a plain function to a function that works on monad values, so our running example can become:

p_expr = liftM EInt p_int

Can we do better? From reading other code I picked up that the prelude's fmap, which is map over the Functor type class, is the same as liftM for a monad. It feels a bit golfy but not so bad: conceptually you're mapping EInt over the result of the parse (but only if it's succesful):

p_expr = fmap EInt p_Int

But recently I gained a newfound understanding of Control.Applicative and found something better. From their docs, Applicative is "a structure intermediate between a functor and a monad: it provides pure expressions and sequencing, but no binding."

To keep it straight, here's a table with increasing abstraction as you go downwards. (Note the types of the columns aren't exactly the same, but maybe it illustrates the point.)

Typeclass	bring value in	primary operation	helpers
Monad	`return`	`>>=`	`liftM`, `ap`
Applicative	`pure`	`<*>`	`<$>`
Functor		`fmap`

Conceptually, Applicative lets you bring values in and combine values, but doesn't let you get back at the values. It suits the parser example just fine:

p_expr = pure EInt <*> p_int

That is, pure brings the whole function into Applicative and then <*> is normal Haskell composition (normally just written with whitespace or $) within Applicative. But there's a helper operator that's just for this sort of thing:

(<$>) :: Functor f => (a -> b) -> f (a -> b)

You'll note it's the same signature as liftM, but in usage it's quite clear:

p_expr = EInt <$> p_int

The name is analagous to the $ operator but the brackets indicate within Applicative. You can read that as "apply EInt to p_int, but within Applicative". Succinct and clear.

The nice thing about Applicative (and sort of the point of it) as compared to monads is that you don't have this continual diving in and out of the monad; you just lift everything in and then combine. Contrast these two binary expressions:

add x y = x + y
normal_expression = add int_1 int_2
applicative_expr  = add <$> p_int <*> p_int

<*> (equivalently ap for monads) is there to let you to do further composition on the right, all within Applicative.

Finally, note that all of this isn't specific to this parser problem; you can use these functions just as well with other monads (like IO) and also to other places involving functors:

(+1) <$> [1, 2, 3]
-- equivalent to map (+1) [1, 2, 3]

See also Perl 6 meta-operators, which is this sort of thing done at the syntax level.

seeking intern for chrome linux

2009-02-26T01:54:46Z

Tony, my officemate, is looking for an intern to join us hacking on Chromium (Chrome) and Linux this summer. Gtk hackers, X hackers, sandbox hackers, WebKit hackers, browser hackers especially welcome. Strong C++ required, open source credz preferred. Mail me at evan at chromium.org.

mime sniffing

2009-02-12T05:19:03Z

I post about (in part) the ancient IE mime sniffing hole over on Chromium Notes, and ten days later a 600-reddit-vote article shows up on the same subject. Coincidence? You decide!

(Unfortunately, the defenses proposed in the article are not sufficient for protecting your site. I leave figuring out why to you.)

zippers

2009-02-02T02:33:42Z

Zippers: there are plenty of docs, so here's some random notes instead. (The original paper is in ML and quite readable.) Zippers are a fine example of the sort of progressive generalization that people play with in Haskell; my favorite blog at the moment is filled with interesting math along those lines.

I think the first zipper I encountered, before I knew the term, was in reading a Brainfuck implementation written in O'Caml. Brainfuck's model is basically a Turing machine, where you have a focal point on a tape and you can move around on the tape and do operations based on the focal point.

The question is: how do you represent this tape data structure with immutable data structures? If you use a list and an offset, examining the focal point is O(n) (to walk down the list to the offset) and modifications are O(n) (to rebuild the list). The solution, which seemed novel enough to me at the time that its obviousness now doesn't dissuade me from describing it for you, is to represent the tape as three parts: cells to the left of you, your current cell, and cells to the right. The trick is that on both sides, the nearest cell is at the front of the list; in some sense the left cell list is backwards.

Now stepping by one cell is O(1): you just pop the top of the relevant side list off and stuff your previous focal point as the new top on the other side. What I didn't fully appreciate at the time is that this is also gentle with memory. You're only popping/adding from the front of a list, so the remainder of the cell lists are constant and shared. Each step is destructuring one cons cell and creating one new one.

So whatever, you might say. You have jump through hoops to get functional data structures and it's a waste of time. Here's where it hopefully gets interesting or at least more complicated.

This idea, called a zipper, can be generalized: your cursor is a pair of the current point along with context representing the remainder of the data structure arranged such that nearby movements are small steps. (In the Brainfuck example the context is the cell lists.)

Most examples discuss movement on a tree. Your current point is an unmodified subtree, and your context is a pointer to the subtree's parent along with the rest of the parent's children (the children other than your current point). This otherwise kind of useless Wikibook article has a nice picture about halfway down visualizing it, where your pointer kinda "grabs" the tree and "lifts it up" so it hangs around that point, which is a nice structural metaphor. In terms of memory, it's sorta like the list zipper in that most of the tree is left alone and can be shared during movement; all the zipper added is a reversed path back to the root.

What struck me recently is that with a tree zipper you've stepped around the problem of parent pointers in a tree while still allowing movement in any direction. The zipper here is effectively just building a stack of parent pointers as you walk down the tree and popping the stack to walk up, which is the sort of trick you've likely used (or at least could have invented) in an imperative programming context to conserve memory while allowing arbitrary tree navigation.

This is again all nice enough -- two related concepts that do similar tricks with data structures -- except it can be generalized even further. A zipper can be defined for any data structure systematically by computing the derivative (in the calculus sense, kinda) of its data type! What's the use? Even the guy who discovered it isn't sure: "While this connection seems unlikely to be a mere coincidence, it is perhaps surprising to find a use for the laws of the infinitesimal calculus in such a discrete setting." More research is warranted, I guess.

inbox out of control

2009-01-31T18:03:58Z

I am critically behind on my inbox, to the point where I believe there are five patches for various bits of software of mine hidden under the mass of email.

For example: someone took my Arabic transliteration JS thingy and made an equivalent system for Urdu. He politely had asked me about the license of my code and that question sat unanswered for weeks.

Today I solve this.

adventures in search optimization

2009-01-25T04:11:33Z

At work we heavily use a tool for rapidly searching the codebase with regexps. (I believe it's the same engine behind Google Code Search.) But it isn't as helpful when your index is stale, which happens more often when your code churns rapidly (as Chromium does -- ~50 commits a day by us and ~50 more commits a day from WebKit). I had been idly thinking about making some sort of full-text indexing system for working with my own code that was smart about watching for file modifications, building generations of indexes, etc. It sounded like a fun project.

Then I just chatted with Matt a bit about it and realized I was thinking about it wrong. His first thought was that there shouldn't be that much data involved here and I should be able to just do a brute-force scan.

But there's too much data, I protested. To start with, find across my tree with a cold disk takes tens of seconds to even enumerate all the files (over 200,000). And Visual Studio's "find in files" is also super-slow, supporting my intuition. But upon second glance that number is immediately suspicious as way too many files -- because it's including all sorts of files I don't care about! As far as source goes it's only around 10,000 source files and under 50mb of data.

So what's faster? Git knows which files are in the repo. (A great trick for finding files by name more quickly than using find is git ls-files <pattern>.) Then I just want to limit my grep to source files. I had thought git grep didn't let you specify a filename pattern (which is why I was fooling around with find and grep in the first place) but after rereading the source I see I overlooked it in the man page; something like git grep foo -- *.cc *.h does exactly what I wanted. It's easy enough to stuff in a one-liner shell script so git gs foo quickly searches my code.

On a cold disk (after flushing the cache), it takes ~11s on my laptop. But as soon as the disk is warm (and it's only 50mb of data to keep around anyway) it's 0.35s, which is plenty fast. I note that whoever wrote the grep support for git was clever enough to shell out to grep (unless your combination of passed-in flags prevents it), because you are unlikely to beat grep.

(PS/update: it turns out that M-x vc-git-grep hooks all this into the existing Emacs grep support, complete with shorthand for specifying "file extensions that look like C++ source or headers". I am humbled.)

megaupload captcha

2009-01-24T20:29:46Z

Someone make a Javascript-based captcha cracker for megaupload. It's strange to see those captchas again because I idly myself wrote a captcha-cracker for the same site. I don't think I ever announced it here because there wasn't any real application for it; I was just playing around.

According to the repo history it was two years ago, and the notes indicate how far I went before losing interest. As I recall their captcha image was quite weak, but you also needed to be able to interpret Javascript and fake out the DOM to skip a timed countdown.

emacs

2009-01-23T04:47:07Z

I've been using vim for a very long time -- over ten years -- but over those years I've envied more and more the way emacs integrates other software. So a few months ago I switched. I don't see it as the religious issue that many people make it out to be, so the only real hurdle was taking the time to read the manuals.

It turns out switching is surprisingly not that painful; any competent user of a Unix is already familiar with the basic emacs keybindings because shells use them too. People have historically complained about emacs being bloated but it was bloated on computers of the 80s; it's tiny compared to even simple Java or web-based software today. I have found my mind has "vi mode" and "other editor" mode, so I keep hitting emacs keybindings in Firefox or Visual Studio.

For comparison (eclipse/emulator are the Android development tools):

 8916 martine   20   0  717m 181m  33m S    0  9.0   0:57.65 eclipse
 9005 martine   20   0  280m 137m  23m S    0  6.8   1:06.25 firefox
 9788 martine   20   0  162m 115m 5416 S    2  5.7   2:13.31 emulator
11260 martine   20   0 32632  19m 9068 S    0  1.0   0:00.52 emacs

What I like:

ediff is great. I had previously used meld but meld is buggy and underpowered.
I use the Python, Haskell, and R integration all the time. It's so convenient to have a REPL that works within your editor. It's pretty much the reason I switched. Type some Haskell code, hit C-C C-l, and your cursor is in a buffer with your code loaded.
The buffer management concept makes a lot of sense. The way you switch between buffers with C-x b matches my workflow well.
magit made my work on improving the git UI effectively obsolete. (But it's really slow for some reason...)
"C-h a" is a useful way to interface with a huge interface surface
The vim scripting language is an abomination and I refused to learn it. I've already made a tiny function in emacs lisp for composing entries for my blog software.

What I don't like:

I believe the basic editing key bindings are just not the way I operate. I made a point of not using viper because it produces the uncanny valley effect, where it's close enough to vi that you get frustrated by esoteric keys not working, and instead forced myself to learn the emacs way before I start changing things. But now I'm again considering trying viper.
Undo. I get how it's intended to work but it never quite feels right. Could be learned through more experience, maybe.
I feel like the different programing modes have different keys do needlessly different things. I understand that a unified UI is contrary to the point of a software system defined by its customizability, but that makes it hard for me to get an intuition for the natural way things ought to work. For example, why doesn't C-h m show me the Haskell-mode specific keybindings while it does in other modes?

I get the feeling there are tons of customization tricks that people use that I'm missing. So please, tell me your emacs tips!

One of the big things I've been wanting is making it so when I load up code under the Chromium directory my editor configures itself with the appropriate defaults (e.g., .h files are C++, and M-x compile should run SCons from the proper directory). I saw some project local variables scripts on the emacs wiki but I couldn't get them to work right, probably because I'm missing understanding.