Finally documented the 2011 midwinter hunt

It's been sitting on my hard drive in more-or-less complete state for a while now, but I've finally gotten around to posting the documentation of the 2011 midwinter puzzle-hunt I wrote.
I hope this is the start of a trend toward actually documenting some of the things I do/make, if just for the sake of my own later self.

Updated Backend

Over the last few days, I've massively restructured the way that tchow.com is served.On the surface, you shouldn't notice any big differences, but under the hood things have gotten dramatically simpler.

How it was

From sometime around 2007 until just a few days ago, tchow.com was hosted on a VPS partition from RapidVPS.The partition was set up running a pretty standard stack: Gentoo GNU/Linux, the Apache web server, and PHP5.
I designed my content serving system to be future-proof, simple to use, and to contain minimal redundancy.This meant that I wanted visitors to see clean URLs (no file extensions; file types determined by Content-Type header), and those that looked at the markup to see nice xhtml with a clear separation of content and chrome.Internally, I wanted to be able to serve the web page from a filesystem, provide content with my favorite text editor (and/or scp), and do so in a not-too-idiosyncratic format.
In order to make this work, the following happened on each page load:first, mod_rewrite rules would take the user's URL and turn it into a query string for a dispatch.php script;this script, in turn, would crawl the backend directory structure to find the proper page (a file containing an html fragment),then paste in appropriate templates for navigation and analytics.

What I didn't like about it

This old setup was nice, but it was also overkill -- php was re-generating the same (static) content over and over again, opening and reading through tens of files on each page view.This seems really inefficient (and, honestly, the "standard" answer of wrapping this in a caching web server seems even sillier).
Additionally, external URLs were clean-ish, but I never really resolved where a trailing slash was appropriate, which led almost everything having absolute links all the time.
Also, I ended up storing my page content in an svn repository (to move edits between my staging and live pages), but since svn has no way of updating all of htdocs at once, this creates potential race conditions in page viewing during updates.Besides, I've been using git for a number of years, and it feels so much snappier than svn that it was getting to be a drag to do page updates.
Added to these design concerns, I'd noticed that Digital Ocean was offering a VPS partition of similar size to my RapidVPS partition (but with unlimited bandwidth) for about half the price.So it was time to change.

How it works now

First off, I ditched Apache -- which provided way more than I need -- and switched to Lighttpd; this also provides way more than I need, but at root it's a lot more comprehensible.Indeed, I get nice clean URLs using just two modules: mod_rewrite for hostname correction (e.g. so that www.tchow.net redirects here properly) and mod_magnet for URL dispatch.I no longer have php re-generating static content over and over again;instead, I have a python script that I call on the (git) content repository which regenerates just those pages that need it.
But mod_magnet is really the key to the whole setup because of the way it works:you provide Lighttpd with the path to a lua script, and it runs that script to determine which physical path corresponds to given query string (the script can also do things like trigger redirects and add headers).The cool thing here is that now every page view goes through dispatch.lua (in my case, nothing more than a url -> file mapping table), and it's one file, which means that if I atomically update it then I've atomically updated the web page.So this setup gives me the satisfaction of being able to update the content on tchow all at once, and without race conditions (as long as my scripts are careful to not overwrite any content referenced by the active dispatch.lua).
I also re-worked a few things around the urls so that the semantics are much clearer, changed the styling a smidge, drew some new calendar icons, and did various other tidying.But mostly, I'm satisfied that I've gotten rid of a lot of systems that tchow.com didn't need, and pared things down to a really slick-n-slim software stack that still satisfyingly serves what it should.

Checkpointable Box2D

While working on rktcr, I've spent a while thinking about how to checkpoint Box2D simulations so that one can rewind and play them back exactly.(Useful for replays, or time control when you really care about doing it right.)This turns out to be both harder and easier than one would expect:harder, because some of the simulation state (e.g. order of contact processing) is hard to access, much less properly save;easier, because one can hook memory allocations in Box2D and just save everything.This is one of those hacks where I'm very glad I use a language that can talk about memory.
I'm releasing this code as public domain -- you can find it from GitHub here.There is a small patch you'll need to apply to Box2D included.If you enjoy (or are confused), please let me know,

Added Sworderation

I've added a game I worked on in 2004 -- Sworderation -- to the games section.I just played through it again, and it's still fun.I hope you think so too.

ICFP 2010

(This legacy post contained a description of what I did for gate encoding in the 2010 ICFP. I have moved this content to the projects section.)

wmiirc and Disks

(From the not-useful-unless-you-use-Linux-like-I-do department.)
So, I've been using wmii as the window manager on my laptop.Other than changing its default meta key to be the winkey (I should really do something about that branding), setting it to launch uxterms, configuring it to show CPU and battery info, and setting up pidgin to change a tag color on new messages, I haven't had to change its default settings at all.
That may seem like a lot, but in the world of "nonstandard" window managers, well, it isn't.Programmers (the probable main consumers of -- and face-slappingly-obviously main producers of -- such WMs) are picky people.Indeed, the fact that I didn't have to edit the source code to do any of this is right civilized!(Unlike switching the meta key in, say, Fluxbox.)
However, I did manage to do something stupid when performing that second-to-last tweak.First some background:wmii has a status line at the bottom of the screen.In the wmiirc script there is a function -- status() -- whose output is pasted into the lower right at some frequency.To display battery information you simply create a program (likely a shell script) to echo the information you'd like displayed and have status() call it.I learned as much from this forum post, which also includes a handy script to produce exactly the sort of info I wanted to display.Thus, I adapted this script to my needs and it has been quietly working ever since.
But my hard drive has been being accessed every five seconds (even when the computer is apparently idle) ever since I set that script up.Turns out that here-documents in bash create temp files.Creating temp files is disk activity.Disk activity makes kjournald decide to write a journal every five seconds.Which is probably not good for the disk, definitely not good for battery life, and quite annoying.Here'my version with that problem fixed:

#!/bin/sh
#based on http://bbs.archlinux.org/viewtopic.php?id=1809

rate=`grep 'present rate' /proc/acpi/battery/BAT0/state | tr -d -c 0-9`
current=`grep 'remaining capacity' /proc/acpi/battery/BAT0/state | tr -d -c 0-9`
total=`grep 'last full capacity' /proc/acpi/battery/BAT0/info | tr -d -c 0-9`
percent=`echo "100 * $current / $total" | bc`

speed=`grep -m1 -o 'cpu MHz.*\.' /proc/cpuinfo | tr -d -c '\n0-9'`

if grep -q discharging /proc/acpi/battery/BAT0/state
then
 minutes=`echo "60 * $current / $rate" | bc`
 echo $speed $minutes $percent'%'
else
 echo $speed $percent'%'
fi

By the way, should you find yourself in a similar pickle, echo '1' > /proc/sys/vm/block_dump.It will produce kernel messages (viewable by dmesg -- you probably want to turn off your system logger while doing this) which will let you know who is causing blocks to need writeback.

SIGGRAPH Panic

(Reported far after the deadline.)
SIGGRAPH is crazy. What do you do when you need more space to edit your video? Well, to be fair, I had those backed up on another computer. Still, it was the most convenient spare 10+ gig.
And kdenlive is quite buggy (not old days cinelerra bad, but...). I've resisted up to now, but I may have to write my own nonlinear video editor. Or maybe try blender's built-in editor. The key is to write up a library that can load and seek video files in a frame-accurate way, with minimal pre-processing and memory overhead.

Added Blots

I added blots, an old animation project, to the art section.

Dead-Bugging a QFP32

This morning I spent a few hours soldering a QT1103.Now, the QT1103 comes in a QFP32 form factor --a form factor which was almost certainly never meant to be soldered by hand,and, just as certainly, never meant to be soldered without a nice little footprint of tinned smd pads.But you can connect to them, so I had read, byflipping 'em on their back and soldering small wires to each pin, the result of which you see above.While this may look like a terrible job (a) there aren't any bridges and (b) this is really really small -- these wires are individual strands of a stranded copper wire.
Of course, I have yet to see if the device will actually work after I went to all this trouble (I could have cooked it, after all).But whatever the outcome, it was certainly the most fiddly soldering job I've ever done. I begin to see why people pay $20-$40 to have a PCB made.

SIGGRAPH Day 5

Well, that's it. The end.
I gave my talk today, and people seemed to appreciate it (and the presentation software). It would be cool if someone else started using my presentation style, but I doubt it will happen.On the plus side, this was a fully morally-correct presentation: linux and my own presentation software -- no non-free taints.
I also got my final sticker sheets: TCHOW in color + some 'redacted' stickers to place over my laptop's hard-to-remove branding.All told I have enough stickers to even give a few away or stick them on stuff.

SIGGRAPH Day 4

Today was exciting. I made stickers (again), and talked with Mat Shlian, who does cut and folded paper sculpture.The thing pictured below impressed me because it has a nice springiness.

I chatted with some interesting folks during the poster session; not so much about the poster and more about life in general.Most interesting was the conversation with a film restorer about how noise is removed and then re-added for old films.
I went to the reception and hung out with various people; found out that some other folks are also considering research much like mine (but I think it will be okay).
I did a talk dry run in here somewhere too. Should be interesting to see how it goes tomorrow.

SIGGRAPH Day 3

Not much happening that I was interested in today.Saw Ton Rosendaal and others talk; resisted the urge to get a photo with him.Picked up my stickers from Studio; I shall definitely do more! Digital Domain's pre-vis for Speed Racer used the Top Gear intro music.
I went to the Computer Animation Festival. There was a long run of films from a French school (supinfocom, if I recall), all of which had a lazy animation style that offended my eyes. It was very much like being slowly crushed by rocks.The other films were good though.

SIGGRAPH Day 2

Sat in sessions in the morning, talked with people about my poster (and about gradient paint) in the afternoon.Brian Curless apparently has something coming out at ECCV worth checking out.Gradient Paint seems to impress and confuse people (though different people are confused by different parts).
In the art gallery, I came across a cleverly cut sheet of steel:

Then came TCHOW logo-stuff.

Thanks to emerging technology (and the guts of an old printer), one can enjoy a TCHOW-branded cup of coffee, as pictured above -- though, unfortunately, they weren't actually allowing people to drink the coffee.So one could look at it, at least.

That's a lenticular print of the tchow logo (that is, a flat image with a lens sheet over it allowing for different views per-eye). Sort of very much like a zig-zag-folded piece of paper really.Upshot being that, in person, it's 3d. Which is nifty!
Finally, I made some TCHOW stickers, but those haven't printed yet.

SIGGRAPH Day 1

Doing a fast-forward is a bit of an adrenaline rush, I must say.
Everything else was pretty much me just being sleepy and hanging out; Alex Evans was going to talk but he canceled (which was disappointing).
Ed's keynote was nice -- I need to catch up with him and see how he reconciles his commitment to produce the best possible content with being a profit-seeking public comapny.

SIGGRAPH day 0

Consider this the first of my SIGGRAPH reports.I figured I'd write things up this year to capture the event for posterity.
Approaching the building, I began to again meet the realization of the sheer enormity of the event.The venue (the LA convention center) is giant, just like all the other venues have been.Somehow I always forget this.
I was early for registration and thus about the third person in line for contributer registration.For some reason I never got the mini-guide that is so useful for finding one's way about the conference. I'll have to get one tomorrow.
Also, I didn't get the ribbons I was entitled to (I got "Paper Author" and had to ask for "Poster Contributer" and "Paper Reviewer" -- by number, in fact)The ribbon-accounting is relatively poor, so I could probably easily have gotten some other ribbons I wasn't quite entitled to (I recall that Wojtan may have managed something of the sort last year).Nonetheless.I think there might also be a speaker ribbon or paper presenter ribbon or something.Really, it's all just silliness anyway.
But, on a positive note, I did get the copy of the proceedings I was entitled to (unlike last year).Of course, maybe it would have been better not to -- given as I spent the next five hours walking around with it in my backpack.
I did manage to relieve myself of one burden: my gigantic 8'x3' poster barely squeezed onto the provided board. (I did see at least one 8'x4' poster already up, so while I was tied for maximum width I was not at maximum area.)I put some post-it notes nearby with a request-for-comments; I'll see if anyone uses this novel system to interact with the poster.
So that's about it for day zero: some items lost, some gained, some not gained that should have been and some gained that perhaps, more prudently, shouldn't have been.

On Milestones

When programming, it helps to have little goals to guide you along the way;for simple projects (like I suspect the server stuff for necklace should be)those goals break down well as:

1. It compiles: code made into object format with no warnings. Apparently all the syntax was correct. (Inevitably I'll find myself back here when I compile with VC++ instead of g++.)
2. It links: there's some sort of simple test program that exercises the code. Here's where I find out that I forgot to write certain functions. (Another way of looking at it is that link errors serve as a to do list.)
3. It works: there's a basic test driver working and things are going swimmingly.
4. It actually works: all the nasty corner cases have been tested (all that I can think of -- I mentally earmark as I'm coding).

I'm currently at the second step, hoping to get to the third eventually.Big systems aren't a cake-walk, and I'm enough of a realist to know that there will be bugs and they will be subtle.

PS3 Gaming

Ladies and gentlemen, the Playstation 3 gaming experience, for me:

Seriously, every time I turn this thing on, it needs to shuffle off to the internet for a few hours to update itself.Perhaps it could download updates in the background; then I'd only need to wait the interminable 15 minutes for the update to install.

Interesting Times

The previous two weeks have been exhausting.I flew to Dublin for the Symposium on Computer Animation, arrived back in the states, and catapulted into a glass blowing intensive class -- five days of nothing but sleep, eat, glass.
Needless to say, progress on almost everything TCHOW-related has been minimal.I am inching the necklace server code forward; in fact, as it stands, the game can submit high scores and learn if they are new.What is needed, however, is a cleaner design -- all this net code is getting confusing! The plus side is that a correct design should basically "drop into" other projects.
Time is running short, and there is research that needs to be done.

Silly Hats On Mars

So, the network stuff for necklace is stretching on and on -- partially because I'm not working on it very much, partially because of other events.Like, for instance, viewing new art while wearing formal clothing and silly hats (yesterday afternoon). I'm second-from-the-right by the way.

Necklace Net code

I'm chugging along on some network code for necklace. The first release will be about the basic pervasive multiplayer element (i.e. a unified solution database). The next milestone after that is a decent level-sharing mechanism-- not that pasting text to your friends over IM doesn't work or anything.
One reason (among many) that this release has been taking a while is that I'm debating what really constitutes a "similar" level.In the most general sense, it's the graph structure on the pieces and the bitmap of the obstacles -- but I think that finding a canonical form here might well begin to look like graph isomorphism.Not to mention that there is plenty more path to travel here (reflections and rotations of levels are pretty much the same level).Perhaps I should just avoid this slope altogether.