Interactive vs Generational Collaboration

My research group has been writing a lot of papers recently. Like any reasonable set of CS folks, we use LaTeX as our typesetting tool of choice. Of course, like any modern set of collaborators, we also all want to be editing the paper at the same time, changing words as others are writing, and completing each-others sentences.

To accomplish this, we have generally turned to Overleaf, which -- like LaTeX itself -- isn't a great solution to the problem, but it is the best solution we have. One of the interesting things about Overleaf is that it supports two completely different models of collaboration, which -- for the purposes of this post and without reference to any external sources -- I'll term "interactive" and "generational".

Interactive Collaboration

Interactive collaboration models a shared workspace. All changes are made simultaneously and with fine-grained (real-time) synchronization. Interactive collaboration seems to work best when edit operations are "local" -- generally modeled by granting participants a notion of a cursor or selection around which their edits are centered.

Interactive collaboration doesn't work very well with dependent or global edits (e.g., trying to make an overarching change to the order of sections). And -- as often happens on Overleaf -- it really doesn't work well when the synchronization system begins to bog down.

Generational Collaboration

Generational collaboration models... well, I'm not exactly sure what it models. Perhaps a series of independent artifacts, individually crafted, but with reference to other previously created artifacts? Regardless, changes are made individually, and merged non-real-time into a final artifact.

Generational collaboration works well for changes that don't overlap and (thus) can be merged automatically. It is, thus, a natural fit for global but distinct changes (e.g., changing all references to a figure; doing a global re-ordering of paragraphs in a the "Results" section while another collaborator works on the "Introduction" section).

Further, generational collaboration can operate in a decentralized and disconnected way, which are distinct advantages, especially on a paper deadline when nobody has time to wait on a slow server. (Or, e.g., on an airplane.)

Combining the Models

So, how do we combine the models to get the benefits of both?

Overleaf's approach is to support the generational collaboration model via git push/pull support, but to require that no interactive edits happen between a git pull and the subsequent git push. This, effectively, makes generational collaboration lower priority than interactive, making it incredibly awkward to leverage this model while others are using the interactive model.

My preference here would, instead, be to start with a decentralized generational model and build in some notion of interactivity. I tend to like disconnected/disjoint tools, so perhaps the way to do this is to have the notion of a multiplayer editor protocol that can work (optionally, in parallel) with a git repository.

Here's a sketch of how that might work:

When a multiplayer (text?) editor opens a file, it would check in the same folder (and parent folders) for a .multiplayer subdirectory and contained config file. (Similar to how git looks for a .git subdirectory.)

The .multiplayer config file would contain either the address of a responsible server and a project token to identify the project to that server, or a list of peers to contact directly. Any multiplayer-capable editor would -- after prompting the user -- get the list of peers to coordinate editing with (possibly via connecting to a server, possibly via direct peer entries). The server could also handle STUN to make peer-peer communications easier.

Regardless, said multiplayer-capable editor would now be connected to a peer cloud and could use peer-to-peer multicast techniques to keep the document synchronized, publish cursor state, and so on.

So that takes care of the interactive collaboration. How about the generational part? We could make it so the .multiplayer directory contains a reference to the most recent point in shared edit history at which an editor had synchronized to the peer cloud. If -- when your multiplayer editor connects -- your current file matches the recent version in the .multiplayer directory, then all changes between that version and the current peer cloud version could be automatically applied.

If, on the other hand, your current file differs from the reference version in the .multiplayer directory, then there's a generational change to merge. This is the classic "three-way merge" problem that (e.g.) git solves reasonably well. In the interactive setting, it might even make sense to allow you to request others stop for a second and help with the merge.

Of course, if your text editor of choice doesn't support multiplayer, you could still edit single-player and use a command-line utility to deal with the three-way merges. (Perhaps with a "hang on, let's do the merge together everyone" option to avoid the same problem that overleaf has.)

So, Who's Building It?

Perhaps the most surprising thing for me in my (scant) research around this topic is that nobody has built an interoperable interactive collaboration system that actually supports, e.g., cross-editor cursor sharing; at least not one that is still in action today (I seem to recall that google docs at some point had an API with cursors).

Perhaps the notion of a cursor is so editor-specific that it takes special sauce to do it. Perhaps the idea that you need to run a server makes people want to lock in users to finance said server. And perhaps I just haven't looked hard enough.

In the latter case, let me know! I'd love to try out something different with my group when we next have a big crop of papers ready to go.

The Rktcr Benchmark

In preparing Rktcr for release on Steam, I've been making some minor adjustments to the code and art (Steam Achievements! Blinking characters!). As I've made a bit of a fuss about before, Rktcr is a game that plays exactly the same across three compiler/OS combinations (g++/Debian, clang++/OSX, cl.exe/Windows). As such, part of any update is confirming determinism with an extensive test suite.

Specifically, I have the game compute SHA1 hashes of relevant state as it plays itself on a hand-crafted set of ~1,600 par time paths. Yes, there are over 1,600 individual paths segments you might want to play in Rktcr, and I have a plan to add cheat-proof leaderboards for all of those. But that's a topic for another post.

The Benchmark

So how do my Windows and Linux partitions stack up against each-other on this (blended filesystem/compiler optimization) benchmark? (I'm not testing clang++/OSX since that's not installed on the same machine.)

OS	Compiler	Time
Windows	VS2013, Update 3	120 seconds
Windows	VS2013, Update 4	121 seconds
Windows	VS2013, Update 4, optimizations off	137 seconds
Linux	g++ 4.8.1	79 seconds
Linux	g++ 4.9.2	76 seconds

I'm pretty sure that the one-second difference between VS updates is within noise, though I dread that it may be a performance regression. Turning optimizations off demonstrates that they are at least doing something.

Under Linux, there is a noticeable improvement between the steam-runtime-targeting g++ 4.8 and my system-wide g++ 4.9; whether it's a compiler or standard library difference isn't clear; but it is certainly nice when newer compilers make old code faster.

Finally, I'm surprised that the Windows/VS version is doing so poorly compared to Debian/g++ -- my previous experience has been that cl.exe produces code that is a fair bit faster than either g++ or clang++. I'm left wondering if this particular code is something that cl.exe fails to optimize properly, or if the Windows stack is failing somewhere else (filesystem read performance? standard library implementation?). There could even be exotic hardware causes -- Windows and Linux are run from different [though identically-branded, purchased-at-the-same-time] SSDs -- perhaps one is slower.

Updated Backend

Over the last few days, I've massively restructured the way that tchow.com is served.On the surface, you shouldn't notice any big differences, but under the hood things have gotten dramatically simpler.

How it was

From sometime around 2007 until just a few days ago, tchow.com was hosted on a VPS partition from RapidVPS.The partition was set up running a pretty standard stack: Gentoo GNU/Linux, the Apache web server, and PHP5.
I designed my content serving system to be future-proof, simple to use, and to contain minimal redundancy.This meant that I wanted visitors to see clean URLs (no file extensions; file types determined by Content-Type header), and those that looked at the markup to see nice xhtml with a clear separation of content and chrome.Internally, I wanted to be able to serve the web page from a filesystem, provide content with my favorite text editor (and/or scp), and do so in a not-too-idiosyncratic format.
In order to make this work, the following happened on each page load:first, mod_rewrite rules would take the user's URL and turn it into a query string for a dispatch.php script;this script, in turn, would crawl the backend directory structure to find the proper page (a file containing an html fragment),then paste in appropriate templates for navigation and analytics.

What I didn't like about it

This old setup was nice, but it was also overkill -- php was re-generating the same (static) content over and over again, opening and reading through tens of files on each page view.This seems really inefficient (and, honestly, the "standard" answer of wrapping this in a caching web server seems even sillier).
Additionally, external URLs were clean-ish, but I never really resolved where a trailing slash was appropriate, which led almost everything having absolute links all the time.
Also, I ended up storing my page content in an svn repository (to move edits between my staging and live pages), but since svn has no way of updating all of htdocs at once, this creates potential race conditions in page viewing during updates.Besides, I've been using git for a number of years, and it feels so much snappier than svn that it was getting to be a drag to do page updates.
Added to these design concerns, I'd noticed that Digital Ocean was offering a VPS partition of similar size to my RapidVPS partition (but with unlimited bandwidth) for about half the price.So it was time to change.

How it works now

First off, I ditched Apache -- which provided way more than I need -- and switched to Lighttpd; this also provides way more than I need, but at root it's a lot more comprehensible.Indeed, I get nice clean URLs using just two modules: mod_rewrite for hostname correction (e.g. so that www.tchow.net redirects here properly) and mod_magnet for URL dispatch.I no longer have php re-generating static content over and over again;instead, I have a python script that I call on the (git) content repository which regenerates just those pages that need it.
But mod_magnet is really the key to the whole setup because of the way it works:you provide Lighttpd with the path to a lua script, and it runs that script to determine which physical path corresponds to given query string (the script can also do things like trigger redirects and add headers).The cool thing here is that now every page view goes through dispatch.lua (in my case, nothing more than a url -> file mapping table), and it's one file, which means that if I atomically update it then I've atomically updated the web page.So this setup gives me the satisfaction of being able to update the content on tchow all at once, and without race conditions (as long as my scripts are careful to not overwrite any content referenced by the active dispatch.lua).
I also re-worked a few things around the urls so that the semantics are much clearer, changed the styling a smidge, drew some new calendar icons, and did various other tidying.But mostly, I'm satisfied that I've gotten rid of a lot of systems that tchow.com didn't need, and pared things down to a really slick-n-slim software stack that still satisfyingly serves what it should.

wmiirc and Disks

(From the not-useful-unless-you-use-Linux-like-I-do department.)
So, I've been using wmii as the window manager on my laptop.Other than changing its default meta key to be the winkey (I should really do something about that branding), setting it to launch uxterms, configuring it to show CPU and battery info, and setting up pidgin to change a tag color on new messages, I haven't had to change its default settings at all.
That may seem like a lot, but in the world of "nonstandard" window managers, well, it isn't.Programmers (the probable main consumers of -- and face-slappingly-obviously main producers of -- such WMs) are picky people.Indeed, the fact that I didn't have to edit the source code to do any of this is right civilized!(Unlike switching the meta key in, say, Fluxbox.)
However, I did manage to do something stupid when performing that second-to-last tweak.First some background:wmii has a status line at the bottom of the screen.In the wmiirc script there is a function -- status() -- whose output is pasted into the lower right at some frequency.To display battery information you simply create a program (likely a shell script) to echo the information you'd like displayed and have status() call it.I learned as much from this forum post, which also includes a handy script to produce exactly the sort of info I wanted to display.Thus, I adapted this script to my needs and it has been quietly working ever since.
But my hard drive has been being accessed every five seconds (even when the computer is apparently idle) ever since I set that script up.Turns out that here-documents in bash create temp files.Creating temp files is disk activity.Disk activity makes kjournald decide to write a journal every five seconds.Which is probably not good for the disk, definitely not good for battery life, and quite annoying.Here'my version with that problem fixed:

#!/bin/sh
#based on http://bbs.archlinux.org/viewtopic.php?id=1809

rate=`grep 'present rate' /proc/acpi/battery/BAT0/state | tr -d -c 0-9`
current=`grep 'remaining capacity' /proc/acpi/battery/BAT0/state | tr -d -c 0-9`
total=`grep 'last full capacity' /proc/acpi/battery/BAT0/info | tr -d -c 0-9`
percent=`echo "100 * $current / $total" | bc`

speed=`grep -m1 -o 'cpu MHz.*\.' /proc/cpuinfo | tr -d -c '\n0-9'`

if grep -q discharging /proc/acpi/battery/BAT0/state
then
 minutes=`echo "60 * $current / $rate" | bc`
 echo $speed $minutes $percent'%'
else
 echo $speed $percent'%'
fi

By the way, should you find yourself in a similar pickle, echo '1' > /proc/sys/vm/block_dump.It will produce kernel messages (viewable by dmesg -- you probably want to turn off your system logger while doing this) which will let you know who is causing blocks to need writeback.