New Home for Dustin's Blog

Hey, if you came here somehow, you probably want the NEW and SHINY home of the blog at:
http://dustinfreeman.org/blog/

Improv tour diary

I went out checked out Jet City Improv in Seattle yesterday. I've been so obsessed with longform the past 2 years that I had pretty much forgotten that short form existed. My two favourite games currently are He Said She Said and Three Way Dub, mostly because they are split-control games. Here's 3 games I saw yesterday that I had never heard of/seen before and really liked:

Double Blind

2 improvisors (C,D) leave the room. The remaining two (A,B) perform a short scene from a suggestion. A leaves the stage and C is called in. C gets a new suggestion and now B and C perform a scene. B says the same lines and performs the same physicalization as before, while C improvises along. At then end, D is called in. D gets a suggestion and performs a scene with C, where C must use the same lines and physicalization as they just performed.

This is a great combination of broken telephone and the game Actor's Nightmare. I could easily see it being extended in a few ways. Like Actor's Nightmare, it is funny when moments align well as well as when they are non-sequiturs. However, the scenes need to be pretty short to be memorized. Bold choices in the scenes are really helpful here, as it makes the subsequent alignment/non-sequiturs much more entertaining.

Game-O-Mystery

This is pretty meta. 1 improvisor (A) leaves the room, while the other improvisors come up with a game. They come up with a name first ("Mish Mash" this time) and then simple rules, from the improvisors only. Then, A comes back, and they get a suggestion and perform a hilarious scene, after which A sees if they can guess the rules.

This time, the rules they made up were:

1. If someone speaks, at least one other person must be squished into them
2. If A raises their voice in pitch, everyone claps and we change scenes.

It makes more sense if the improvisors come up with the rules, since game rules are very important to make a scene "work." Also, its more entertaining if the rules have some dependency on what A does. Rule 2 didn't get triggered for a while, so one improvisor came in with the feed of "sex-changing powder" (the suggestion was "Renaissance", from yours truly) and in improv-land, the way to act female is to just pitch your voice up 3 octaves. This game is very familiar to Interrogation.

Backwards-Forwards

They played a game many people have seen, where an improvisor sits on the side of the stage and plays the scene like a video, "backwards" or "forwards". (Aside: do people ever say "ahead by one frame"). When going backwards, the improvisors would say their lines forwards. This could be a point of contention if you cared enough. Personally, I enjoy trying to make backwards vocalizations, but it helps for remembering where you're going if you actually say the lines.

After establishing the scene for about a minute, they did something I had never seen before: they rewinded to before the beginning of the scene! This provided some interesting exposition about one of the characters (he was undead, and could come back to life by inhabiting someone else's body) which "explained" some of the apparent intentions of the characters. The exposition was made more interesting because the first time we (the audience and improvisors) saw it, we were going through it backwards. This seems like a good reason for not making backwards vocalizations. 

The show also had a dedicated musician and lights controller. The lights didn't go just on and off either, but they had "disco", "horror", etc. modes. Check them out if you're ever in Seattle.

Diamond Touch Projects

While working on 3 courses last semester, I got a head-start on some research ideas. When I was starting out, I was simply interested in the idea of "Miming and Mimicry" as an inspiration. After many discussions, I've developed further ideas, but still with Miming and Mimicry as  the seed. Here's the most tangible of what I've worked on: 

This first one, which I titled "Space Shift" certainly doesn't count as a finished Contribution To Research, but I whipped it together quickly to try an idea out.

This second video is more about manipulating objects on  desktop. Most people, when thinking of multitouch, think of the ubiquitous "Look ma, I'm moving photos" demo. This uses very literally realistic physics, which is great because it takes advantage of the physical intuition we already have. I played with a few very simple ways of using non-realistic physics. I am also working on (in progress) creating new "types" of physics by miming. At the end of this video, we see a copy gesture inspired by miming.

Where I'm going, where I've been

Near the end of the first semester of my Master's, my research topic was becoming clearer - along the lines of "teaching the use of gestural interfaces". This was motivated by the proliferation of gestural interaction in devices of many form factors. Much of my thoughts on this were driven by my improv and theatre background, especially with respect to miming and mimicry. After some emails were sent, and calls made, I have been hired an an intern at Microsoft Research in Redmond, Washington from January to April to work with the Microsoft Surface team on teaching gestural interface use, and to measure how well a user has learned. With that in mind, here's the courses I took last semester and projects I did.

Computational Biology
This course takes problems solved in computer science and maps them onto problems in biology that are becoming harder as more data becomes available. I initially only took this course because I am required to take "breadth" in my courses, but it turned out to be pretty enjoyable. I was part of a sub-group that examined current research into codon bias. My final project was on modifying a gene from one organism to another to make it perform better, with respect to codon bias. I wanted a picture from each course I did, but since I don't really have a good one for Comp. Bio., here's a bunch of completely unexplained equations I made for my final project.


Ubiquitous Computing
Ubicomp helped me get more coverage in the literature of my field, although more on the ubiquitous, rather than interactive, side. I was interested in ambient displays, and my project partner was interested in household devices, so we came up with the humourous yet sincere project title: "Devices that Bruise: Battered Device Syndrome". The idea was to give everyday items the ability to give feedback if they received damage, either causing immediate harm, or causing harm if this behaviour was continued in the long run. Below is what we imagined a door would look like if it was slammed shut.


And here is a storyboard of some devices that have been "bruised".

In addition to giving the user's feedback about unintentional misuse, these devices could let you know when you are in a bad mood, as the user above realizes. This is much like a good friend would alert you to your mood. For our prototype, we wired an accelerometer up to some LEDs, which isn't visually interesting enough for me to show here. One concern with this sort of behaviour feedback is that it might encourage the inverse response to what we are looking for, like in the art project love hate punch, which was one of our inspirations. User testing would have to be done.

Machine Learning
This was mentioned in an earlier post. Upon learning about the ability of Restricted Boltzmann Machines, which are pretty good digit classifiers, to generate new digits, I immediately wanted to apply it to HCI somehow. These animations were a large part of my inspiration. Given a user's ambiguous digit, I wanted to show a user how they could improve their writing to make it easier to recognize by the device. The philosophy here isn't to show the user the "perfect" digit, but rather to show how their given digit could be improved. Personalized feedback. I was afraid that the animations wouldn't be smooth enough to be pedagogical, but my professor said he would be surprised if they weren't. It turns out that the animations were very smooth, but did not always work perfectly. Below shows examples of digits that were improved, starting with the original digit at the left and evolving right.

For the above digits, the classifier can easily tell what the digits were, but improvement is still possible. However, for some digits in the dataset, it isn't really clear what was meant, and feedback on how to disambiguate towards either alternative must be shown. Below, we see an ambiguous digit in the centre. To make it a better 1, the evolution goes left. To make it a better 2, the evolution goes right.


Although it doesn't look that cool, I'm pretty exciting by these results. Classifiers in machine learning haven't been used in this way before, and I would like to think of more interactive ways to apply them. Generally, Artificial Intelligence and Human-Computer Interaction are not as closely related as they should be, and I think they should be if we want to makes computers worth caring about.

Evolving Ambiguous Digits

The above image represents my final Machine Learning project, Evolving Ambiguous Digits. I have just finished writing it now.

Yeah! First semester of Master's finished!

More details are to come (I will likely make a project page). But the above is the output of a neural network as represented by a Restricted Boltzmann Machine, when it is prodded in the direction of different digits. The top row is how the digit evolves if you poke the 'zero' neuron (think of it as poking the 'burnt toast' neuron) and each row down represents the next digit. Poking five works pretty well since it is a five already.

Blog Entry Ideas

I keep a bunch of text files on my cellphone that contain random ideas I get while on the go, categorized such as 'Books I want to write', 'Funny ideas', 'Directives', 'Improv Ideas', 'To read/watch/play'. One of these files is 'Blog entry ideas', a collection of stuff I wanted to write or rant about but haven't yet. Well, I've had it for a year now, and still haven't written these posts. Possibly because I could never come up with a coherent way to express what I meant, I was too lazy, or I forgot what the hell I was talking about. Instead, let's do it the easy way and let your imagination fill in the blanks. This way, I don't have to write something, and you can imagine exactly what you would want to read about. Except where I've noted, these are completely unedited:


Academia vs industry: you always maximize stockholder value. In aca, that's you! But, you're hired on the presumption that stockholder value corelates [sic] highly with personal value.

An argument against knives

I've been going at it wrong. What is expressable [sic] through mime is viewable by the power of imagination and anything that tries to recreate that will destroy that effect.

Blog name: follow these instructions

Observation: movies like tropic thunder, kiss kiss bang bang and adaptation mock the film conventions of Hollywood initially, but then use all of them very effectively in their final acts. Is this out of homage, or mockery? Is it true that these ridiculous things are, inevitably, the most effective way to tell stories on the large screen? As an audience, are we so entrenched in film language that we can't expect otherwise?

The honour of the grind. Granddad Rahnasto [on my mom's side] once built the foundation of a house by himself. Conversation I had with a guy after the play copenhagen.

Dumb thoughts on clean hydrogen fuel. Where does it come from? Clean?

Numbing in the transfer from actor to audience.

On my approach to objectivity and art, specifically by script-reading. Why do I seek to be objective? Does it matter?

The implicit nature of geometrical shapes

Difference in interactive media between part of the 'plot' and something it is possible for the player to do. Who is responsible for their actions, ie drink driving in GTA or spousal abuse in fable.

Catch 22s and strange loops in godel's incompleteness theorem.

On genuineness and science. On the sliding scale of carefulness whenit comes to truth, where should you be? Not admitting a chance of failure. 'Religious' thinking.

User responsibility:
My viewpoint on things that can be "addictive", such as videogames and drugs, and the roles/responsibilities of users and designers. how my viewpoint has changed recently with my awareness of HCI. Acting like a baby as a user. Also applies to pieces of art that need to be understood. Eg. The knife in "no exit" - the audience is expected to notice it.

Coping with tunnel vision - epiphanies are important

Rationalizing decision time. Monetary. The value of my time and reducing stress

Improv Math: Military Clock

A few weeks ago, I went to see PROJECTproject, an improv show in the Toronto area. They had just come back from a tour to Western Canada, and one of the things they brought back with them was a structure called Military Clock. I think they said it came from Winnipeg. I'm always on the look for interesting improv structures, and this has a mathematical tilt to it, so that's why I have written it up.

The way it works is:
- split the team up into two groups
- have each group form a circle on either side of the stage
- the people from either circle who are closest to the audience (two of them) are in the scene
- whenever anyone not in the scene feels that it is getting stale, they yell "Clock!" and both circles rotate
- a new scene starts with the two people who are now at the front
- repeat until you feel you're done, or everyone has had a scene

When I saw PROJECTproject do it, they had one circle of three and another circle of four, like below.

Go ahead! Click on the stage to make the clock run! Also, click on the arrows to add or subtract improvisors to each circle.



However, this only "works" for certain numbers of improvisors in the circles. Ideally, you want to be able to run the military clock around without having any repeats, and have every possible pairing of improvisors show up. When I saw it, the improvisors would keep their character for future scenes, so we saw all possible character combinations (7 choose 2).

So what numbers does it work for? It is possible to find these numbers for any troupe size?

To rotate through every possible combination, the size of the circles must be mutually prime.

Can every troupe size be expressed as a sum of two mutually prime numbers? NO. I'm also assuming we can't have circles of size one, since that isn't the spirit of the game.
It works for
5 - 3&2
7 - 3&4 (PROJECTproject)
8 - 3&5
9 - 4&5
10 - 3&7
11 - 5&6

It doesn't work for 6, sadly.

Does it NOT work for any troupe that is larger than 11? I am not sure. But seriously, who has an improv troupe larger than 11? Think of the egos!

Playing your part

One of the problem with video games that have a story component is that the player you represent can behave inconsistently with the "story of the game". Games that do not have a story component, such as most Will Wright-style games, are the exception. An example of this problem is the very detailed story and character development of Grand Theft Auto IV, and yet the cartoonish violence which is actually fun and deviates from the character that the story has built for you. Also, in some levels of the Thief Series, it is sometimes just easier to sprint through an entire level instead of being stealthy, and thus having the entire complement of guard and monster AIs making a racket trying to find you.

Elsewhere, video games have adapted strange conventions to prevent this problem, such as the "silent protagonist" seen in many first-person shooters that avoids the sticky issue by simply shutting up.

One potential cool way to synthesize story and gameplay is to somehow reward or punish the player for acting more or less like their part. I hadn't yet seen this in a game yet, until I was reading the Wikipedia entry for Valve's upcoming game, Left 4 Dead. The Tank, concept art seen below, is one of the playable zombie characters in the game.

Obviously, this character is not meant to just sit still and remain pretty. Strategic thinking also doesn't seem to be suggested by its form. To quote the article:
"Also, the tank has a "Frustration Meter" which means that control will be taken away from a human player if the Tank does not engage the survivors for a certain time."
Cool! There are other varieties of playable infected that have different characteristics, and they don't have this same description. So, the game designers are suggesting a certain "play character" in their design. I can't wait to see what more comes of this.

Suggestion: make the physical appearance and gameplay characteristics change automatically based on the game learning a player's "style".

Note: The most famous silent protagonist is Gordon Freeman, the main character of Valve's Half-Life series. Valve is great.

How can computers use tacit knowledge?

I'm seven weeks into my Master's, and the clouds are beginning to clear around my research topic. But first, an aside...



In my improv class at Impatient Theatre on Saturday, there was a scene that was described as "incredibly complex". We were playing with the idea of an "analogous scene" that explores and expands the themes of a previous scene in a different situation, with different characters.

The first scene was an old couple going through storage in their basement. As the scene progressed, there was subtle tension between the two, with the woman wanting to bring something up, and the man ignoring her, and instead bringing attention to some old toys. He passed one to her (a plastic Pony) and held one himself (a He-Man). He held the He-Man in front of his face and said "Hey there little Pony!" and so on. They weren't interacting as a couple anymore, but as two little toys. Finally, the Pony said that she wasn't sure if it was working out between her and He-Man, and that maybe they should re-evaluate their relationship. Both the woman and the man froze, the conversation between the toys having become about their relationship. The scene ended.

The second scene was two friends exploring the garage of a house that one of them was house-sitting. Strangely, there were a bunch of sports equipment and other costumes in the garage. The house-sitter started putting on and taking off equipment for football and hockey, etcetera, while the friend starting playing with some Star-Wars costumes. The more equipment the house-sitter had on, the angrier they would be about some issue with the friend (a car getting scratched, poster in their house being thrown out). Meanwhile, the friend thought they were just play-acting scenes from the Star Wars movies. At one point, the friend put a Yoda head on the house-sitter to try to get him out of his angry mode, but the house-sitter took it off, saying it didn't feel natural. Finally, the friend put the Darth Vader costume back on and got really angry about an insignificant housemate issue, attempting to show the house-sitter how he was acting. The house-sitter responded with "Did that happen in the movie?" and the scene ended.

The first scene took about 2 minutes, the second one took 4 minutes. We analyzed both of them for about 5 minutes apiece, and I still find myself thinking about them. When the improvisors were asked why they made certain choices, they couldn't actually verbalize their thinking or reasoning, yet all of them made amazing scenes. It turned out that the improvisors in the second scene had two very different in-the-moment interpretations of one part of the scene, yet they together made something that was interesting to watch from the audience.


In my continuing research about learning by mimicry, I've become interested in the idea of tacit knowledge, knowledge that is defined by its difficulty of communication. Tacit knowledge is usually passed by a learner observing someone else, whether they are explicitly a teacher or not. I assert that interactions that require or use tacit knowledge are more powerful and expressive than those that are not. While my initial motivation to explore this space was miming, a form of physical expression frequently used by people when words are not enough, tacit knowledge exists in the use of any sufficiently complex interaction. I use the metaphor of drafting while riding a bicycle as the way that a user can explore the space of existing tacit knowledge:

1. Advance tentatively
2. Have the space of previous interactions shown in a useful representation
3. Made a choice
4. GoTo 1

A small philosophical problem I have with this is that it seems to suggest that all possible interactions are pre-determined. Well, on a regular basis, the vast majority of interactions are pre-determined. New space can be explored by a different mechanism, but frequently it is built on existing knowledge from different areas that is synthesized, filling in convexities.

I want to focus this down to how tacit knowledge about physical interactions is communicated and processed by users. In particular, I want to focus on newer gestural interfaces, which are currently baffling to people that haven't seen them (I played with one, and it was trial and error). If communicating gestural information is possible, it also means we can move beyond the "list of (x,y) finger points" as the only possible interaction, which is embarassingly limiting given the possibilities afforded.

In the improv scene I wrote about above, the improvisors in the second beat didn't know explicitly what pattern they were following, but they did know it, and they could expand on it. This is a power that is hard to express in existing interfaces. If computers can be made to express tacit knowledge, then I believe we've taken another step to the final goal: making computers worth caring about.

Image above is a hand painting that was made each night for a play I was in this summer, "For a Better World" or "Für eine bessere Welt", for Single Thread Theatre Company. One theme of the play, as I saw it, was the evolution of culture.

Stuck in Real Time

At Friday's DGP party, a projector and a webcam was set up. The webcam was aimed at the party, and the projector was aimed at a large white wall, displaying the party. This is all good and cool, and if you aim the webcam so that it sees part of the projection, then you get fun feedback.



However, the clincher was that the projector displayed what happened on the webcam 2 minutes in the past. This led to the party itself becoming self-aware, where much of the discussion led to what had happened 2 minutes ago. Since the webcam was aimed to produce feedback, then this included what happened 4, 6, 8 minutes ago too.



As the party continued and my own personal inebriation increased, combined with my recent play-through of Braid, it occurred to me that the projection on the wall represented a barrier between this world and the world from 2 minutes ago. I wanted to cross this barrier, and be in both worlds at once.

Me taking a picture of myself taking a picture:


I created a sequence of movement of myself in front of the projector that lasted about 2 minutes, and kept repeating it and repeating it. The more I did it, the more I aligned with the past version of myself. As the party kept going on in parallel worlds, I was the only person in every world at once.