Where I'm going, where I've been

Near the end of the first semester of my Master's, my research topic was becoming clearer - along the lines of "teaching the use of gestural interfaces". This was motivated by the proliferation of gestural interaction in devices of many form factors. Much of my thoughts on this were driven by my improv and theatre background, especially with respect to miming and mimicry. After some emails were sent, and calls made, I have been hired an an intern at Microsoft Research in Redmond, Washington from January to April to work with the Microsoft Surface team on teaching gestural interface use, and to measure how well a user has learned. With that in mind, here's the courses I took last semester and projects I did.

Computational Biology
This course takes problems solved in computer science and maps them onto problems in biology that are becoming harder as more data becomes available. I initially only took this course because I am required to take "breadth" in my courses, but it turned out to be pretty enjoyable. I was part of a sub-group that examined current research into codon bias. My final project was on modifying a gene from one organism to another to make it perform better, with respect to codon bias. I wanted a picture from each course I did, but since I don't really have a good one for Comp. Bio., here's a bunch of completely unexplained equations I made for my final project.


Ubiquitous Computing
Ubicomp helped me get more coverage in the literature of my field, although more on the ubiquitous, rather than interactive, side. I was interested in ambient displays, and my project partner was interested in household devices, so we came up with the humourous yet sincere project title: "Devices that Bruise: Battered Device Syndrome". The idea was to give everyday items the ability to give feedback if they received damage, either causing immediate harm, or causing harm if this behaviour was continued in the long run. Below is what we imagined a door would look like if it was slammed shut.


And here is a storyboard of some devices that have been "bruised".

In addition to giving the user's feedback about unintentional misuse, these devices could let you know when you are in a bad mood, as the user above realizes. This is much like a good friend would alert you to your mood. For our prototype, we wired an accelerometer up to some LEDs, which isn't visually interesting enough for me to show here. One concern with this sort of behaviour feedback is that it might encourage the inverse response to what we are looking for, like in the art project love hate punch, which was one of our inspirations. User testing would have to be done.

Machine Learning
This was mentioned in an earlier post. Upon learning about the ability of Restricted Boltzmann Machines, which are pretty good digit classifiers, to generate new digits, I immediately wanted to apply it to HCI somehow. These animations were a large part of my inspiration. Given a user's ambiguous digit, I wanted to show a user how they could improve their writing to make it easier to recognize by the device. The philosophy here isn't to show the user the "perfect" digit, but rather to show how their given digit could be improved. Personalized feedback. I was afraid that the animations wouldn't be smooth enough to be pedagogical, but my professor said he would be surprised if they weren't. It turns out that the animations were very smooth, but did not always work perfectly. Below shows examples of digits that were improved, starting with the original digit at the left and evolving right.

For the above digits, the classifier can easily tell what the digits were, but improvement is still possible. However, for some digits in the dataset, it isn't really clear what was meant, and feedback on how to disambiguate towards either alternative must be shown. Below, we see an ambiguous digit in the centre. To make it a better 1, the evolution goes left. To make it a better 2, the evolution goes right.


Although it doesn't look that cool, I'm pretty exciting by these results. Classifiers in machine learning haven't been used in this way before, and I would like to think of more interactive ways to apply them. Generally, Artificial Intelligence and Human-Computer Interaction are not as closely related as they should be, and I think they should be if we want to makes computers worth caring about.

Improv Math: Military Clock

A few weeks ago, I went to see PROJECTproject, an improv show in the Toronto area. They had just come back from a tour to Western Canada, and one of the things they brought back with them was a structure called Military Clock. I think they said it came from Winnipeg. I'm always on the look for interesting improv structures, and this has a mathematical tilt to it, so that's why I have written it up.

The way it works is:
- split the team up into two groups
- have each group form a circle on either side of the stage
- the people from either circle who are closest to the audience (two of them) are in the scene
- whenever anyone not in the scene feels that it is getting stale, they yell "Clock!" and both circles rotate
- a new scene starts with the two people who are now at the front
- repeat until you feel you're done, or everyone has had a scene

When I saw PROJECTproject do it, they had one circle of three and another circle of four, like below.

Go ahead! Click on the stage to make the clock run! Also, click on the arrows to add or subtract improvisors to each circle.



However, this only "works" for certain numbers of improvisors in the circles. Ideally, you want to be able to run the military clock around without having any repeats, and have every possible pairing of improvisors show up. When I saw it, the improvisors would keep their character for future scenes, so we saw all possible character combinations (7 choose 2).

So what numbers does it work for? It is possible to find these numbers for any troupe size?

To rotate through every possible combination, the size of the circles must be mutually prime.

Can every troupe size be expressed as a sum of two mutually prime numbers? NO. I'm also assuming we can't have circles of size one, since that isn't the spirit of the game.
It works for
5 - 3&2
7 - 3&4 (PROJECTproject)
8 - 3&5
9 - 4&5
10 - 3&7
11 - 5&6

It doesn't work for 6, sadly.

Does it NOT work for any troupe that is larger than 11? I am not sure. But seriously, who has an improv troupe larger than 11? Think of the egos!

Stuck in Real Time

At Friday's DGP party, a projector and a webcam was set up. The webcam was aimed at the party, and the projector was aimed at a large white wall, displaying the party. This is all good and cool, and if you aim the webcam so that it sees part of the projection, then you get fun feedback.



However, the clincher was that the projector displayed what happened on the webcam 2 minutes in the past. This led to the party itself becoming self-aware, where much of the discussion led to what had happened 2 minutes ago. Since the webcam was aimed to produce feedback, then this included what happened 4, 6, 8 minutes ago too.



As the party continued and my own personal inebriation increased, combined with my recent play-through of Braid, it occurred to me that the projection on the wall represented a barrier between this world and the world from 2 minutes ago. I wanted to cross this barrier, and be in both worlds at once.

Me taking a picture of myself taking a picture:


I created a sequence of movement of myself in front of the projector that lasted about 2 minutes, and kept repeating it and repeating it. The more I did it, the more I aligned with the past version of myself. As the party kept going on in parallel worlds, I was the only person in every world at once.

A conversation with the Google Earth API

Dustin: Oh hey there, Google Earth, it looks like you have an API so that we can actually talk to each other.
GE: Yeah, hello! It's really useful.
Dustin: Well, we'll see about that. So, I've turned you on. Can you tell me what layers you have?
GE: Of course! I've got the Geographic Web, Roads, 3D Buildings, Street View.. [goes on a long list]
Dustin: Oh that's great. So, can you tell me what is in each layer?
GE: ...
Dustin: Hello?
GE: ...
Dustin: Oh. I see. Well if I zoom in, I can see that panoramas from Street View are now rendered as spheres floating in space where the picture was taken. That's really cool! But, I guess you can't tell me where they are. I'm looking at a few now. Which ones are they?
GE: ...
Dustin: Okay. What about this one? 482 Avenue of the Americas in NY, NY.
GE: Oh yes. I've memorized that one now.
Dustin: Okay. What do you see now?
GE: 482 Avenue of the Americas in NY, NY. That's all.
Dustin: Oh. Well, can you memorize that street view locations for 478, 486, 490 too?
GE: Yep! Done.
Dustin: Okay, what do you see?
GE: 478, 482, 486, 490 Avenue of the Americas in NY, NY.
Dustin: So you'll only tell me you see something if I specifically tell you to look for it?
GE: Yes.
Dustin: Well what about that panorama over there? 94 W 12th Street?
GE: Sorry I don't know what you're talking about.
Dustin: Really. Memorize 94 W 12th Street.
GE: Ah, yes. So now I see 478, 482, 486, 490 Avenue of the Americas and 94 W 12th Street in NY, NY.
Dustin: Okay fine, you're young so it's okay. Now, can you tell me in latitude and longitude where 482 Avenue of the Americas is?
GE: ...
Dustin: No? But its right there on the screen. I can click on it to make you fly to it!
GE: Oh, I can fly there!
Dustin: Where is 482 Avenue of the Americas?
GE: ...
Dustin: I said where is 482 Avenue of the Americas?
GE: ...
Dustin: Please fly to 482 Avenue of the Americas.
GE: Done.
Dustin: Okay, where are you now?
GE: [gives a series of latitude and longitude coordinates of where it's looking]
Dustin: I see. So, you can't tell me where 482 Avenue of the Americas is, but I can ask you to fly right to it and then you'll tell me where you are. It seems like you're trying to make my life difficult.
GE: [shrug]
Dustin: Okay, now that I'm sitting at 482 Avenue of the Americas and looking towards the horizon like I'm driving. Cool! So, let's turn to the left to look over there.
GE: [turns slightly to the left]
Dustin: Wait, hold on, I'm not in the sphere anymore. I've moved over to the right of the road somehow.
GE: Yes, I turned you around the Focus point!
Dustin: What? What is that?
GE: It's where you're looking at.
Dustin: I thought I was looking at the panorama at 482 Avenue of the Americas.
GE: Actually, that's what you told me to do, but I decided to fake it by making you look at a point 60 meters to the north at a really shallow angle. You weren't actually looking at the panorama at all.
Dustin: Can I look directly at the panorama at all?
GE: ...
Dustin: ...
GE: When I look at something, its defined by the point on the earth itself, and then the compass angle, vertical tilt, and distance I am from it. The panorama is just floating above the Earth, so you can never truly look at it.
Dustin: But there's little controls I have on you that let me move you around whatever way I want. However, when I actually want to talk to you I can only tell you to move in certain ways.
GE: Yes.
Dustin: Well okay. Let me think. I know! How about, instead of looking at a point 60 meters to the north of the panorama, I tell you to look at a point 60 meters to the northwest, and then make the compass angle so that the panorama is in the center of your view.
GE: Like this?
Dustin: Wow! It's like I rotated 45 degrees around the panorama.
GE: No it isn't. Not exactly. That's not what I did.
Dustin: Shut up, it looks like I did, and that's all that matters.
GE: Okay.
Dustin: So, in order to rotate around a panorama as if I'm viewing it, I need to exactly tell you the panorama I want (I can't ask which panoramas are available). Then, I have to tell you to fly to it. Then, I have to ask you where I am, and the answer you give back to me is some point 60 meters away from the panorama. Then, I have to do some clever trigonometry to figure out where on earth the actual panorama is. Then, in order to rotate around the panorama to look at it from different angles I have to make up different points on the Earth in latitude and longitude which are on the opposite side of the panorama I want to look at. Then I can tell you to fly to those points, which don't explicitly look at the panorama but panorama happens to exactly be in the way of the point on the ground I told you to look at.
GE: ...
Dustin: Right, you're not much help here are you?