Where I'm going, where I've been

Near the end of the first semester of my Master's, my research topic was becoming clearer - along the lines of "teaching the use of gestural interfaces". This was motivated by the proliferation of gestural interaction in devices of many form factors. Much of my thoughts on this were driven by my improv and theatre background, especially with respect to miming and mimicry. After some emails were sent, and calls made, I have been hired an an intern at Microsoft Research in Redmond, Washington from January to April to work with the Microsoft Surface team on teaching gestural interface use, and to measure how well a user has learned. With that in mind, here's the courses I took last semester and projects I did.

Computational Biology
This course takes problems solved in computer science and maps them onto problems in biology that are becoming harder as more data becomes available. I initially only took this course because I am required to take "breadth" in my courses, but it turned out to be pretty enjoyable. I was part of a sub-group that examined current research into codon bias. My final project was on modifying a gene from one organism to another to make it perform better, with respect to codon bias. I wanted a picture from each course I did, but since I don't really have a good one for Comp. Bio., here's a bunch of completely unexplained equations I made for my final project.


Ubiquitous Computing
Ubicomp helped me get more coverage in the literature of my field, although more on the ubiquitous, rather than interactive, side. I was interested in ambient displays, and my project partner was interested in household devices, so we came up with the humourous yet sincere project title: "Devices that Bruise: Battered Device Syndrome". The idea was to give everyday items the ability to give feedback if they received damage, either causing immediate harm, or causing harm if this behaviour was continued in the long run. Below is what we imagined a door would look like if it was slammed shut.


And here is a storyboard of some devices that have been "bruised".

In addition to giving the user's feedback about unintentional misuse, these devices could let you know when you are in a bad mood, as the user above realizes. This is much like a good friend would alert you to your mood. For our prototype, we wired an accelerometer up to some LEDs, which isn't visually interesting enough for me to show here. One concern with this sort of behaviour feedback is that it might encourage the inverse response to what we are looking for, like in the art project love hate punch, which was one of our inspirations. User testing would have to be done.

Machine Learning
This was mentioned in an earlier post. Upon learning about the ability of Restricted Boltzmann Machines, which are pretty good digit classifiers, to generate new digits, I immediately wanted to apply it to HCI somehow. These animations were a large part of my inspiration. Given a user's ambiguous digit, I wanted to show a user how they could improve their writing to make it easier to recognize by the device. The philosophy here isn't to show the user the "perfect" digit, but rather to show how their given digit could be improved. Personalized feedback. I was afraid that the animations wouldn't be smooth enough to be pedagogical, but my professor said he would be surprised if they weren't. It turns out that the animations were very smooth, but did not always work perfectly. Below shows examples of digits that were improved, starting with the original digit at the left and evolving right.

For the above digits, the classifier can easily tell what the digits were, but improvement is still possible. However, for some digits in the dataset, it isn't really clear what was meant, and feedback on how to disambiguate towards either alternative must be shown. Below, we see an ambiguous digit in the centre. To make it a better 1, the evolution goes left. To make it a better 2, the evolution goes right.


Although it doesn't look that cool, I'm pretty exciting by these results. Classifiers in machine learning haven't been used in this way before, and I would like to think of more interactive ways to apply them. Generally, Artificial Intelligence and Human-Computer Interaction are not as closely related as they should be, and I think they should be if we want to makes computers worth caring about.

Evolving Ambiguous Digits

The above image represents my final Machine Learning project, Evolving Ambiguous Digits. I have just finished writing it now.

Yeah! First semester of Master's finished!

More details are to come (I will likely make a project page). But the above is the output of a neural network as represented by a Restricted Boltzmann Machine, when it is prodded in the direction of different digits. The top row is how the digit evolves if you poke the 'zero' neuron (think of it as poking the 'burnt toast' neuron) and each row down represents the next digit. Poking five works pretty well since it is a five already.

How can computers use tacit knowledge?

I'm seven weeks into my Master's, and the clouds are beginning to clear around my research topic. But first, an aside...



In my improv class at Impatient Theatre on Saturday, there was a scene that was described as "incredibly complex". We were playing with the idea of an "analogous scene" that explores and expands the themes of a previous scene in a different situation, with different characters.

The first scene was an old couple going through storage in their basement. As the scene progressed, there was subtle tension between the two, with the woman wanting to bring something up, and the man ignoring her, and instead bringing attention to some old toys. He passed one to her (a plastic Pony) and held one himself (a He-Man). He held the He-Man in front of his face and said "Hey there little Pony!" and so on. They weren't interacting as a couple anymore, but as two little toys. Finally, the Pony said that she wasn't sure if it was working out between her and He-Man, and that maybe they should re-evaluate their relationship. Both the woman and the man froze, the conversation between the toys having become about their relationship. The scene ended.

The second scene was two friends exploring the garage of a house that one of them was house-sitting. Strangely, there were a bunch of sports equipment and other costumes in the garage. The house-sitter started putting on and taking off equipment for football and hockey, etcetera, while the friend starting playing with some Star-Wars costumes. The more equipment the house-sitter had on, the angrier they would be about some issue with the friend (a car getting scratched, poster in their house being thrown out). Meanwhile, the friend thought they were just play-acting scenes from the Star Wars movies. At one point, the friend put a Yoda head on the house-sitter to try to get him out of his angry mode, but the house-sitter took it off, saying it didn't feel natural. Finally, the friend put the Darth Vader costume back on and got really angry about an insignificant housemate issue, attempting to show the house-sitter how he was acting. The house-sitter responded with "Did that happen in the movie?" and the scene ended.

The first scene took about 2 minutes, the second one took 4 minutes. We analyzed both of them for about 5 minutes apiece, and I still find myself thinking about them. When the improvisors were asked why they made certain choices, they couldn't actually verbalize their thinking or reasoning, yet all of them made amazing scenes. It turned out that the improvisors in the second scene had two very different in-the-moment interpretations of one part of the scene, yet they together made something that was interesting to watch from the audience.


In my continuing research about learning by mimicry, I've become interested in the idea of tacit knowledge, knowledge that is defined by its difficulty of communication. Tacit knowledge is usually passed by a learner observing someone else, whether they are explicitly a teacher or not. I assert that interactions that require or use tacit knowledge are more powerful and expressive than those that are not. While my initial motivation to explore this space was miming, a form of physical expression frequently used by people when words are not enough, tacit knowledge exists in the use of any sufficiently complex interaction. I use the metaphor of drafting while riding a bicycle as the way that a user can explore the space of existing tacit knowledge:

1. Advance tentatively
2. Have the space of previous interactions shown in a useful representation
3. Made a choice
4. GoTo 1

A small philosophical problem I have with this is that it seems to suggest that all possible interactions are pre-determined. Well, on a regular basis, the vast majority of interactions are pre-determined. New space can be explored by a different mechanism, but frequently it is built on existing knowledge from different areas that is synthesized, filling in convexities.

I want to focus this down to how tacit knowledge about physical interactions is communicated and processed by users. In particular, I want to focus on newer gestural interfaces, which are currently baffling to people that haven't seen them (I played with one, and it was trial and error). If communicating gestural information is possible, it also means we can move beyond the "list of (x,y) finger points" as the only possible interaction, which is embarassingly limiting given the possibilities afforded.

In the improv scene I wrote about above, the improvisors in the second beat didn't know explicitly what pattern they were following, but they did know it, and they could expand on it. This is a power that is hard to express in existing interfaces. If computers can be made to express tacit knowledge, then I believe we've taken another step to the final goal: making computers worth caring about.

Image above is a hand painting that was made each night for a play I was in this summer, "For a Better World" or "Für eine bessere Welt", for Single Thread Theatre Company. One theme of the play, as I saw it, was the evolution of culture.

Implicitly Explicit

For the last 15 years or so, the state of the art in Human-Computer Interaction, and Computer Science as a whole, has been riding the line between the explicit and the implicit. I am trying very hard to understand this.

When we meet a new Thing, we apply very powerful algorithms subconsciously to figure out its behaviour until, finally, it becomes part of Our Understanding. We are highly unaware of how we do this, even if we are told to think it through. The most exciting applications in this area improve the ease of adoption of a new Thing. We give up quickly on Things that fail early, so making sure the first few dates between Us and Them go smoothly is very important. Long term success, termed "expert use" is harder to describe.

How do we define something Good?

Time is an important quantity to Us. Let us use Time to as a metric to evaluate the Good-ness of a Thing.

For what Goals shall we measure the Time?

A Goal should be something easy to measure, that can be done with both the old Thing and the new Thing. Let us compare them to determine the optimal Thing.

This process frustrates Me. 

Jump back up a level.

When a Thing fails for Us a few times in a new way, our interaction paradigm switches from the Implicit to the Explicit. We construct rules and try to inference them. Sometimes this is good, and sometimes this is bad. Most of the time, Explicit reasoning should not be necessary. I am a fan of thinking that when a Thing is able to take over a part of our normal life, it frees more of us up to do more better and interesting activities. However, sometimes switching to Explicit reasoning is important. Feynman made a point, when speaking to physics students, that quantum mechanics is like nothing they have ever thought of before, not particles, not waves, not dice. It is entirely axiomatic.

Explicit understanding takes a very long time, but it helps build Implicit understanding (for some reason I am able to think of multi-dimensional vector spaces). It becomes a part of Us. 

I have started taking classes with Toronto's Impatient Theatre Company, a improvisation group focused on the Harold, a type of longform improvisation comedy. The thesis here is (using unusual words) to organically discover the Pattern of Behaviour present in the suggestion given and the improvisors on stage. Organically, Implicit, means not imposed, Explicit. Why is this only restricted to comedy? 

The best Things do not dissapear, they are consumed by Us. When we discover a Pattern of Behaviour, it quickly becomes interesting cool accepted normal required obligatory and we have moved on. Then, the Pattern of Behaviour may be extended and explored to more Patterns of Behaviour. Abusing a definition here, it is like there is a fractal structure to behaviour.

Is there a non-grey difference between "lots of Implicitness" and "Explicitness"?

How can one consciously empower oneself to use Implicit and Explicit modes? Can a Thing be designed to tell one how?

Can a Thing that

animates

draws

files

schedules

makes other programs

learn your Pattern of Behaviour and continue it for You?

Blood-Powered Multi-touch Sensing

My summer research at the Human Media Lab involves large investigations into multitouch sensing. For large interfaces, this involves the user touching the surface under some sort of illumination while a video camera captures the entire surface, and then filtering is applied to determine the touch points of the finger. There are often issues with large amounts of noise. The particular technique I'm working with is Frustrated Total Internal Reflection, but there are many other ways to do it; Microsoft uses Diffused Illumination as well. For these surfaces, it is also convenient if you can tell the difference between a human body part touching the sensor, or if someone put a mug down on top of it.

Someone brought a Plasma Ball in last week, and it looks like it makes a great multi-touch sensor too:


If you've ever played with a Plasma Ball, you'll know that the strings of energy (I have absolutely no idea what the proper name is) dissipate out from the central sphere evenly, until one touches the edge of the outer glass sphere with a finger or other body part. Then, the point of contact glows bright pink and a much brighter strand attaches that point and the central sphere. This effect only works with objects that are properly grounded, such as a human body, which is even better.

Finally, I was playing with our acrylic surface, and noticed another technique that could work even better, and would also only work body parts: blood! Look at the pair of images below:


The point of contact between my fingers and the plexiglass goes from healthy red to pale white as the pressure makes the blood gush away! With colour filtering, this effect would actually be incredibly easy to pick up, given ideal skin and light conditions. I was not even applying that much pressure. The rings of frosting around my fingers in the second picture are from condensation.

3 Days left in Kingston...