How can computers use tacit knowledge?

I'm seven weeks into my Master's, and the clouds are beginning to clear around my research topic. But first, an aside...



In my improv class at Impatient Theatre on Saturday, there was a scene that was described as "incredibly complex". We were playing with the idea of an "analogous scene" that explores and expands the themes of a previous scene in a different situation, with different characters.

The first scene was an old couple going through storage in their basement. As the scene progressed, there was subtle tension between the two, with the woman wanting to bring something up, and the man ignoring her, and instead bringing attention to some old toys. He passed one to her (a plastic Pony) and held one himself (a He-Man). He held the He-Man in front of his face and said "Hey there little Pony!" and so on. They weren't interacting as a couple anymore, but as two little toys. Finally, the Pony said that she wasn't sure if it was working out between her and He-Man, and that maybe they should re-evaluate their relationship. Both the woman and the man froze, the conversation between the toys having become about their relationship. The scene ended.

The second scene was two friends exploring the garage of a house that one of them was house-sitting. Strangely, there were a bunch of sports equipment and other costumes in the garage. The house-sitter started putting on and taking off equipment for football and hockey, etcetera, while the friend starting playing with some Star-Wars costumes. The more equipment the house-sitter had on, the angrier they would be about some issue with the friend (a car getting scratched, poster in their house being thrown out). Meanwhile, the friend thought they were just play-acting scenes from the Star Wars movies. At one point, the friend put a Yoda head on the house-sitter to try to get him out of his angry mode, but the house-sitter took it off, saying it didn't feel natural. Finally, the friend put the Darth Vader costume back on and got really angry about an insignificant housemate issue, attempting to show the house-sitter how he was acting. The house-sitter responded with "Did that happen in the movie?" and the scene ended.

The first scene took about 2 minutes, the second one took 4 minutes. We analyzed both of them for about 5 minutes apiece, and I still find myself thinking about them. When the improvisors were asked why they made certain choices, they couldn't actually verbalize their thinking or reasoning, yet all of them made amazing scenes. It turned out that the improvisors in the second scene had two very different in-the-moment interpretations of one part of the scene, yet they together made something that was interesting to watch from the audience.


In my continuing research about learning by mimicry, I've become interested in the idea of tacit knowledge, knowledge that is defined by its difficulty of communication. Tacit knowledge is usually passed by a learner observing someone else, whether they are explicitly a teacher or not. I assert that interactions that require or use tacit knowledge are more powerful and expressive than those that are not. While my initial motivation to explore this space was miming, a form of physical expression frequently used by people when words are not enough, tacit knowledge exists in the use of any sufficiently complex interaction. I use the metaphor of drafting while riding a bicycle as the way that a user can explore the space of existing tacit knowledge:

1. Advance tentatively
2. Have the space of previous interactions shown in a useful representation
3. Made a choice
4. GoTo 1

A small philosophical problem I have with this is that it seems to suggest that all possible interactions are pre-determined. Well, on a regular basis, the vast majority of interactions are pre-determined. New space can be explored by a different mechanism, but frequently it is built on existing knowledge from different areas that is synthesized, filling in convexities.

I want to focus this down to how tacit knowledge about physical interactions is communicated and processed by users. In particular, I want to focus on newer gestural interfaces, which are currently baffling to people that haven't seen them (I played with one, and it was trial and error). If communicating gestural information is possible, it also means we can move beyond the "list of (x,y) finger points" as the only possible interaction, which is embarassingly limiting given the possibilities afforded.

In the improv scene I wrote about above, the improvisors in the second beat didn't know explicitly what pattern they were following, but they did know it, and they could expand on it. This is a power that is hard to express in existing interfaces. If computers can be made to express tacit knowledge, then I believe we've taken another step to the final goal: making computers worth caring about.

Image above is a hand painting that was made each night for a play I was in this summer, "For a Better World" or "Für eine bessere Welt", for Single Thread Theatre Company. One theme of the play, as I saw it, was the evolution of culture.

Stuck in Real Time

At Friday's DGP party, a projector and a webcam was set up. The webcam was aimed at the party, and the projector was aimed at a large white wall, displaying the party. This is all good and cool, and if you aim the webcam so that it sees part of the projection, then you get fun feedback.



However, the clincher was that the projector displayed what happened on the webcam 2 minutes in the past. This led to the party itself becoming self-aware, where much of the discussion led to what had happened 2 minutes ago. Since the webcam was aimed to produce feedback, then this included what happened 4, 6, 8 minutes ago too.



As the party continued and my own personal inebriation increased, combined with my recent play-through of Braid, it occurred to me that the projection on the wall represented a barrier between this world and the world from 2 minutes ago. I wanted to cross this barrier, and be in both worlds at once.

Me taking a picture of myself taking a picture:


I created a sequence of movement of myself in front of the projector that lasted about 2 minutes, and kept repeating it and repeating it. The more I did it, the more I aligned with the past version of myself. As the party kept going on in parallel worlds, I was the only person in every world at once.

Mathematical Recreations - Tetris

So, I was doodling while watching an episode of The Wire, and I came across this curiousity. Let's look at Tetris pieces, from the popular video game Tetris. Another way to define these is the set of arrangements of four squares where you can travel from one square to any other given square by shared sides. Go ahead and try to think of any that are in that set that aren't in the above picture, except by changes in rotation.

Now, since we're doodling, lets see if you can draw all of the Tetris pieces without taking your pen off the paper, and without crossing over or drawing on top of an existing line. Try it. Well, it works for some, and it doesn't for others.

Now why?! It's a little baffling. But, like any mathematical curiousity, it is made simple by breaking things down into constituent parts. The parts of the pieces where the rules are applied directly is at the joints/crossroads, so let's try to categorize them:


[Below here there are spoilers, beware]

Going from left to right:

One: In all of the pieces, there is no joint with only one line coming in, since this wouldn't be able to make a square.

Two: There are many joints with two lines attached to them, but they have to be at corners, not like I've drawn it above. When you're drawing the continuous line, you come in one way and go out another.

Three: If there are three lines attached to a joint, it must mean that it is an endpoint. See what I mean? Since you can't come in to an intersection more than you go out, and vice versa, then that means you need to stop in the intersection. This will be made clearer when we look at four.

Four: Since there is an even number of lines, you can go in twice and go out twice.

Now, let's number the joints in our Tetris shapes:

See? Only the top two shapes (I wish I had names for them) have exactly two endpoints, the points labeled with a 3. All the other pieces have more than two, and if we're trying to draw a continuous line you can't really stop or start more than twice, can you?

Implicitly Explicit

For the last 15 years or so, the state of the art in Human-Computer Interaction, and Computer Science as a whole, has been riding the line between the explicit and the implicit. I am trying very hard to understand this.

When we meet a new Thing, we apply very powerful algorithms subconsciously to figure out its behaviour until, finally, it becomes part of Our Understanding. We are highly unaware of how we do this, even if we are told to think it through. The most exciting applications in this area improve the ease of adoption of a new Thing. We give up quickly on Things that fail early, so making sure the first few dates between Us and Them go smoothly is very important. Long term success, termed "expert use" is harder to describe.

How do we define something Good?

Time is an important quantity to Us. Let us use Time to as a metric to evaluate the Good-ness of a Thing.

For what Goals shall we measure the Time?

A Goal should be something easy to measure, that can be done with both the old Thing and the new Thing. Let us compare them to determine the optimal Thing.

This process frustrates Me. 

Jump back up a level.

When a Thing fails for Us a few times in a new way, our interaction paradigm switches from the Implicit to the Explicit. We construct rules and try to inference them. Sometimes this is good, and sometimes this is bad. Most of the time, Explicit reasoning should not be necessary. I am a fan of thinking that when a Thing is able to take over a part of our normal life, it frees more of us up to do more better and interesting activities. However, sometimes switching to Explicit reasoning is important. Feynman made a point, when speaking to physics students, that quantum mechanics is like nothing they have ever thought of before, not particles, not waves, not dice. It is entirely axiomatic.

Explicit understanding takes a very long time, but it helps build Implicit understanding (for some reason I am able to think of multi-dimensional vector spaces). It becomes a part of Us. 

I have started taking classes with Toronto's Impatient Theatre Company, a improvisation group focused on the Harold, a type of longform improvisation comedy. The thesis here is (using unusual words) to organically discover the Pattern of Behaviour present in the suggestion given and the improvisors on stage. Organically, Implicit, means not imposed, Explicit. Why is this only restricted to comedy? 

The best Things do not dissapear, they are consumed by Us. When we discover a Pattern of Behaviour, it quickly becomes interesting cool accepted normal required obligatory and we have moved on. Then, the Pattern of Behaviour may be extended and explored to more Patterns of Behaviour. Abusing a definition here, it is like there is a fractal structure to behaviour.

Is there a non-grey difference between "lots of Implicitness" and "Explicitness"?

How can one consciously empower oneself to use Implicit and Explicit modes? Can a Thing be designed to tell one how?

Can a Thing that

animates

draws

files

schedules

makes other programs

learn your Pattern of Behaviour and continue it for You?